[Notes] [Git][BuildStream/buildstream][tiagogomes/issue-520] 11 commits:

Tiago Gomes pushed to branch tiagogomes/issue-520 at BuildStream / buildstream

Commits:

6d349610

by Javier Jardón at 2018-08-07T10:53:23Z

source/install_linux_distro.rst: Fix formatting

493d19d2

by James Ennis at 2018-08-07T11:08:36Z

install_linux_distro.rst: Change install instructions for python-arpy package on Arch

  - python-arpy is unobtainable via pacman
  - Also a minor change to how we display install instructions
  from Fedora's dnf package manager

fc9869e2

by James Ennis at 2018-08-07T11:56:21Z

Merge branch 'jennis/change_arch_install_instructions' into 'master'

Change install instructions for python-arpy package on Arch

See merge request BuildStream/buildstream!614

42aa3999

by William Salmon at 2018-08-07T13:41:02Z

Add warning to git track if track and ref are not present

This is to address https://gitlab.com/BuildStream/buildstream/issues/471 that
documented unhelpfull behavour when tracking git sources.

2ceb5dec

by Will Salmon at 2018-08-07T14:46:36Z

Merge branch 'willsalmon/trackWarning' into 'master'

Add warning to git track if track and ref are not present

Closes #471

See merge request BuildStream/buildstream!580

eee4b674

by Jürg Billeter at 2018-08-07T15:36:35Z

Revert "Restrict version of pylint"

This reverts commit 4f168b9b6a02216e2fae24d758ae6b778e545869.

The latest version of pytest_pylint works fine with pylint 2, which
means there is no longer a reason to restrict the pylint version.

pylint 2 is required for Python 3.7.

ea27e389

by Jürg Billeter at 2018-08-07T15:36:35Z

_artifactcache/cascache.py: Fix for PEP 479 / Python 3.7

Do not rely on `StopIteration` bubbling up.

https://www.python.org/dev/peps/pep-0479/

fa5a59f0

by Jürg Billeter at 2018-08-07T16:48:21Z

Merge branch 'juerg/python3.7' into 'master'

Python 3.7 support

See merge request BuildStream/buildstream!615

ed653fbc

by Chandan Singh at 2018-08-08T00:06:28Z

.gitlab-ci.yml: Remove unused script "install.sh"

65f382f1

by Chandan Singh at 2018-08-08T10:52:32Z

Merge branch 'chandan/remove-dead-code' into 'master'

.gitlab-ci.yml: Remove unused script "install.sh"

See merge request BuildStream/buildstream!617

64268608

by Tiago Gomes at 2018-08-08T14:01:10Z

cascache: move tmp directory one level up

The tmp directory is filled when adding an artifact with temporary
files. This causes problems on calculate_cache_size() because we could
try to a attemp to do a stat() on a temporary file that meanwhile has
been reaped.

Handle this by not considering the tmp directory when calculating the
cache size.

Changes:

.gitlab-ci.yml

@@ -26,15 +26,6 @@ source_dist:
    - tar -ztf dist/*
    - tarball=$(cd dist && echo $(ls *))
 -  # Create an installer script
 -  - |
 -    cat > dist/install.sh << EOF
 -    #!/bin/sh
 -    tar -zxf ${tarball}
 -    cd ${tarball%.tar.gz}
 -    pip3 install --no-index .
 -    EOF
+-
    # unpack tarball as `dist/buildstream` directory
    - |
      cat > dist/unpack.sh << EOF
@@ -44,7 +35,6 @@ source_dist:
      EOF
    # Make our helpers executable
 -  - chmod +x dist/install.sh
    - chmod +x dist/unpack.sh
    artifacts:
      paths:

buildstream/_artifactcache/cascache.py

@@ -56,7 +56,10 @@ class CASCache(ArtifactCache):
          super().__init__(context)
          self.casdir = os.path.join(context.artifactdir, 'cas')
 -        os.makedirs(os.path.join(self.casdir, 'tmp'), exist_ok=True)
 +        os.makedirs(self.casdir, exist_ok=True)
++
 +        self.tmpdir = os.path.join(context.artifactdir, 'tmp')
 +        os.makedirs(self.tmpdir, exist_ok=True)
          self._enable_push = enable_push
@@ -394,7 +397,7 @@ class CASCache(ArtifactCache):
          try:
              h = hashlib.sha256()
              # Always write out new file to avoid corruption if input file is modified
 -            with tempfile.NamedTemporaryFile(dir=os.path.join(self.casdir, 'tmp')) as out:
 +            with tempfile.NamedTemporaryFile(dir=self.tmpdir) as out:
                  # Set mode bits to 0644
                  os.chmod(out.name, stat.S_IRUSR | stat.S_IWUSR | stat.S_IRGRP | stat.S_IROTH)
@@ -764,7 +767,7 @@ class CASCache(ArtifactCache):
              # already in local cache
              return
 -        with tempfile.NamedTemporaryFile(dir=os.path.join(self.casdir, 'tmp')) as out:
 +        with tempfile.NamedTemporaryFile(dir=self.tmpdir) as out:
              self._fetch_blob(remote, tree, out)
              directory = remote_execution_pb2.Directory()
@@ -778,7 +781,7 @@ class CASCache(ArtifactCache):
                      # already in local cache
                      continue
 -                with tempfile.NamedTemporaryFile(dir=os.path.join(self.casdir, 'tmp')) as f:
 +                with tempfile.NamedTemporaryFile(dir=self.tmpdir) as f:
                      self._fetch_blob(remote, filenode.digest, f)
                      digest = self.add_object(path=f.name)
@@ -846,6 +849,9 @@ class _CASRemote():
  def _grouper(iterable, n):
 -    # pylint: disable=stop-iteration-return
      while True:
 -        yield itertools.chain([next(iterable)], itertools.islice(iterable, n - 1))
 +        try:
 +            current = next(iterable)
 +        except StopIteration:
 +            return
 +        yield itertools.chain([current], itertools.islice(iterable, n - 1))

buildstream/_artifactcache/casserver.py

@@ -161,7 +161,7 @@ class _ByteStreamServicer(bytestream_pb2_grpc.ByteStreamServicer):
          offset = 0
          finished = False
          resource_name = None
 -        with tempfile.NamedTemporaryFile(dir=os.path.join(self.cas.casdir, 'tmp')) as out:
 +        with tempfile.NamedTemporaryFile(dir=self.cas.tmpdir) as out:
              for request in request_iterator:
                  assert not finished
                  assert request.write_offset == offset

buildstream/_pipeline.py

@@ -358,10 +358,24 @@ class Pipeline():
                      inconsistent.append(element)
          if inconsistent:
 -            detail = "Exact versions are missing for the following elements\n" + \
 -                     "Try tracking these elements first with `bst track`\n\n"
 +            detail = "Exact versions are missing for the following elements:\n\n"
++
 +            missingTrack = 0
              for element in inconsistent:
 -                detail += "  " + element._get_full_name() + "\n"
 +                detail += "  " + element._get_full_name()
 +                for source in element.sources():
 +                    if not source._get_consistency() and not source.get_ref():
 +                        if hasattr(source, 'tracking') and source.tracking is None:
 +                            detail += ": Source {} is missing ref and track. ".format(source._get_full_name()) + \
 +                                      "Please specify a ref or branch/tag to track."
 +                            missingTrack = 1
++
 +                detail += "\n"
++
 +            if missingTrack:
 +                detail += "\nThen track these elements with `bst track`\n"
 +            else:
 +                detail += "\nTry tracking these elements first with `bst track`\n"
              raise PipelineError("Inconsistent pipeline", detail=detail, reason="inconsistent-pipeline")
      #############################################################

buildstream/plugins/sources/git.py

@@ -363,6 +363,12 @@ class GitSource(Source):
          # If self.tracking is not specified it's not an error, just silently return
          if not self.tracking:
 +            # Is there a better way to check if a ref is given.
 +            if self.mirror.ref is None:
 +                detail = 'Without a tracking branch ref can not be updated. Please ' + \
 +                         'provide a ref or a track.'
 +                raise SourceError("{}: No track or ref".format(self),
 +                                  detail=detail, reason="track-attempt-no-track")
              return None
          with self.timed_activity("Tracking {} from {}"

doc/source/install_linux_distro.rst

@@ -57,9 +57,20 @@ Install the dependencies with::
  For the default plugins::
    sudo pacman -S \
 -      bzr git lzip ostree patch python-arpy python-gobject
 +      bzr git lzip ostree patch python-gobject
 +The package *python-arpy* is required by the deb source plugin. This is not
 +obtainable via `pacman`, you must get *python-arpy* from AUR:
 +https://aur.archlinux.org/packages/python-arpy/
++
 +To install::
++
 +  wget https://aur.archlinux.org/cgit/aur.git/snapshot/python-arpy.tar.gz
 +  tar -xvf python-arpy.tar.gz
 +  cd python-arpy
 +  makepkg -si
++
  Debian
  ++++++
  Install the dependencies with::
@@ -218,14 +229,16 @@ Arch Linux
  ~~~~~~~~~~
  Packages for Arch exist in `AUR <https://wiki.archlinux.org/index.php/Arch_User_Repository#Installing_packages>`_.
  Two different package versions are available:
++
  * Latest release: `buildstream <https://aur.archlinux.org/packages/buildstream>`_
  * Latest development snapshot: `buildstream-git <https://aur.archlinux.org/packages/buildstream-git>`_
++
  Fedora
  ~~~~~~
  BuildStream is not yet in the official Fedora repositories, but you can
 -install it from a Copr:
 +install it from a Copr::
    sudo dnf copr enable bochecha/buildstream
    sudo dnf install buildstream

setup.py

@@ -272,6 +272,5 @@ setup(name='BuildStream',
                       'pytest-cov >= 2.5.0',
                       # Provide option to run tests in parallel, less reliable
                       'pytest-xdist',
 -                     'pytest >= 3.1.0',
 -                     'pylint >= 1.8 , < 2'],
 +                     'pytest >= 3.1.0'],
        zip_safe=False)

tests/sources/git.py

@@ -359,3 +359,45 @@ def test_submodule_track_ignore_inconsistent(cli, tmpdir, datafiles):
      # Assert that we are just fine without it, and emit a warning to the user.
      assert "Ignoring inconsistent submodule" in result.stderr
++
++
 +@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
 +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
 +def test_submodule_track_no_ref_or_track(cli, tmpdir, datafiles):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
++
 +    # Create the repo from 'repofiles' subdir
 +    repo = create_repo('git', str(tmpdir))
 +    ref = repo.create(os.path.join(project, 'repofiles'))
++
 +    # Write out our test target
 +    gitsource = repo.source_config(ref=None)
 +    gitsource.pop('track')
 +    element = {
 +        'kind': 'import',
 +        'sources': [
 +            gitsource
 +        ]
 +    }
++
 +    _yaml.dump(element, os.path.join(project, 'target.bst'))
++
 +    # Track will encounter an inconsistent submodule without any ref
 +    result = cli.run(project=project, args=['track', 'target.bst'])
 +    result.assert_main_error(ErrorDomain.STREAM, None)
 +    result.assert_task_error(ErrorDomain.SOURCE, 'track-attempt-no-track')
++
 +    # Assert that we are just fine without it, and emit a warning to the user.
 +    assert "FAILURE git source at" in result.stderr
 +    assert "Without a tracking branch ref can not be updated. Please " + \
 +        "provide a ref or a track." in result.stderr
++
 +    # Track will encounter an inconsistent submodule without any ref
 +    result = cli.run(project=project, args=['build', 'target.bst'])
 +    result.assert_main_error(ErrorDomain.PIPELINE, 'inconsistent-pipeline')
 +    result.assert_task_error(None, None)
++
 +    # Assert that we are just fine without it, and emit a warning to the user.
 +    assert "Exact versions are missing for the following elements" in result.stderr
 +    assert "is missing ref and track." in result.stderr
 +    assert "Then track these elements with `bst track`" in result.stderr

[Notes] [Git][BuildStream/buildstream][tiagogomes/issue-520] 11 commits: source/install_linux_distro.rst: Fix formatting

Tiago Gomes pushed to branch tiagogomes/issue-520 at BuildStream / buildstream

Commits:

8 changed files:

Changes: