[Notes] [Git][BuildStream/buildstream][jmac/stop-caching-vdirs] 14 commi

Jim MacArthur pushed to branch jmac/stop-caching-vdirs at BuildStream / buildstream

Commits:

461a0588

by Jim MacArthur at 2018-09-21T10:53:11Z

element.py: Remove early warning and check directory is not None

The check for directory should have always been here. The early warning
test doesn't work for `bst checkout` because caches haven't been configured
at the point of the test, so I've had to remove it.

aa9caaac

by Jim MacArthur at 2018-09-21T10:53:11Z

_sandboxremote.py: Ignore return value of push_directory

The return value is always upload_vdir.ref.

2aae68c7

by Jim MacArthur at 2018-09-21T10:53:11Z

cascache.py: make push_remote raise exception if no push remotes exist

Also add docstring to method and remove return value since it was useless

ca1bb72c

by Jim MacArthur at 2018-09-21T10:53:11Z

tests/artifactcache/push.py: Ignore return value of push_directory

55c93a82

by Jim MacArthur at 2018-09-21T11:26:55Z

Merge branch 'jmac/remote_exec_checkout_fix' into 'master'

Remote exec: Remove early warning and check directory is not None

See merge request BuildStream/buildstream!800

e209beb0

by Chandan Singh at 2018-09-21T13:10:08Z

_stream.py: Ensure source-bundle's source directory exists

Currently, `source-bundle` command is entirely broken as it tries to stage the
sources in a directory that doesn't exist. Fix it by ensuring that we create
the necessary directories before calling any methods that try to use those
directories.

This fix comes with a regression test to ensure that the basic use-case
of `source-bundle` continues to work in future.

Fixes https://gitlab.com/BuildStream/buildstream/issues/651.

0b000518

by Chandan Singh at 2018-09-21T13:56:55Z

Merge branch 'chandan/fix-source-bundle' into 'master'

Fix source-bundle command

Closes #651

See merge request BuildStream/buildstream!807

ef26043a
by Chandan Singh at 2018-09-21T17:14:16Z
```
setup.py: Make website the primary homepage
```

1b2aed40

by Chandan Singh at 2018-09-21T17:40:11Z

Merge branch 'chandan/update-project-homepage' into 'master'

setup.py: Make website the primary homepage

See merge request BuildStream/buildstream!795

da7e038b

by Jürg Billeter at 2018-09-24T13:58:55Z

element.py: Fix cache check in non-strict mode

The behavior of __assert_cached() was changed when the keystrength
parameter was introduced. This restores the previous behavior.

Fixes #607.

Fixes: 3d308894 ("element.py: Cache the result of checking whether...")

86ea1173

by Jürg Billeter at 2018-09-24T13:59:59Z

tests/frontend/rebuild.py: Add strict/non-strict rebuild test

Test rebuild that requires staging of weakly cached dependency.

This is a regression test for #607.

a76339de

by Jürg Billeter at 2018-09-24T16:02:38Z

Merge branch 'juerg/rebuild' into 'master'

element.py: Fix cache check in non-strict mode

Closes #607

See merge request BuildStream/buildstream!822

31b3f5be

by Jim MacArthur at 2018-09-24T17:02:29Z

sandbox.py: Stop caching get_virtual_directory if get_directory is used

Also update the documentation for get_virtual_directory to make the
implications of using both methods clearer.

82cf1d69

by Jim MacArthur at 2018-09-24T17:02:29Z

element.py/_assemble(): Call get_virtual_directory later.

12 changed files:

buildstream/_artifactcache/cascache.py
buildstream/_stream.py
buildstream/element.py
buildstream/sandbox/_sandboxremote.py
buildstream/sandbox/sandbox.py
setup.py
tests/artifactcache/push.py
+ tests/frontend/project/elements/rebuild-target.bst
+ tests/frontend/project/elements/source-bundle/source-bundle-hello.bst
+ tests/frontend/project/files/source-bundle/llamas.txt
+ tests/frontend/rebuild.py
+ tests/frontend/source_bundle.py

Changes:

buildstream/_artifactcache/cascache.py

@@ -348,19 +348,29 @@ class CASCache(ArtifactCache):
          return pushed
      def push_directory(self, project, directory):
 +        """ Push the given virtual directory to all remotes.
++
 +        Args:
 +            project (Project): The current project
 +            directory (Directory): A virtual directory object to push.
++
 +        Raises: ArtifactError if no push remotes are configured.
 +        """
          push_remotes = [r for r in self._remotes[project] if r.spec.push]
 +        if not push_remotes:
 +            raise ArtifactError("CASCache: push_directory was called, but no remote artifact " +
 +                                "servers are configured as push remotes.")
++
          if directory.ref is None:
 -            return None
 +            return
          for remote in push_remotes:
              remote.init()
              self._send_directory(remote, directory.ref)
 -        return directory.ref
+-
      def push_message(self, project, message):
          push_remotes = [r for r in self._remotes[project] if r.spec.push]

buildstream/_stream.py

@@ -703,6 +703,7 @@ class Stream():
          # Create a temporary directory to build the source tree in
          builddir = self._context.builddir
 +        os.makedirs(builddir, exist_ok=True)
          prefix = "{}-".format(target.normal_name)
          with TemporaryDirectory(prefix=prefix, dir=builddir) as tempdir:
@@ -1085,6 +1086,7 @@ class Stream():
          for element in elements:
              source_dir = os.path.join(directory, "source")
              element_source_dir = os.path.join(source_dir, element.normal_name)
 +            os.makedirs(element_source_dir)
              element._stage_sources_at(element_source_dir)

buildstream/element.py

@@ -1532,8 +1532,6 @@ class Element(Plugin):
              with _signals.terminator(cleanup_rootdir), \
                  self.__sandbox(rootdir, output_file, output_file, self.__sandbox_config) as sandbox:  # nopep8
 -                sandbox_vroot = sandbox.get_virtual_directory()
+-
                  # By default, the dynamic public data is the same as the static public data.
                  # The plugin's assemble() method may modify this, though.
                  self.__dynamic_public = _yaml.node_copy(self.__public)
@@ -1581,7 +1579,6 @@ class Element(Plugin):
                  finally:
                      if collect is not None:
                          try:
 -                            # Sandbox will probably have replaced its virtual directory, so get it again
                              sandbox_vroot = sandbox.get_virtual_directory()
                              collectvdir = sandbox_vroot.descend(collect.lstrip(os.sep).split(os.sep))
                          except VirtualDirectoryError:
@@ -1606,6 +1603,7 @@ class Element(Plugin):
                          collectvdir.export_files(filesdir, can_link=True)
                      try:
 +                        sandbox_vroot = sandbox.get_virtual_directory()
                          sandbox_build_dir = sandbox_vroot.descend(
                              self.get_variable('build-root').lstrip(os.sep).split(os.sep))
                          # Hard link files from build-root dir to buildtreedir directory
@@ -2084,7 +2082,7 @@ class Element(Plugin):
+     #
      # Raises an error if the artifact is not cached.
+     #
 -    def __assert_cached(self, keystrength=_KeyStrength.STRONG):
 +    def __assert_cached(self, keystrength=None):
          assert self.__is_cached(keystrength=keystrength), "{}: Missing artifact {}".format(
              self, self._get_brief_display_key())
@@ -2137,14 +2135,11 @@ class Element(Plugin):
          project = self._get_project()
          platform = Platform.get_platform()
 -        if self.__remote_execution_url and self.BST_VIRTUAL_DIRECTORY:
 -            if not self.__artifacts.has_push_remotes(element=self):
 -                # Give an early warning if remote execution will not work
 -                raise ElementError("Artifact {} is configured to use remote execution but has no push remotes. "
 -                                   .format(self.name) +
 -                                   "The remote artifact server(s) may not be correctly configured or contactable.")
 +        if (directory is not None and
 +            self.__remote_execution_url and
 +            self.BST_VIRTUAL_DIRECTORY):
 -            self.info("Using a remote sandbox for artifact {}".format(self.name))
 +            self.info("Using a remote sandbox for artifact {} with directory '{}'".format(self.name, directory))
              sandbox = SandboxRemote(context, project,
                                      directory,

buildstream/sandbox/_sandboxremote.py

@@ -173,8 +173,8 @@ class SandboxRemote(Sandbox):
          platform = Platform.get_platform()
          cascache = platform.artifactcache
          # Now, push that key (without necessarily needing a ref) to the remote.
 -        vdir_digest = cascache.push_directory(self._get_project(), upload_vdir)
 -        if not vdir_digest or not cascache.verify_digest_pushed(self._get_project(), vdir_digest):
 +        cascache.push_directory(self._get_project(), upload_vdir)
 +        if not cascache.verify_digest_pushed(self._get_project(), upload_vdir.ref):
              raise SandboxError("Failed to verify that source has been pushed to the remote artifact cache.")
          # Set up environment and working directory

buildstream/sandbox/sandbox.py

@@ -110,6 +110,10 @@ class Sandbox():
              os.makedirs(directory_, exist_ok=True)
          self._vdir = None
 +        # This is set if anyone requests access to the underlying
 +        # directory via get_directory.
 +        self._never_cache_vdirs = False
++
      def get_directory(self):
          """Fetches the sandbox root directory
@@ -122,24 +126,28 @@ class Sandbox():
          """
          if self.__allow_real_directory:
 +            self._never_cache_vdirs = True
              return self._root
          else:
              raise BstError("You can't use get_directory")
      def get_virtual_directory(self):
 -        """Fetches the sandbox root directory
 +        """Fetches the sandbox root directory as a virtual Directory.
          The root directory is where artifacts for the base
 -        runtime environment should be staged. Only works if
 -        BST_VIRTUAL_DIRECTORY is not set.
 +        runtime environment should be staged.
++
 +        Use caution if you use get_directory and
 +        get_virtual_directory.  If you alter the contents of the
 +        directory returned by get_directory, all objects returned by
 +        get_virtual_directory or derived from them are invalid and you
 +        must call get_virtual_directory again to get a new copy.
          Returns:
 -           (str): The sandbox root directory
 +           (Directory): The sandbox root directory
          """
 -        if not self._vdir:
 -            # BST_CAS_DIRECTORIES is a deliberately hidden environment variable which
 -            # can be used to switch on CAS-based directories for testing.
 +        if self._vdir is None or self._never_cache_vdirs:
              if 'BST_CAS_DIRECTORIES' in os.environ:
                  self._vdir = CasBasedDirectory(self.__context, ref=None)
              else:

setup.py

@@ -264,8 +264,9 @@ setup(name='BuildStream',
        license='LGPL',
        long_description=long_description,
        long_description_content_type='text/x-rst; charset=UTF-8',
 -      url='https://gitlab.com/BuildStream/buildstream',
 +      url='https://buildstream.build',
        project_urls={
 +          'Source': 'https://gitlab.com/BuildStream/buildstream',
            'Documentation': 'https://buildstream.gitlab.io/buildstream/',
            'Tracker': 'https://gitlab.com/BuildStream/buildstream/issues',
            'Mailing List': 'https://mail.gnome.org/mailman/listinfo/buildstream-list'

tests/artifactcache/push.py

@@ -228,9 +228,9 @@ def _test_push_directory(user_config_file, project_dir, artifact_dir, artifact_d
          directory = CasBasedDirectory(context, ref=artifact_digest)
          # Push the CasBasedDirectory object
 -        directory_digest = cas.push_directory(project, directory)
 +        cas.push_directory(project, directory)
 -        queue.put(directory_digest.hash)
 +        queue.put(directory.ref.hash)
      else:
          queue.put("No remote configured")

tests/frontend/project/elements/rebuild-target.bst

 +kind: compose
++
 +build-depends:
 +- target.bst

tests/frontend/project/elements/source-bundle/source-bundle-hello.bst

 +kind: import
 +description: the kind of this element must implement generate_script() method
++
 +sources:
 +- kind: local
 +  path: files/source-bundle

tests/frontend/project/files/source-bundle/llamas.txt

+llamas

tests/frontend/rebuild.py

 +import os
 +import pytest
 +from tests.testutils import cli
++
 +# Project directory
 +DATA_DIR = os.path.join(
 +    os.path.dirname(os.path.realpath(__file__)),
 +    "project",
 +)
++
++
 +def strict_args(args, strict):
 +    if strict != "strict":
 +        return ['--no-strict'] + args
 +    return args
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize("strict", ["strict", "non-strict"])
 +def test_rebuild(datafiles, cli, strict):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    checkout = os.path.join(cli.directory, 'checkout')
++
 +    # First build intermediate target.bst
 +    result = cli.run(project=project, args=strict_args(['build', 'target.bst'], strict))
 +    result.assert_success()
++
 +    # Modify base import
 +    with open(os.path.join(project, 'files', 'dev-files', 'usr', 'include', 'new.h'), "w") as f:
 +        f.write("#define NEW")
++
 +    # Rebuild base import and build top-level rebuild-target.bst
 +    # In non-strict mode, this does not rebuild intermediate target.bst,
 +    # which means that a weakly cached target.bst will be staged as dependency.
 +    result = cli.run(project=project, args=strict_args(['build', 'rebuild-target.bst'], strict))
 +    result.assert_success()

tests/frontend/source_bundle.py

 +#
 +#  Copyright (C) 2018 Bloomberg Finance LP
 +#
 +#  This program is free software; you can redistribute it and/or
 +#  modify it under the terms of the GNU Lesser General Public
 +#  License as published by the Free Software Foundation; either
 +#  version 2 of the License, or (at your option) any later version.
 +#
 +#  This library is distributed in the hope that it will be useful,
 +#  but WITHOUT ANY WARRANTY; without even the implied warranty of
 +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 +#  Lesser General Public License for more details.
 +#
 +#  You should have received a copy of the GNU Lesser General Public
 +#  License along with this library. If not, see <http://www.gnu.org/licenses/>.
 +#
 +#  Authors: Chandan Singh <csingh43 bloomberg net>
 +#
++
 +import os
 +import tarfile
++
 +import pytest
++
 +from tests.testutils import cli
++
 +# Project directory
 +DATA_DIR = os.path.join(
 +    os.path.dirname(os.path.realpath(__file__)),
 +    "project",
 +)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_source_bundle(cli, tmpdir, datafiles):
 +    project_path = os.path.join(datafiles.dirname, datafiles.basename)
 +    element_name = 'source-bundle/source-bundle-hello.bst'
 +    normal_name = 'source-bundle-source-bundle-hello'
++
 +    # Verify that we can correctly produce a source-bundle
 +    args = ['source-bundle', element_name, '--directory', str(tmpdir)]
 +    result = cli.run(project=project_path, args=args)
 +    result.assert_success()
++
 +    # Verify that the source-bundle contains our sources and a build script
 +    with tarfile.open(os.path.join(str(tmpdir), '{}.tar.gz'.format(normal_name))) as bundle:
 +        assert os.path.join(normal_name, 'source', normal_name, 'llamas.txt') in bundle.getnames()
 +        assert os.path.join(normal_name, 'build.sh') in bundle.getnames()

[Notes] [Git][BuildStream/buildstream][jmac/stop-caching-vdirs] 14 commits: element.py: Remove early warning and check directory is not None

Jim MacArthur pushed to branch jmac/stop-caching-vdirs at BuildStream / buildstream

Commits:

12 changed files:

Changes: