[Notes] [Git][BuildStream/buildstream][jonathan/wsl-tests-allow-fail] 17

Jonathan Maw pushed to branch jonathan/wsl-tests-allow-fail at BuildStream / buildstream

Commits:

3b889eb8

by Javier Jardón at 2019-02-15T20:22:08Z

README.rst: Add table with distros with packaged buildstream

[ci skip]

232906b1

by Javier Jardón at 2019-02-15T23:08:50Z

Merge branch 'jjardon/distros' into 'master'

README.rst: Add table with distros with packaged buildstream

See merge request BuildStream/buildstream!1143

ee2296ab

by Tristan Van Berkom at 2019-02-18T08:41:57Z

CONTRIBUTING.rst: Documenting the release process

So that other people can also make releases.

c79696fe

by Tristan Van Berkom at 2019-02-18T09:43:22Z

Merge branch 'tristan/document-release-process' into 'master'

CONTRIBUTING.rst: Documenting the release process

See merge request BuildStream/buildstream!1155

fb65af6c

by Jürg Billeter at 2019-02-18T10:29:25Z

_casbaseddirectory.py: Fix order in _recalculate_recursing_up()

The directory needs to be serialized after the Digest for the
subdirectory `caller` has been updated.

f5660fa0

by Jim MacArthur at 2019-02-18T10:29:25Z

_casbaseddirectory.py: Correct typo for _get_underlying_directory

cdcf0dc5

by Jürg Billeter at 2019-02-18T10:29:25Z

_casbaseddirectory.py: Remove unused find_root() method

f9dd6ea2
by Jim MacArthur at 2019-02-18T10:29:25Z
```
storage: Add Directory.get_size()
```

050249bb

by Jürg Billeter at 2019-02-18T10:29:25Z

_casbaseddirectory.py: Add _get_digest() method

3b881efc

by Jürg Billeter at 2019-02-18T10:29:25Z

element.py: Expand scope of timed_activity in _cache_artifact()

ArtifactCache.commit() is no longer the only expensive part and even
more so with the upcoming change to use CASBasedDirectory.

3832c0d1

by Jim MacArthur at 2019-02-18T10:29:25Z

element.py: Alter _cache_artifact to use CasBasedDirectory

ef85e3b2

by Jürg Billeter at 2019-02-18T11:18:10Z

Merge branch 'jmac/cache_artifacts_with_vdir' into 'master'

Cache artifacts with virtual directories instead of filesystem.

Closes #787

See merge request BuildStream/buildstream!991

e0f0a01d

by Chandan Singh at 2019-02-18T12:41:31Z

requirements/Makefile: exclude pkg-resources

Having `pkg-resources` in requirements files is never the right thing as
it is not really a package that one can install. This is basically an
artifact of Debian-based systems providing incorrect metadata to pip.
See [this pip issue](https://github.com/pypa/pip/issues/4022) and
[this Ubuntu issue](https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1635463)
for more background.

f149fed2

by Tristan Van Berkom at 2019-02-18T13:43:28Z

Merge branch 'chandan/remove-pkg-resources' into 'master'

requirements/Makefile: exclude pkg-resources

Closes #918

See merge request BuildStream/buildstream!1156

4022234e

by Tristan Van Berkom at 2019-02-18T13:44:29Z

.gitlab-ci.yml: Some new test environments were added, but not collecting coverage.

The test jobs which create coverage reports need to be listed
in the dependencies of the coverage job, as highlighted in
issue #356.

86466e7e

by Tristan Van Berkom at 2019-02-18T14:27:08Z

Merge branch 'tristan/collect-missing-coverage' into 'master'

.gitlab-ci.yml: Some new test environments were added, but not collecting coverage.

See merge request BuildStream/buildstream!1157

c8cd24b9

by Jonathan Maw at 2019-02-18T17:37:00Z

.gitlab-ci: make wsl-test automatic, allow failure

9 changed files:

.gitlab-ci.yml
CONTRIBUTING.rst
README.rst
buildstream/_artifactcache.py
buildstream/element.py
buildstream/storage/_casbaseddirectory.py
buildstream/storage/_filebaseddirectory.py
buildstream/storage/directory.py
requirements/Makefile

Changes:

.gitlab-ci.yml

@@ -160,7 +160,7 @@ tests-wsl:
    script:
    - "${TEST_COMMAND}"
 -  when: manual
 +  allow_failure: true
  # Automatically build documentation for every commit, we want to know
  # if building documentation fails even if we're not deploying it.
@@ -281,10 +281,12 @@ coverage:
      - tox -e coverage
      - cp -a .coverage-reports/ ./coverage-report
    dependencies:
 +  - tests-centos-7.6
    - tests-debian-9
    - tests-fedora-28
    - tests-fedora-29
    - tests-fedora-missing-deps
 +  - tests-fedora-update-deps
    - tests-ubuntu-18.04
    - tests-unix
    except:

CONTRIBUTING.rst

@@ -1234,6 +1234,8 @@ This will give you a ``doc/build/html`` directory with the html docs which
  you can view in your browser locally to test.
 +.. _contributing_session_html:
++
  Regenerating session html
  '''''''''''''''''''''''''
  The documentation build will build the session files if they are missing,
@@ -1252,6 +1254,8 @@ To force rebuild session html while building the doc, simply run `tox` with the
    env BST_FORCE_SESSION_REBUILD=1 tox -e docs
 +.. _contributing_man_pages:
++
  Man pages
  ~~~~~~~~~
  Unfortunately it is quite difficult to integrate the man pages build
@@ -1779,3 +1783,257 @@ changing the ``.in`` file, run the following to update the matching ``.txt``
  file::
     make -C requirements
++
++
 +Making releases
 +---------------
 +This is a checklist of activities which must be observed when creating
 +BuildStream releases, it is important to keep this section up to date
 +whenever the release process changes.
++
++
 +Requirements
 +~~~~~~~~~~~~
 +There are a couple of requirements and accounts required in order
 +to publish a release.
++
 +* Ability to send email to ``buildstream-list gnome org`` and
 +  to ``gnome-announce-list gnome org``.
++
 +* Shell account at ``master.gnome.org``.
++
 +* Access to the `BuildStream project on PyPI <https://pypi.org/project/BuildStream/>`_
++
 +* An email client which still knows how to send emails in plain text.
++
++
 +Pre-release changes
 +~~~~~~~~~~~~~~~~~~~
 +Before actually rolling the release, here is a list of changes which
 +might need to be done in preparation of the release.
++
 +* Ensure that the man pages are up to date
++
 +  The man pages are committed to the repository because we are
 +  currently unable to integrate this generation into the setuptools
 +  build phase, as outlined in issue #8.
++
 +  If any of the user facing CLI has changed, or if any of the
 +  related docstrings have changed, then you should
 +  :ref:`regenerate the man pages <contributing_man_pages>` and
 +  add/commit the results before wrapping a release.
++
 +* Ensure the documentation session HTML is up to date
++
 +  The session HTML files are committed to the repository for multiple
 +  reasons, one of them being that the documentation must be buildable
 +  from within a release build environment so that downstream distribution
 +  packagers can easily create the docs package.
++
 +  This is currently only needed for the first stable release
 +  in a stable line of releases, after this point the API is frozen
 +  and will not change for the remainder of the stable release lifetime,
 +  so nothing interesting will have changed in these session files.
++
 +  If regeneration is needed, follow :ref:`the instructions above <contributing_session_html>`.
++
 +* Ensure the NEWS entry is up to date and ready
++
 +  For a stable release where features have not been added, we
 +  should at least add some entries about the issues which have
 +  been fixed since the last stable release.
++
 +  For development releases, it is worthwhile going over the
 +  existing entries and ensuring all the major feature additions
 +  are mentioned and there are no redundancies.
++
 +* Push pre-release changes
++
 +  Now that any final pre-release changes to generated files or NEWS have
 +  been made, push these directly to the upstream repository.
++
 +  Do not sit around waiting for CI or approval, these superficial changes
 +  do not affect CI and you are intended to push these changes directly
 +  to the upstream repository.
++
++
 +Release process
 +~~~~~~~~~~~~~~~
++
 +* Ensure that the latest commit is passing in CI
++
 +  Of course, we do not release software which does not pass it's own
 +  tests.
++
 +* Get the list of contributors
++
 +  The list of contributors for a given list is a list of
 +  any contributors who have landed any patches since the
 +  last release.
++
 +  An easy way to get this list is to ask git to summarize
 +  the authors of commits since the *last release tag*. For
 +  example, if we are about to create the ``1.1.1`` release, then
 +  we need to observe all of the commits since the ``1.1.0``
 +  release:
++
 +  .. code:: shell
++
 +     git shortlog -s 1.1.0...@
++
 +  At times, the same contributor might make contributions from different
 +  machines which they have setup their author names differently, you
 +  can see that some of the authors are actually duplicates, then
 +  remove the duplicates.
++
 +* Start composing the release announcement email
++
 +  The first thing to do when composing the release email is to
 +  ensure your mail client has disabled any HTML formatting and will
 +  safely use plain text only.
++
 +  Try to make the release announcement consistent with other release
 +  announcements as much as possible, an example of the email
 +  can be `found here <https://mail.gnome.org/archives/buildstream-list/2019-February/msg00039.html>`_.
++
 +  The recipients of the email are ``buildstream-list gnome org`` and
 +  ``gnome-announce-list gnome org`` and the title of the email should
 +  be of the form: ``BuildStream 1.1.1 released``, without any exclamation point.
++
 +  The format of the email is essentially::
++
 +    Hi all,
++
 +    This is the personalized message written to you about this
 +    release.
++
 +    If this is an unstable release, this should include a warning
 +    to this effect and an invitation to users to please help us
 +    test this release.
++
 +    This is also a good place to highlight specific bug fixes which
 +    users may have been waiting for, or highlight a new feature we
 +    want users to try out.
++
++
 +    What is BuildStream ?
 +    =====================
 +    This is a concise blurb which describes BuildStream in a couple of
 +    sentences, and is taken from the the README.rst.
++
 +    The easiest thing is to just copy this over from the last release email.
++
++
 +    =================
 +    buildstream 1.1.1
 +    =================
 +    This section is directly copy pasted from the top of the NEWS file
++
++
 +    Contributors
 +    ============
 +     - This is Where
 +     - You Put
 +     - The Contributor
 +     - Names Which
 +     - You Extracted
 +     - Using git shortlog -s
++
++
 +    Where can I get it ?
 +    ====================
 +    https://download.gnome.org/sources/BuildStream/1.1/
++
 +    For more information on the BuildStream project, visit our home page
 +    at https://buildstream.build/
++
 +* Publish the release tag
++
 +  Now that any pre-release changes are upstream, create and push the
 +  signed release tag like so:
++
 +  .. code:: shell
++
 +     git tag -s 1.1.1
 +     git push origin 1.1.1
++
 +* Upload the release tarball
++
 +  First get yourself into a clean repository state, ensure that you
 +  don't have any unfinished work or precious, uncommitted files lying
 +  around in your checkout and then run:
++
 +  .. code:: shell
++
 +     git clean -xdff
++
 +  Create the tarball with the following command:
++
 +  .. code:: shell
++
 +     python3 setup.py sdist
++
 +  And upload the resulting tarball to the master GNOME server:
++
 +  .. code:: shell
++
 +     scp dist/BuildStream-1.1.1.tar.gz <user>@master.gnome.org:
++
 +  And finally login to your account at ``master.gnome.org`` and run
 +  the install scripts to publish the tarball and update the mirrors:
++
 +  .. code:: shell
++
 +     ftpadmin install BuildStream-1.1.1.tar.gz
++
 +* Send the release email
++
 +  Now that the release tag is up and the tarball is published,
 +  you can send the release email.
++
++
 +Post-release activities
 +~~~~~~~~~~~~~~~~~~~~~~~
 +Once the release has been published, there are some activities
 +which need to be done to ensure everything is up to date.
++
 +* If this is a stable release, then the tarball should also be
 +  uploaded to PyPI.
++
 +  Make sure you have ``twine`` installed and upload the tarball
 +  like so:
++
 +  .. code:: shell
++
 +     pip3 install --user twine
 +     twine upload -r pypi dist/BuildStream-1.0.1.tar.gz
++
 +* Update the topic line in the #buildstream IRC channel if needed
++
 +  The IRC channel usually advertizes the latest stable release
 +  in the topic line, now is the right time to update it.
++
 +* Update the website repository
++
 +  The website wants to link to release announcements, but this
 +  cannot be automated because we cannot guess what the link to
 +  the release email will be in the mailing list archive.
++
 +  Find the URL to the announcement you just published
 +  `in the mailing list archives <https://mail.gnome.org/archives/buildstream-list/>`_,
 +  and use that URL to update the ``anouncements.json`` file in the website
 +  repository.
++
 +  Commit and push this change to the the ``anouncements.json`` file to
 +  the upstream website repository, and gitlab will take care of automatically
 +  updating the website accordingly.
++
 +* Regenerate BuildStream documentation
++
 +  In order to update the badges we use in various documentation
 +  which reflects what is the latest stable releases and the latest
 +  development snapshots, we simply need to ensure a pipeline runs
 +  for the master branch in the BuildStream repository.
++
 +  You can do this by using the "Run Pipeline" feature on the
 +  `pipelines page in the gitlab UI <https://gitlab.com/BuildStream/buildstream/pipelines>`_.

README.rst

@@ -100,3 +100,9 @@ We also recommend exploring some existing BuildStream projects:
  * https://gitlab.com/baserock/definitions
  If you have any questions please ask on our `#buildstream <irc://irc.gnome.org/buildstream>`_ channel in `irc.gnome.org <irc://irc.gnome.org>`_
++
++
 +Availability in distros
 +=======================
 +.. image:: https://repology.org/badge/vertical-allrepos/buildstream.svg
 +   :target: https://repology.org/metapackage/buildstream/versions

buildstream/_artifactcache.py

@@ -588,13 +588,16 @@ class ArtifactCache():
+     #
      # Args:
      #     element (Element): The Element commit an artifact for
 -    #     content (str): The element's content directory
 +    #     content (Directory): The element's content directory
      #     keys (list): The cache keys to use
+     #
      def commit(self, element, content, keys):
          refs = [element.get_artifact_name(key) for key in keys]
 -        self.cas.commit(refs, content)
 +        tree = content._get_digest()
++
 +        for ref in refs:
 +            self.cas.set_ref(ref, tree)
      # diff():
+     #

buildstream/element.py

@@ -103,6 +103,7 @@ from .types import _KeyStrength, CoreWarnings
  from .storage.directory import Directory
  from .storage._filebaseddirectory import FileBasedDirectory
 +from .storage._casbaseddirectory import CasBasedDirectory
  from .storage.directory import VirtualDirectoryError
@@ -1670,106 +1671,109 @@ class Element(Plugin):
                      cleanup_rootdir()
      def _cache_artifact(self, rootdir, sandbox, collect):
 -        if collect is not None:
 -            try:
 -                sandbox_vroot = sandbox.get_virtual_directory()
 -                collectvdir = sandbox_vroot.descend(collect.lstrip(os.sep).split(os.sep))
 -            except VirtualDirectoryError:
 -                # No collect directory existed
 -                collectvdir = None
 +        with self.timed_activity("Caching artifact"):
 +            if collect is not None:
 +                try:
 +                    sandbox_vroot = sandbox.get_virtual_directory()
 +                    collectvdir = sandbox_vroot.descend(collect.lstrip(os.sep).split(os.sep))
 +                except VirtualDirectoryError:
 +                    # No collect directory existed
 +                    collectvdir = None
 -        context = self._get_context()
 +            context = self._get_context()
 -        # Create artifact directory structure
 -        assembledir = os.path.join(rootdir, 'artifact')
 -        filesdir = os.path.join(assembledir, 'files')
 -        logsdir = os.path.join(assembledir, 'logs')
 -        metadir = os.path.join(assembledir, 'meta')
 -        buildtreedir = os.path.join(assembledir, 'buildtree')
 -        os.mkdir(assembledir)
 -        if collect is not None and collectvdir is not None:
 -            os.mkdir(filesdir)
 -        os.mkdir(logsdir)
 -        os.mkdir(metadir)
 -        os.mkdir(buildtreedir)
+-
 -        # Hard link files from collect dir to files directory
 -        if collect is not None and collectvdir is not None:
 -            collectvdir.export_files(filesdir, can_link=True)
+-
 -        cache_buildtrees = context.cache_buildtrees
 -        build_success = self.__build_result[0]
+-
 -        # cache_buildtrees defaults to 'always', as such the
 -        # default behaviour is to attempt to cache them. If only
 -        # caching failed artifact buildtrees, then query the build
 -        # result. Element types without a build-root dir will be cached
 -        # with an empty buildtreedir regardless of this configuration.
+-
 -        if cache_buildtrees == 'always' or (cache_buildtrees == 'failure' and not build_success):
 -            try:
 +            assemblevdir = CasBasedDirectory(cas_cache=context.artifactcache.cas, ref=None)
 +            logsvdir = assemblevdir.descend("logs", create=True)
 +            metavdir = assemblevdir.descend("meta", create=True)
 +            buildtreevdir = assemblevdir.descend("buildtree", create=True)
++
 +            # Create artifact directory structure
 +            assembledir = os.path.join(rootdir, 'artifact')
 +            logsdir = os.path.join(assembledir, 'logs')
 +            metadir = os.path.join(assembledir, 'meta')
 +            os.mkdir(assembledir)
 +            os.mkdir(logsdir)
 +            os.mkdir(metadir)
++
 +            if collect is not None and collectvdir is not None:
 +                filesvdir = assemblevdir.descend("files", create=True)
 +                filesvdir.import_files(collectvdir)
++
 +            cache_buildtrees = context.cache_buildtrees
 +            build_success = self.__build_result[0]
++
 +            # cache_buildtrees defaults to 'always', as such the
 +            # default behaviour is to attempt to cache them. If only
 +            # caching failed artifact buildtrees, then query the build
 +            # result. Element types without a build-root dir will be cached
 +            # with an empty buildtreedir regardless of this configuration.
++
 +            if cache_buildtrees == 'always' or (cache_buildtrees == 'failure' and not build_success):
                  sandbox_vroot = sandbox.get_virtual_directory()
 -                sandbox_build_dir = sandbox_vroot.descend(
 -                    self.get_variable('build-root').lstrip(os.sep).split(os.sep))
 -                # Hard link files from build-root dir to buildtreedir directory
 -                sandbox_build_dir.export_files(buildtreedir)
 -            except VirtualDirectoryError:
 -                # Directory could not be found. Pre-virtual
 -                # directory behaviour was to continue silently
 -                # if the directory could not be found.
 -                pass
 +                try:
 +                    sandbox_build_dir = sandbox_vroot.descend(
 +                        self.get_variable('build-root').lstrip(os.sep).split(os.sep))
 +                    buildtreevdir.import_files(sandbox_build_dir)
 +                except VirtualDirectoryError:
 +                    # Directory could not be found. Pre-virtual
 +                    # directory behaviour was to continue silently
 +                    # if the directory could not be found.
 +                    pass
++
 +            # Write some logs out to normal directories: logsdir and metadir
 +            # Copy build log
 +            log_filename = context.get_log_filename()
 +            self._build_log_path = os.path.join(logsdir, 'build.log')
 +            if log_filename:
 +                shutil.copyfile(log_filename, self._build_log_path)
++
 +            # Store public data
 +            _yaml.dump(_yaml.node_sanitize(self.__dynamic_public), os.path.join(metadir, 'public.yaml'))
++
 +            # Store result
 +            build_result_dict = {"success": self.__build_result[0], "description": self.__build_result[1]}
 +            if self.__build_result[2] is not None:
 +                build_result_dict["detail"] = self.__build_result[2]
 +            _yaml.dump(build_result_dict, os.path.join(metadir, 'build-result.yaml'))
++
 +            # ensure we have cache keys
 +            self._assemble_done()
++
 +            # Store keys.yaml
 +            _yaml.dump(_yaml.node_sanitize({
 +                'strong': self._get_cache_key(),
 +                'weak': self._get_cache_key(_KeyStrength.WEAK),
 +            }), os.path.join(metadir, 'keys.yaml'))
++
 +            # Store dependencies.yaml
 +            _yaml.dump(_yaml.node_sanitize({
 +                e.name: e._get_cache_key() for e in self.dependencies(Scope.BUILD)
 +            }), os.path.join(metadir, 'dependencies.yaml'))
++
 +            # Store workspaced.yaml
 +            _yaml.dump(_yaml.node_sanitize({
 +                'workspaced': bool(self._get_workspace())
 +            }), os.path.join(metadir, 'workspaced.yaml'))
++
 +            # Store workspaced-dependencies.yaml
 +            _yaml.dump(_yaml.node_sanitize({
 +                'workspaced-dependencies': [
 +                    e.name for e in self.dependencies(Scope.BUILD)
 +                    if e._get_workspace()
 +                ]
 +            }), os.path.join(metadir, 'workspaced-dependencies.yaml'))
 -        # Copy build log
 -        log_filename = context.get_log_filename()
 -        self._build_log_path = os.path.join(logsdir, 'build.log')
 -        if log_filename:
 -            shutil.copyfile(log_filename, self._build_log_path)
+-
 -        # Store public data
 -        _yaml.dump(_yaml.node_sanitize(self.__dynamic_public), os.path.join(metadir, 'public.yaml'))
+-
 -        # Store result
 -        build_result_dict = {"success": self.__build_result[0], "description": self.__build_result[1]}
 -        if self.__build_result[2] is not None:
 -            build_result_dict["detail"] = self.__build_result[2]
 -        _yaml.dump(build_result_dict, os.path.join(metadir, 'build-result.yaml'))
+-
 -        # ensure we have cache keys
 -        self._assemble_done()
+-
 -        # Store keys.yaml
 -        _yaml.dump(_yaml.node_sanitize({
 -            'strong': self._get_cache_key(),
 -            'weak': self._get_cache_key(_KeyStrength.WEAK),
 -        }), os.path.join(metadir, 'keys.yaml'))
+-
 -        # Store dependencies.yaml
 -        _yaml.dump(_yaml.node_sanitize({
 -            e.name: e._get_cache_key() for e in self.dependencies(Scope.BUILD)
 -        }), os.path.join(metadir, 'dependencies.yaml'))
+-
 -        # Store workspaced.yaml
 -        _yaml.dump(_yaml.node_sanitize({
 -            'workspaced': bool(self._get_workspace())
 -        }), os.path.join(metadir, 'workspaced.yaml'))
+-
 -        # Store workspaced-dependencies.yaml
 -        _yaml.dump(_yaml.node_sanitize({
 -            'workspaced-dependencies': [
 -                e.name for e in self.dependencies(Scope.BUILD)
 -                if e._get_workspace()
 -            ]
 -        }), os.path.join(metadir, 'workspaced-dependencies.yaml'))
 +            metavdir.import_files(metadir)
 +            logsvdir.import_files(logsdir)
 -        with self.timed_activity("Caching artifact"):
 -            artifact_size = utils._get_dir_size(assembledir)
 -            self.__artifacts.commit(self, assembledir, self.__get_cache_keys_for_commit())
+-
 -        if collect is not None and collectvdir is None:
 -            raise ElementError(
 -                "Directory '{}' was not found inside the sandbox, "
 -                "unable to collect artifact contents"
 -                .format(collect))
 +            artifact_size = assemblevdir.get_size()
 +            self.__artifacts.commit(self, assemblevdir, self.__get_cache_keys_for_commit())
++
 +            if collect is not None and collectvdir is None:
 +                raise ElementError(
 +                    "Directory '{}' was not found inside the sandbox, "
 +                    "unable to collect artifact contents"
 +                    .format(collect))
          return artifact_size

buildstream/storage/_casbaseddirectory.py

@@ -136,10 +136,10 @@ class CasBasedDirectory(Directory):
          the parent).
          """
 -        self.ref = self.cas_cache.add_object(buffer=self.pb2_directory.SerializeToString())
          if caller:
              old_dir = self._find_pb2_entry(caller.filename)
              self.cas_cache.add_object(digest=old_dir.digest, buffer=caller.pb2_directory.SerializeToString())
 +        self.ref = self.cas_cache.add_object(buffer=self.pb2_directory.SerializeToString())
          if self.parent:
              self.parent._recalculate_recursing_up(self)
@@ -277,14 +277,6 @@ class CasBasedDirectory(Directory):
                                                           directory_list))
          return None
 -    def find_root(self):
 -        """ Finds the root of this directory tree by following 'parent' until there is
 -        no parent. """
 -        if self.parent:
 -            return self.parent.find_root()
 -        else:
 -            return self
+-
      def _check_replacement(self, name, path_prefix, fileListResult):
          """ Checks whether 'name' exists, and if so, whether we can overwrite it.
          If we can, add the name to 'overwritten_files' and delete the existing entry.
@@ -451,7 +443,7 @@ class CasBasedDirectory(Directory):
                  files = external_pathspec.list_relative_paths()
          if isinstance(external_pathspec, FileBasedDirectory):
 -            source_directory = external_pathspec.get_underlying_directory()
 +            source_directory = external_pathspec._get_underlying_directory()
              result = self._import_files_from_directory(source_directory, files=files)
          elif isinstance(external_pathspec, str):
              source_directory = external_pathspec
@@ -635,6 +627,18 @@ class CasBasedDirectory(Directory):
          self._recalculate_recursing_up()
          self._recalculate_recursing_down()
 +    def get_size(self):
 +        total = len(self.pb2_directory.SerializeToString())
 +        for i in self.index.values():
 +            if isinstance(i.buildstream_object, CasBasedDirectory):
 +                total += i.buildstream_object.get_size()
 +            elif isinstance(i.pb_object, remote_execution_pb2.FileNode):
 +                src_name = self.cas_cache.objpath(i.pb_object.digest)
 +                filesize = os.stat(src_name).st_size
 +                total += filesize
 +            # Symlink nodes are encoded as part of the directory serialization.
 +        return total
++
      def _get_identifier(self):
          path = ""
          if self.parent:
@@ -653,3 +657,15 @@ class CasBasedDirectory(Directory):
          throw an exception. """
          raise VirtualDirectoryError("_get_underlying_directory was called on a CAS-backed directory," +
                                      " which has no underlying directory.")
++
 +    # _get_digest():
 +    #
 +    # Return the Digest for this directory.
 +    #
 +    # Returns:
 +    #   (Digest): The Digest protobuf object for the Directory protobuf
 +    #
 +    def _get_digest(self):
 +        if not self.ref:
 +            self.ref = self.cas_cache.add_object(buffer=self.pb2_directory.SerializeToString())
 +        return self.ref

buildstream/storage/_filebaseddirectory.py

@@ -30,6 +30,7 @@ See also: :ref:`sandboxing`.
  import os
  import time
  from .directory import Directory, VirtualDirectoryError
 +from .. import utils
  from ..utils import link_files, copy_files, list_relative_paths, _get_link_mtime, _magic_timestamp
  from ..utils import _set_deterministic_user, _set_deterministic_mtime
@@ -201,6 +202,9 @@ class FileBasedDirectory(Directory):
          return list_relative_paths(self.external_directory)
 +    def get_size(self):
 +        return utils._get_dir_size(self.external_directory)
++
      def __str__(self):
          # This returns the whole path (since we don't know where the directory started)
          # which exposes the sandbox directory; we will have to assume for the time being

buildstream/storage/directory.py

@@ -177,3 +177,9 @@ class Directory():
          """
          raise NotImplementedError()
++
 +    def get_size(self):
 +        """ Get an approximation of the storage space in bytes used by this directory
 +        and all files and subdirectories in it. Storage space varies by implementation
 +        and effective space used may be lower than this number due to deduplication. """
 +        raise NotImplementedError()

requirements/Makefile

@@ -17,5 +17,5 @@ all: $(REQUIREMENTS_TXT)
  	$(eval VENVDIR := $(shell mktemp -d $(CURDIR)/.bst-venv.XXXXXX))
  	$(VENV) $(VENVDIR)
  	$(VENV_PIP) install -r $^
 -	$(VENV_PIP) freeze -r $^ > $@
 +	$(VENV_PIP) freeze -r $^ | grep -v pkg-resources > $@
  	rm -rf $(VENVDIR)

[Notes] [Git][BuildStream/buildstream][jonathan/wsl-tests-allow-fail] 17 commits: README.rst: Add table with distros with packaged buildstream

Jonathan Maw pushed to branch jonathan/wsl-tests-allow-fail at BuildStream / buildstream

Commits:

9 changed files:

Changes: