[Notes] [Git][BuildStream/buildstream][jmac/cache_artifacts_with

Jürg Billeter pushed to branch jmac/cache_artifacts_with_vdir at BuildStream / buildstream

Commits:

ee2296ab

by Tristan Van Berkom at 2019-02-18T08:41:57Z

CONTRIBUTING.rst: Documenting the release process

So that other people can also make releases.

c79696fe

by Tristan Van Berkom at 2019-02-18T09:43:22Z

Merge branch 'tristan/document-release-process' into 'master'

CONTRIBUTING.rst: Documenting the release process

See merge request BuildStream/buildstream!1155

fb65af6c

by Jürg Billeter at 2019-02-18T10:29:25Z

_casbaseddirectory.py: Fix order in _recalculate_recursing_up()

The directory needs to be serialized after the Digest for the
subdirectory `caller` has been updated.

f5660fa0

by Jim MacArthur at 2019-02-18T10:29:25Z

_casbaseddirectory.py: Correct typo for _get_underlying_directory

cdcf0dc5

by Jürg Billeter at 2019-02-18T10:29:25Z

_casbaseddirectory.py: Remove unused find_root() method

f9dd6ea2
by Jim MacArthur at 2019-02-18T10:29:25Z
```
storage: Add Directory.get_size()
```

050249bb

by Jürg Billeter at 2019-02-18T10:29:25Z

_casbaseddirectory.py: Add _get_digest() method

3b881efc

by Jürg Billeter at 2019-02-18T10:29:25Z

element.py: Expand scope of timed_activity in _cache_artifact()

ArtifactCache.commit() is no longer the only expensive part and even
more so with the upcoming change to use CASBasedDirectory.

3832c0d1

by Jim MacArthur at 2019-02-18T10:29:25Z

element.py: Alter _cache_artifact to use CasBasedDirectory

6 changed files:

CONTRIBUTING.rst
buildstream/_artifactcache.py
buildstream/element.py
buildstream/storage/_casbaseddirectory.py
buildstream/storage/_filebaseddirectory.py
buildstream/storage/directory.py

Changes:

CONTRIBUTING.rst

@@ -1234,6 +1234,8 @@ This will give you a ``doc/build/html`` directory with the html docs which
  you can view in your browser locally to test.
 +.. _contributing_session_html:
++
  Regenerating session html
  '''''''''''''''''''''''''
  The documentation build will build the session files if they are missing,
@@ -1252,6 +1254,8 @@ To force rebuild session html while building the doc, simply run `tox` with the
    env BST_FORCE_SESSION_REBUILD=1 tox -e docs
 +.. _contributing_man_pages:
++
  Man pages
  ~~~~~~~~~
  Unfortunately it is quite difficult to integrate the man pages build
@@ -1779,3 +1783,257 @@ changing the ``.in`` file, run the following to update the matching ``.txt``
  file::
     make -C requirements
++
++
 +Making releases
 +---------------
 +This is a checklist of activities which must be observed when creating
 +BuildStream releases, it is important to keep this section up to date
 +whenever the release process changes.
++
++
 +Requirements
 +~~~~~~~~~~~~
 +There are a couple of requirements and accounts required in order
 +to publish a release.
++
 +* Ability to send email to ``buildstream-list gnome org`` and
 +  to ``gnome-announce-list gnome org``.
++
 +* Shell account at ``master.gnome.org``.
++
 +* Access to the `BuildStream project on PyPI <https://pypi.org/project/BuildStream/>`_
++
 +* An email client which still knows how to send emails in plain text.
++
++
 +Pre-release changes
 +~~~~~~~~~~~~~~~~~~~
 +Before actually rolling the release, here is a list of changes which
 +might need to be done in preparation of the release.
++
 +* Ensure that the man pages are up to date
++
 +  The man pages are committed to the repository because we are
 +  currently unable to integrate this generation into the setuptools
 +  build phase, as outlined in issue #8.
++
 +  If any of the user facing CLI has changed, or if any of the
 +  related docstrings have changed, then you should
 +  :ref:`regenerate the man pages <contributing_man_pages>` and
 +  add/commit the results before wrapping a release.
++
 +* Ensure the documentation session HTML is up to date
++
 +  The session HTML files are committed to the repository for multiple
 +  reasons, one of them being that the documentation must be buildable
 +  from within a release build environment so that downstream distribution
 +  packagers can easily create the docs package.
++
 +  This is currently only needed for the first stable release
 +  in a stable line of releases, after this point the API is frozen
 +  and will not change for the remainder of the stable release lifetime,
 +  so nothing interesting will have changed in these session files.
++
 +  If regeneration is needed, follow :ref:`the instructions above <contributing_session_html>`.
++
 +* Ensure the NEWS entry is up to date and ready
++
 +  For a stable release where features have not been added, we
 +  should at least add some entries about the issues which have
 +  been fixed since the last stable release.
++
 +  For development releases, it is worthwhile going over the
 +  existing entries and ensuring all the major feature additions
 +  are mentioned and there are no redundancies.
++
 +* Push pre-release changes
++
 +  Now that any final pre-release changes to generated files or NEWS have
 +  been made, push these directly to the upstream repository.
++
 +  Do not sit around waiting for CI or approval, these superficial changes
 +  do not affect CI and you are intended to push these changes directly
 +  to the upstream repository.
++
++
 +Release process
 +~~~~~~~~~~~~~~~
++
 +* Ensure that the latest commit is passing in CI
++
 +  Of course, we do not release software which does not pass it's own
 +  tests.
++
 +* Get the list of contributors
++
 +  The list of contributors for a given list is a list of
 +  any contributors who have landed any patches since the
 +  last release.
++
 +  An easy way to get this list is to ask git to summarize
 +  the authors of commits since the *last release tag*. For
 +  example, if we are about to create the ``1.1.1`` release, then
 +  we need to observe all of the commits since the ``1.1.0``
 +  release:
++
 +  .. code:: shell
++
 +     git shortlog -s 1.1.0...@
++
 +  At times, the same contributor might make contributions from different
 +  machines which they have setup their author names differently, you
 +  can see that some of the authors are actually duplicates, then
 +  remove the duplicates.
++
 +* Start composing the release announcement email
++
 +  The first thing to do when composing the release email is to
 +  ensure your mail client has disabled any HTML formatting and will
 +  safely use plain text only.
++
 +  Try to make the release announcement consistent with other release
 +  announcements as much as possible, an example of the email
 +  can be `found here <https://mail.gnome.org/archives/buildstream-list/2019-February/msg00039.html>`_.
++
 +  The recipients of the email are ``buildstream-list gnome org`` and
 +  ``gnome-announce-list gnome org`` and the title of the email should
 +  be of the form: ``BuildStream 1.1.1 released``, without any exclamation point.
++
 +  The format of the email is essentially::
++
 +    Hi all,
++
 +    This is the personalized message written to you about this
 +    release.
++
 +    If this is an unstable release, this should include a warning
 +    to this effect and an invitation to users to please help us
 +    test this release.
++
 +    This is also a good place to highlight specific bug fixes which
 +    users may have been waiting for, or highlight a new feature we
 +    want users to try out.
++
++
 +    What is BuildStream ?
 +    =====================
 +    This is a concise blurb which describes BuildStream in a couple of
 +    sentences, and is taken from the the README.rst.
++
 +    The easiest thing is to just copy this over from the last release email.
++
++
 +    =================
 +    buildstream 1.1.1
 +    =================
 +    This section is directly copy pasted from the top of the NEWS file
++
++
 +    Contributors
 +    ============
 +     - This is Where
 +     - You Put
 +     - The Contributor
 +     - Names Which
 +     - You Extracted
 +     - Using git shortlog -s
++
++
 +    Where can I get it ?
 +    ====================
 +    https://download.gnome.org/sources/BuildStream/1.1/
++
 +    For more information on the BuildStream project, visit our home page
 +    at https://buildstream.build/
++
 +* Publish the release tag
++
 +  Now that any pre-release changes are upstream, create and push the
 +  signed release tag like so:
++
 +  .. code:: shell
++
 +     git tag -s 1.1.1
 +     git push origin 1.1.1
++
 +* Upload the release tarball
++
 +  First get yourself into a clean repository state, ensure that you
 +  don't have any unfinished work or precious, uncommitted files lying
 +  around in your checkout and then run:
++
 +  .. code:: shell
++
 +     git clean -xdff
++
 +  Create the tarball with the following command:
++
 +  .. code:: shell
++
 +     python3 setup.py sdist
++
 +  And upload the resulting tarball to the master GNOME server:
++
 +  .. code:: shell
++
 +     scp dist/BuildStream-1.1.1.tar.gz <user>@master.gnome.org:
++
 +  And finally login to your account at ``master.gnome.org`` and run
 +  the install scripts to publish the tarball and update the mirrors:
++
 +  .. code:: shell
++
 +     ftpadmin install BuildStream-1.1.1.tar.gz
++
 +* Send the release email
++
 +  Now that the release tag is up and the tarball is published,
 +  you can send the release email.
++
++
 +Post-release activities
 +~~~~~~~~~~~~~~~~~~~~~~~
 +Once the release has been published, there are some activities
 +which need to be done to ensure everything is up to date.
++
 +* If this is a stable release, then the tarball should also be
 +  uploaded to PyPI.
++
 +  Make sure you have ``twine`` installed and upload the tarball
 +  like so:
++
 +  .. code:: shell
++
 +     pip3 install --user twine
 +     twine upload -r pypi dist/BuildStream-1.0.1.tar.gz
++
 +* Update the topic line in the #buildstream IRC channel if needed
++
 +  The IRC channel usually advertizes the latest stable release
 +  in the topic line, now is the right time to update it.
++
 +* Update the website repository
++
 +  The website wants to link to release announcements, but this
 +  cannot be automated because we cannot guess what the link to
 +  the release email will be in the mailing list archive.
++
 +  Find the URL to the announcement you just published
 +  `in the mailing list archives <https://mail.gnome.org/archives/buildstream-list/>`_,
 +  and use that URL to update the ``anouncements.json`` file in the website
 +  repository.
++
 +  Commit and push this change to the the ``anouncements.json`` file to
 +  the upstream website repository, and gitlab will take care of automatically
 +  updating the website accordingly.
++
 +* Regenerate BuildStream documentation
++
 +  In order to update the badges we use in various documentation
 +  which reflects what is the latest stable releases and the latest
 +  development snapshots, we simply need to ensure a pipeline runs
 +  for the master branch in the BuildStream repository.
++
 +  You can do this by using the "Run Pipeline" feature on the
 +  `pipelines page in the gitlab UI <https://gitlab.com/BuildStream/buildstream/pipelines>`_.

buildstream/_artifactcache.py

@@ -588,13 +588,16 @@ class ArtifactCache():
+     #
      # Args:
      #     element (Element): The Element commit an artifact for
 -    #     content (str): The element's content directory
 +    #     content (Directory): The element's content directory
      #     keys (list): The cache keys to use
+     #
      def commit(self, element, content, keys):
          refs = [element.get_artifact_name(key) for key in keys]
 -        self.cas.commit(refs, content)
 +        tree = content._get_digest()
++
 +        for ref in refs:
 +            self.cas.set_ref(ref, tree)
      # diff():
+     #

buildstream/element.py

@@ -103,6 +103,7 @@ from .types import _KeyStrength, CoreWarnings
  from .storage.directory import Directory
  from .storage._filebaseddirectory import FileBasedDirectory
 +from .storage._casbaseddirectory import CasBasedDirectory
  from .storage.directory import VirtualDirectoryError
@@ -1670,106 +1671,109 @@ class Element(Plugin):
                      cleanup_rootdir()
      def _cache_artifact(self, rootdir, sandbox, collect):
 -        if collect is not None:
 -            try:
 -                sandbox_vroot = sandbox.get_virtual_directory()
 -                collectvdir = sandbox_vroot.descend(collect.lstrip(os.sep).split(os.sep))
 -            except VirtualDirectoryError:
 -                # No collect directory existed
 -                collectvdir = None
 +        with self.timed_activity("Caching artifact"):
 +            if collect is not None:
 +                try:
 +                    sandbox_vroot = sandbox.get_virtual_directory()
 +                    collectvdir = sandbox_vroot.descend(collect.lstrip(os.sep).split(os.sep))
 +                except VirtualDirectoryError:
 +                    # No collect directory existed
 +                    collectvdir = None
 -        context = self._get_context()
 +            context = self._get_context()
 -        # Create artifact directory structure
 -        assembledir = os.path.join(rootdir, 'artifact')
 -        filesdir = os.path.join(assembledir, 'files')
 -        logsdir = os.path.join(assembledir, 'logs')
 -        metadir = os.path.join(assembledir, 'meta')
 -        buildtreedir = os.path.join(assembledir, 'buildtree')
 -        os.mkdir(assembledir)
 -        if collect is not None and collectvdir is not None:
 -            os.mkdir(filesdir)
 -        os.mkdir(logsdir)
 -        os.mkdir(metadir)
 -        os.mkdir(buildtreedir)
+-
 -        # Hard link files from collect dir to files directory
 -        if collect is not None and collectvdir is not None:
 -            collectvdir.export_files(filesdir, can_link=True)
+-
 -        cache_buildtrees = context.cache_buildtrees
 -        build_success = self.__build_result[0]
+-
 -        # cache_buildtrees defaults to 'always', as such the
 -        # default behaviour is to attempt to cache them. If only
 -        # caching failed artifact buildtrees, then query the build
 -        # result. Element types without a build-root dir will be cached
 -        # with an empty buildtreedir regardless of this configuration.
+-
 -        if cache_buildtrees == 'always' or (cache_buildtrees == 'failure' and not build_success):
 -            try:
 +            assemblevdir = CasBasedDirectory(cas_cache=context.artifactcache.cas, ref=None)
 +            logsvdir = assemblevdir.descend("logs", create=True)
 +            metavdir = assemblevdir.descend("meta", create=True)
 +            buildtreevdir = assemblevdir.descend("buildtree", create=True)
++
 +            # Create artifact directory structure
 +            assembledir = os.path.join(rootdir, 'artifact')
 +            logsdir = os.path.join(assembledir, 'logs')
 +            metadir = os.path.join(assembledir, 'meta')
 +            os.mkdir(assembledir)
 +            os.mkdir(logsdir)
 +            os.mkdir(metadir)
++
 +            if collect is not None and collectvdir is not None:
 +                filesvdir = assemblevdir.descend("files", create=True)
 +                filesvdir.import_files(collectvdir)
++
 +            cache_buildtrees = context.cache_buildtrees
 +            build_success = self.__build_result[0]
++
 +            # cache_buildtrees defaults to 'always', as such the
 +            # default behaviour is to attempt to cache them. If only
 +            # caching failed artifact buildtrees, then query the build
 +            # result. Element types without a build-root dir will be cached
 +            # with an empty buildtreedir regardless of this configuration.
++
 +            if cache_buildtrees == 'always' or (cache_buildtrees == 'failure' and not build_success):
                  sandbox_vroot = sandbox.get_virtual_directory()
 -                sandbox_build_dir = sandbox_vroot.descend(
 -                    self.get_variable('build-root').lstrip(os.sep).split(os.sep))
 -                # Hard link files from build-root dir to buildtreedir directory
 -                sandbox_build_dir.export_files(buildtreedir)
 -            except VirtualDirectoryError:
 -                # Directory could not be found. Pre-virtual
 -                # directory behaviour was to continue silently
 -                # if the directory could not be found.
 -                pass
 +                try:
 +                    sandbox_build_dir = sandbox_vroot.descend(
 +                        self.get_variable('build-root').lstrip(os.sep).split(os.sep))
 +                    buildtreevdir.import_files(sandbox_build_dir)
 +                except VirtualDirectoryError:
 +                    # Directory could not be found. Pre-virtual
 +                    # directory behaviour was to continue silently
 +                    # if the directory could not be found.
 +                    pass
++
 +            # Write some logs out to normal directories: logsdir and metadir
 +            # Copy build log
 +            log_filename = context.get_log_filename()
 +            self._build_log_path = os.path.join(logsdir, 'build.log')
 +            if log_filename:
 +                shutil.copyfile(log_filename, self._build_log_path)
++
 +            # Store public data
 +            _yaml.dump(_yaml.node_sanitize(self.__dynamic_public), os.path.join(metadir, 'public.yaml'))
++
 +            # Store result
 +            build_result_dict = {"success": self.__build_result[0], "description": self.__build_result[1]}
 +            if self.__build_result[2] is not None:
 +                build_result_dict["detail"] = self.__build_result[2]
 +            _yaml.dump(build_result_dict, os.path.join(metadir, 'build-result.yaml'))
++
 +            # ensure we have cache keys
 +            self._assemble_done()
++
 +            # Store keys.yaml
 +            _yaml.dump(_yaml.node_sanitize({
 +                'strong': self._get_cache_key(),
 +                'weak': self._get_cache_key(_KeyStrength.WEAK),
 +            }), os.path.join(metadir, 'keys.yaml'))
++
 +            # Store dependencies.yaml
 +            _yaml.dump(_yaml.node_sanitize({
 +                e.name: e._get_cache_key() for e in self.dependencies(Scope.BUILD)
 +            }), os.path.join(metadir, 'dependencies.yaml'))
++
 +            # Store workspaced.yaml
 +            _yaml.dump(_yaml.node_sanitize({
 +                'workspaced': bool(self._get_workspace())
 +            }), os.path.join(metadir, 'workspaced.yaml'))
++
 +            # Store workspaced-dependencies.yaml
 +            _yaml.dump(_yaml.node_sanitize({
 +                'workspaced-dependencies': [
 +                    e.name for e in self.dependencies(Scope.BUILD)
 +                    if e._get_workspace()
 +                ]
 +            }), os.path.join(metadir, 'workspaced-dependencies.yaml'))
 -        # Copy build log
 -        log_filename = context.get_log_filename()
 -        self._build_log_path = os.path.join(logsdir, 'build.log')
 -        if log_filename:
 -            shutil.copyfile(log_filename, self._build_log_path)
+-
 -        # Store public data
 -        _yaml.dump(_yaml.node_sanitize(self.__dynamic_public), os.path.join(metadir, 'public.yaml'))
+-
 -        # Store result
 -        build_result_dict = {"success": self.__build_result[0], "description": self.__build_result[1]}
 -        if self.__build_result[2] is not None:
 -            build_result_dict["detail"] = self.__build_result[2]
 -        _yaml.dump(build_result_dict, os.path.join(metadir, 'build-result.yaml'))
+-
 -        # ensure we have cache keys
 -        self._assemble_done()
+-
 -        # Store keys.yaml
 -        _yaml.dump(_yaml.node_sanitize({
 -            'strong': self._get_cache_key(),
 -            'weak': self._get_cache_key(_KeyStrength.WEAK),
 -        }), os.path.join(metadir, 'keys.yaml'))
+-
 -        # Store dependencies.yaml
 -        _yaml.dump(_yaml.node_sanitize({
 -            e.name: e._get_cache_key() for e in self.dependencies(Scope.BUILD)
 -        }), os.path.join(metadir, 'dependencies.yaml'))
+-
 -        # Store workspaced.yaml
 -        _yaml.dump(_yaml.node_sanitize({
 -            'workspaced': bool(self._get_workspace())
 -        }), os.path.join(metadir, 'workspaced.yaml'))
+-
 -        # Store workspaced-dependencies.yaml
 -        _yaml.dump(_yaml.node_sanitize({
 -            'workspaced-dependencies': [
 -                e.name for e in self.dependencies(Scope.BUILD)
 -                if e._get_workspace()
 -            ]
 -        }), os.path.join(metadir, 'workspaced-dependencies.yaml'))
 +            metavdir.import_files(metadir)
 +            logsvdir.import_files(logsdir)
 -        with self.timed_activity("Caching artifact"):
 -            artifact_size = utils._get_dir_size(assembledir)
 -            self.__artifacts.commit(self, assembledir, self.__get_cache_keys_for_commit())
+-
 -        if collect is not None and collectvdir is None:
 -            raise ElementError(
 -                "Directory '{}' was not found inside the sandbox, "
 -                "unable to collect artifact contents"
 -                .format(collect))
 +            artifact_size = assemblevdir.get_size()
 +            self.__artifacts.commit(self, assemblevdir, self.__get_cache_keys_for_commit())
++
 +            if collect is not None and collectvdir is None:
 +                raise ElementError(
 +                    "Directory '{}' was not found inside the sandbox, "
 +                    "unable to collect artifact contents"
 +                    .format(collect))
          return artifact_size

buildstream/storage/_casbaseddirectory.py

@@ -136,10 +136,10 @@ class CasBasedDirectory(Directory):
          the parent).
          """
 -        self.ref = self.cas_cache.add_object(buffer=self.pb2_directory.SerializeToString())
          if caller:
              old_dir = self._find_pb2_entry(caller.filename)
              self.cas_cache.add_object(digest=old_dir.digest, buffer=caller.pb2_directory.SerializeToString())
 +        self.ref = self.cas_cache.add_object(buffer=self.pb2_directory.SerializeToString())
          if self.parent:
              self.parent._recalculate_recursing_up(self)
@@ -277,14 +277,6 @@ class CasBasedDirectory(Directory):
                                                           directory_list))
          return None
 -    def find_root(self):
 -        """ Finds the root of this directory tree by following 'parent' until there is
 -        no parent. """
 -        if self.parent:
 -            return self.parent.find_root()
 -        else:
 -            return self
+-
      def _check_replacement(self, name, path_prefix, fileListResult):
          """ Checks whether 'name' exists, and if so, whether we can overwrite it.
          If we can, add the name to 'overwritten_files' and delete the existing entry.
@@ -451,7 +443,7 @@ class CasBasedDirectory(Directory):
                  files = external_pathspec.list_relative_paths()
          if isinstance(external_pathspec, FileBasedDirectory):
 -            source_directory = external_pathspec.get_underlying_directory()
 +            source_directory = external_pathspec._get_underlying_directory()
              result = self._import_files_from_directory(source_directory, files=files)
          elif isinstance(external_pathspec, str):
              source_directory = external_pathspec
@@ -635,6 +627,18 @@ class CasBasedDirectory(Directory):
          self._recalculate_recursing_up()
          self._recalculate_recursing_down()
 +    def get_size(self):
 +        total = len(self.pb2_directory.SerializeToString())
 +        for i in self.index.values():
 +            if isinstance(i.buildstream_object, CasBasedDirectory):
 +                total += i.buildstream_object.get_size()
 +            elif isinstance(i.pb_object, remote_execution_pb2.FileNode):
 +                src_name = self.cas_cache.objpath(i.pb_object.digest)
 +                filesize = os.stat(src_name).st_size
 +                total += filesize
 +            # Symlink nodes are encoded as part of the directory serialization.
 +        return total
++
      def _get_identifier(self):
          path = ""
          if self.parent:
@@ -653,3 +657,15 @@ class CasBasedDirectory(Directory):
          throw an exception. """
          raise VirtualDirectoryError("_get_underlying_directory was called on a CAS-backed directory," +
                                      " which has no underlying directory.")
++
 +    # _get_digest():
 +    #
 +    # Return the Digest for this directory.
 +    #
 +    # Returns:
 +    #   (Digest): The Digest protobuf object for the Directory protobuf
 +    #
 +    def _get_digest(self):
 +        if not self.ref:
 +            self.ref = self.cas_cache.add_object(buffer=self.pb2_directory.SerializeToString())
 +        return self.ref

buildstream/storage/_filebaseddirectory.py

@@ -30,6 +30,7 @@ See also: :ref:`sandboxing`.
  import os
  import time
  from .directory import Directory, VirtualDirectoryError
 +from .. import utils
  from ..utils import link_files, copy_files, list_relative_paths, _get_link_mtime, _magic_timestamp
  from ..utils import _set_deterministic_user, _set_deterministic_mtime
@@ -201,6 +202,9 @@ class FileBasedDirectory(Directory):
          return list_relative_paths(self.external_directory)
 +    def get_size(self):
 +        return utils._get_dir_size(self.external_directory)
++
      def __str__(self):
          # This returns the whole path (since we don't know where the directory started)
          # which exposes the sandbox directory; we will have to assume for the time being

buildstream/storage/directory.py

@@ -177,3 +177,9 @@ class Directory():
          """
          raise NotImplementedError()
++
 +    def get_size(self):
 +        """ Get an approximation of the storage space in bytes used by this directory
 +        and all files and subdirectories in it. Storage space varies by implementation
 +        and effective space used may be lower than this number due to deduplication. """
 +        raise NotImplementedError()

[Notes] [Git][BuildStream/buildstream][jmac/cache_artifacts_with_vdir] 9 commits: CONTRIBUTING.rst: Documenting the release process

Jürg Billeter pushed to branch jmac/cache_artifacts_with_vdir at BuildStream / buildstream

Commits:

6 changed files:

Changes: