[Notes] [Git][BuildStream/buildstream][adamjones/labels] 30 commits: use

Adam Jones pushed to branch adamjones/labels at BuildStream / buildstream

Commits:

c24f2971

by Angelos Evripiotis at 2019-02-15T13:56:48Z

userconfig: rm really-workspace-close-project-inaccessible

Remove the need for the 'really-workspace-close-project-inaccessible'
config option, as well as the option itself.

As agreed on the mailing list [1], all the 'are you sure?' prompts on
workspace reset and close were removed. While that discussion was going
on, this new prompt and option was added. At the 2019 BuildStream
Gathering, it was verbally agreed between myself and Tristan VB that we
would also remove this instance.

It was also agreed that we should have a notice to let the user know
what they'd done, this was already in place if interactive. Moved it to
be unconditional so that there's no difference in non-interactive
behaviour. Made it output to stderr, as it's diagnostic meant for the
user. Made it the last thing echo'd so it's next to the prompt - it's
very relevant to what they type next. Added a test to make sure the text
makes it to stderr in the appropriate case, and not in an inappropriate
one.

This is the last instance of any prompt configuration, so BuildStream
can also forget all of that machinery.

[1] https://mail.gnome.org/archives/buildstream-list/2018-December/msg00111.html

91eb29a6

by Angelos Evripiotis at 2019-02-15T14:55:06Z

Merge branch 'are_you_sure2' into 'master'

userconfig: rm really-workspace-close-project-inaccessible

Closes #726 and #744

See merge request BuildStream/buildstream!1130

b8561fb1

by Chandan Singh at 2019-02-15T14:58:22Z

contrib/bst-here: Change default image to buildstream/buildstream

See https://gitlab.com/BuildStream/buildstream-docker-images/issues/26
for detailed discussion around this.

`buildstream/buildstream-fedora` is now considered deprecated. Switch to
`buildstream/buildstream` image. This image also offers more tags that
will provide users more flexibility.

2699c77c

by Chandan Singh at 2019-02-15T14:58:31Z

contrib/bst-here: Allow users to specify image variant

Now that the `buildstream/buildstream` image has 9 variants, let's make
it easier to choose the desired tag, using a command-line option.

This is otherwise possible by specifying the full image name `image:tag`
using the `-i` option. But, this will make it easier to specify just the
tag using `-j`.

The following two invocations of `bst-here` are now equivalent:

    bst-here -i buildstream/buildstream:dev
    bst-here -j dev

677fc6c5

by Chandan Singh at 2019-02-15T16:09:21Z

Merge branch 'chandan/bst-here-tags' into 'master'

contrib/bst-here: Allow users to specify image variant

See merge request BuildStream/buildstream!1153

3b889eb8

by Javier Jardón at 2019-02-15T20:22:08Z

README.rst: Add table with distros with packaged buildstream

[ci skip]

232906b1

by Javier Jardón at 2019-02-15T23:08:50Z

Merge branch 'jjardon/distros' into 'master'

README.rst: Add table with distros with packaged buildstream

See merge request BuildStream/buildstream!1143

ee2296ab

by Tristan Van Berkom at 2019-02-18T08:41:57Z

CONTRIBUTING.rst: Documenting the release process

So that other people can also make releases.

c79696fe

by Tristan Van Berkom at 2019-02-18T09:43:22Z

Merge branch 'tristan/document-release-process' into 'master'

CONTRIBUTING.rst: Documenting the release process

See merge request BuildStream/buildstream!1155

fb65af6c

by Jürg Billeter at 2019-02-18T10:29:25Z

_casbaseddirectory.py: Fix order in _recalculate_recursing_up()

The directory needs to be serialized after the Digest for the
subdirectory `caller` has been updated.

f5660fa0

by Jim MacArthur at 2019-02-18T10:29:25Z

_casbaseddirectory.py: Correct typo for _get_underlying_directory

cdcf0dc5

by Jürg Billeter at 2019-02-18T10:29:25Z

_casbaseddirectory.py: Remove unused find_root() method

f9dd6ea2
by Jim MacArthur at 2019-02-18T10:29:25Z
```
storage: Add Directory.get_size()
```

050249bb

by Jürg Billeter at 2019-02-18T10:29:25Z

_casbaseddirectory.py: Add _get_digest() method

3b881efc

by Jürg Billeter at 2019-02-18T10:29:25Z

element.py: Expand scope of timed_activity in _cache_artifact()

ArtifactCache.commit() is no longer the only expensive part and even
more so with the upcoming change to use CASBasedDirectory.

3832c0d1

by Jim MacArthur at 2019-02-18T10:29:25Z

element.py: Alter _cache_artifact to use CasBasedDirectory

ef85e3b2

by Jürg Billeter at 2019-02-18T11:18:10Z

Merge branch 'jmac/cache_artifacts_with_vdir' into 'master'

Cache artifacts with virtual directories instead of filesystem.

Closes #787

See merge request BuildStream/buildstream!991

e0f0a01d

by Chandan Singh at 2019-02-18T12:41:31Z

requirements/Makefile: exclude pkg-resources

Having `pkg-resources` in requirements files is never the right thing as
it is not really a package that one can install. This is basically an
artifact of Debian-based systems providing incorrect metadata to pip.
See [this pip issue](https://github.com/pypa/pip/issues/4022) and
[this Ubuntu issue](https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1635463)
for more background.

f149fed2

by Tristan Van Berkom at 2019-02-18T13:43:28Z

Merge branch 'chandan/remove-pkg-resources' into 'master'

requirements/Makefile: exclude pkg-resources

Closes #918

See merge request BuildStream/buildstream!1156

4022234e

by Tristan Van Berkom at 2019-02-18T13:44:29Z

.gitlab-ci.yml: Some new test environments were added, but not collecting coverage.

The test jobs which create coverage reports need to be listed
in the dependencies of the coverage job, as highlighted in
issue #356.

86466e7e

by Tristan Van Berkom at 2019-02-18T14:27:08Z

Merge branch 'tristan/collect-missing-coverage' into 'master'

.gitlab-ci.yml: Some new test environments were added, but not collecting coverage.

See merge request BuildStream/buildstream!1157

c8cd24b9

by Jonathan Maw at 2019-02-18T17:37:00Z

.gitlab-ci: make wsl-test automatic, allow failure

de70f8c0

by Jonathan Maw at 2019-02-18T18:57:45Z

Merge branch 'jonathan/wsl-tests-allow-fail' into 'master'

Allow WSL tests to run, but they don't block the CI

See merge request BuildStream/buildstream!1159

00b86b19

by Chandan Singh at 2019-02-18T18:59:31Z

doc/badges.py: Fix catching of CalledProcessError

Since `CalledProcessError` is defined by the `subprocess` module, it
must be caught as `subprocess.CalledProcessError`, and not as simply
`CalledProcessError`.

067a0c6b

by Chandan Singh at 2019-02-18T20:31:10Z

Merge branch 'chandan/fix-badge-subprocess-error' into 'master'

doc/badges.py: Fix catching of CalledProcessError

See merge request BuildStream/buildstream!1160

fca37d0d

by Chandan Singh at 2019-02-19T06:37:04Z

buildstream/_yaml.py: Import Mapping from collections.abc

Instead of importing `Mapping` from `collections`, it must be imported
from `collections.abc`, as the old method will stop working from Python
3.8.

5f0571d1

by Jürg Billeter at 2019-02-19T07:37:21Z

Merge branch 'chandan/fix-mapping-import' into 'master'

buildstream/_yaml.py: Import Mapping from collections.abc

See merge request BuildStream/buildstream!1161

e2074693

by Gökçen Nurlu at 2019-02-19T07:44:54Z

Prevent caching when Element.stage() fails

That step is not actually part of the build phase should not be cached and
causes hard to find bugs.

Fixes #791

cfbe409d

by Jürg Billeter at 2019-02-19T09:00:31Z

Merge branch 'gokcennurlu/dont_cache_if_stage_fails' into 'master'

Prevent caching when Element.stage() fails

Closes #791

See merge request BuildStream/buildstream!1012

0e52ee44

by Adam Jones at 2019-02-19T09:53:35Z

Add a link to the label descriptions in the issue templates

18 changed files:

.gitlab-ci.yml
.gitlab/issue_templates/bst_bug.md
.gitlab/issue_templates/bst_task.md
CONTRIBUTING.rst
README.rst
buildstream/_artifactcache.py
buildstream/_context.py
buildstream/_frontend/cli.py
buildstream/_yaml.py
buildstream/data/userconfig.yaml
buildstream/element.py
buildstream/storage/_casbaseddirectory.py
buildstream/storage/_filebaseddirectory.py
buildstream/storage/directory.py
contrib/bst-here
doc/badges.py
requirements/Makefile
tests/frontend/workspace.py

Changes:

.gitlab-ci.yml

@@ -160,7 +160,7 @@ tests-wsl:
    script:
    - "${TEST_COMMAND}"
 -  when: manual
 +  allow_failure: true
  # Automatically build documentation for every commit, we want to know
  # if building documentation fails even if we're not deploying it.
@@ -281,10 +281,12 @@ coverage:
      - tox -e coverage
      - cp -a .coverage-reports/ ./coverage-report
    dependencies:
 +  - tests-centos-7.6
    - tests-debian-9
    - tests-fedora-28
    - tests-fedora-29
    - tests-fedora-missing-deps
 +  - tests-fedora-update-deps
    - tests-ubuntu-18.04
    - tests-unix
    except:

.gitlab/issue_templates/bst_bug.md

@@ -33,4 +33,6 @@
  * BuildStream version affected: /milestone %BuildStream_v1.x
  ----
 +[//]: # (To review information about possible relevant labels for this issue please view the list of labels: https://gitlab.com/BuildStream/buildstream/labels)
++
  /label ~bug

.gitlab/issue_templates/bst_task.md

@@ -15,3 +15,5 @@
  [//]: # (Acceptance criteria should follow the S.M.A.R.T. principle https://en.wikipedia.org/wiki/SMART_criteria )
  ----
 +[//]: # (To review information about possible relevant labels for this issue please view the list of labels: https://gitlab.com/BuildStream/buildstream/labels)
++

CONTRIBUTING.rst

@@ -1234,6 +1234,8 @@ This will give you a ``doc/build/html`` directory with the html docs which
  you can view in your browser locally to test.
 +.. _contributing_session_html:
++
  Regenerating session html
  '''''''''''''''''''''''''
  The documentation build will build the session files if they are missing,
@@ -1252,6 +1254,8 @@ To force rebuild session html while building the doc, simply run `tox` with the
    env BST_FORCE_SESSION_REBUILD=1 tox -e docs
 +.. _contributing_man_pages:
++
  Man pages
  ~~~~~~~~~
  Unfortunately it is quite difficult to integrate the man pages build
@@ -1779,3 +1783,257 @@ changing the ``.in`` file, run the following to update the matching ``.txt``
  file::
     make -C requirements
++
++
 +Making releases
 +---------------
 +This is a checklist of activities which must be observed when creating
 +BuildStream releases, it is important to keep this section up to date
 +whenever the release process changes.
++
++
 +Requirements
 +~~~~~~~~~~~~
 +There are a couple of requirements and accounts required in order
 +to publish a release.
++
 +* Ability to send email to ``buildstream-list gnome org`` and
 +  to ``gnome-announce-list gnome org``.
++
 +* Shell account at ``master.gnome.org``.
++
 +* Access to the `BuildStream project on PyPI <https://pypi.org/project/BuildStream/>`_
++
 +* An email client which still knows how to send emails in plain text.
++
++
 +Pre-release changes
 +~~~~~~~~~~~~~~~~~~~
 +Before actually rolling the release, here is a list of changes which
 +might need to be done in preparation of the release.
++
 +* Ensure that the man pages are up to date
++
 +  The man pages are committed to the repository because we are
 +  currently unable to integrate this generation into the setuptools
 +  build phase, as outlined in issue #8.
++
 +  If any of the user facing CLI has changed, or if any of the
 +  related docstrings have changed, then you should
 +  :ref:`regenerate the man pages <contributing_man_pages>` and
 +  add/commit the results before wrapping a release.
++
 +* Ensure the documentation session HTML is up to date
++
 +  The session HTML files are committed to the repository for multiple
 +  reasons, one of them being that the documentation must be buildable
 +  from within a release build environment so that downstream distribution
 +  packagers can easily create the docs package.
++
 +  This is currently only needed for the first stable release
 +  in a stable line of releases, after this point the API is frozen
 +  and will not change for the remainder of the stable release lifetime,
 +  so nothing interesting will have changed in these session files.
++
 +  If regeneration is needed, follow :ref:`the instructions above <contributing_session_html>`.
++
 +* Ensure the NEWS entry is up to date and ready
++
 +  For a stable release where features have not been added, we
 +  should at least add some entries about the issues which have
 +  been fixed since the last stable release.
++
 +  For development releases, it is worthwhile going over the
 +  existing entries and ensuring all the major feature additions
 +  are mentioned and there are no redundancies.
++
 +* Push pre-release changes
++
 +  Now that any final pre-release changes to generated files or NEWS have
 +  been made, push these directly to the upstream repository.
++
 +  Do not sit around waiting for CI or approval, these superficial changes
 +  do not affect CI and you are intended to push these changes directly
 +  to the upstream repository.
++
++
 +Release process
 +~~~~~~~~~~~~~~~
++
 +* Ensure that the latest commit is passing in CI
++
 +  Of course, we do not release software which does not pass it's own
 +  tests.
++
 +* Get the list of contributors
++
 +  The list of contributors for a given list is a list of
 +  any contributors who have landed any patches since the
 +  last release.
++
 +  An easy way to get this list is to ask git to summarize
 +  the authors of commits since the *last release tag*. For
 +  example, if we are about to create the ``1.1.1`` release, then
 +  we need to observe all of the commits since the ``1.1.0``
 +  release:
++
 +  .. code:: shell
++
 +     git shortlog -s 1.1.0...@
++
 +  At times, the same contributor might make contributions from different
 +  machines which they have setup their author names differently, you
 +  can see that some of the authors are actually duplicates, then
 +  remove the duplicates.
++
 +* Start composing the release announcement email
++
 +  The first thing to do when composing the release email is to
 +  ensure your mail client has disabled any HTML formatting and will
 +  safely use plain text only.
++
 +  Try to make the release announcement consistent with other release
 +  announcements as much as possible, an example of the email
 +  can be `found here <https://mail.gnome.org/archives/buildstream-list/2019-February/msg00039.html>`_.
++
 +  The recipients of the email are ``buildstream-list gnome org`` and
 +  ``gnome-announce-list gnome org`` and the title of the email should
 +  be of the form: ``BuildStream 1.1.1 released``, without any exclamation point.
++
 +  The format of the email is essentially::
++
 +    Hi all,
++
 +    This is the personalized message written to you about this
 +    release.
++
 +    If this is an unstable release, this should include a warning
 +    to this effect and an invitation to users to please help us
 +    test this release.
++
 +    This is also a good place to highlight specific bug fixes which
 +    users may have been waiting for, or highlight a new feature we
 +    want users to try out.
++
++
 +    What is BuildStream ?
 +    =====================
 +    This is a concise blurb which describes BuildStream in a couple of
 +    sentences, and is taken from the the README.rst.
++
 +    The easiest thing is to just copy this over from the last release email.
++
++
 +    =================
 +    buildstream 1.1.1
 +    =================
 +    This section is directly copy pasted from the top of the NEWS file
++
++
 +    Contributors
 +    ============
 +     - This is Where
 +     - You Put
 +     - The Contributor
 +     - Names Which
 +     - You Extracted
 +     - Using git shortlog -s
++
++
 +    Where can I get it ?
 +    ====================
 +    https://download.gnome.org/sources/BuildStream/1.1/
++
 +    For more information on the BuildStream project, visit our home page
 +    at https://buildstream.build/
++
 +* Publish the release tag
++
 +  Now that any pre-release changes are upstream, create and push the
 +  signed release tag like so:
++
 +  .. code:: shell
++
 +     git tag -s 1.1.1
 +     git push origin 1.1.1
++
 +* Upload the release tarball
++
 +  First get yourself into a clean repository state, ensure that you
 +  don't have any unfinished work or precious, uncommitted files lying
 +  around in your checkout and then run:
++
 +  .. code:: shell
++
 +     git clean -xdff
++
 +  Create the tarball with the following command:
++
 +  .. code:: shell
++
 +     python3 setup.py sdist
++
 +  And upload the resulting tarball to the master GNOME server:
++
 +  .. code:: shell
++
 +     scp dist/BuildStream-1.1.1.tar.gz <user>@master.gnome.org:
++
 +  And finally login to your account at ``master.gnome.org`` and run
 +  the install scripts to publish the tarball and update the mirrors:
++
 +  .. code:: shell
++
 +     ftpadmin install BuildStream-1.1.1.tar.gz
++
 +* Send the release email
++
 +  Now that the release tag is up and the tarball is published,
 +  you can send the release email.
++
++
 +Post-release activities
 +~~~~~~~~~~~~~~~~~~~~~~~
 +Once the release has been published, there are some activities
 +which need to be done to ensure everything is up to date.
++
 +* If this is a stable release, then the tarball should also be
 +  uploaded to PyPI.
++
 +  Make sure you have ``twine`` installed and upload the tarball
 +  like so:
++
 +  .. code:: shell
++
 +     pip3 install --user twine
 +     twine upload -r pypi dist/BuildStream-1.0.1.tar.gz
++
 +* Update the topic line in the #buildstream IRC channel if needed
++
 +  The IRC channel usually advertizes the latest stable release
 +  in the topic line, now is the right time to update it.
++
 +* Update the website repository
++
 +  The website wants to link to release announcements, but this
 +  cannot be automated because we cannot guess what the link to
 +  the release email will be in the mailing list archive.
++
 +  Find the URL to the announcement you just published
 +  `in the mailing list archives <https://mail.gnome.org/archives/buildstream-list/>`_,
 +  and use that URL to update the ``anouncements.json`` file in the website
 +  repository.
++
 +  Commit and push this change to the the ``anouncements.json`` file to
 +  the upstream website repository, and gitlab will take care of automatically
 +  updating the website accordingly.
++
 +* Regenerate BuildStream documentation
++
 +  In order to update the badges we use in various documentation
 +  which reflects what is the latest stable releases and the latest
 +  development snapshots, we simply need to ensure a pipeline runs
 +  for the master branch in the BuildStream repository.
++
 +  You can do this by using the "Run Pipeline" feature on the
 +  `pipelines page in the gitlab UI <https://gitlab.com/BuildStream/buildstream/pipelines>`_.

README.rst

@@ -100,3 +100,9 @@ We also recommend exploring some existing BuildStream projects:
  * https://gitlab.com/baserock/definitions
  If you have any questions please ask on our `#buildstream <irc://irc.gnome.org/buildstream>`_ channel in `irc.gnome.org <irc://irc.gnome.org>`_
++
++
 +Availability in distros
 +=======================
 +.. image:: https://repology.org/badge/vertical-allrepos/buildstream.svg
 +   :target: https://repology.org/metapackage/buildstream/versions

buildstream/_artifactcache.py

@@ -588,13 +588,16 @@ class ArtifactCache():
+     #
      # Args:
      #     element (Element): The Element commit an artifact for
 -    #     content (str): The element's content directory
 +    #     content (Directory): The element's content directory
      #     keys (list): The cache keys to use
+     #
      def commit(self, element, content, keys):
          refs = [element.get_artifact_name(key) for key in keys]
 -        self.cas.commit(refs, content)
 +        tree = content._get_digest()
++
 +        for ref in refs:
 +            self.cas.set_ref(ref, tree)
      # diff():
+     #

buildstream/_context.py

@@ -124,10 +124,6 @@ class Context():
          # Whether or not to cache build trees on artifact creation
          self.cache_buildtrees = None
 -        # Boolean, whether we double-check with the user that they meant to
 -        # close the workspace when they're using it to access the project.
 -        self.prompt_workspace_close_project_inaccessible = None
+-
          # Whether elements must be rebuilt when their dependencies have changed
          self._strict_build_plan = None
@@ -248,22 +244,6 @@ class Context():
          self.sched_pushers = _yaml.node_get(scheduler, int, 'pushers')
          self.sched_network_retries = _yaml.node_get(scheduler, int, 'network-retries')
 -        # Load prompt preferences
 -        #
 -        # We convert string options to booleans here, so we can be both user
 -        # and coder-friendly. The string options are worded to match the
 -        # responses the user would give at the cli, for least surprise. The
 -        # booleans are converted here because it's easiest to eyeball that the
 -        # strings are right.
 -        #
 -        prompt = _yaml.node_get(
 -            defaults, Mapping, 'prompt')
 -        _yaml.node_validate(prompt, [
 -            'really-workspace-close-project-inaccessible',
 -        ])
 -        self.prompt_workspace_close_project_inaccessible = _node_get_option_str(
 -            prompt, 'really-workspace-close-project-inaccessible', ['ask', 'yes']) == 'ask'
+-
          # Load per-projects overrides
          self._project_overrides = _yaml.node_get(defaults, Mapping, 'projects', default_value={})

buildstream/_frontend/cli.py

@@ -814,6 +814,8 @@ def workspace_open(app, no_checkout, force, track_, directory, elements):
  def workspace_close(app, remove_dir, all_, elements):
      """Close a workspace"""
 +    removed_required_element = False
++
      with app.initialized():
          if not (all_ or elements):
              # NOTE: I may need to revisit this when implementing multiple projects
@@ -840,18 +842,20 @@ def workspace_close(app, remove_dir, all_, elements):
          for element_name in elements:
              if not app.stream.workspace_exists(element_name):
                  nonexisting.append(element_name)
 -            if (app.stream.workspace_is_required(element_name) and app.interactive and
 -                    app.context.prompt_workspace_close_project_inaccessible):
 -                click.echo("Removing '{}' will prevent you from running "
 -                           "BuildStream commands from the current directory".format(element_name))
 -                if not click.confirm('Are you sure you want to close this workspace?'):
 -                    click.echo('Aborting', err=True)
 -                    sys.exit(-1)
          if nonexisting:
              raise AppError("Workspace does not exist", detail="\n".join(nonexisting))
          for element_name in elements:
              app.stream.workspace_close(element_name, remove_dir=remove_dir)
 +            if app.stream.workspace_is_required(element_name):
 +                removed_required_element = True
++
 +    # This message is echo'd last, as it's most relevant to the next
 +    # thing the user will type.
 +    if removed_required_element:
 +        click.echo(
 +            "Removed '{}', therefore you can no longer run BuildStream "
 +            "commands from the current directory.".format(element_name), err=True)
  ##################################################################

buildstream/_yaml.py

@@ -940,7 +940,7 @@ def node_sanitize(node):
          return [node_sanitize(elt) for elt in node]
      # Finally ChainMap and dict, and other Mappings need special handling
 -    if node_type in (dict, ChainMap) or isinstance(node, collections.Mapping):
 +    if node_type in (dict, ChainMap) or isinstance(node, collections.abc.Mapping):
          result = SanitizedDict()
          key_list = [key for key, _ in node_items(node)]

buildstream/data/userconfig.yaml

@@ -111,20 +111,3 @@ logging:
    message-format: |
      [%{elapsed}][%{key}][%{element}] %{action} %{message}
+-
 -#
 -#    Prompt overrides
 -#
 -# Here you can suppress 'are you sure?' and other kinds of prompts by supplying
 -# override values. Note that e.g. 'yes' and 'no' have the same meaning here as
 -# they do in the actual cli prompt.
 -#
 -prompt:
+-
 -  # Whether to really proceed with 'bst workspace close' when doing so would
 -  # stop them from running bst commands in this workspace.
 -  #
 -  #  ask - Ask the user if they are sure.
 -  #  yes - Always close, without asking.
 -  #
 -  really-workspace-close-project-inaccessible: ask

buildstream/element.py

@@ -103,6 +103,7 @@ from .types import _KeyStrength, CoreWarnings
  from .storage.directory import Directory
  from .storage._filebaseddirectory import FileBasedDirectory
 +from .storage._casbaseddirectory import CasBasedDirectory
  from .storage.directory import VirtualDirectoryError
@@ -1619,12 +1620,12 @@ class Element(Plugin):
                  self.__dynamic_public = _yaml.node_copy(self.__public)
                  # Call the abstract plugin methods
 -                try:
 -                    # Step 1 - Configure
 -                    self.__configure_sandbox(sandbox)
 -                    # Step 2 - Stage
 -                    self.stage(sandbox)
 +                # Step 1 - Configure
 +                self.__configure_sandbox(sandbox)
 +                # Step 2 - Stage
 +                self.stage(sandbox)
 +                try:
                      if self.__batch_prepare_assemble:
                          cm = sandbox.batch(self.__batch_prepare_assemble_flags,
                                             collect=self.__batch_prepare_assemble_collect)
@@ -1670,106 +1671,109 @@ class Element(Plugin):
                      cleanup_rootdir()
      def _cache_artifact(self, rootdir, sandbox, collect):
 -        if collect is not None:
 -            try:
 -                sandbox_vroot = sandbox.get_virtual_directory()
 -                collectvdir = sandbox_vroot.descend(collect.lstrip(os.sep).split(os.sep))
 -            except VirtualDirectoryError:
 -                # No collect directory existed
 -                collectvdir = None
 +        with self.timed_activity("Caching artifact"):
 +            if collect is not None:
 +                try:
 +                    sandbox_vroot = sandbox.get_virtual_directory()
 +                    collectvdir = sandbox_vroot.descend(collect.lstrip(os.sep).split(os.sep))
 +                except VirtualDirectoryError:
 +                    # No collect directory existed
 +                    collectvdir = None
 -        context = self._get_context()
 +            context = self._get_context()
 -        # Create artifact directory structure
 -        assembledir = os.path.join(rootdir, 'artifact')
 -        filesdir = os.path.join(assembledir, 'files')
 -        logsdir = os.path.join(assembledir, 'logs')
 -        metadir = os.path.join(assembledir, 'meta')
 -        buildtreedir = os.path.join(assembledir, 'buildtree')
 -        os.mkdir(assembledir)
 -        if collect is not None and collectvdir is not None:
 -            os.mkdir(filesdir)
 -        os.mkdir(logsdir)
 -        os.mkdir(metadir)
 -        os.mkdir(buildtreedir)
+-
 -        # Hard link files from collect dir to files directory
 -        if collect is not None and collectvdir is not None:
 -            collectvdir.export_files(filesdir, can_link=True)
+-
 -        cache_buildtrees = context.cache_buildtrees
 -        build_success = self.__build_result[0]
+-
 -        # cache_buildtrees defaults to 'always', as such the
 -        # default behaviour is to attempt to cache them. If only
 -        # caching failed artifact buildtrees, then query the build
 -        # result. Element types without a build-root dir will be cached
 -        # with an empty buildtreedir regardless of this configuration.
+-
 -        if cache_buildtrees == 'always' or (cache_buildtrees == 'failure' and not build_success):
 -            try:
 +            assemblevdir = CasBasedDirectory(cas_cache=context.artifactcache.cas, ref=None)
 +            logsvdir = assemblevdir.descend("logs", create=True)
 +            metavdir = assemblevdir.descend("meta", create=True)
 +            buildtreevdir = assemblevdir.descend("buildtree", create=True)
++
 +            # Create artifact directory structure
 +            assembledir = os.path.join(rootdir, 'artifact')
 +            logsdir = os.path.join(assembledir, 'logs')
 +            metadir = os.path.join(assembledir, 'meta')
 +            os.mkdir(assembledir)
 +            os.mkdir(logsdir)
 +            os.mkdir(metadir)
++
 +            if collect is not None and collectvdir is not None:
 +                filesvdir = assemblevdir.descend("files", create=True)
 +                filesvdir.import_files(collectvdir)
++
 +            cache_buildtrees = context.cache_buildtrees
 +            build_success = self.__build_result[0]
++
 +            # cache_buildtrees defaults to 'always', as such the
 +            # default behaviour is to attempt to cache them. If only
 +            # caching failed artifact buildtrees, then query the build
 +            # result. Element types without a build-root dir will be cached
 +            # with an empty buildtreedir regardless of this configuration.
++
 +            if cache_buildtrees == 'always' or (cache_buildtrees == 'failure' and not build_success):
                  sandbox_vroot = sandbox.get_virtual_directory()
 -                sandbox_build_dir = sandbox_vroot.descend(
 -                    self.get_variable('build-root').lstrip(os.sep).split(os.sep))
 -                # Hard link files from build-root dir to buildtreedir directory
 -                sandbox_build_dir.export_files(buildtreedir)
 -            except VirtualDirectoryError:
 -                # Directory could not be found. Pre-virtual
 -                # directory behaviour was to continue silently
 -                # if the directory could not be found.
 -                pass
 +                try:
 +                    sandbox_build_dir = sandbox_vroot.descend(
 +                        self.get_variable('build-root').lstrip(os.sep).split(os.sep))
 +                    buildtreevdir.import_files(sandbox_build_dir)
 +                except VirtualDirectoryError:
 +                    # Directory could not be found. Pre-virtual
 +                    # directory behaviour was to continue silently
 +                    # if the directory could not be found.
 +                    pass
++
 +            # Write some logs out to normal directories: logsdir and metadir
 +            # Copy build log
 +            log_filename = context.get_log_filename()
 +            self._build_log_path = os.path.join(logsdir, 'build.log')
 +            if log_filename:
 +                shutil.copyfile(log_filename, self._build_log_path)
++
 +            # Store public data
 +            _yaml.dump(_yaml.node_sanitize(self.__dynamic_public), os.path.join(metadir, 'public.yaml'))
++
 +            # Store result
 +            build_result_dict = {"success": self.__build_result[0], "description": self.__build_result[1]}
 +            if self.__build_result[2] is not None:
 +                build_result_dict["detail"] = self.__build_result[2]
 +            _yaml.dump(build_result_dict, os.path.join(metadir, 'build-result.yaml'))
++
 +            # ensure we have cache keys
 +            self._assemble_done()
++
 +            # Store keys.yaml
 +            _yaml.dump(_yaml.node_sanitize({
 +                'strong': self._get_cache_key(),
 +                'weak': self._get_cache_key(_KeyStrength.WEAK),
 +            }), os.path.join(metadir, 'keys.yaml'))
++
 +            # Store dependencies.yaml
 +            _yaml.dump(_yaml.node_sanitize({
 +                e.name: e._get_cache_key() for e in self.dependencies(Scope.BUILD)
 +            }), os.path.join(metadir, 'dependencies.yaml'))
++
 +            # Store workspaced.yaml
 +            _yaml.dump(_yaml.node_sanitize({
 +                'workspaced': bool(self._get_workspace())
 +            }), os.path.join(metadir, 'workspaced.yaml'))
++
 +            # Store workspaced-dependencies.yaml
 +            _yaml.dump(_yaml.node_sanitize({
 +                'workspaced-dependencies': [
 +                    e.name for e in self.dependencies(Scope.BUILD)
 +                    if e._get_workspace()
 +                ]
 +            }), os.path.join(metadir, 'workspaced-dependencies.yaml'))
 -        # Copy build log
 -        log_filename = context.get_log_filename()
 -        self._build_log_path = os.path.join(logsdir, 'build.log')
 -        if log_filename:
 -            shutil.copyfile(log_filename, self._build_log_path)
+-
 -        # Store public data
 -        _yaml.dump(_yaml.node_sanitize(self.__dynamic_public), os.path.join(metadir, 'public.yaml'))
+-
 -        # Store result
 -        build_result_dict = {"success": self.__build_result[0], "description": self.__build_result[1]}
 -        if self.__build_result[2] is not None:
 -            build_result_dict["detail"] = self.__build_result[2]
 -        _yaml.dump(build_result_dict, os.path.join(metadir, 'build-result.yaml'))
+-
 -        # ensure we have cache keys
 -        self._assemble_done()
+-
 -        # Store keys.yaml
 -        _yaml.dump(_yaml.node_sanitize({
 -            'strong': self._get_cache_key(),
 -            'weak': self._get_cache_key(_KeyStrength.WEAK),
 -        }), os.path.join(metadir, 'keys.yaml'))
+-
 -        # Store dependencies.yaml
 -        _yaml.dump(_yaml.node_sanitize({
 -            e.name: e._get_cache_key() for e in self.dependencies(Scope.BUILD)
 -        }), os.path.join(metadir, 'dependencies.yaml'))
+-
 -        # Store workspaced.yaml
 -        _yaml.dump(_yaml.node_sanitize({
 -            'workspaced': bool(self._get_workspace())
 -        }), os.path.join(metadir, 'workspaced.yaml'))
+-
 -        # Store workspaced-dependencies.yaml
 -        _yaml.dump(_yaml.node_sanitize({
 -            'workspaced-dependencies': [
 -                e.name for e in self.dependencies(Scope.BUILD)
 -                if e._get_workspace()
 -            ]
 -        }), os.path.join(metadir, 'workspaced-dependencies.yaml'))
 +            metavdir.import_files(metadir)
 +            logsvdir.import_files(logsdir)
 -        with self.timed_activity("Caching artifact"):
 -            artifact_size = utils._get_dir_size(assembledir)
 -            self.__artifacts.commit(self, assembledir, self.__get_cache_keys_for_commit())
+-
 -        if collect is not None and collectvdir is None:
 -            raise ElementError(
 -                "Directory '{}' was not found inside the sandbox, "
 -                "unable to collect artifact contents"
 -                .format(collect))
 +            artifact_size = assemblevdir.get_size()
 +            self.__artifacts.commit(self, assemblevdir, self.__get_cache_keys_for_commit())
++
 +            if collect is not None and collectvdir is None:
 +                raise ElementError(
 +                    "Directory '{}' was not found inside the sandbox, "
 +                    "unable to collect artifact contents"
 +                    .format(collect))
          return artifact_size

buildstream/storage/_casbaseddirectory.py

@@ -136,10 +136,10 @@ class CasBasedDirectory(Directory):
          the parent).
          """
 -        self.ref = self.cas_cache.add_object(buffer=self.pb2_directory.SerializeToString())
          if caller:
              old_dir = self._find_pb2_entry(caller.filename)
              self.cas_cache.add_object(digest=old_dir.digest, buffer=caller.pb2_directory.SerializeToString())
 +        self.ref = self.cas_cache.add_object(buffer=self.pb2_directory.SerializeToString())
          if self.parent:
              self.parent._recalculate_recursing_up(self)
@@ -277,14 +277,6 @@ class CasBasedDirectory(Directory):
                                                           directory_list))
          return None
 -    def find_root(self):
 -        """ Finds the root of this directory tree by following 'parent' until there is
 -        no parent. """
 -        if self.parent:
 -            return self.parent.find_root()
 -        else:
 -            return self
+-
      def _check_replacement(self, name, path_prefix, fileListResult):
          """ Checks whether 'name' exists, and if so, whether we can overwrite it.
          If we can, add the name to 'overwritten_files' and delete the existing entry.
@@ -451,7 +443,7 @@ class CasBasedDirectory(Directory):
                  files = external_pathspec.list_relative_paths()
          if isinstance(external_pathspec, FileBasedDirectory):
 -            source_directory = external_pathspec.get_underlying_directory()
 +            source_directory = external_pathspec._get_underlying_directory()
              result = self._import_files_from_directory(source_directory, files=files)
          elif isinstance(external_pathspec, str):
              source_directory = external_pathspec
@@ -635,6 +627,18 @@ class CasBasedDirectory(Directory):
          self._recalculate_recursing_up()
          self._recalculate_recursing_down()
 +    def get_size(self):
 +        total = len(self.pb2_directory.SerializeToString())
 +        for i in self.index.values():
 +            if isinstance(i.buildstream_object, CasBasedDirectory):
 +                total += i.buildstream_object.get_size()
 +            elif isinstance(i.pb_object, remote_execution_pb2.FileNode):
 +                src_name = self.cas_cache.objpath(i.pb_object.digest)
 +                filesize = os.stat(src_name).st_size
 +                total += filesize
 +            # Symlink nodes are encoded as part of the directory serialization.
 +        return total
++
      def _get_identifier(self):
          path = ""
          if self.parent:
@@ -653,3 +657,15 @@ class CasBasedDirectory(Directory):
          throw an exception. """
          raise VirtualDirectoryError("_get_underlying_directory was called on a CAS-backed directory," +
                                      " which has no underlying directory.")
++
 +    # _get_digest():
 +    #
 +    # Return the Digest for this directory.
 +    #
 +    # Returns:
 +    #   (Digest): The Digest protobuf object for the Directory protobuf
 +    #
 +    def _get_digest(self):
 +        if not self.ref:
 +            self.ref = self.cas_cache.add_object(buffer=self.pb2_directory.SerializeToString())
 +        return self.ref

buildstream/storage/_filebaseddirectory.py

@@ -30,6 +30,7 @@ See also: :ref:`sandboxing`.
  import os
  import time
  from .directory import Directory, VirtualDirectoryError
 +from .. import utils
  from ..utils import link_files, copy_files, list_relative_paths, _get_link_mtime, _magic_timestamp
  from ..utils import _set_deterministic_user, _set_deterministic_mtime
@@ -201,6 +202,9 @@ class FileBasedDirectory(Directory):
          return list_relative_paths(self.external_directory)
 +    def get_size(self):
 +        return utils._get_dir_size(self.external_directory)
++
      def __str__(self):
          # This returns the whole path (since we don't know where the directory started)
          # which exposes the sandbox directory; we will have to assume for the time being

buildstream/storage/directory.py

@@ -177,3 +177,9 @@ class Directory():
          """
          raise NotImplementedError()
++
 +    def get_size(self):
 +        """ Get an approximation of the storage space in bytes used by this directory
 +        and all files and subdirectories in it. Storage space varies by implementation
 +        and effective space used may be lower than this number due to deduplication. """
 +        raise NotImplementedError()

contrib/bst-here

@@ -25,16 +25,22 @@
  usage() {
      cat <<EOF
 -USAGE: $(basename "$0") [-i BST_HERE_IMAGE] [-p] [-t] [-T] [-v VOLUME ...] [-h] [COMMAND [ARG..]]
 +USAGE: $(basename "$0") [-i BST_HERE_IMAGE] [-j TAG] [-p] [-t] [-T] [-v VOLUME ...] [-h] [COMMAND [ARG..]]
  Run a bst command in a new BuildStream container.
  If no command is specified, an interactive shell is launched
  using "/bin/bash -i".
 +See https://hub.docker.com/r/buildstream/buildstream for details on image
 +variants.
++
  OPTIONS:
      -i IMAGE      Specify Docker image to use; can also be specified by setting
                    BST_HERE_IMAGE environment variable.
 +                  (default: buildstream/buildstream)
 +    -j TAG        Specify the tag of the Docker image to use.
 +                  (default: latest)
      -p            Pull the latest buildstream image before running.
      -t            Force pseudo-terminal allocation.
      -T            Disable pseudo-terminal allocation.
@@ -46,7 +52,8 @@ EOF
      exit "$1"
+ }
 -bst_here_image="${BST_HERE_IMAGE:-buildstream/buildstream-fedora:latest}"
 +bst_here_image="${BST_HERE_IMAGE:-buildstream/buildstream}"
 +bst_here_tag=
  is_tty=
  update=false
@@ -57,12 +64,15 @@ then
      is_tty=y
  fi
 -while getopts i:ptTv:h arg
 +while getopts i:j:ptTv:h arg
  do
      case $arg in
      i)
          bst_here_image="$OPTARG"
          ;;
 +    j)
 +        bst_here_tag="$OPTARG"
 +        ;;
      p)
          update=true
          ;;
@@ -83,6 +93,10 @@ do
      esac
  done
 +if [ -n "$bst_here_tag" ]; then
 +    bst_here_image="$bst_here_image:$bst_here_tag"
 +fi
++
  test "$OPTIND" -gt 1 &&
      shift $(( OPTIND - 1 ))

doc/badges.py

@@ -96,7 +96,7 @@ def parse_tag(tag):
  def guess_version(release):
      try:
          tags_output = subprocess.check_output(['git', 'tag'])
 -    except CalledProcessError:
 +    except subprocess.CalledProcessError:
          return (0, 0, 0)
      # Parse the `git tag` output into a list of integer tuples

requirements/Makefile

@@ -17,5 +17,5 @@ all: $(REQUIREMENTS_TXT)
  	$(eval VENVDIR := $(shell mktemp -d $(CURDIR)/.bst-venv.XXXXXX))
  	$(VENV) $(VENVDIR)
  	$(VENV_PIP) install -r $^
 -	$(VENV_PIP) freeze -r $^ > $@
 +	$(VENV_PIP) freeze -r $^ | grep -v pkg-resources > $@
  	rm -rf $(VENVDIR)

tests/frontend/workspace.py

@@ -1184,6 +1184,7 @@ def test_external_close_other(cli, datafiles, tmpdir_factory):
      result = cli.run(project=project, args=['-C', alpha_workspace, 'workspace', 'close', beta_element])
      result.assert_success()
 +    assert 'you can no longer run BuildStream' not in result.stderr
  @pytest.mark.datafiles(DATA_DIR)
@@ -1199,6 +1200,7 @@ def test_external_close_self(cli, datafiles, tmpdir_factory, guess_element):
      result = cli.run(project=project, args=['-C', alpha_workspace, 'workspace', 'close'] + arg_elm)
      result.assert_success()
 +    assert 'you can no longer run BuildStream' in result.stderr
  @pytest.mark.datafiles(DATA_DIR)

[Notes] [Git][BuildStream/buildstream][adamjones/labels] 30 commits: userconfig: rm really-workspace-close-project-inaccessible

Adam Jones pushed to branch adamjones/labels at BuildStream / buildstream

Commits:

18 changed files:

Changes: