[Notes] [Git][BuildStream/buildstream][tpollard/494] 17 commits: Merge b

Tom Pollard pushed to branch tpollard/494 at BuildStream / buildstream

Commits:

35ac26a7

by Chandan Singh at 2018-10-10T00:59:17Z

Merge branch 'chandan/bst-and-docker' into 'master'

Add documentation and NEWS entry about bst-docker-import

See merge request BuildStream/buildstream!864

7d96333f
by Laurence Urhegyi at 2018-10-10T19:24:14Z
```
README: Update to add link to website
```

39492db8

by devcurmudgeon at 2018-10-10T19:24:14Z

Merge branch 'patch-2' into 'master'

README: Update to add link to website

See merge request BuildStream/buildstream!859

4f0bfb4a

by Tristan Van Berkom at 2018-10-11T15:37:51Z

Rename element_enums.py -> types.py

This will be the place to store low level data types used
throughout the core, for now this includes public and private
types.

9dc10cc7

by Tristan Van Berkom at 2018-10-11T15:45:36Z

doc/source/core_framework.rst: Include the foundation types in the API docs.

efc0d4cc

by Tristan Van Berkom at 2018-10-11T15:48:44Z

types.py: Moved Consistency definition here from source.py

And slightly touch up the doc strings for Consistency and Scope.

a0712ead

by Tristan Van Berkom at 2018-10-11T16:23:54Z

Merge branch 'tristan/refactor-types-api' into 'master'

Refactor types api

See merge request BuildStream/buildstream!870

ce55b9a0

by Tiago Gomes at 2018-10-12T10:15:46Z

doc: updates considering website being live now

* Add a link to the website on the main page.
* Remove install instructions as they are now on the website.
* Remove Resources section as that information can be found at the
  website, and also looks bad.
* Move artifact server setup from the no longer existing Install section
  to the Using section.

10b092e1

by Tristan Van Berkom at 2018-10-12T10:21:27Z

doc/source/additional_docker.rst: Fix link to refer to website

The docker guide, which is part of the install guide, has moved
to the website.

1c7ba9a0

by Tristan Van Berkom at 2018-10-12T10:24:31Z

doc/source/index.rst: Moved references to the website below the simple ToC.

3738dd06

by Tristan Van Berkom at 2018-10-12T10:49:39Z

Merge branch 'tristan/remove-install-guide' into 'master'

Remove install guide

See merge request BuildStream/buildstream!872

c31ed138

by Tristan Van Berkom at 2018-10-14T15:24:53Z

CONTRIBUTING.rst: Added more guidelines about documenting the user guide

b39d87b4

by Jürg Billeter at 2018-10-15T11:07:05Z

_artifactcache: Fix crash in push_directory() without artifact server

Fixes #708.

4e3ec89e

by Jürg Billeter at 2018-10-15T11:48:16Z

Merge branch 'juerg/remote-config' into 'master'

_artifactcache: Fix crash in push_directory() without artifact server

Closes #708

See merge request BuildStream/buildstream!871

62794857

by Tom Pollard at 2018-10-15T14:57:41Z

_frontend/cli.py: Add --pull-buildtrees to bst build & pull

The addition of the pull-buildtrees flag will be used to override
the default behaviour of remote artifact cache pulling. Included
is the required changes to completions tests.

6fadcfa0

by Tom Pollard at 2018-10-15T14:57:41Z

buildstream/_context.py: Add pullbuildtrees global user option

The addition of the pullbuildtrees option to the users global
buildstream.conf will be used to override the default behaviour
of remote artifact cache pulling.

e3ca1018

by Tom Pollard at 2018-10-15T14:57:41Z

Don't pull artifact buildtrees by default.

The addition of cached buildtrees being included in element
artifacts has led to mostly redundant download overheads when
pulling from a remote artifact server. As such the default behaviour
of pull shouldn't fetch the buildtree object if available.

_stream.py: propagate if pullbuildtrees has been set via cli or
user config to the PullQueue.

pullqueue.py: add extensible attributes to determine required/
excluded directories for element pull jobs.

element.py: extend relevant pull logic for specified subdir
consideration and ensure push logic does not lead to partial
artifact pushes.

_artifactcache/: artifactcache.py & cascache.py inclusion of
helper functions for subdir artifact checking & fetching, fetch
logic extended to only pull required artifact directories.

tests/: integration test pullbuildtrees.py & fix for testutils/
artifactshare.py pathing.

29 changed files:

CONTRIBUTING.rst
README.rst
buildstream/__init__.py
buildstream/_artifactcache/artifactcache.py
buildstream/_artifactcache/cascache.py
buildstream/_context.py
buildstream/_frontend/cli.py
buildstream/_scheduler/queues/pullqueue.py
buildstream/_stream.py
buildstream/element.py
buildstream/source.py
buildstream/element_enums.py → buildstream/types.py
doc/source/additional_docker.rst
doc/source/core_framework.rst
doc/source/index.rst
− doc/source/install_docker.rst
− doc/source/install_linux_distro.rst
− doc/source/install_source.rst
− doc/source/install_versions.rst
− doc/source/main_install.rst
doc/source/main_using.rst
− doc/source/release-badge.rst
− doc/source/snapshot-badge.rst
doc/source/install_artifacts.rst → doc/source/using_configuring_artifact_server.rst
doc/source/using_examples.rst
doc/source/using_tutorial.rst
tests/completions/completions.py
+ tests/integration/pullbuildtrees.py
tests/testutils/artifactshare.py

Changes:

CONTRIBUTING.rst

@@ -1147,7 +1147,7 @@ To actually regenerate the code::
  Documenting
  -----------
  BuildStream starts out as a documented project from day one and uses
 -sphinx to document itself.
 +`sphinx <www.sphinx-doc.org>`_ to document itself.
  This section discusses formatting policies for editing files in the
  ``doc/source`` directory, and describes the details of how the docs are
@@ -1194,10 +1194,8 @@ The BuildStream documentation style is as follows:
      the target, e.g.: ``:ref:`Link text <anchor_name>```.
      Note that the "_" prefix is not used when referring to the target.
 -Useful links:
+-
 -For further information, please see the `Sphinx Documentation
 -<http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_.
 +For further information about using the reStructuredText with sphinx, please see the
 +`Sphinx Documentation <http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_.
  Building Docs
@@ -1266,10 +1264,101 @@ the ``man/`` subdirectory, which will be automatically included
  in the buildstream distribution.
 -Documentation Examples
 -~~~~~~~~~~~~~~~~~~~~~~
 -The examples section of the documentation contains a series of standalone
 -examples, here are the criteria for an example addition.
 +User guide
 +~~~~~~~~~~
 +The :ref:`user guide <using>` is comprised of free form documentation
 +in manually written ``.rst`` files and is split up into a few sections,
 +of main interest are the :ref:`tutorial <tutorial>` and the
 +:ref:`examples <examples>`.
++
 +The distinction of the two categories of user guides is important to
 +understand too.
++
 +* **Tutorial**
++
 +  The tutorial is structured as a series of exercises which start with
 +  the most basic concepts and build upon the previous chapters in order
 +  to arrive at a basic understanding of how to create BuildStream projects.
++
 +  This series of examples should be easy enough to complete in a matter
 +  of a few hours for a new user, and should provide just enough insight to
 +  get the user started in creating their own projects.
++
 +  Going through the tutorial step by step should also result in the user
 +  becoming proficient enough with the reference manual to get by on their own.
++
 +* **Examples**
++
 +  These exist to demonstrate how to accomplish more advanced tasks which
 +  are not always obvious and discoverable.
++
 +  Alternatively, these also demonstrate elegant and recommended ways of
 +  accomplishing some tasks which could be done in various ways.
++
++
 +Guidelines
 +''''''''''
 +Here are some general guidelines for adding new free form documentation
 +to the user guide.
++
 +* **Focus on a single subject**
++
 +  It is important to stay focused on a single subject and avoid getting
 +  into tangential material when creating a new entry, so that the articles
 +  remain concise and the user is not distracted by unrelated subject material.
++
 +  A single tutorial chapter or example should not introduce any additional
 +  subject material than the material being added for the given example.
++
 +* **Reuse existing sample project elements**
++
 +  To help avoid distracting from the topic at hand, it is always preferable to
 +  reuse the same project sample material from other examples and only deviate
 +  slightly to demonstrate the new material, than to create completely new projects.
++
 +  This helps us remain focused on a single topic at a time, and reduces the amount
 +  of unrelated material the reader needs to learn in order to digest the new
 +  example.
++
 +* **Don't be redundant**
++
 +  When something has already been explained in the tutorial or in another example,
 +  it is best to simply refer to the other user guide entry in a new example.
++
 +  Always prefer to link to the tutorial if an explanation exists in the tutorial,
 +  rather than linking to another example, where possible.
++
 +* **Link into the reference manual at every opportunity**
++
 +  The format and plugin API is 100% documented at all times. Whenever discussing
 +  anything about the format or plugin API, always do so while providing a link
 +  into the more terse reference material.
++
 +  We don't want users to have to search for the material themselves, and we also
 +  want the user to become proficient at navigating the reference material over
 +  time.
++
 +* **Use concise terminology**
++
 +  As developers, we tend to come up with code names for features we develop, and
 +  then end up documenting a new feature in an example.
++
 +  Never use a code name or shorthand to refer to a feature in the user guide, instead
 +  always use fully qualified sentences outlining very explicitly what we are doing
 +  in the example, or what the example is for in the case of a title.
++
 +  We need to be considerate that the audience of our user guide is probably a
 +  proficient developer or integrator, but has no idea what we might have decided
 +  to name a given activity.
++
++
 +Structure of an example
 +'''''''''''''''''''''''
 +The :ref:`tutorial <tutorial>` and the :ref:`examples <examples>` sections
 +of the documentation contain a series of sample projects, each chapter in
 +the tutoral, or standalone example uses a sample project.
++
 +Here is the the structure for adding new examples and tutorial chapters.
  * The example has a ``${name}``.
@@ -1280,11 +1369,17 @@ examples, here are the criteria for an example addition.
  * The example has a documentation component.
    * This is added at ``doc/source/examples/${name}.rst``
 -  * A reference to ``examples/${name}`` is added to the toctree in ``doc/source/examples.rst``
 +  * An entry for ``examples/${name}`` is added to the toctree in ``doc/source/using_examples.rst``
    * This documentation discusses the project elements declared in the project and may
      provide some BuildStream command examples.
    * This documentation links out to the reference manual at every opportunity.
 +  .. note::
++
 +     In the case of a tutorial chapter, the ``.rst`` file is added in at
 +     ``doc/source/tutorial/${name}.rst`` and an entry for ``tutorial/${name}``
 +     is added to ``doc/source/using_tutorial.rst``.
++
  * The example has a CI test component.
    * This is an integration test added at ``tests/examples/${name}``.

README.rst

@@ -19,7 +19,8 @@ About
  What is BuildStream?
  ====================
 -BuildStream is a Free Software tool for building/integrating software stacks.
 +`BuildStream <https://buildstream.build>`_ is a Free Software tool for
 +building/integrating software stacks.
  It takes inspiration, lessons and use-cases from various projects including
  OBS, Reproducible Builds, Yocto, Baserock, Buildroot, Aboriginal, GNOME Continuous,
  JHBuild, Flatpak Builder and Android repo.

buildstream/__init__.py

@@ -28,9 +28,9 @@ if "_BST_COMPLETION" not in os.environ:
      from .utils import UtilError, ProgramNotFoundError
      from .sandbox import Sandbox, SandboxFlags
 +    from .types import Scope, Consistency
      from .plugin import Plugin
 -    from .source import Source, SourceError, Consistency, SourceFetcher
 +    from .source import Source, SourceError, SourceFetcher
      from .element import Element, ElementError
 -    from .element_enums import Scope
      from .buildelement import BuildElement
      from .scriptelement import ScriptElement

buildstream/_artifactcache/artifactcache.py

@@ -21,7 +21,7 @@ import os
  import string
  from collections import Mapping, namedtuple
 -from ..element_enums import _KeyStrength
 +from ..types import _KeyStrength
  from .._exceptions import ArtifactError, ImplError, LoadError, LoadErrorReason
  from .._message import Message, MessageType
  from .. import utils
@@ -426,6 +426,22 @@ class ArtifactCache():
          raise ImplError("Cache '{kind}' does not implement contains()"
                          .format(kind=type(self).__name__))
 +    # contains_subdir_artifact():
 +    #
 +    # Check whether an artifact element contains a digest for a subdir
 +    # which is populated in the cache, i.e non dangling.
 +    #
 +    # Args:
 +    #     element (Element): The Element to check
 +    #     key (str): The cache key to use
 +    #     subdir (str): The subdir to check
 +    #
 +    # Returns: True if the subdir exists & is populated in the cache, False otherwise
 +    #
 +    def contains_subdir_artifact(self, element, key, subdir):
 +        raise ImplError("Cache '{kind}' does not implement contains_subdir_artifact()"
 +                        .format(kind=type(self).__name__))
++
      # list_artifacts():
+     #
      # List artifacts in this cache in LRU order.
@@ -551,11 +567,12 @@ class ArtifactCache():
      #     element (Element): The Element whose artifact is to be fetched
      #     key (str): The cache key to use
      #     progress (callable): The progress callback, if any
 +    #     subdir (str): The optional specific subdir to pull
+     #
      # Returns:
      #   (bool): True if pull was successful, False if artifact was not available
+     #
 -    def pull(self, element, key, *, progress=None):
 +    def pull(self, element, key, *, progress=None, subdir=None, excluded_subdirs=None):
          raise ImplError("Cache '{kind}' does not implement pull()"
                          .format(kind=type(self).__name__))

buildstream/_artifactcache/cascache.py

@@ -92,6 +92,16 @@ class CASCache(ArtifactCache):
          # This assumes that the repository doesn't have any dangling pointers
          return os.path.exists(refpath)
 +    def contains_subdir_artifact(self, element, key, subdir):
 +        tree = self.resolve_ref(self.get_artifact_fullname(element, key))
++
 +        # This assumes that the subdir digest is present in the element tree
 +        subdirdigest = self._get_subdir(tree, subdir)
 +        objpath = self.objpath(subdirdigest)
++
 +        # True if subdir content is cached or if empty as expected
 +        return os.path.exists(objpath)
++
      def extract(self, element, key):
          ref = self.get_artifact_fullname(element, key)
@@ -228,7 +238,7 @@ class CASCache(ArtifactCache):
              remotes_for_project = self._remotes[element._get_project()]
              return any(remote.spec.push for remote in remotes_for_project)
 -    def pull(self, element, key, *, progress=None):
 +    def pull(self, element, key, *, progress=None, subdir=None, excluded_subdirs=None):
          ref = self.get_artifact_fullname(element, key)
          project = element._get_project()
@@ -247,8 +257,14 @@ class CASCache(ArtifactCache):
                  tree.hash = response.digest.hash
                  tree.size_bytes = response.digest.size_bytes
 -                self._fetch_directory(remote, tree)
 +                # Check if the element artifact is present, if so just fetch subdir
 +                if subdir and os.path.exists(self.objpath(tree)):
 +                    self._fetch_subdir(remote, tree, subdir)
 +                else:
 +                    # Fetch artifact, excluded_subdirs determined in pullqueue
 +                    self._fetch_directory(remote, tree, excluded_subdirs=excluded_subdirs)
 +                # tree is the remote value, so is the same without or without dangling ref locally
                  self.set_ref(ref, tree)
                  element.info("Pulled artifact {} <- {}".format(display_key, remote.spec.url))
@@ -365,7 +381,10 @@ class CASCache(ArtifactCache):
          Raises: ArtifactError if no push remotes are configured.
          """
 -        push_remotes = [r for r in self._remotes[project] if r.spec.push]
 +        if self._has_push_remotes:
 +            push_remotes = [r for r in self._remotes[project] if r.spec.push]
 +        else:
 +            push_remotes = []
          if not push_remotes:
              raise ArtifactError("CASCache: push_directory was called, but no remote artifact " +
@@ -668,8 +687,10 @@ class CASCache(ArtifactCache):
                           stat.S_IRGRP | stat.S_IXGRP | stat.S_IROTH | stat.S_IXOTH)
          for dirnode in directory.directories:
 -            fullpath = os.path.join(dest, dirnode.name)
 -            self._checkout(fullpath, dirnode.digest)
 +            # Don't try to checkout a dangling ref
 +            if os.path.exists(self.objpath(dirnode.digest)):
 +                fullpath = os.path.join(dest, dirnode.name)
 +                self._checkout(fullpath, dirnode.digest)
          for symlinknode in directory.symlinks:
              # symlink
@@ -948,10 +969,12 @@ class CASCache(ArtifactCache):
      #     remote (Remote): The remote to use.
      #     dir_digest (Digest): Digest object for the directory to fetch.
+     #
 -    def _fetch_directory(self, remote, dir_digest):
 +    def _fetch_directory(self, remote, dir_digest, *, excluded_subdirs=None):
          fetch_queue = [dir_digest]
          fetch_next_queue = []
          batch = _CASBatchRead(remote)
 +        if not excluded_subdirs:
 +            excluded_subdirs = []
          while len(fetch_queue) + len(fetch_next_queue) > 0:
              if len(fetch_queue) == 0:
@@ -966,8 +989,9 @@ class CASCache(ArtifactCache):
                  directory.ParseFromString(f.read())
              for dirnode in directory.directories:
 -                batch = self._fetch_directory_node(remote, dirnode.digest, batch,
 -                                                   fetch_queue, fetch_next_queue, recursive=True)
 +                if dirnode.name not in excluded_subdirs:
 +                    batch = self._fetch_directory_node(remote, dirnode.digest, batch,
 +                                                       fetch_queue, fetch_next_queue, recursive=True)
              for filenode in directory.files:
                  batch = self._fetch_directory_node(remote, filenode.digest, batch,
@@ -976,6 +1000,10 @@ class CASCache(ArtifactCache):
          # Fetch final batch
          self._fetch_directory_batch(remote, batch, fetch_queue, fetch_next_queue)
 +    def _fetch_subdir(self, remote, tree, subdir):
 +        subdirdigest = self._get_subdir(tree, subdir)
 +        self._fetch_directory(remote, subdirdigest)
++
      def _fetch_tree(self, remote, digest):
          # download but do not store the Tree object
          with tempfile.NamedTemporaryFile(dir=self.tmpdir) as out:

buildstream/_context.py

@@ -110,6 +110,9 @@ class Context():
          # Make sure the XDG vars are set in the environment before loading anything
          self._init_xdg()
 +        # Whether or not to attempt to pull buildtrees globally
 +        self.pullbuildtrees = False
++
          # Private variables
          self._cache_key = None
          self._message_handler = None
@@ -160,7 +163,7 @@ class Context():
          _yaml.node_validate(defaults, [
              'sourcedir', 'builddir', 'artifactdir', 'logdir',
              'scheduler', 'artifacts', 'logging', 'projects',
 -            'cache'
 +            'cache', 'pullbuildtrees'
          ])
          for directory in ['sourcedir', 'builddir', 'artifactdir', 'logdir']:
@@ -185,6 +188,9 @@ class Context():
          # Load artifact share configuration
          self.artifact_cache_specs = ArtifactCache.specs_from_config_node(defaults)
 +        # Load pull buildtrees configuration
 +        self.pullbuildtrees = _yaml.node_get(defaults, bool, 'pullbuildtrees', default_value='False')
++
          # Load logging config
          logging = _yaml.node_get(defaults, Mapping, 'logging')
          _yaml.node_validate(logging, [

buildstream/_frontend/cli.py

@@ -305,10 +305,12 @@ def init(app, project_name, format_version, element_path, force):
                help="Allow tracking to cross junction boundaries")
  @click.option('--track-save', default=False, is_flag=True,
                help="Deprecated: This is ignored")
 +@click.option('--pull-buildtrees', default=False, is_flag=True,
 +              help="Pull buildtrees from a remote cache server")
  @click.argument('elements', nargs=-1,
                  type=click.Path(readable=False))
  @click.pass_obj
 -def build(app, elements, all_, track_, track_save, track_all, track_except, track_cross_junctions):
 +def build(app, elements, all_, track_, track_save, track_all, track_except, track_cross_junctions, pull_buildtrees):
      """Build elements in a pipeline"""
      if (track_except or track_cross_junctions) and not (track_ or track_all):
@@ -327,7 +329,8 @@ def build(app, elements, all_, track_, track_save, track_all, track_except, trac
                           track_targets=track_,
                           track_except=track_except,
                           track_cross_junctions=track_cross_junctions,
 -                         build_all=all_)
 +                         build_all=all_,
 +                         pull_buildtrees=pull_buildtrees)
  ##################################################################
@@ -429,10 +432,12 @@ def track(app, elements, deps, except_, cross_junctions):
                help='The dependency artifacts to pull (default: none)')
  @click.option('--remote', '-r',
                help="The URL of the remote cache (defaults to the first configured cache)")
 +@click.option('--pull-buildtrees', default=False, is_flag=True,
 +              help="Pull buildtrees from a remote cache server")
  @click.argument('elements', nargs=-1,
                  type=click.Path(readable=False))
  @click.pass_obj
 -def pull(app, elements, deps, remote):
 +def pull(app, elements, deps, remote, pull_buildtrees):
      """Pull a built artifact from the configured remote artifact cache.
      By default the artifact will be pulled one of the configured caches
@@ -446,7 +451,7 @@ def pull(app, elements, deps, remote):
          all:   All dependencies
      """
      with app.initialized(session_name="Pull"):
 -        app.stream.pull(elements, selection=deps, remote=remote)
 +        app.stream.pull(elements, selection=deps, remote=remote, pull_buildtrees=pull_buildtrees)
  ##################################################################

buildstream/_scheduler/queues/pullqueue.py

@@ -32,9 +32,20 @@ class PullQueue(Queue):
      complete_name = "Pulled"
      resources = [ResourceType.DOWNLOAD, ResourceType.CACHE]
 +    def __init__(self, scheduler, buildtrees=False):
 +        super().__init__(scheduler)
++
 +        # Current default exclusions on pull
 +        self._excluded_subdirs = ["buildtree"]
 +        self._subdir = None
 +        # If buildtrees are to be pulled, remove the value from exclusion list
 +        if buildtrees:
 +            self._subdir = "buildtree"
 +            self._excluded_subdirs.remove(self._subdir)
++
      def process(self, element):
          # returns whether an artifact was downloaded or not
 -        if not element._pull():
 +        if not element._pull(subdir=self._subdir, excluded_subdirs=self._excluded_subdirs):
              raise SkipJob(self.action_name)
      def status(self, element):
@@ -49,7 +60,7 @@ class PullQueue(Queue):
          if not element._can_query_cache():
              return QueueStatus.WAIT
 -        if element._pull_pending():
 +        if element._pull_pending(subdir=self._subdir):
              return QueueStatus.READY
          else:
              return QueueStatus.SKIP

buildstream/_stream.py

@@ -160,12 +160,14 @@ class Stream():
      #    track_cross_junctions (bool): Whether tracking should cross junction boundaries
      #    build_all (bool): Whether to build all elements, or only those
      #                      which are required to build the target.
 +    #    pull_buildtrees (bool): Whether to pull buildtrees from a remote cache server
+     #
      def build(self, targets, *,
                track_targets=None,
                track_except=None,
                track_cross_junctions=False,
 -              build_all=False):
 +              build_all=False,
 +              pull_buildtrees=False):
          if build_all:
              selection = PipelineSelection.ALL
@@ -195,7 +197,10 @@ class Stream():
              self._add_queue(track_queue, track=True)
          if self._artifacts.has_fetch_remotes():
 -            self._add_queue(PullQueue(self._scheduler))
 +            # Query if pullbuildtrees has been set globally in user config
 +            if self._context.pullbuildtrees:
 +                pull_buildtrees = True
 +            self._add_queue(PullQueue(self._scheduler, buildtrees=pull_buildtrees))
          self._add_queue(FetchQueue(self._scheduler, skip_cached=True))
          self._add_queue(BuildQueue(self._scheduler))
@@ -295,7 +300,8 @@ class Stream():
+     #
      def pull(self, targets, *,
               selection=PipelineSelection.NONE,
 -             remote=None):
 +             remote=None,
 +             pull_buildtrees=False):
          use_config = True
          if remote:
@@ -310,8 +316,12 @@ class Stream():
          if not self._artifacts.has_fetch_remotes():
              raise StreamError("No artifact caches available for pulling artifacts")
 +        # Query if pullbuildtrees has been set globally in user config
 +        if self._context.pullbuildtrees:
 +            pull_buildtrees = True
++
          self._pipeline.assert_consistent(elements)
 -        self._add_queue(PullQueue(self._scheduler))
 +        self._add_queue(PullQueue(self._scheduler, buildtrees=pull_buildtrees))
          self._enqueue_plan(elements)
          self._run()

buildstream/element.py

@@ -86,7 +86,7 @@ from ._variables import Variables
  from ._versions import BST_CORE_ARTIFACT_VERSION
  from ._exceptions import BstError, LoadError, LoadErrorReason, ImplError, ErrorDomain
  from .utils import UtilError
 -from . import Plugin, Consistency
 +from . import Plugin, Consistency, Scope
  from . import SandboxFlags
  from . import utils
  from . import _cachekey
@@ -96,11 +96,11 @@ from ._platform import Platform
  from .plugin import CoreWarnings
  from .sandbox._config import SandboxConfig
  from .sandbox._sandboxremote import SandboxRemote
 +from .types import _KeyStrength
  from .storage.directory import Directory
  from .storage._filebaseddirectory import FileBasedDirectory
  from .storage.directory import VirtualDirectoryError
 -from .element_enums import _KeyStrength, Scope
  class ElementError(BstError):
@@ -1692,18 +1692,26 @@ class Element(Plugin):
      # _pull_pending()
+     #
 -    # Check whether the artifact will be pulled.
 +    # Check whether the artifact will be pulled. If the pull operation is to
 +    # include a specific subdir of the element artifact (from cli or user conf)
 +    # then the local cache is queried for the subdirs existence.
 +    #
 +    # Args:
 +    #    subdir (str): Whether the pull has been invoked with a specific subdir set
+     #
      # Returns:
      #   (bool): Whether a pull operation is pending
+     #
 -    def _pull_pending(self):
 +    def _pull_pending(self, subdir=None):
          if self._get_workspace():
              # Workspace builds are never pushed to artifact servers
              return False
 -        if self.__strong_cached:
 -            # Artifact already in local cache
 +        if self.__strong_cached and subdir:
 +            # If we've specified a subdir, check if the subdir is cached locally
 +            if self.__artifacts.contains_subdir_artifact(self, self.__strict_cache_key, subdir):
 +                return False
 +        elif self.__strong_cached:
              return False
          # Pull is pending if artifact remote server available
@@ -1725,11 +1733,10 @@ class Element(Plugin):
          self._update_state()
 -    def _pull_strong(self, *, progress=None):
 +    def _pull_strong(self, *, progress=None, subdir=None, excluded_subdirs=None):
          weak_key = self._get_cache_key(strength=_KeyStrength.WEAK)
+-
          key = self.__strict_cache_key
 -        if not self.__artifacts.pull(self, key, progress=progress):
 +        if not self.__artifacts.pull(self, key, progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs):
              return False
          # update weak ref by pointing it to this newly fetched artifact
@@ -1737,10 +1744,10 @@ class Element(Plugin):
          return True
 -    def _pull_weak(self, *, progress=None):
 +    def _pull_weak(self, *, progress=None, subdir=None, excluded_subdirs=None):
          weak_key = self._get_cache_key(strength=_KeyStrength.WEAK)
+-
 -        if not self.__artifacts.pull(self, weak_key, progress=progress):
 +        if not self.__artifacts.pull(self, weak_key, progress=progress, subdir=subdir,
 +                                     excluded_subdirs=excluded_subdirs):
              return False
          # extract strong cache key from this newly fetched artifact
@@ -1758,17 +1765,17 @@ class Element(Plugin):
+     #
      # Returns: True if the artifact has been downloaded, False otherwise
+     #
 -    def _pull(self):
 +    def _pull(self, subdir=None, excluded_subdirs=None):
          context = self._get_context()
          def progress(percent, message):
              self.status(message)
          # Attempt to pull artifact without knowing whether it's available
 -        pulled = self._pull_strong(progress=progress)
 +        pulled = self._pull_strong(progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs)
          if not pulled and not self._cached() and not context.get_strict():
 -            pulled = self._pull_weak(progress=progress)
 +            pulled = self._pull_weak(progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs)
          if not pulled:
              return False
@@ -1791,10 +1798,21 @@ class Element(Plugin):
          if not self._cached():
              return True
 -        # Do not push tained artifact
 +        # Do not push tainted artifact
          if self.__get_tainted():
              return True
 +        # Do not push elements that have a dangling buildtree artifact unless element type is
 +        # expected to have an empty buildtree directory
 +        if not self.__artifacts.contains_subdir_artifact(self, self.__strict_cache_key, 'buildtree'):
 +            return True
++
 +        # strict_cache_key can't be relied on to be available when running in non strict mode
 +        context = self._get_context()
 +        if not context.get_strict():
 +            if not self.__artifacts.contains_subdir_artifact(self, self.__weak_cache_key, 'buildtree'):
 +                return True
++
          return False
      # _push():

buildstream/source.py

@@ -145,35 +145,12 @@ import os
  from collections import Mapping
  from contextlib import contextmanager
 -from . import Plugin
 +from . import Plugin, Consistency
  from . import _yaml, utils
  from ._exceptions import BstError, ImplError, ErrorDomain
  from ._projectrefs import ProjectRefStorage
 -class Consistency():
 -    INCONSISTENT = 0
 -    """Inconsistent
+-
 -    Inconsistent sources have no explicit reference set. They cannot
 -    produce a cache key, be fetched or staged. They can only be tracked.
 -    """
+-
 -    RESOLVED = 1
 -    """Resolved
+-
 -    Resolved sources have a reference and can produce a cache key and
 -    be fetched, however they cannot be staged.
 -    """
+-
 -    CACHED = 2
 -    """Cached
+-
 -    Cached sources have a reference which is present in the local
 -    source cache. Only cached sources can be staged.
 -    """
+-
+-
  class SourceError(BstError):
      """This exception should be raised by :class:`.Source` implementations
      to report errors to the user.

buildstream/element_enums.py → buildstream/types.py

@@ -19,31 +19,19 @@
  #        Jim MacArthur <jim macarthur codethink co uk>
  """
 -Element - Globally visible enumerations
 -=======================================
 +Foundation types
 +================
  """
  from enum import Enum
 -# _KeyStrength():
 -#
 -# Strength of cache key
 -#
 -class _KeyStrength(Enum):
+-
 -    # Includes strong cache keys of all build dependencies and their
 -    # runtime dependencies.
 -    STRONG = 1
+-
 -    # Includes names of direct build dependencies but does not include
 -    # cache keys of dependencies.
 -    WEAK = 2
+-
+-
  class Scope(Enum):
 -    """Types of scope for a given element"""
 +    """Defines the scope of dependencies to include for a given element
 +    when iterating over the dependency graph in APIs like
 +    :func:`Element.dependencies() <buildstream.element.Element.dependencies>`
 +    """
      ALL = 1
      """All elements which the given element depends on, following
@@ -59,3 +47,44 @@ class Scope(Enum):
      """All elements required for running the element. Including the element
      itself.
      """
++
++
 +class Consistency():
 +    """Defines the various consistency states of a :class:`.Source`.
 +    """
++
 +    INCONSISTENT = 0
 +    """Inconsistent
++
 +    Inconsistent sources have no explicit reference set. They cannot
 +    produce a cache key, be fetched or staged. They can only be tracked.
 +    """
++
 +    RESOLVED = 1
 +    """Resolved
++
 +    Resolved sources have a reference and can produce a cache key and
 +    be fetched, however they cannot be staged.
 +    """
++
 +    CACHED = 2
 +    """Cached
++
 +    Cached sources have a reference which is present in the local
 +    source cache. Only cached sources can be staged.
 +    """
++
++
 +# _KeyStrength():
 +#
 +# Strength of cache key
 +#
 +class _KeyStrength(Enum):
++
 +    # Includes strong cache keys of all build dependencies and their
 +    # runtime dependencies.
 +    STRONG = 1
++
 +    # Includes names of direct build dependencies but does not include
 +    # cache keys of dependencies.
 +    WEAK = 2

doc/source/additional_docker.rst

@@ -4,19 +4,18 @@
  BuildStream and Docker
  ======================
+-
  BuildStream integrates with Docker in multiple ways. Here are some ways in
  which these integrations work.
++
  Run BuildStream inside Docker
  -----------------------------
 +Refer to the `BuildStream inside Docker <https://buildstream.build/docker_install.html>`_
 +documentation for instructions on how to run BuildStream as a Docker container.
 -Refer to the :ref:`BuildStream inside Docker <docker>` documentation for
 -instructions on how to run BuildStream as a Docker container.
  Generate Docker images
  ----------------------
+-
  The
  `bst-docker-import script <https://gitlab.com/BuildStream/buildstream/blob/master/contrib/bst-docker-import>`_
  can be used to generate a Docker image from built artifacts.

doc/source/core_framework.rst

@@ -12,6 +12,7 @@ useful for working on BuildStream itself.
  .. toctree::
     :maxdepth: 1
 +   buildstream.types
     buildstream.plugin
     buildstream.source
     buildstream.element

doc/source/index.rst

@@ -13,20 +13,13 @@ They begin with a basic introduction to BuildStream, background
  information on basic concepts, and a guide to the BuildStream command line interface.
  Later sections provide detailed information on BuildStream internals.
+-
  .. toctree::
     :maxdepth: 1
     main_about
 -   main_install
     main_using
     main_core
     CONTRIBUTING
+-
 -Resources
 ----------
 -* GitLab repository: https://gitlab.com/BuildStream/buildstream
 -* Bug Tracking: https://gitlab.com/BuildStream/buildstream/issues
 -* Mailing list: https://mail.gnome.org/mailman/listinfo/buildstream-list
 -* IRC Channel: irc://irc.gnome.org/#buildstream
 +For any other information, including `how to install BuildStream <https://buildstream.build/install.html>`_,
 +refer to `the BuildStream website <https://buildstream.build>`_.

doc/source/install_docker.rst deleted

+-
+-
 -.. _docker:
+-
 -BuildStream inside Docker
 --------------------------
 -If your system cannot provide the base system requirements for BuildStream, then it is possible to run buildstream within a Docker image.
+-
 -The BuildStream project provides
 -`Docker images <https://hub.docker.com/r/buildstream/buildstream-fedora>`_
 -containing BuildStream and its dependencies.
 -This gives you an easy way to get started using BuildStream on any Unix-like
 -platform where Docker is available, including Mac OS X.
+-
 -We recommend using the
 -`bst-here wrapper script <https://gitlab.com/BuildStream/buildstream/blob/master/contrib/bst-here>`_
 -which automates the necessary container setup. You can download it and make
 -it executable like this:
+-
 -.. code:: bash
+-
 -  mkdir -p ~/.local/bin
 -  curl --get https://gitlab.com/BuildStream/buildstream/raw/master/contrib/bst-here > ~/.local/bin/bst-here
 -  chmod +x ~/.local/bin/bst-here
+-
 -Check if ``~/.local/bin`` appears in your PATH environment variable -- if it
 -doesn't, you should
 -`edit your ~/.profile so that it does <https://stackoverflow.com/questions/14637979/>`_.
+-
 -Once the script is available in your PATH, you can run ``bst-here`` to open a
 -shell session inside a new container based off the latest version of the
 -buildstream-fedora Docker image. The current working directory will be mounted
 -inside the container at ``/src``.
+-
 -You can also run individual BuildStream commands as ``bst-here COMMAND``. For
 -example: ``bst-here show systems/my-system.bst``. Note that BuildStream won't
 -be able to integrate with Bash tab-completion if you invoke it in this way.
+-
 -Two Docker volumes are set up by the ``bst-here`` script:
+-
 - * ``buildstream-cache --`` mounted at ``~/.cache/buildstream``
 - * ``buildstream-config --`` mounted at ``~/.config/``
+-
 -These are necessary so that your BuildStream cache and configuration files
 -persist between invocations of ``bst-here``.

doc/source/install_linux_distro.rst deleted

+-
+-
 -.. _install_linux_distro:
+-
 -Installing from distro packages
 -===============================
 -BuildStream is available on some linux distributions, here are
 -some install instructions for the linux distributions which
 -have packaged BuildStream.
+-
+-
 -Arch Linux
 -----------
 -Packages for Arch exist in `AUR <https://wiki.archlinux.org/index.php/Arch_User_Repository#Installing_packages>`_.
 -Two different package versions are available:
+-
 -* Latest release: `buildstream <https://aur.archlinux.org/packages/buildstream>`_
 -* Latest development snapshot: `buildstream-git <https://aur.archlinux.org/packages/buildstream-git>`_
+-
+-
 -Fedora
 -------
 -BuildStream is not yet in the official Fedora repositories, but you can
 -install it from a Copr::
+-
 -  sudo dnf copr enable bochecha/buildstream
 -  sudo dnf install buildstream
+-
 -Optionally, install the ``buildstream-docs`` package to have the BuildStream
 -documentation in Devhelp or GNOME Builder.

doc/source/install_source.rst deleted

+-
+-
 -Installing from source
 -======================
 -Until BuildStream is available in :ref:`your distro <install_linux_distro>`, you will
 -need to install it yourself from source.
+-
+-
 -Installing dependencies
 ------------------------
 -Before installing BuildStream from source, it is necessary to first install
 -the system dependencies. Below are some linux distribution specific instructions
 -for installing these dependencies.
+-
 -BuildStream requires the following base system requirements:
+-
 -* python3 >= 3.5
 -* bubblewrap >= 0.1.2
 -* fuse2
+-
 -BuildStream also depends on the host tools for the :mod:`Source <buildstream.source>` plugins.
 -Refer to the respective :ref:`source plugin <plugins_sources>` documentation for host tool
 -requirements of specific plugins.
+-
 -The default plugins with extra host dependencies are:
+-
 -* bzr
 -* deb
 -* git
 -* ostree
 -* patch
 -* pip
 -* tar
+-
 -If you intend to push built artifacts to a remote artifact server,
 -which requires special permissions, you will also need:
+-
 -* ssh
+-
+-
 -Arch Linux
 -~~~~~~~~~~
 -Install the dependencies with::
+-
 -  sudo pacman -S \
 -      python fuse2 bubblewrap \
 -      python-pip
+-
 -For the default plugins::
+-
 -  sudo pacman -S \
 -      bzr git lzip ostree patch python-gobject
+-
+-
 -The package *python-arpy* is required by the deb source plugin. This is not
 -obtainable via `pacman`, you must get *python-arpy* from AUR:
 -https://aur.archlinux.org/packages/python-arpy/
+-
 -To install::
+-
 -  wget https://aur.archlinux.org/cgit/aur.git/snapshot/python-arpy.tar.gz
 -  tar -xvf python-arpy.tar.gz
 -  cd python-arpy
 -  makepkg -si
+-
+-
 -Debian
 -~~~~~~
 -Install the dependencies with::
+-
 -  sudo apt-get install \
 -      python3 fuse bubblewrap \
 -      python3-pip python3-dev
+-
 -For the default plugins:
+-
+-
 -Stretch
 -+++++++
 -With stretch, you first need to ensure that you have the backports repository
 -setup as described `here <https://backports.debian.org/Instructions/>`_
+-
 -By adding the following line to your sources.list::
+-
 -  deb http://deb.debian.org/debian stretch-backports main
+-
 -And then running::
+-
 -  sudo apt update
+-
 -At this point you should be able to get the system requirements for the default plugins with::
+-
 -  sudo apt install \
 -      bzr git lzip patch python3-arpy python3-gi
 -  sudo apt install -t stretch-backports \
 -      gir1.2-ostree-1.0 ostree
+-
+-
 -Buster or Sid
 -+++++++++++++
 -For debian unstable or testing, only the following line should be enough
 -to get the system requirements for the default plugins installed::
+-
 -  sudo apt-get install \
 -      lzip gir1.2-ostree-1.0 git bzr ostree patch python3-arpy python3-gi
+-
+-
 -Fedora
 -~~~~~~
 -For recent fedora systems, the following line should get you the system
 -requirements you need::
+-
 -  dnf install -y \
 -      python3 fuse bubblewrap \
 -      python3-pip python3-devel
+-
 -For the default plugins::
+-
 -  dnf install -y \
 -      bzr git lzip patch ostree python3-gobject
 -  pip3 install --user arpy
+-
+-
 -Ubuntu
 -~~~~~~
+-
+-
 -Ubuntu 18.04 LTS or later
 -+++++++++++++++++++++++++
 -Install the dependencies with::
+-
 -  sudo apt install \
 -      python3 fuse bubblewrap \
 -      python3-pip python3-dev
+-
 -For the default plugins::
+-
 -  sudo apt install \
 -      bzr gir1.2-ostree-1.0 git lzip ostree patch python3-arpy python3-gi
+-
+-
 -Ubuntu 16.04 LTS
 -++++++++++++++++
 -On Ubuntu 16.04, neither `bubblewrap <https://github.com/projectatomic/bubblewrap/>`_
 -or `ostree <https://github.com/ostreedev/ostree>`_ are available in the official repositories.
 -You will need to install them in whichever way you see fit. Refer the the upstream documentation
 -for advice on this.
+-
+-
 -Installing
 -----------
 -Once you have the base system dependencies, you can install the BuildStream
 -python package as a regular user.
+-
+-
 -Installing from PyPI (recommended)
 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 -Since we only ever publish :ref:`release versions <install_semantic_versioning>` on
 -PyPI, it is currently recommended to use this installation path. This will
 -ensure that you always have the latest recommended version of BuildStream that
 -we recommend.
+-
 -To install from PyPI, you will additionally require:
+-
 -* pip for python3 (only required for setup)
 -* Python 3 development libraries and headers
+-
 -Simply run the following command::
+-
 -  pip3 install --user BuildStream
+-
 -This will install latest stable version of BuildStream and its pure python
 -dependencies into your user's homedir in ``~/.local``.
+-
 -Keep following the instructions below to ensure that the ``bst``
 -command is in your ``PATH`` and to enable bash completions for it.
+-
 -.. note::
+-
 -  If you want a specific version of BuildStream, you can install it using
 -  ``pip install --user BuildStream==<version-number>``
+-
+-
 -Upgrading from PyPI
 -+++++++++++++++++++
 -Once you have already installed BuildStream from PyPI, you can later update
 -to the latest recommended version like so::
+-
 -  pip install --user --upgrade BuildStream
+-
+-
 -.. _install_git_checkout:
+-
 -Installing from a git checkout
 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 -To install directly from the `git repository <https://gitlab.com/BuildStream/buildstream.git>`_
 -using python's ``pip`` package manager, you will additionally require:
+-
 -* pip for python3 (only required for setup)
 -* Python 3 development libraries and headers
 -* git (to checkout BuildStream)
+-
 -Before installing, please check the existing tags in the git repository
 -and determine which version you want to install, and whether you want
 -to install an official release version (recommended), or a development snapshot
 -to help us out testing the bleeding edge of development. Follow the
 -:ref:`semantic versioning guide <install_semantic_versioning>` to determine
 -which tag you intend to install.
+-
 -Run the following commands::
+-
 -  git clone https://gitlab.com/BuildStream/buildstream.git
 -  cd buildstream
 -  git checkout <desired release tag>
 -  pip3 install --user -e .
+-
 -This will install buildstream's pure python dependencies into
 -your user's homedir in ``~/.local`` and will run BuildStream directly
 -from the git checkout directory.
+-
 -Keep following the instructions below to ensure that the ``bst``
 -command is in your ``PATH`` and to enable bash completions for it.
+-
 -.. note::
+-
 -   We recommend the ``-e`` option because you can upgrade your
 -   installation by simply updating the checked out git repository.
+-
 -   If you want a full installation that is not linked to your
 -   git checkout, just omit the ``-e`` option from the above commands.
+-
+-
 -Upgrading from a git checkout
 -+++++++++++++++++++++++++++++
 -If you installed BuildStream from a local git checkout using ``-e`` option, all
 -you need to do to upgrade BuildStream is to update your local git checkout::
+-
 -  cd /path/to/buildstream
 -  git pull --rebase
+-
 -If you did not specify the ``-e`` option at install time or the dependancies
 -have changed, you will need to cleanly reinstall BuildStream::
+-
 -  pip3 uninstall buildstream
 -  cd /path/to/buildstream
 -  git pull --rebase
 -  pip3 install --user .
+-
 -.. note::
+-
 -   If BuildStream has added any dependencies since the last upgrade,
 -   you will need to uninstall and reinstall to ensure those dependencies
 -   are met, regardless of whether you have used the ``-e`` option at
 -   install time.
+-
+-
 -Post install setup
 -------------------
 -After having installed from source using any of the above methods, some
 -setup will be required to use BuildStream.
+-
+-
 -Adjust PATH
 -~~~~~~~~~~~
 -Since BuildStream is now installed under your local user's install directories,
 -you need to ensure that ``PATH`` is adjusted.
+-
 -A regular way to do this is to add the following line to the end of your ``~/.bashrc``::
+-
 -  export PATH="${PATH}:${HOME}/.local/bin"
+-
 -.. note::
+-
 -   You will have to restart your terminal in order for these changes to take effect.
+-
+-
 -Bash completions
 -~~~~~~~~~~~~~~~~
 -Bash completions are supported by sourcing the ``buildstream/data/bst``
 -script found in the BuildStream repository. On many systems this script
 -can be installed into a completions directory but when installing BuildStream
 -without a package manager this is not an option.
+-
 -To enable completions for an installation of BuildStream you
 -installed yourself from git, just append the script verbatim
 -to your ``~/.bash_completion``:
+-
 -.. literalinclude:: ../../buildstream/data/bst
 -   :language: yaml

doc/source/install_versions.rst deleted

+-
+-
 -.. _install_semantic_versioning:
+-
 -Semantic Versioning
 -===================
 -BuildStream follows the Semantic Versioning Convention `(SemVer) <https://semver.org/>`_,
 -and uses even minor point numbers to denote releases intended for users while
 -odd minor point numbers represent development snapshops.
+-
 -For example, for a given version number ``X.Y.Z``
 - * The ``X.<even number>.*`` versions are releases intended for users.
 - * The ``X.<odd number>.*`` versions are development spanshots intended for testing.
+-
 -If you are :ref:`installing from git <install_git_checkout>`, please look for the latest
 -tag to ensure you're getting the latest release.
+-
 -* Latest release:
+-
 -  .. include:: release-badge.rst
+-
 -* Latest development snapshot:
+-
 -  .. include:: snapshot-badge.rst

doc/source/main_install.rst deleted

+-
+-
 -.. _install:
+-
 -Install
 -=======
+-
 -.. include:: release-badge.rst
+-
 -.. include:: snapshot-badge.rst
+-
 -This section provides instructions for installing BuildStream and its
 -companion artifact server on various platforms, along with any installation
 -related materials.
+-
 -.. note::
+-
 -   BuildStream is currently only supported natively on Linux. Users of Unix-like
 -   systems where Docker is available can still use BuildStream by following the
 -   :ref:`Docker install guide <docker>`
+-
 -.. toctree::
 -   :maxdepth: 1
+-
 -   install_source
 -   install_linux_distro
 -   install_docker
 -   install_artifacts
 -   install_versions

doc/source/main_using.rst

 +.. _using:
++
  Using
  =====
  This section includes user facing documentation including tutorials,
@@ -15,3 +17,4 @@ guides and information on user preferences and configuration.
     using_examples
     using_config
     using_commands
 +   using_configuring_artifact_server

doc/source/release-badge.rst deleted

+-
 -.. Use this file to include the badge in the documentation, but not in
 -   the README.rst or gitlab rendered materials, that doesnt work.
+-
 -   This is partly a workaround for a sphinx issue, we will be able
 -   to avoid the raw html once this is implemented in sphinx:
+-
 -       https://github.com/sphinx-doc/sphinx/issues/2240
+-
 -   Using the <object> tag instead of the <img> tag which sphinx generates
 -   allows the svg to be "interactive", for us this basically means that
 -   the link we encode in the badge svg is used, rather than static urls
 -   which need to be used around the <img> tag.
+-
 -   WARNING: The custom CSS on the style tag will need to change if we
 -            change the theme, so that the <object> tag behaves similar
 -	    to how the <img> tag is themed by the style sheets.
+-
 -.. raw:: html
+-
 -   <a class="reference external image-reference">
 -     <object style="margin-bottom:24px;vertical-align:middle"
 -             data=""
 -	     type="image/svg+xml"/>
 -     </object>
 -   </a>

doc/source/snapshot-badge.rst deleted

+-
 -.. Use this file to include the badge in the documentation, but not in
 -   the README.rst or gitlab rendered materials, that doesnt work.
+-
 -   This is partly a workaround for a sphinx issue, we will be able
 -   to avoid the raw html once this is implemented in sphinx:
+-
 -       https://github.com/sphinx-doc/sphinx/issues/2240
+-
 -   Using the <object> tag instead of the <img> tag which sphinx generates
 -   allows the svg to be "interactive", for us this basically means that
 -   the link we encode in the badge svg is used, rather than static urls
 -   which need to be used around the <img> tag.
+-
 -   WARNING: The custom CSS on the style tag will need to change if we
 -            change the theme, so that the <object> tag behaves similar
 -	    to how the <img> tag is themed by the style sheets.
+-
 -.. raw:: html
+-
 -   <a class="reference external image-reference">
 -     <object style="margin-bottom:24px;vertical-align:middle"
 -             data=""
 -	     type="image/svg+xml"/>
 -     </object>
 -   </a>

doc/source/install_artifacts.rst → doc/source/using_configuring_artifact_server.rst

@@ -2,8 +2,8 @@
  .. _artifacts:
 -Installing an artifact server
 -=============================
 +Configuring Artifact Server
 +===========================
  BuildStream caches the results of builds in a local artifact cache, and will
  avoid building an element if there is a suitable build already present in the
  local artifact cache.
@@ -72,7 +72,7 @@ Installing the server
  ~~~~~~~~~~~~~~~~~~~~~
  You will also need to install BuildStream on the artifact server in order
  to receive uploaded artifacts over ssh. Follow the instructions for installing
 -BuildStream :ref:`here <install>`
 +BuildStream `here <https://buildstream.build/install.html>`_.
  When installing BuildStream on the artifact server, it must be installed
  in a system wide location, with ``pip3 install .`` in the BuildStream

doc/source/using_examples.rst

++
 +.. _examples:
++
  Examples
  ========
  This page contains documentation for real examples of BuildStream projects,

doc/source/using_tutorial.rst

++
 +.. _tutorial:
++
  Tutorial
  ========
  This is a step by step walkthrough meant help the user quickly get

tests/completions/completions.py

@@ -103,7 +103,7 @@ def test_commands(cli, cmd, word_idx, expected):
      ('bst --no-colors build -', 3, ['--all ', '--track ', '--track-all ',
                                      '--track-except ',
                                      '--track-cross-junctions ', '-J ',
 -                                    '--track-save ']),
 +                                    '--track-save ', '--pull-buildtrees ']),
      # Test the behavior of completing after an option that has a
      # parameter that cannot be completed, vs an option that has

tests/integration/pullbuildtrees.py

 +import os
 +import shutil
 +import pytest
++
 +from tests.testutils import cli_integration as cli, create_artifact_share
 +from tests.testutils.integration import assert_contains
++
++
 +DATA_DIR = os.path.join(
 +    os.path.dirname(os.path.realpath(__file__)),
 +    "project"
 +)
++
++
 +# Remove artifact cache & set cli.config value of pullbuildtrees
 +# to false, which is the default user context
 +def default_state(cli, integration_cache, share):
 +    shutil.rmtree(os.path.join(integration_cache, 'artifacts'))
 +    cli.configure({'pullbuildtrees': False, 'artifacts': {'url': share.repo, 'push': False}})
++
++
 +# A test to capture the integration of the pullbuildtrees
 +# behaviour, which by default is to not include the buildtree
 +# directory of an element
 +@pytest.mark.integration
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_pullbuildtrees(cli, tmpdir, datafiles, integration_cache):
++
 +    # Ensure artifact cache is purged, as we can't have dangling refs/objects
 +    shutil.rmtree(os.path.join(integration_cache, 'artifacts'))
++
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    element_name = 'autotools/amhello.bst'
++
 +    # Create artifact shares for pull & push testing
 +    with create_artifact_share(os.path.join(str(tmpdir), 'share1')) as share1,\
 +        create_artifact_share(os.path.join(str(tmpdir), 'share2')) as share2:
 +        cli.configure({
 +            'artifacts': {'url': share1.repo, 'push': True},
 +        })
++
 +        # Build autotools element, checked pushed, delete local
 +        result = cli.run(project=project, args=['build', element_name])
 +        assert result.exit_code == 0
 +        assert cli.get_element_state(project, element_name) == 'cached'
 +        assert share1.has_artifact('test', element_name, cli.get_element_key(project, element_name))
 +        default_state(cli, integration_cache, share1)
++
 +        # Pull artifact with default config, assert that pulling again
 +        # doesn't create a pull job, then assert with buildtrees user
 +        # config set creates a pull job.
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name not in result.get_pulled_elements()
 +        cli.configure({'pullbuildtrees': True})
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        default_state(cli, integration_cache, share1)
++
 +        # Pull artifact with default config, then assert that pulling
 +        # with buildtrees cli flag set creates a pull job.
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        result = cli.run(project=project, args=['pull', '--pull-buildtrees', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        default_state(cli, integration_cache, share1)
++
 +        # Pull artifact with pullbuildtrees set in user config, then assert
 +        # that pulling with the same user config doesn't creates a pull job,
 +        # or when buildtrees cli flag is set.
 +        cli.configure({'pullbuildtrees': True})
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name not in result.get_pulled_elements()
 +        result = cli.run(project=project, args=['pull', '--pull-buildtrees', element_name])
 +        assert element_name not in result.get_pulled_elements()
 +        default_state(cli, integration_cache, share1)
++
 +        # Pull artifact with default config and buildtrees cli flag set, then assert
 +        # that pulling with pullbuildtrees set in user config doesn't create a pull
 +        # job.
 +        result = cli.run(project=project, args=['pull', '--pull-buildtrees', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        cli.configure({'pullbuildtrees': True})
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name not in result.get_pulled_elements()
 +        default_state(cli, integration_cache, share1)
++
 +        # Assert that a partial build element (not containing a populated buildtree dir)
 +        # can't be pushed to an artifact share, then assert that a complete build element
 +        # can be. This will attempt a partial pull from share1 and then a partial push
 +        # to share2
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        cli.configure({'artifacts': {'url': share2.repo, 'push': True}})
 +        result = cli.run(project=project, args=['push', element_name])
 +        assert element_name not in result.get_pushed_elements()
 +        assert not share2.has_artifact('test', element_name, cli.get_element_key(project, element_name))
++
 +        # Assert that after pulling the missing buildtree the element artifact can be
 +        # successfully pushed to the remote. This will attempt to pull the buildtree
 +        # from share1 and then a 'complete' push to share2
 +        cli.configure({'artifacts': {'url': share1.repo, 'push': False}})
 +        result = cli.run(project=project, args=['pull', '--pull-buildtrees', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        cli.configure({'artifacts': {'url': share2.repo, 'push': True}})
 +        result = cli.run(project=project, args=['push', element_name])
 +        assert element_name in result.get_pushed_elements()
 +        assert share2.has_artifact('test', element_name, cli.get_element_key(project, element_name))

tests/testutils/artifactshare.py

@@ -128,7 +128,7 @@ class ArtifactShare():
          valid_chars = string.digits + string.ascii_letters + '-._'
          element_name = ''.join([
 -            x if x in valid_chars else '_'
 +            x if x in valid_chars else '-'
              for x in element_name
          ])
          artifact_key = '{0}/{1}/{2}'.format(project_name, element_name, cache_key)

[Notes] [Git][BuildStream/buildstream][tpollard/494] 17 commits: Merge branch 'chandan/bst-and-docker' into 'master'

Tom Pollard pushed to branch tpollard/494 at BuildStream / buildstream

Commits:

29 changed files:

Changes: