[Notes] [Git][BuildStream/buildstream][tpollard/494] 10 commits: yaml: A

Tom Pollard pushed to branch tpollard/494 at BuildStream / buildstream

Commits:

d15ed8fc

by Jonathan Maw at 2018-10-15T15:54:59Z

yaml: Add a cache of parsed and provenanced yaml

Note that the ProvenanceFile's names will be incorrect after loading
from the cache, but this is currently only used for writeback, which
isn't used in junctions.

f4eb685a
by Jonathan Maw at 2018-10-15T15:54:59Z
```
yamlcache.py: Add yaml cache tests
```

26dfa882

by Lachlan Mackenzie at 2018-10-15T15:54:59Z

yaml.py: Add loader for yaml test files cache option

* Fix to CWD issue provided by Jonathan Maw

0fbff08b

by Lachlan Mackenzie at 2018-10-15T15:54:59Z

yaml.py: Add yaml cache testing to yaml list composition test

3d58d7c5

by Lachlan Mackenzie at 2018-10-15T15:54:59Z

yaml.py: Add yaml cache test to yaml list composition twice

363cfc39

by Lachlan Mackenzie at 2018-10-15T15:54:59Z

yamlcache.py: Add YAML cache changed file test

* Test in same style as test_yamlcache_used
* Move of project not required so removed

086c47c4

by Jonathan Maw at 2018-10-15T16:19:01Z

Merge branch 'lachlan/pickle-yaml-test-list-composite' into 'master'

Add YAM cache testing to yaml tests and further YAML cache refinements

See merge request BuildStream/buildstream!836

9df404b2

by Tom Pollard at 2018-10-16T12:21:03Z

_frontend/cli.py: Add --pull-buildtrees to bst build & pull

The addition of the pull-buildtrees flag will be used to override
the default behaviour of remote artifact cache pulling. Included
is the required changes to completions tests.

9fddada0

by Tom Pollard at 2018-10-16T12:21:03Z

buildstream/_context.py: Add pullbuildtrees global user option

The addition of the pullbuildtrees option to the users global
buildstream.conf will be used to override the default behaviour
of remote artifact cache pulling.

b3df2835

by Tom Pollard at 2018-10-16T13:06:35Z

Don't pull artifact buildtrees by default.

The addition of cached buildtrees being included in element
artifacts has led to mostly redundant download overheads when
pulling from a remote artifact server. As such the default behaviour
of pull shouldn't fetch the buildtree object if available.

_stream.py: propagate if pullbuildtrees has been set via cli or
user config to the PullQueue.

pullqueue.py: add extensible attributes to determine required/
excluded directories for element pull jobs.

element.py: extend relevant pull logic for specified subdir
consideration and ensure push logic does not lead to partial
artifact pushes.

_artifactcache/: artifactcache.py & cascache.py inclusion of
helper functions for subdir artifact checking & fetching, fetch
logic extended to only pull required artifact directories.

tests/: integration test pullbuildtrees.py & fix for testutils/
artifactshare.py pathing.

16 changed files:

NEWS
buildstream/_artifactcache/artifactcache.py
buildstream/_artifactcache/cascache.py
buildstream/_context.py
buildstream/_frontend/cli.py
buildstream/_loader/loader.py
buildstream/_scheduler/queues/pullqueue.py
buildstream/_stream.py
buildstream/_yaml.py
+ buildstream/_yamlcache.py
buildstream/element.py
tests/completions/completions.py
+ tests/frontend/yamlcache.py
+ tests/integration/pullbuildtrees.py
tests/testutils/artifactshare.py
tests/yaml/yaml.py

Changes:

NEWS

@@ -27,6 +27,16 @@ buildstream 1.3.1
    o Generate Docker images from built artifacts using
      `contrib/bst-docker-import` script.
 +  o Due to the element `buildtree` being cached in the respective artifact their
 +    size in some cases has significantly increased. In *most* cases the buildtree
 +    is not utilised when building targets, as such by default bst 'pull' & 'build'
 +    will not fetch buildtrees from remotes. This behaviour can be overriden with
 +    the cli option '--pull-buildtrees', or the user configuration option
 +    'pullbuildtrees = True'. The override will also add the buildtree to already
 +    cached artifacts. When attempting to populate an artifactcache server with
 +    cached artifacts, only 'complete' elements can be pushed. If the element is
 +    expected to have a populated buildtree then it must be cached before pushing.
++
  =================
  buildstream 1.1.5

buildstream/_artifactcache/artifactcache.py

@@ -426,6 +426,22 @@ class ArtifactCache():
          raise ImplError("Cache '{kind}' does not implement contains()"
                          .format(kind=type(self).__name__))
 +    # contains_subdir_artifact():
 +    #
 +    # Check whether an artifact element contains a digest for a subdir
 +    # which is populated in the cache, i.e non dangling.
 +    #
 +    # Args:
 +    #     element (Element): The Element to check
 +    #     key (str): The cache key to use
 +    #     subdir (str): The subdir to check
 +    #
 +    # Returns: True if the subdir exists & is populated in the cache, False otherwise
 +    #
 +    def contains_subdir_artifact(self, element, key, subdir):
 +        raise ImplError("Cache '{kind}' does not implement contains_subdir_artifact()"
 +                        .format(kind=type(self).__name__))
++
      # list_artifacts():
+     #
      # List artifacts in this cache in LRU order.
@@ -551,11 +567,12 @@ class ArtifactCache():
      #     element (Element): The Element whose artifact is to be fetched
      #     key (str): The cache key to use
      #     progress (callable): The progress callback, if any
 +    #     subdir (str): The optional specific subdir to pull
+     #
      # Returns:
      #   (bool): True if pull was successful, False if artifact was not available
+     #
 -    def pull(self, element, key, *, progress=None):
 +    def pull(self, element, key, *, progress=None, subdir=None, excluded_subdirs=None):
          raise ImplError("Cache '{kind}' does not implement pull()"
                          .format(kind=type(self).__name__))

buildstream/_artifactcache/cascache.py

@@ -92,6 +92,16 @@ class CASCache(ArtifactCache):
          # This assumes that the repository doesn't have any dangling pointers
          return os.path.exists(refpath)
 +    def contains_subdir_artifact(self, element, key, subdir):
 +        tree = self.resolve_ref(self.get_artifact_fullname(element, key))
++
 +        # This assumes that the subdir digest is present in the element tree
 +        subdirdigest = self._get_subdir(tree, subdir)
 +        objpath = self.objpath(subdirdigest)
++
 +        # True if subdir content is cached or if empty as expected
 +        return os.path.exists(objpath)
++
      def extract(self, element, key):
          ref = self.get_artifact_fullname(element, key)
@@ -228,7 +238,7 @@ class CASCache(ArtifactCache):
              remotes_for_project = self._remotes[element._get_project()]
              return any(remote.spec.push for remote in remotes_for_project)
 -    def pull(self, element, key, *, progress=None):
 +    def pull(self, element, key, *, progress=None, subdir=None, excluded_subdirs=None):
          ref = self.get_artifact_fullname(element, key)
          project = element._get_project()
@@ -247,8 +257,14 @@ class CASCache(ArtifactCache):
                  tree.hash = response.digest.hash
                  tree.size_bytes = response.digest.size_bytes
 -                self._fetch_directory(remote, tree)
 +                # Check if the element artifact is present, if so just fetch subdir
 +                if subdir and os.path.exists(self.objpath(tree)):
 +                    self._fetch_subdir(remote, tree, subdir)
 +                else:
 +                    # Fetch artifact, excluded_subdirs determined in pullqueue
 +                    self._fetch_directory(remote, tree, excluded_subdirs=excluded_subdirs)
 +                # tree is the remote value, so is the same without or without dangling ref locally
                  self.set_ref(ref, tree)
                  element.info("Pulled artifact {} <- {}".format(display_key, remote.spec.url))
@@ -671,8 +687,10 @@ class CASCache(ArtifactCache):
                           stat.S_IRGRP | stat.S_IXGRP | stat.S_IROTH | stat.S_IXOTH)
          for dirnode in directory.directories:
 -            fullpath = os.path.join(dest, dirnode.name)
 -            self._checkout(fullpath, dirnode.digest)
 +            # Don't try to checkout a dangling ref
 +            if os.path.exists(self.objpath(dirnode.digest)):
 +                fullpath = os.path.join(dest, dirnode.name)
 +                self._checkout(fullpath, dirnode.digest)
          for symlinknode in directory.symlinks:
              # symlink
@@ -951,10 +969,12 @@ class CASCache(ArtifactCache):
      #     remote (Remote): The remote to use.
      #     dir_digest (Digest): Digest object for the directory to fetch.
+     #
 -    def _fetch_directory(self, remote, dir_digest):
 +    def _fetch_directory(self, remote, dir_digest, *, excluded_subdirs=None):
          fetch_queue = [dir_digest]
          fetch_next_queue = []
          batch = _CASBatchRead(remote)
 +        if not excluded_subdirs:
 +            excluded_subdirs = []
          while len(fetch_queue) + len(fetch_next_queue) > 0:
              if len(fetch_queue) == 0:
@@ -969,8 +989,9 @@ class CASCache(ArtifactCache):
                  directory.ParseFromString(f.read())
              for dirnode in directory.directories:
 -                batch = self._fetch_directory_node(remote, dirnode.digest, batch,
 -                                                   fetch_queue, fetch_next_queue, recursive=True)
 +                if dirnode.name not in excluded_subdirs:
 +                    batch = self._fetch_directory_node(remote, dirnode.digest, batch,
 +                                                       fetch_queue, fetch_next_queue, recursive=True)
              for filenode in directory.files:
                  batch = self._fetch_directory_node(remote, filenode.digest, batch,
@@ -979,6 +1000,10 @@ class CASCache(ArtifactCache):
          # Fetch final batch
          self._fetch_directory_batch(remote, batch, fetch_queue, fetch_next_queue)
 +    def _fetch_subdir(self, remote, tree, subdir):
 +        subdirdigest = self._get_subdir(tree, subdir)
 +        self._fetch_directory(remote, subdirdigest)
++
      def _fetch_tree(self, remote, digest):
          # download but do not store the Tree object
          with tempfile.NamedTemporaryFile(dir=self.tmpdir) as out:

buildstream/_context.py

@@ -110,6 +110,9 @@ class Context():
          # Make sure the XDG vars are set in the environment before loading anything
          self._init_xdg()
 +        # Whether or not to attempt to pull buildtrees globally
 +        self.pullbuildtrees = False
++
          # Private variables
          self._cache_key = None
          self._message_handler = None
@@ -160,7 +163,7 @@ class Context():
          _yaml.node_validate(defaults, [
              'sourcedir', 'builddir', 'artifactdir', 'logdir',
              'scheduler', 'artifacts', 'logging', 'projects',
 -            'cache'
 +            'cache', 'pullbuildtrees'
          ])
          for directory in ['sourcedir', 'builddir', 'artifactdir', 'logdir']:
@@ -185,6 +188,9 @@ class Context():
          # Load artifact share configuration
          self.artifact_cache_specs = ArtifactCache.specs_from_config_node(defaults)
 +        # Load pull buildtrees configuration
 +        self.pullbuildtrees = _yaml.node_get(defaults, bool, 'pullbuildtrees', default_value='False')
++
          # Load logging config
          logging = _yaml.node_get(defaults, Mapping, 'logging')
          _yaml.node_validate(logging, [

buildstream/_frontend/cli.py

@@ -305,10 +305,12 @@ def init(app, project_name, format_version, element_path, force):
                help="Allow tracking to cross junction boundaries")
  @click.option('--track-save', default=False, is_flag=True,
                help="Deprecated: This is ignored")
 +@click.option('--pull-buildtrees', default=False, is_flag=True,
 +              help="Pull buildtrees from a remote cache server")
  @click.argument('elements', nargs=-1,
                  type=click.Path(readable=False))
  @click.pass_obj
 -def build(app, elements, all_, track_, track_save, track_all, track_except, track_cross_junctions):
 +def build(app, elements, all_, track_, track_save, track_all, track_except, track_cross_junctions, pull_buildtrees):
      """Build elements in a pipeline"""
      if (track_except or track_cross_junctions) and not (track_ or track_all):
@@ -327,7 +329,8 @@ def build(app, elements, all_, track_, track_save, track_all, track_except, trac
                           track_targets=track_,
                           track_except=track_except,
                           track_cross_junctions=track_cross_junctions,
 -                         build_all=all_)
 +                         build_all=all_,
 +                         pull_buildtrees=pull_buildtrees)
  ##################################################################
@@ -429,10 +432,12 @@ def track(app, elements, deps, except_, cross_junctions):
                help='The dependency artifacts to pull (default: none)')
  @click.option('--remote', '-r',
                help="The URL of the remote cache (defaults to the first configured cache)")
 +@click.option('--pull-buildtrees', default=False, is_flag=True,
 +              help="Pull buildtrees from a remote cache server")
  @click.argument('elements', nargs=-1,
                  type=click.Path(readable=False))
  @click.pass_obj
 -def pull(app, elements, deps, remote):
 +def pull(app, elements, deps, remote, pull_buildtrees):
      """Pull a built artifact from the configured remote artifact cache.
      By default the artifact will be pulled one of the configured caches
@@ -446,7 +451,7 @@ def pull(app, elements, deps, remote):
          all:   All dependencies
      """
      with app.initialized(session_name="Pull"):
 -        app.stream.pull(elements, selection=deps, remote=remote)
 +        app.stream.pull(elements, selection=deps, remote=remote, pull_buildtrees=pull_buildtrees)
  ##################################################################

buildstream/_loader/loader.py

@@ -29,6 +29,7 @@ from .. import _yaml
  from ..element import Element
  from .._profile import Topics, profile_start, profile_end
  from .._includes import Includes
 +from .._yamlcache import YamlCache
  from .types import Symbol, Dependency
  from .loadelement import LoadElement
@@ -108,13 +109,19 @@ class Loader():
+         #
          deps = []
 -        for target in targets:
 -            profile_start(Topics.LOAD_PROJECT, target)
 -            junction, name, loader = self._parse_name(target, rewritable, ticker,
 -                                                      fetch_subprojects=fetch_subprojects)
 -            loader._load_file(name, rewritable, ticker, fetch_subprojects)
 -            deps.append(Dependency(name, junction=junction))
 -            profile_end(Topics.LOAD_PROJECT, target)
 +        # XXX This will need to be changed to the context's top-level project if this method
 +        # is ever used for subprojects
 +        top_dir = self.project.directory
++
 +        cache_file = YamlCache.get_cache_file(top_dir)
 +        with YamlCache.open(self._context, cache_file) as yaml_cache:
 +            for target in targets:
 +                profile_start(Topics.LOAD_PROJECT, target)
 +                junction, name, loader = self._parse_name(target, rewritable, ticker,
 +                                                          fetch_subprojects=fetch_subprojects)
 +                loader._load_file(name, rewritable, ticker, fetch_subprojects, yaml_cache)
 +                deps.append(Dependency(name, junction=junction))
 +                profile_end(Topics.LOAD_PROJECT, target)
+         #
          # Now that we've resolve the dependencies, scan them for circular dependencies
@@ -201,11 +208,12 @@ class Loader():
      #    rewritable (bool): Whether we should load in round trippable mode
      #    ticker (callable): A callback to report loaded filenames to the frontend
      #    fetch_subprojects (bool): Whether to fetch subprojects while loading
 +    #    yaml_cache (YamlCache): A yaml cache
+     #
      # Returns:
      #    (LoadElement): A loaded LoadElement
+     #
 -    def _load_file(self, filename, rewritable, ticker, fetch_subprojects):
 +    def _load_file(self, filename, rewritable, ticker, fetch_subprojects, yaml_cache=None):
          # Silently ignore already loaded files
          if filename in self._elements:
@@ -218,7 +226,8 @@ class Loader():
          # Load the data and process any conditional statements therein
          fullpath = os.path.join(self._basedir, filename)
          try:
 -            node = _yaml.load(fullpath, shortname=filename, copy_tree=rewritable, project=self.project)
 +            node = _yaml.load(fullpath, shortname=filename, copy_tree=rewritable,
 +                              project=self.project, yaml_cache=yaml_cache)
          except LoadError as e:
              if e.reason == LoadErrorReason.MISSING_FILE:
                  # If we can't find the file, try to suggest plausible
@@ -261,13 +270,13 @@ class Loader():
          # Load all dependency files for the new LoadElement
          for dep in element.deps:
              if dep.junction:
 -                self._load_file(dep.junction, rewritable, ticker, fetch_subprojects)
 +                self._load_file(dep.junction, rewritable, ticker, fetch_subprojects, yaml_cache)
                  loader = self._get_loader(dep.junction, rewritable=rewritable, ticker=ticker,
                                            fetch_subprojects=fetch_subprojects)
              else:
                  loader = self
 -            dep_element = loader._load_file(dep.name, rewritable, ticker, fetch_subprojects)
 +            dep_element = loader._load_file(dep.name, rewritable, ticker, fetch_subprojects, yaml_cache)
              if _yaml.node_get(dep_element.node, str, Symbol.KIND) == 'junction':
                  raise LoadError(LoadErrorReason.INVALID_DATA,

buildstream/_scheduler/queues/pullqueue.py

@@ -32,9 +32,20 @@ class PullQueue(Queue):
      complete_name = "Pulled"
      resources = [ResourceType.DOWNLOAD, ResourceType.CACHE]
 +    def __init__(self, scheduler, buildtrees=False):
 +        super().__init__(scheduler)
++
 +        # Current default exclusions on pull
 +        self._excluded_subdirs = ["buildtree"]
 +        self._subdir = None
 +        # If buildtrees are to be pulled, remove the value from exclusion list
 +        if buildtrees:
 +            self._subdir = "buildtree"
 +            self._excluded_subdirs.remove(self._subdir)
++
      def process(self, element):
          # returns whether an artifact was downloaded or not
 -        if not element._pull():
 +        if not element._pull(subdir=self._subdir, excluded_subdirs=self._excluded_subdirs):
              raise SkipJob(self.action_name)
      def status(self, element):
@@ -49,7 +60,7 @@ class PullQueue(Queue):
          if not element._can_query_cache():
              return QueueStatus.WAIT
 -        if element._pull_pending():
 +        if element._pull_pending(subdir=self._subdir):
              return QueueStatus.READY
          else:
              return QueueStatus.SKIP

buildstream/_stream.py

@@ -160,12 +160,14 @@ class Stream():
      #    track_cross_junctions (bool): Whether tracking should cross junction boundaries
      #    build_all (bool): Whether to build all elements, or only those
      #                      which are required to build the target.
 +    #    pull_buildtrees (bool): Whether to pull buildtrees from a remote cache server
+     #
      def build(self, targets, *,
                track_targets=None,
                track_except=None,
                track_cross_junctions=False,
 -              build_all=False):
 +              build_all=False,
 +              pull_buildtrees=False):
          if build_all:
              selection = PipelineSelection.ALL
@@ -195,7 +197,10 @@ class Stream():
              self._add_queue(track_queue, track=True)
          if self._artifacts.has_fetch_remotes():
 -            self._add_queue(PullQueue(self._scheduler))
 +            # Query if pullbuildtrees has been set globally in user config
 +            if self._context.pullbuildtrees:
 +                pull_buildtrees = True
 +            self._add_queue(PullQueue(self._scheduler, buildtrees=pull_buildtrees))
          self._add_queue(FetchQueue(self._scheduler, skip_cached=True))
          self._add_queue(BuildQueue(self._scheduler))
@@ -295,7 +300,8 @@ class Stream():
+     #
      def pull(self, targets, *,
               selection=PipelineSelection.NONE,
 -             remote=None):
 +             remote=None,
 +             pull_buildtrees=False):
          use_config = True
          if remote:
@@ -310,8 +316,12 @@ class Stream():
          if not self._artifacts.has_fetch_remotes():
              raise StreamError("No artifact caches available for pulling artifacts")
 +        # Query if pullbuildtrees has been set globally in user config
 +        if self._context.pullbuildtrees:
 +            pull_buildtrees = True
++
          self._pipeline.assert_consistent(elements)
 -        self._add_queue(PullQueue(self._scheduler))
 +        self._add_queue(PullQueue(self._scheduler, buildtrees=pull_buildtrees))
          self._enqueue_plan(elements)
          self._run()

buildstream/_yaml.py

@@ -183,20 +183,32 @@ class CompositeTypeError(CompositeError):
  #    shortname (str): The filename in shorthand for error reporting (or None)
  #    copy_tree (bool): Whether to make a copy, preserving the original toplevels
  #                      for later serialization
 +#    yaml_cache (YamlCache): A yaml cache to consult rather than parsing
+ #
  # Returns (dict): A loaded copy of the YAML file with provenance information
+ #
  # Raises: LoadError
+ #
 -def load(filename, shortname=None, copy_tree=False, *, project=None):
 +def load(filename, shortname=None, copy_tree=False, *, project=None, yaml_cache=None):
      if not shortname:
          shortname = filename
      file = ProvenanceFile(filename, shortname, project)
      try:
 +        data = None
          with open(filename) as f:
 -            return load_data(f, file, copy_tree=copy_tree)
 +            contents = f.read()
 +        if yaml_cache:
 +            data, key = yaml_cache.get(project, filename, contents, copy_tree)
++
 +        if not data:
 +            data = load_data(contents, file, copy_tree=copy_tree)
++
 +        if yaml_cache:
 +            yaml_cache.put_from_key(project, filename, key, data)
++
 +        return data
      except FileNotFoundError as e:
          raise LoadError(LoadErrorReason.MISSING_FILE,
                          "Could not find file at {}".format(filename)) from e

buildstream/_yamlcache.py

 +#
 +#  Copyright 2018 Bloomberg Finance LP
 +#
 +#  This program is free software; you can redistribute it and/or
 +#  modify it under the terms of the GNU Lesser General Public
 +#  License as published by the Free Software Foundation; either
 +#  version 2 of the License, or (at your option) any later version.
 +#
 +#  This library is distributed in the hope that it will be useful,
 +#  but WITHOUT ANY WARRANTY; without even the implied warranty of
 +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 +#  Lesser General Public License for more details.
 +#
 +#  You should have received a copy of the GNU Lesser General Public
 +#  License along with this library. If not, see <http://www.gnu.org/licenses/>.
 +#
 +#  Authors:
 +#        Jonathan Maw <jonathan maw codethink co uk>
++
 +import os
 +import pickle
 +import hashlib
 +import io
++
 +import sys
++
 +from contextlib import contextmanager
 +from collections import namedtuple
++
 +from ._cachekey import generate_key
 +from ._context import Context
 +from . import utils, _yaml
++
++
 +YAML_CACHE_FILENAME = "yaml_cache.pickle"
++
++
 +# YamlCache()
 +#
 +# A cache that wraps around the loading of yaml in projects.
 +#
 +# The recommended way to use a YamlCache is:
 +#   with YamlCache.open(context) as yamlcache:
 +#     # Load all the yaml
 +#     ...
 +#
 +# Args:
 +#    context (Context): The invocation Context
 +#
 +class YamlCache():
++
 +    def __init__(self, context):
 +        self._project_caches = {}
 +        self._context = context
++
 +    ##################
 +    # Public Methods #
 +    ##################
++
 +    # is_cached():
 +    #
 +    # Checks whether a file is cached.
 +    #
 +    # Args:
 +    #    project (Project): The project this file is in.
 +    #    filepath (str): The path to the file, *relative to the project's directory*.
 +    #
 +    # Returns:
 +    #    (bool): Whether the file is cached.
 +    def is_cached(self, project, filepath):
 +        cache_path = self._get_filepath(project, filepath)
 +        project_name = project.name if project else ""
 +        try:
 +            project_cache = self._project_caches[project_name]
 +            if cache_path in project_cache.elements:
 +                return True
 +        except KeyError:
 +            pass
 +        return False
++
 +    # open():
 +    #
 +    # Return an instance of the YamlCache which writes to disk when it leaves scope.
 +    #
 +    # Args:
 +    #    context (Context): The context.
 +    #    cachefile (str): The path to the cache file.
 +    #
 +    # Returns:
 +    #    (YamlCache): A YamlCache.
 +    @staticmethod
 +    @contextmanager
 +    def open(context, cachefile):
 +        # Try to load from disk first
 +        cache = None
 +        if os.path.exists(cachefile):
 +            try:
 +                with open(cachefile, "rb") as f:
 +                    cache = BstUnpickler(f, context).load()
 +            except EOFError:
 +                # The file was empty
 +                pass
 +            except pickle.UnpicklingError as e:
 +                sys.stderr.write("Failed to load YamlCache, {}\n".format(e))
++
 +        # Failed to load from disk, create a new one
 +        if not cache:
 +            cache = YamlCache(context)
++
 +        yield cache
++
 +        cache._write(cachefile)
++
 +    # get_cache_file():
 +    #
 +    # Retrieves a path to the yaml cache file.
 +    #
 +    # Returns:
 +    #   (str): The path to the cache file
 +    @staticmethod
 +    def get_cache_file(top_dir):
 +        return os.path.join(top_dir, ".bst", YAML_CACHE_FILENAME)
++
 +    # get():
 +    #
 +    # Gets a parsed file from the cache.
 +    #
 +    # Args:
 +    #    project (Project) or None: The project this file is in, if it exists.
 +    #    filepath (str): The absolute path to the file.
 +    #    contents (str): The contents of the file to be cached
 +    #    copy_tree (bool): Whether the data should make a copy when it's being generated
 +    #                      (i.e. exactly as when called in yaml)
 +    #
 +    # Returns:
 +    #    (decorated dict): The parsed yaml from the cache, or None if the file isn't in the cache.
 +    #    (str):            The key used to look up the parsed yaml in the cache
 +    def get(self, project, filepath, contents, copy_tree):
 +        key = self._calculate_key(contents, copy_tree)
 +        data = self._get(project, filepath, key)
 +        return data, key
++
 +    # put():
 +    #
 +    # Puts a parsed file into the cache.
 +    #
 +    # Args:
 +    #    project (Project): The project this file is in.
 +    #    filepath (str): The path to the file.
 +    #    contents (str): The contents of the file that has been cached
 +    #    copy_tree (bool): Whether the data should make a copy when it's being generated
 +    #                      (i.e. exactly as when called in yaml)
 +    #    value (decorated dict): The data to put into the cache.
 +    def put(self, project, filepath, contents, copy_tree, value):
 +        key = self._calculate_key(contents, copy_tree)
 +        self.put_from_key(project, filepath, key, value)
++
 +    # put_from_key():
 +    #
 +    # Put a parsed file into the cache when given a key.
 +    #
 +    # Args:
 +    #    project (Project): The project this file is in.
 +    #    filepath (str): The path to the file.
 +    #    key (str): The key to the file within the cache. Typically, this is the
 +    #               value of `calculate_key()` with the file's unparsed contents
 +    #               and any relevant metadata passed in.
 +    #    value (decorated dict): The data to put into the cache.
 +    def put_from_key(self, project, filepath, key, value):
 +        cache_path = self._get_filepath(project, filepath)
 +        project_name = project.name if project else ""
 +        try:
 +            project_cache = self._project_caches[project_name]
 +        except KeyError:
 +            project_cache = self._project_caches[project_name] = CachedProject({})
++
 +        project_cache.elements[cache_path] = CachedYaml(key, value)
++
 +    ###################
 +    # Private Methods #
 +    ###################
++
 +    # Writes the yaml cache to the specified path.
 +    #
 +    # Args:
 +    #    path (str): The path to the cache file.
 +    def _write(self, path):
 +        parent_dir = os.path.dirname(path)
 +        os.makedirs(parent_dir, exist_ok=True)
 +        with open(path, "wb") as f:
 +            BstPickler(f).dump(self)
++
 +    # _get_filepath():
 +    #
 +    # Returns a file path relative to a project if passed, or the original path if
 +    # the project is None
 +    #
 +    # Args:
 +    #    project (Project) or None: The project the filepath exists within
 +    #    full_path (str): The path that the returned path is based on
 +    #
 +    # Returns:
 +    #    (str): The path to the file, relative to a project if it exists
 +    def _get_filepath(self, project, full_path):
 +        if project:
 +            assert full_path.startswith(project.directory)
 +            filepath = os.path.relpath(full_path, project.directory)
 +        else:
 +            filepath = full_path
 +        return full_path
++
 +    # _calculate_key():
 +    #
 +    # Calculates a key for putting into the cache.
 +    #
 +    # Args:
 +    #    (basic object)... : Any number of strictly-ordered basic objects
 +    #
 +    # Returns:
 +    #   (str): A key made out of every arg passed in
 +    @staticmethod
 +    def _calculate_key(*args):
 +        string = pickle.dumps(args)
 +        return hashlib.sha1(string).hexdigest()
++
 +    # _get():
 +    #
 +    # Gets a parsed file from the cache when given a key.
 +    #
 +    # Args:
 +    #    project (Project): The project this file is in.
 +    #    filepath (str): The path to the file.
 +    #    key (str): The key to the file within the cache. Typically, this is the
 +    #               value of `calculate_key()` with the file's unparsed contents
 +    #               and any relevant metadata passed in.
 +    #
 +    # Returns:
 +    #    (decorated dict): The parsed yaml from the cache, or None if the file isn't in the cache.
 +    def _get(self, project, filepath, key):
 +        cache_path = self._get_filepath(project, filepath)
 +        project_name = project.name if project else ""
 +        try:
 +            project_cache = self._project_caches[project_name]
 +            try:
 +                cachedyaml = project_cache.elements[cache_path]
 +                if cachedyaml._key == key:
 +                    # We've unpickled the YamlCache, but not the specific file
 +                    if cachedyaml._contents is None:
 +                        cachedyaml._contents = BstUnpickler.loads(cachedyaml._pickled_contents, self._context)
 +                    return cachedyaml._contents
 +            except KeyError:
 +                pass
 +        except KeyError:
 +            pass
 +        return None
++
++
 +CachedProject = namedtuple('CachedProject', ['elements'])
++
++
 +class CachedYaml():
 +    def __init__(self, key, contents):
 +        self._key = key
 +        self.set_contents(contents)
++
 +    # Sets the contents of the CachedYaml.
 +    #
 +    # Args:
 +    #    contents (provenanced dict): The contents to put in the cache.
 +    #
 +    def set_contents(self, contents):
 +        self._contents = contents
 +        self._pickled_contents = BstPickler.dumps(contents)
++
 +    # Pickling helper method, prevents 'contents' from being serialised
 +    def __getstate__(self):
 +        data = self.__dict__.copy()
 +        data['_contents'] = None
 +        return data
++
++
 +# In _yaml.load, we have a ProvenanceFile that stores the project the file
 +# came from. Projects can't be pickled, but it's always going to be the same
 +# project between invocations (unless the entire project is moved but the
 +# file stayed in the same place)
 +class BstPickler(pickle.Pickler):
 +    def persistent_id(self, obj):
 +        if isinstance(obj, _yaml.ProvenanceFile):
 +            if obj.project:
 +                # ProvenanceFile's project object cannot be stored as it is.
 +                project_tag = obj.project.name
 +                # ProvenanceFile's filename must be stored relative to the
 +                # project, as the project dir may move.
 +                name = os.path.relpath(obj.name, obj.project.directory)
 +            else:
 +                project_tag = None
 +                name = obj.name
 +            return ("ProvenanceFile", name, obj.shortname, project_tag)
 +        elif isinstance(obj, Context):
 +            return ("Context",)
 +        else:
 +            return None
++
 +    @staticmethod
 +    def dumps(obj):
 +        stream = io.BytesIO()
 +        BstPickler(stream).dump(obj)
 +        stream.seek(0)
 +        return stream.read()
++
++
 +class BstUnpickler(pickle.Unpickler):
 +    def __init__(self, file, context):
 +        super().__init__(file)
 +        self._context = context
++
 +    def persistent_load(self, pid):
 +        if pid[0] == "ProvenanceFile":
 +            _, tagged_name, shortname, project_tag = pid
++
 +            if project_tag is not None:
 +                for p in self._context.get_projects():
 +                    if project_tag == p.name:
 +                        project = p
 +                        break
++
 +                name = os.path.join(project.directory, tagged_name)
++
 +                if not project:
 +                    projects = [p.name for p in self._context.get_projects()]
 +                    raise pickle.UnpicklingError("No project with name {} found in {}"
 +                                                 .format(key_id, projects))
 +            else:
 +                project = None
 +                name = tagged_name
++
 +            return _yaml.ProvenanceFile(name, shortname, project)
 +        elif pid[0] == "Context":
 +            return self._context
 +        else:
 +            raise pickle.UnpicklingError("Unsupported persistent object, {}".format(pid))
++
 +    @staticmethod
 +    def loads(text, context):
 +        stream = io.BytesIO()
 +        stream.write(bytes(text))
 +        stream.seek(0)
 +        return BstUnpickler(stream, context).load()

buildstream/element.py

@@ -1692,18 +1692,26 @@ class Element(Plugin):
      # _pull_pending()
+     #
 -    # Check whether the artifact will be pulled.
 +    # Check whether the artifact will be pulled. If the pull operation is to
 +    # include a specific subdir of the element artifact (from cli or user conf)
 +    # then the local cache is queried for the subdirs existence.
 +    #
 +    # Args:
 +    #    subdir (str): Whether the pull has been invoked with a specific subdir set
+     #
      # Returns:
      #   (bool): Whether a pull operation is pending
+     #
 -    def _pull_pending(self):
 +    def _pull_pending(self, subdir=None):
          if self._get_workspace():
              # Workspace builds are never pushed to artifact servers
              return False
 -        if self.__strong_cached:
 -            # Artifact already in local cache
 +        if self.__strong_cached and subdir:
 +            # If we've specified a subdir, check if the subdir is cached locally
 +            if self.__artifacts.contains_subdir_artifact(self, self.__strict_cache_key, subdir):
 +                return False
 +        elif self.__strong_cached:
              return False
          # Pull is pending if artifact remote server available
@@ -1725,11 +1733,10 @@ class Element(Plugin):
          self._update_state()
 -    def _pull_strong(self, *, progress=None):
 +    def _pull_strong(self, *, progress=None, subdir=None, excluded_subdirs=None):
          weak_key = self._get_cache_key(strength=_KeyStrength.WEAK)
+-
          key = self.__strict_cache_key
 -        if not self.__artifacts.pull(self, key, progress=progress):
 +        if not self.__artifacts.pull(self, key, progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs):
              return False
          # update weak ref by pointing it to this newly fetched artifact
@@ -1737,10 +1744,10 @@ class Element(Plugin):
          return True
 -    def _pull_weak(self, *, progress=None):
 +    def _pull_weak(self, *, progress=None, subdir=None, excluded_subdirs=None):
          weak_key = self._get_cache_key(strength=_KeyStrength.WEAK)
+-
 -        if not self.__artifacts.pull(self, weak_key, progress=progress):
 +        if not self.__artifacts.pull(self, weak_key, progress=progress, subdir=subdir,
 +                                     excluded_subdirs=excluded_subdirs):
              return False
          # extract strong cache key from this newly fetched artifact
@@ -1758,17 +1765,17 @@ class Element(Plugin):
+     #
      # Returns: True if the artifact has been downloaded, False otherwise
+     #
 -    def _pull(self):
 +    def _pull(self, subdir=None, excluded_subdirs=None):
          context = self._get_context()
          def progress(percent, message):
              self.status(message)
          # Attempt to pull artifact without knowing whether it's available
 -        pulled = self._pull_strong(progress=progress)
 +        pulled = self._pull_strong(progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs)
          if not pulled and not self._cached() and not context.get_strict():
 -            pulled = self._pull_weak(progress=progress)
 +            pulled = self._pull_weak(progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs)
          if not pulled:
              return False
@@ -1791,10 +1798,21 @@ class Element(Plugin):
          if not self._cached():
              return True
 -        # Do not push tained artifact
 +        # Do not push tainted artifact
          if self.__get_tainted():
              return True
 +        # Do not push elements that have a dangling buildtree artifact unless element type is
 +        # expected to have an empty buildtree directory
 +        if not self.__artifacts.contains_subdir_artifact(self, self.__strict_cache_key, 'buildtree'):
 +            return True
++
 +        # strict_cache_key can't be relied on to be available when running in non strict mode
 +        context = self._get_context()
 +        if not context.get_strict():
 +            if not self.__artifacts.contains_subdir_artifact(self, self.__weak_cache_key, 'buildtree'):
 +                return True
++
          return False
      # _push():

tests/completions/completions.py

@@ -103,7 +103,7 @@ def test_commands(cli, cmd, word_idx, expected):
      ('bst --no-colors build -', 3, ['--all ', '--track ', '--track-all ',
                                      '--track-except ',
                                      '--track-cross-junctions ', '-J ',
 -                                    '--track-save ']),
 +                                    '--track-save ', '--pull-buildtrees ']),
      # Test the behavior of completing after an option that has a
      # parameter that cannot be completed, vs an option that has

tests/frontend/yamlcache.py

 +import os
 +import pytest
 +import hashlib
 +import tempfile
 +from ruamel import yaml
++
 +from tests.testutils import cli, generate_junction, create_element_size, create_repo
 +from buildstream import _yaml
 +from buildstream._yamlcache import YamlCache
 +from buildstream._project import Project
 +from buildstream._context import Context
 +from contextlib import contextmanager
++
++
 +def generate_project(tmpdir, ref_storage, with_junction, name="test"):
 +    if with_junction == 'junction':
 +        subproject_dir = generate_project(
 +            tmpdir, ref_storage,
 +            'no-junction', name='test-subproject'
 +        )
++
 +    project_dir = os.path.join(tmpdir, name)
 +    os.makedirs(project_dir)
 +    # project.conf
 +    project_conf_path = os.path.join(project_dir, 'project.conf')
 +    elements_path = 'elements'
 +    project_conf = {
 +        'name': name,
 +        'element-path': elements_path,
 +        'ref-storage': ref_storage,
 +    }
 +    _yaml.dump(project_conf, project_conf_path)
++
 +    # elements
 +    if with_junction == 'junction':
 +        junction_name = 'junction.bst'
 +        junction_dir = os.path.join(project_dir, elements_path)
 +        junction_path = os.path.join(project_dir, elements_path, junction_name)
 +        os.makedirs(junction_dir)
 +        generate_junction(tmpdir, subproject_dir, junction_path)
 +        element_depends = [{'junction': junction_name, 'filename': 'test.bst'}]
 +    else:
 +        element_depends = []
++
 +    element_name = 'test.bst'
 +    create_element_size(element_name, project_dir, elements_path, element_depends, 1)
++
 +    return project_dir
++
++
 +@contextmanager
 +def with_yamlcache(project_dir):
 +    context = Context()
 +    project = Project(project_dir, context)
 +    cache_file = YamlCache.get_cache_file(project_dir)
 +    with YamlCache.open(context, cache_file) as yamlcache:
 +        yield yamlcache, project
++
++
 +def yamlcache_key(yamlcache, in_file, copy_tree=False):
 +    with open(in_file) as f:
 +        key = yamlcache._calculate_key(f.read(), copy_tree)
 +    return key
++
++
 +def modified_file(input_file, tmpdir):
 +    with open(input_file) as f:
 +        data = f.read()
 +    assert 'variables' not in data
 +    data += '\nvariables: {modified: True}\n'
 +    _, temppath = tempfile.mkstemp(dir=tmpdir, text=True)
 +    with open(temppath, 'w') as f:
 +        f.write(data)
++
 +    return temppath
++
++
 +@pytest.mark.parametrize('ref_storage', ['inline', 'project.refs'])
 +@pytest.mark.parametrize('with_junction', ['no-junction', 'junction'])
 +@pytest.mark.parametrize('move_project', ['move', 'no-move'])
 +def test_yamlcache_used(cli, tmpdir, ref_storage, with_junction, move_project):
 +    # Generate the project
 +    project = generate_project(str(tmpdir), ref_storage, with_junction)
 +    if with_junction == 'junction':
 +        result = cli.run(project=project, args=['fetch', '--track', 'junction.bst'])
 +        result.assert_success()
++
 +    # bst show to put it in the cache
 +    result = cli.run(project=project, args=['show', 'test.bst'])
 +    result.assert_success()
++
 +    element_path = os.path.join(project, 'elements', 'test.bst')
 +    with with_yamlcache(project) as (yc, prj):
 +        # Check that it's in the cache
 +        assert yc.is_cached(prj, element_path)
++
 +        # *Absolutely* horrible cache corruption to check it's being used
 +        # Modifying the data from the cache is fraught with danger,
 +        # so instead I'll load a modified version of the original file
 +        temppath = modified_file(element_path, str(tmpdir))
 +        contents = _yaml.load(temppath, copy_tree=False, project=prj)
 +        key = yamlcache_key(yc, element_path)
 +        yc.put_from_key(prj, element_path, key, contents)
++
 +    # Show that a variable has been added
 +    result = cli.run(project=project, args=['show', '--format', '%{vars}', 'test.bst'])
 +    result.assert_success()
 +    data = yaml.safe_load(result.output)
 +    assert 'modified' in data
 +    assert data['modified'] == 'True'
++
++
 +@pytest.mark.parametrize('ref_storage', ['inline', 'project.refs'])
 +@pytest.mark.parametrize('with_junction', ['junction', 'no-junction'])
 +def test_yamlcache_changed_file(cli, tmpdir, ref_storage, with_junction):
 +    # i.e. a file is cached, the file is changed, loading the file (with cache) returns new data
 +    # inline and junction can only be changed by opening a workspace
 +    # Generate the project
 +    project = generate_project(str(tmpdir), ref_storage, with_junction)
 +    if with_junction == 'junction':
 +        result = cli.run(project=project, args=['fetch', '--track', 'junction.bst'])
 +        result.assert_success()
++
 +    # bst show to put it in the cache
 +    result = cli.run(project=project, args=['show', 'test.bst'])
 +    result.assert_success()
++
 +    element_path = os.path.join(project, 'elements', 'test.bst')
 +    with with_yamlcache(project) as (yc, prj):
 +        # Check that it's in the cache then modify
 +        assert yc.is_cached(prj, element_path)
 +        with open(element_path, "a") as f:
 +            f.write('\nvariables: {modified: True}\n')
 +        # Load modified yaml cache file into cache
 +        _yaml.load(element_path, copy_tree=False, project=prj, yaml_cache=yc)
++
 +    # Show that a variable has been added
 +    result = cli.run(project=project, args=['show', '--format', '%{vars}', 'test.bst'])
 +    result.assert_success()
 +    data = yaml.safe_load(result.output)
 +    assert 'modified' in data
 +    assert data['modified'] == 'True'

tests/integration/pullbuildtrees.py

 +import os
 +import shutil
 +import pytest
++
 +from tests.testutils import cli_integration as cli, create_artifact_share
 +from tests.testutils.integration import assert_contains
++
++
 +DATA_DIR = os.path.join(
 +    os.path.dirname(os.path.realpath(__file__)),
 +    "project"
 +)
++
++
 +# Remove artifact cache & set cli.config value of pullbuildtrees
 +# to false, which is the default user context
 +def default_state(cli, integration_cache, share):
 +    shutil.rmtree(os.path.join(integration_cache, 'artifacts'))
 +    cli.configure({'pullbuildtrees': False, 'artifacts': {'url': share.repo, 'push': False}})
++
++
 +# A test to capture the integration of the pullbuildtrees
 +# behaviour, which by default is to not include the buildtree
 +# directory of an element
 +@pytest.mark.integration
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_pullbuildtrees(cli, tmpdir, datafiles, integration_cache):
++
 +    # Ensure artifact cache is purged, as we can't have dangling refs/objects
 +    shutil.rmtree(os.path.join(integration_cache, 'artifacts'))
++
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    element_name = 'autotools/amhello.bst'
++
 +    # Create artifact shares for pull & push testing
 +    with create_artifact_share(os.path.join(str(tmpdir), 'share1')) as share1,\
 +        create_artifact_share(os.path.join(str(tmpdir), 'share2')) as share2:
 +        cli.configure({
 +            'artifacts': {'url': share1.repo, 'push': True},
 +        })
++
 +        # Build autotools element, checked pushed, delete local
 +        result = cli.run(project=project, args=['build', element_name])
 +        assert result.exit_code == 0
 +        assert cli.get_element_state(project, element_name) == 'cached'
 +        assert share1.has_artifact('test', element_name, cli.get_element_key(project, element_name))
 +        default_state(cli, integration_cache, share1)
++
 +        # Pull artifact with default config, assert that pulling again
 +        # doesn't create a pull job, then assert with buildtrees user
 +        # config set creates a pull job.
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name not in result.get_pulled_elements()
 +        cli.configure({'pullbuildtrees': True})
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        default_state(cli, integration_cache, share1)
++
 +        # Pull artifact with default config, then assert that pulling
 +        # with buildtrees cli flag set creates a pull job.
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        result = cli.run(project=project, args=['pull', '--pull-buildtrees', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        default_state(cli, integration_cache, share1)
++
 +        # Pull artifact with pullbuildtrees set in user config, then assert
 +        # that pulling with the same user config doesn't creates a pull job,
 +        # or when buildtrees cli flag is set.
 +        cli.configure({'pullbuildtrees': True})
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name not in result.get_pulled_elements()
 +        result = cli.run(project=project, args=['pull', '--pull-buildtrees', element_name])
 +        assert element_name not in result.get_pulled_elements()
 +        default_state(cli, integration_cache, share1)
++
 +        # Pull artifact with default config and buildtrees cli flag set, then assert
 +        # that pulling with pullbuildtrees set in user config doesn't create a pull
 +        # job.
 +        result = cli.run(project=project, args=['pull', '--pull-buildtrees', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        cli.configure({'pullbuildtrees': True})
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name not in result.get_pulled_elements()
 +        default_state(cli, integration_cache, share1)
++
 +        # Assert that a partial build element (not containing a populated buildtree dir)
 +        # can't be pushed to an artifact share, then assert that a complete build element
 +        # can be. This will attempt a partial pull from share1 and then a partial push
 +        # to share2
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        cli.configure({'artifacts': {'url': share2.repo, 'push': True}})
 +        result = cli.run(project=project, args=['push', element_name])
 +        assert element_name not in result.get_pushed_elements()
 +        assert not share2.has_artifact('test', element_name, cli.get_element_key(project, element_name))
++
 +        # Assert that after pulling the missing buildtree the element artifact can be
 +        # successfully pushed to the remote. This will attempt to pull the buildtree
 +        # from share1 and then a 'complete' push to share2
 +        cli.configure({'artifacts': {'url': share1.repo, 'push': False}})
 +        result = cli.run(project=project, args=['pull', '--pull-buildtrees', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        cli.configure({'artifacts': {'url': share2.repo, 'push': True}})
 +        result = cli.run(project=project, args=['push', element_name])
 +        assert element_name in result.get_pushed_elements()
 +        assert share2.has_artifact('test', element_name, cli.get_element_key(project, element_name))

tests/testutils/artifactshare.py

@@ -128,7 +128,7 @@ class ArtifactShare():
          valid_chars = string.digits + string.ascii_letters + '-._'
          element_name = ''.join([
 -            x if x in valid_chars else '_'
 +            x if x in valid_chars else '-'
              for x in element_name
          ])
          artifact_key = '{0}/{1}/{2}'.format(project_name, element_name, cache_key)

tests/yaml/yaml.py

  import os
  import pytest
 +import tempfile
  from collections import Mapping
  from buildstream import _yaml
  from buildstream._exceptions import LoadError, LoadErrorReason
 +from buildstream._context import Context
 +from buildstream._yamlcache import YamlCache
  DATA_DIR = os.path.join(
      os.path.dirname(os.path.realpath(__file__)),
@@ -150,6 +153,21 @@ def test_composite_preserve_originals(datafiles):
      assert(_yaml.node_get(orig_extra, str, 'old') == 'new')
 +def load_yaml_file(filename, *, cache_path, shortname=None, from_cache='raw'):
++
 +    _, temppath = tempfile.mkstemp(dir=os.path.join(cache_path.dirname, cache_path.basename), text=True)
 +    context = Context()
++
 +    with YamlCache.open(context, temppath) as yc:
 +        if from_cache == 'raw':
 +            return _yaml.load(filename, shortname)
 +        elif from_cache == 'cached':
 +            _yaml.load(filename, shortname, yaml_cache=yc)
 +            return _yaml.load(filename, shortname, yaml_cache=yc)
 +        else:
 +            assert False
++
++
  # Tests for list composition
+ #
  # Each test composits a filename on top of basics.yaml, and tests
@@ -165,6 +183,7 @@ def test_composite_preserve_originals(datafiles):
  #    prov_col: The expected provenance column of "mood"
+ #
  @pytest.mark.datafiles(os.path.join(DATA_DIR))
 +@pytest.mark.parametrize('caching', [('raw'), ('cached')])
  @pytest.mark.parametrize("filename,index,length,mood,prov_file,prov_line,prov_col", [
      # Test results of compositing with the (<) prepend directive
@@ -195,14 +214,15 @@ def test_composite_preserve_originals(datafiles):
      ('implicitoverwrite.yaml', 0, 2, 'overwrite1', 'implicitoverwrite.yaml', 4, 8),
      ('implicitoverwrite.yaml', 1, 2, 'overwrite2', 'implicitoverwrite.yaml', 6, 8),
  ])
 -def test_list_composition(datafiles, filename,
 +def test_list_composition(datafiles, filename, tmpdir,
                            index, length, mood,
 -                          prov_file, prov_line, prov_col):
 -    base = os.path.join(datafiles.dirname, datafiles.basename, 'basics.yaml')
 -    overlay = os.path.join(datafiles.dirname, datafiles.basename, filename)
 +                          prov_file, prov_line, prov_col, caching):
 +    base_file = os.path.join(datafiles.dirname, datafiles.basename, 'basics.yaml')
 +    overlay_file = os.path.join(datafiles.dirname, datafiles.basename, filename)
++
 +    base = load_yaml_file(base_file, cache_path=tmpdir, shortname='basics.yaml', from_cache=caching)
 +    overlay = load_yaml_file(overlay_file, cache_path=tmpdir, shortname=filename, from_cache=caching)
 -    base = _yaml.load(base, shortname='basics.yaml')
 -    overlay = _yaml.load(overlay, shortname=filename)
      _yaml.composite_dict(base, overlay)
      children = _yaml.node_get(base, list, 'children')
@@ -254,6 +274,7 @@ def test_list_deletion(datafiles):
  #    prov_col: The expected provenance column of "mood"
+ #
  @pytest.mark.datafiles(os.path.join(DATA_DIR))
 +@pytest.mark.parametrize('caching', [('raw'), ('cached')])
  @pytest.mark.parametrize("filename1,filename2,index,length,mood,prov_file,prov_line,prov_col", [
      # Test results of compositing literal list with (>) and then (<)
@@ -310,9 +331,9 @@ def test_list_deletion(datafiles):
      ('listoverwrite.yaml', 'listprepend.yaml', 2, 4, 'overwrite1', 'listoverwrite.yaml', 5, 10),
      ('listoverwrite.yaml', 'listprepend.yaml', 3, 4, 'overwrite2', 'listoverwrite.yaml', 7, 10),
  ])
 -def test_list_composition_twice(datafiles, filename1, filename2,
 +def test_list_composition_twice(datafiles, tmpdir, filename1, filename2,
                                  index, length, mood,
 -                                prov_file, prov_line, prov_col):
 +                                prov_file, prov_line, prov_col, caching):
      file_base = os.path.join(datafiles.dirname, datafiles.basename, 'basics.yaml')
      file1 = os.path.join(datafiles.dirname, datafiles.basename, filename1)
      file2 = os.path.join(datafiles.dirname, datafiles.basename, filename2)
@@ -320,9 +341,9 @@ def test_list_composition_twice(datafiles, filename1, filename2,
      #####################
      # Round 1 - Fight !
      #####################
 -    base = _yaml.load(file_base, shortname='basics.yaml')
 -    overlay1 = _yaml.load(file1, shortname=filename1)
 -    overlay2 = _yaml.load(file2, shortname=filename2)
 +    base = load_yaml_file(file_base, cache_path=tmpdir, shortname='basics.yaml', from_cache=caching)
 +    overlay1 = load_yaml_file(file1, cache_path=tmpdir, shortname=filename1, from_cache=caching)
 +    overlay2 = load_yaml_file(file2, cache_path=tmpdir, shortname=filename2, from_cache=caching)
      _yaml.composite_dict(base, overlay1)
      _yaml.composite_dict(base, overlay2)
@@ -337,9 +358,9 @@ def test_list_composition_twice(datafiles, filename1, filename2,
      #####################
      # Round 2 - Fight !
      #####################
 -    base = _yaml.load(file_base, shortname='basics.yaml')
 -    overlay1 = _yaml.load(file1, shortname=filename1)
 -    overlay2 = _yaml.load(file2, shortname=filename2)
 +    base = load_yaml_file(file_base, cache_path=tmpdir, shortname='basics.yaml', from_cache=caching)
 +    overlay1 = load_yaml_file(file1, cache_path=tmpdir, shortname=filename1, from_cache=caching)
 +    overlay2 = load_yaml_file(file2, cache_path=tmpdir, shortname=filename2, from_cache=caching)
      _yaml.composite_dict(overlay1, overlay2)
      _yaml.composite_dict(base, overlay1)

[Notes] [Git][BuildStream/buildstream][tpollard/494] 10 commits: yaml: Add a cache of parsed and provenanced yaml

Tom Pollard pushed to branch tpollard/494 at BuildStream / buildstream

Commits:

16 changed files:

Changes: