[Notes] [Git][BuildStream/buildstream][lachlan/pickle-yaml-test-list-com

Lachlan pushed to branch lachlan/pickle-yaml-test-list-composite at BuildStream / buildstream

Commits:

ce55b9a0

by Tiago Gomes at 2018-10-12T10:15:46Z

doc: updates considering website being live now

* Add a link to the website on the main page.
* Remove install instructions as they are now on the website.
* Remove Resources section as that information can be found at the
  website, and also looks bad.
* Move artifact server setup from the no longer existing Install section
  to the Using section.

10b092e1

by Tristan Van Berkom at 2018-10-12T10:21:27Z

doc/source/additional_docker.rst: Fix link to refer to website

The docker guide, which is part of the install guide, has moved
to the website.

1c7ba9a0

by Tristan Van Berkom at 2018-10-12T10:24:31Z

doc/source/index.rst: Moved references to the website below the simple ToC.

3738dd06

by Tristan Van Berkom at 2018-10-12T10:49:39Z

Merge branch 'tristan/remove-install-guide' into 'master'

Remove install guide

See merge request BuildStream/buildstream!872

c31ed138

by Tristan Van Berkom at 2018-10-14T15:24:53Z

CONTRIBUTING.rst: Added more guidelines about documenting the user guide

0be26464

by Jonathan Maw at 2018-10-15T13:18:33Z

yaml: Add a cache of parsed and provenanced yaml

Note that the ProvenanceFile's names will be incorrect after loading
from the cache, but this is currently only used for writeback, which
isn't used in junctions.

ef5cead0
by Jonathan Maw at 2018-10-15T13:19:36Z
```
yamlcache.py: Add yaml cache tests
```

7aecc1d0

by Lachlan Mackenzie at 2018-10-15T13:19:47Z

yaml.py: Add loader for yaml test files cache option

* Fix to CWD issue provided by Jonathan Maw

653f660b

by Lachlan Mackenzie at 2018-10-15T13:19:53Z

yaml.py: Add yaml cache testing to yaml list composition test

fc59802d

by Lachlan Mackenzie at 2018-10-15T13:19:59Z

yaml.py: Add yaml cache test to yaml list composition twice

3744c6d6

by Lachlan Mackenzie at 2018-10-15T13:20:04Z

yamlcache.py: Add YAML cache changed file test

* Test in same style as test_yamlcache_used
* Move of project not required so removed

19 changed files:

CONTRIBUTING.rst
buildstream/_loader/loader.py
buildstream/_yaml.py
+ buildstream/_yamlcache.py
doc/source/additional_docker.rst
doc/source/index.rst
− doc/source/install_docker.rst
− doc/source/install_linux_distro.rst
− doc/source/install_source.rst
− doc/source/install_versions.rst
− doc/source/main_install.rst
doc/source/main_using.rst
− doc/source/release-badge.rst
− doc/source/snapshot-badge.rst
doc/source/install_artifacts.rst → doc/source/using_configuring_artifact_server.rst
doc/source/using_examples.rst
doc/source/using_tutorial.rst
+ tests/frontend/yamlcache.py
tests/yaml/yaml.py

Changes:

CONTRIBUTING.rst

@@ -1147,7 +1147,7 @@ To actually regenerate the code::
  Documenting
  -----------
  BuildStream starts out as a documented project from day one and uses
 -sphinx to document itself.
 +`sphinx <www.sphinx-doc.org>`_ to document itself.
  This section discusses formatting policies for editing files in the
  ``doc/source`` directory, and describes the details of how the docs are
@@ -1194,10 +1194,8 @@ The BuildStream documentation style is as follows:
      the target, e.g.: ``:ref:`Link text <anchor_name>```.
      Note that the "_" prefix is not used when referring to the target.
 -Useful links:
+-
 -For further information, please see the `Sphinx Documentation
 -<http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_.
 +For further information about using the reStructuredText with sphinx, please see the
 +`Sphinx Documentation <http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_.
  Building Docs
@@ -1266,10 +1264,101 @@ the ``man/`` subdirectory, which will be automatically included
  in the buildstream distribution.
 -Documentation Examples
 -~~~~~~~~~~~~~~~~~~~~~~
 -The examples section of the documentation contains a series of standalone
 -examples, here are the criteria for an example addition.
 +User guide
 +~~~~~~~~~~
 +The :ref:`user guide <using>` is comprised of free form documentation
 +in manually written ``.rst`` files and is split up into a few sections,
 +of main interest are the :ref:`tutorial <tutorial>` and the
 +:ref:`examples <examples>`.
++
 +The distinction of the two categories of user guides is important to
 +understand too.
++
 +* **Tutorial**
++
 +  The tutorial is structured as a series of exercises which start with
 +  the most basic concepts and build upon the previous chapters in order
 +  to arrive at a basic understanding of how to create BuildStream projects.
++
 +  This series of examples should be easy enough to complete in a matter
 +  of a few hours for a new user, and should provide just enough insight to
 +  get the user started in creating their own projects.
++
 +  Going through the tutorial step by step should also result in the user
 +  becoming proficient enough with the reference manual to get by on their own.
++
 +* **Examples**
++
 +  These exist to demonstrate how to accomplish more advanced tasks which
 +  are not always obvious and discoverable.
++
 +  Alternatively, these also demonstrate elegant and recommended ways of
 +  accomplishing some tasks which could be done in various ways.
++
++
 +Guidelines
 +''''''''''
 +Here are some general guidelines for adding new free form documentation
 +to the user guide.
++
 +* **Focus on a single subject**
++
 +  It is important to stay focused on a single subject and avoid getting
 +  into tangential material when creating a new entry, so that the articles
 +  remain concise and the user is not distracted by unrelated subject material.
++
 +  A single tutorial chapter or example should not introduce any additional
 +  subject material than the material being added for the given example.
++
 +* **Reuse existing sample project elements**
++
 +  To help avoid distracting from the topic at hand, it is always preferable to
 +  reuse the same project sample material from other examples and only deviate
 +  slightly to demonstrate the new material, than to create completely new projects.
++
 +  This helps us remain focused on a single topic at a time, and reduces the amount
 +  of unrelated material the reader needs to learn in order to digest the new
 +  example.
++
 +* **Don't be redundant**
++
 +  When something has already been explained in the tutorial or in another example,
 +  it is best to simply refer to the other user guide entry in a new example.
++
 +  Always prefer to link to the tutorial if an explanation exists in the tutorial,
 +  rather than linking to another example, where possible.
++
 +* **Link into the reference manual at every opportunity**
++
 +  The format and plugin API is 100% documented at all times. Whenever discussing
 +  anything about the format or plugin API, always do so while providing a link
 +  into the more terse reference material.
++
 +  We don't want users to have to search for the material themselves, and we also
 +  want the user to become proficient at navigating the reference material over
 +  time.
++
 +* **Use concise terminology**
++
 +  As developers, we tend to come up with code names for features we develop, and
 +  then end up documenting a new feature in an example.
++
 +  Never use a code name or shorthand to refer to a feature in the user guide, instead
 +  always use fully qualified sentences outlining very explicitly what we are doing
 +  in the example, or what the example is for in the case of a title.
++
 +  We need to be considerate that the audience of our user guide is probably a
 +  proficient developer or integrator, but has no idea what we might have decided
 +  to name a given activity.
++
++
 +Structure of an example
 +'''''''''''''''''''''''
 +The :ref:`tutorial <tutorial>` and the :ref:`examples <examples>` sections
 +of the documentation contain a series of sample projects, each chapter in
 +the tutoral, or standalone example uses a sample project.
++
 +Here is the the structure for adding new examples and tutorial chapters.
  * The example has a ``${name}``.
@@ -1280,11 +1369,17 @@ examples, here are the criteria for an example addition.
  * The example has a documentation component.
    * This is added at ``doc/source/examples/${name}.rst``
 -  * A reference to ``examples/${name}`` is added to the toctree in ``doc/source/examples.rst``
 +  * An entry for ``examples/${name}`` is added to the toctree in ``doc/source/using_examples.rst``
    * This documentation discusses the project elements declared in the project and may
      provide some BuildStream command examples.
    * This documentation links out to the reference manual at every opportunity.
 +  .. note::
++
 +     In the case of a tutorial chapter, the ``.rst`` file is added in at
 +     ``doc/source/tutorial/${name}.rst`` and an entry for ``tutorial/${name}``
 +     is added to ``doc/source/using_tutorial.rst``.
++
  * The example has a CI test component.
    * This is an integration test added at ``tests/examples/${name}``.

buildstream/_loader/loader.py

@@ -29,6 +29,7 @@ from .. import _yaml
  from ..element import Element
  from .._profile import Topics, profile_start, profile_end
  from .._includes import Includes
 +from .._yamlcache import YamlCache
  from .types import Symbol, Dependency
  from .loadelement import LoadElement
@@ -108,13 +109,19 @@ class Loader():
+         #
          deps = []
 -        for target in targets:
 -            profile_start(Topics.LOAD_PROJECT, target)
 -            junction, name, loader = self._parse_name(target, rewritable, ticker,
 -                                                      fetch_subprojects=fetch_subprojects)
 -            loader._load_file(name, rewritable, ticker, fetch_subprojects)
 -            deps.append(Dependency(name, junction=junction))
 -            profile_end(Topics.LOAD_PROJECT, target)
 +        # XXX This will need to be changed to the context's top-level project if this method
 +        # is ever used for subprojects
 +        top_dir = self.project.directory
++
 +        cache_file = YamlCache.get_cache_file(top_dir)
 +        with YamlCache.open(self._context, cache_file) as yaml_cache:
 +            for target in targets:
 +                profile_start(Topics.LOAD_PROJECT, target)
 +                junction, name, loader = self._parse_name(target, rewritable, ticker,
 +                                                          fetch_subprojects=fetch_subprojects)
 +                loader._load_file(name, rewritable, ticker, fetch_subprojects, yaml_cache)
 +                deps.append(Dependency(name, junction=junction))
 +                profile_end(Topics.LOAD_PROJECT, target)
+         #
          # Now that we've resolve the dependencies, scan them for circular dependencies
@@ -201,11 +208,12 @@ class Loader():
      #    rewritable (bool): Whether we should load in round trippable mode
      #    ticker (callable): A callback to report loaded filenames to the frontend
      #    fetch_subprojects (bool): Whether to fetch subprojects while loading
 +    #    yaml_cache (YamlCache): A yaml cache
+     #
      # Returns:
      #    (LoadElement): A loaded LoadElement
+     #
 -    def _load_file(self, filename, rewritable, ticker, fetch_subprojects):
 +    def _load_file(self, filename, rewritable, ticker, fetch_subprojects, yaml_cache=None):
          # Silently ignore already loaded files
          if filename in self._elements:
@@ -218,7 +226,8 @@ class Loader():
          # Load the data and process any conditional statements therein
          fullpath = os.path.join(self._basedir, filename)
          try:
 -            node = _yaml.load(fullpath, shortname=filename, copy_tree=rewritable, project=self.project)
 +            node = _yaml.load(fullpath, shortname=filename, copy_tree=rewritable,
 +                              project=self.project, yaml_cache=yaml_cache)
          except LoadError as e:
              if e.reason == LoadErrorReason.MISSING_FILE:
                  # If we can't find the file, try to suggest plausible
@@ -261,13 +270,13 @@ class Loader():
          # Load all dependency files for the new LoadElement
          for dep in element.deps:
              if dep.junction:
 -                self._load_file(dep.junction, rewritable, ticker, fetch_subprojects)
 +                self._load_file(dep.junction, rewritable, ticker, fetch_subprojects, yaml_cache)
                  loader = self._get_loader(dep.junction, rewritable=rewritable, ticker=ticker,
                                            fetch_subprojects=fetch_subprojects)
              else:
                  loader = self
 -            dep_element = loader._load_file(dep.name, rewritable, ticker, fetch_subprojects)
 +            dep_element = loader._load_file(dep.name, rewritable, ticker, fetch_subprojects, yaml_cache)
              if _yaml.node_get(dep_element.node, str, Symbol.KIND) == 'junction':
                  raise LoadError(LoadErrorReason.INVALID_DATA,

buildstream/_yaml.py

@@ -183,20 +183,32 @@ class CompositeTypeError(CompositeError):
  #    shortname (str): The filename in shorthand for error reporting (or None)
  #    copy_tree (bool): Whether to make a copy, preserving the original toplevels
  #                      for later serialization
 +#    yaml_cache (YamlCache): A yaml cache to consult rather than parsing
+ #
  # Returns (dict): A loaded copy of the YAML file with provenance information
+ #
  # Raises: LoadError
+ #
 -def load(filename, shortname=None, copy_tree=False, *, project=None):
 +def load(filename, shortname=None, copy_tree=False, *, project=None, yaml_cache=None):
      if not shortname:
          shortname = filename
      file = ProvenanceFile(filename, shortname, project)
      try:
 +        data = None
          with open(filename) as f:
 -            return load_data(f, file, copy_tree=copy_tree)
 +            contents = f.read()
 +        if yaml_cache:
 +            data, key = yaml_cache.get(project, filename, contents, copy_tree)
++
 +        if not data:
 +            data = load_data(contents, file, copy_tree=copy_tree)
++
 +        if yaml_cache:
 +            yaml_cache.put_from_key(project, filename, key, data)
++
 +        return data
      except FileNotFoundError as e:
          raise LoadError(LoadErrorReason.MISSING_FILE,
                          "Could not find file at {}".format(filename)) from e

buildstream/_yamlcache.py

 +#
 +#  Copyright 2018 Bloomberg Finance LP
 +#
 +#  This program is free software; you can redistribute it and/or
 +#  modify it under the terms of the GNU Lesser General Public
 +#  License as published by the Free Software Foundation; either
 +#  version 2 of the License, or (at your option) any later version.
 +#
 +#  This library is distributed in the hope that it will be useful,
 +#  but WITHOUT ANY WARRANTY; without even the implied warranty of
 +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 +#  Lesser General Public License for more details.
 +#
 +#  You should have received a copy of the GNU Lesser General Public
 +#  License along with this library. If not, see <http://www.gnu.org/licenses/>.
 +#
 +#  Authors:
 +#        Jonathan Maw <jonathan maw codethink co uk>
++
 +import os
 +import pickle
 +import hashlib
 +import io
++
 +import sys
++
 +from contextlib import contextmanager
 +from collections import namedtuple
++
 +from ._cachekey import generate_key
 +from ._context import Context
 +from . import utils, _yaml
++
++
 +YAML_CACHE_FILENAME = "yaml_cache.pickle"
++
++
 +# YamlCache()
 +#
 +# A cache that wraps around the loading of yaml in projects.
 +#
 +# The recommended way to use a YamlCache is:
 +#   with YamlCache.open(context) as yamlcache:
 +#     # Load all the yaml
 +#     ...
 +#
 +# Args:
 +#    context (Context): The invocation Context
 +#
 +class YamlCache():
++
 +    def __init__(self, context):
 +        self._project_caches = {}
 +        self._context = context
++
 +    ##################
 +    # Public Methods #
 +    ##################
++
 +    # is_cached():
 +    #
 +    # Checks whether a file is cached.
 +    #
 +    # Args:
 +    #    project (Project): The project this file is in.
 +    #    filepath (str): The path to the file, *relative to the project's directory*.
 +    #
 +    # Returns:
 +    #    (bool): Whether the file is cached.
 +    def is_cached(self, project, filepath):
 +        cache_path = self._get_filepath(project, filepath)
 +        project_name = project.name if project else ""
 +        try:
 +            project_cache = self._project_caches[project_name]
 +            if cache_path in project_cache.elements:
 +                return True
 +        except KeyError:
 +            pass
 +        return False
++
 +    # open():
 +    #
 +    # Return an instance of the YamlCache which writes to disk when it leaves scope.
 +    #
 +    # Args:
 +    #    context (Context): The context.
 +    #    cachefile (str): The path to the cache file.
 +    #
 +    # Returns:
 +    #    (YamlCache): A YamlCache.
 +    @staticmethod
 +    @contextmanager
 +    def open(context, cachefile):
 +        # Try to load from disk first
 +        cache = None
 +        if os.path.exists(cachefile):
 +            try:
 +                with open(cachefile, "rb") as f:
 +                    cache = BstUnpickler(f, context).load()
 +            except EOFError:
 +                # The file was empty
 +                pass
 +            except pickle.UnpicklingError as e:
 +                sys.stderr.write("Failed to load YamlCache, {}\n".format(e))
++
 +        # Failed to load from disk, create a new one
 +        if not cache:
 +            cache = YamlCache(context)
++
 +        yield cache
++
 +        cache._write(cachefile)
++
 +    # get_cache_file():
 +    #
 +    # Retrieves a path to the yaml cache file.
 +    #
 +    # Returns:
 +    #   (str): The path to the cache file
 +    @staticmethod
 +    def get_cache_file(top_dir):
 +        return os.path.join(top_dir, ".bst", YAML_CACHE_FILENAME)
++
 +    # get():
 +    #
 +    # Gets a parsed file from the cache.
 +    #
 +    # Args:
 +    #    project (Project) or None: The project this file is in, if it exists.
 +    #    filepath (str): The absolute path to the file.
 +    #    contents (str): The contents of the file to be cached
 +    #    copy_tree (bool): Whether the data should make a copy when it's being generated
 +    #                      (i.e. exactly as when called in yaml)
 +    #
 +    # Returns:
 +    #    (decorated dict): The parsed yaml from the cache, or None if the file isn't in the cache.
 +    #    (str):            The key used to look up the parsed yaml in the cache
 +    def get(self, project, filepath, contents, copy_tree):
 +        key = self._calculate_key(contents, copy_tree)
 +        data = self._get(project, filepath, key)
 +        return data, key
++
 +    # put():
 +    #
 +    # Puts a parsed file into the cache.
 +    #
 +    # Args:
 +    #    project (Project): The project this file is in.
 +    #    filepath (str): The path to the file.
 +    #    contents (str): The contents of the file that has been cached
 +    #    copy_tree (bool): Whether the data should make a copy when it's being generated
 +    #                      (i.e. exactly as when called in yaml)
 +    #    value (decorated dict): The data to put into the cache.
 +    def put(self, project, filepath, contents, copy_tree, value):
 +        key = self._calculate_key(contents, copy_tree)
 +        self.put_from_key(project, filepath, key, value)
++
 +    # put_from_key():
 +    #
 +    # Put a parsed file into the cache when given a key.
 +    #
 +    # Args:
 +    #    project (Project): The project this file is in.
 +    #    filepath (str): The path to the file.
 +    #    key (str): The key to the file within the cache. Typically, this is the
 +    #               value of `calculate_key()` with the file's unparsed contents
 +    #               and any relevant metadata passed in.
 +    #    value (decorated dict): The data to put into the cache.
 +    def put_from_key(self, project, filepath, key, value):
 +        cache_path = self._get_filepath(project, filepath)
 +        project_name = project.name if project else ""
 +        try:
 +            project_cache = self._project_caches[project_name]
 +        except KeyError:
 +            project_cache = self._project_caches[project_name] = CachedProject({})
++
 +        project_cache.elements[cache_path] = CachedYaml(key, value)
++
 +    ###################
 +    # Private Methods #
 +    ###################
++
 +    # Writes the yaml cache to the specified path.
 +    #
 +    # Args:
 +    #    path (str): The path to the cache file.
 +    def _write(self, path):
 +        parent_dir = os.path.dirname(path)
 +        os.makedirs(parent_dir, exist_ok=True)
 +        with open(path, "wb") as f:
 +            BstPickler(f).dump(self)
++
 +    # _get_filepath():
 +    #
 +    # Returns a file path relative to a project if passed, or the original path if
 +    # the project is None
 +    #
 +    # Args:
 +    #    project (Project) or None: The project the filepath exists within
 +    #    full_path (str): The path that the returned path is based on
 +    #
 +    # Returns:
 +    #    (str): The path to the file, relative to a project if it exists
 +    def _get_filepath(self, project, full_path):
 +        if project:
 +            assert full_path.startswith(project.directory)
 +            filepath = os.path.relpath(full_path, project.directory)
 +        else:
 +            filepath = full_path
 +        return full_path
++
 +    # _calculate_key():
 +    #
 +    # Calculates a key for putting into the cache.
 +    #
 +    # Args:
 +    #    (basic object)... : Any number of strictly-ordered basic objects
 +    #
 +    # Returns:
 +    #   (str): A key made out of every arg passed in
 +    @staticmethod
 +    def _calculate_key(*args):
 +        string = pickle.dumps(args)
 +        return hashlib.sha1(string).hexdigest()
++
 +    # _get():
 +    #
 +    # Gets a parsed file from the cache when given a key.
 +    #
 +    # Args:
 +    #    project (Project): The project this file is in.
 +    #    filepath (str): The path to the file.
 +    #    key (str): The key to the file within the cache. Typically, this is the
 +    #               value of `calculate_key()` with the file's unparsed contents
 +    #               and any relevant metadata passed in.
 +    #
 +    # Returns:
 +    #    (decorated dict): The parsed yaml from the cache, or None if the file isn't in the cache.
 +    def _get(self, project, filepath, key):
 +        cache_path = self._get_filepath(project, filepath)
 +        project_name = project.name if project else ""
 +        try:
 +            project_cache = self._project_caches[project_name]
 +            try:
 +                cachedyaml = project_cache.elements[cache_path]
 +                if cachedyaml._key == key:
 +                    # We've unpickled the YamlCache, but not the specific file
 +                    if cachedyaml._contents is None:
 +                        cachedyaml._contents = BstUnpickler.loads(cachedyaml._pickled_contents, self._context)
 +                    return cachedyaml._contents
 +            except KeyError:
 +                pass
 +        except KeyError:
 +            pass
 +        return None
++
++
 +CachedProject = namedtuple('CachedProject', ['elements'])
++
++
 +class CachedYaml():
 +    def __init__(self, key, contents):
 +        self._key = key
 +        self.set_contents(contents)
++
 +    # Sets the contents of the CachedYaml.
 +    #
 +    # Args:
 +    #    contents (provenanced dict): The contents to put in the cache.
 +    #
 +    def set_contents(self, contents):
 +        self._contents = contents
 +        self._pickled_contents = BstPickler.dumps(contents)
++
 +    # Pickling helper method, prevents 'contents' from being serialised
 +    def __getstate__(self):
 +        data = self.__dict__.copy()
 +        data['_contents'] = None
 +        return data
++
++
 +# In _yaml.load, we have a ProvenanceFile that stores the project the file
 +# came from. Projects can't be pickled, but it's always going to be the same
 +# project between invocations (unless the entire project is moved but the
 +# file stayed in the same place)
 +class BstPickler(pickle.Pickler):
 +    def persistent_id(self, obj):
 +        if isinstance(obj, _yaml.ProvenanceFile):
 +            if obj.project:
 +                # ProvenanceFile's project object cannot be stored as it is.
 +                project_tag = obj.project.name
 +                # ProvenanceFile's filename must be stored relative to the
 +                # project, as the project dir may move.
 +                name = os.path.relpath(obj.name, obj.project.directory)
 +            else:
 +                project_tag = None
 +                name = obj.name
 +            return ("ProvenanceFile", name, obj.shortname, project_tag)
 +        elif isinstance(obj, Context):
 +            return ("Context",)
 +        else:
 +            return None
++
 +    @staticmethod
 +    def dumps(obj):
 +        stream = io.BytesIO()
 +        BstPickler(stream).dump(obj)
 +        stream.seek(0)
 +        return stream.read()
++
++
 +class BstUnpickler(pickle.Unpickler):
 +    def __init__(self, file, context):
 +        super().__init__(file)
 +        self._context = context
++
 +    def persistent_load(self, pid):
 +        if pid[0] == "ProvenanceFile":
 +            _, tagged_name, shortname, project_tag = pid
++
 +            if project_tag is not None:
 +                for p in self._context.get_projects():
 +                    if project_tag == p.name:
 +                        project = p
 +                        break
++
 +                name = os.path.join(project.directory, tagged_name)
++
 +                if not project:
 +                    projects = [p.name for p in self._context.get_projects()]
 +                    raise pickle.UnpicklingError("No project with name {} found in {}"
 +                                                 .format(key_id, projects))
 +            else:
 +                project = None
 +                name = tagged_name
++
 +            return _yaml.ProvenanceFile(name, shortname, project)
 +        elif pid[0] == "Context":
 +            return self._context
 +        else:
 +            raise pickle.UnpicklingError("Unsupported persistent object, {}".format(pid))
++
 +    @staticmethod
 +    def loads(text, context):
 +        stream = io.BytesIO()
 +        stream.write(bytes(text))
 +        stream.seek(0)
 +        return BstUnpickler(stream, context).load()

doc/source/additional_docker.rst

@@ -4,19 +4,18 @@
  BuildStream and Docker
  ======================
+-
  BuildStream integrates with Docker in multiple ways. Here are some ways in
  which these integrations work.
++
  Run BuildStream inside Docker
  -----------------------------
 +Refer to the `BuildStream inside Docker <https://buildstream.build/docker_install.html>`_
 +documentation for instructions on how to run BuildStream as a Docker container.
 -Refer to the :ref:`BuildStream inside Docker <docker>` documentation for
 -instructions on how to run BuildStream as a Docker container.
  Generate Docker images
  ----------------------
+-
  The
  `bst-docker-import script <https://gitlab.com/BuildStream/buildstream/blob/master/contrib/bst-docker-import>`_
  can be used to generate a Docker image from built artifacts.

doc/source/index.rst

@@ -13,20 +13,13 @@ They begin with a basic introduction to BuildStream, background
  information on basic concepts, and a guide to the BuildStream command line interface.
  Later sections provide detailed information on BuildStream internals.
+-
  .. toctree::
     :maxdepth: 1
     main_about
 -   main_install
     main_using
     main_core
     CONTRIBUTING
+-
 -Resources
 ----------
 -* GitLab repository: https://gitlab.com/BuildStream/buildstream
 -* Bug Tracking: https://gitlab.com/BuildStream/buildstream/issues
 -* Mailing list: https://mail.gnome.org/mailman/listinfo/buildstream-list
 -* IRC Channel: irc://irc.gnome.org/#buildstream
 +For any other information, including `how to install BuildStream <https://buildstream.build/install.html>`_,
 +refer to `the BuildStream website <https://buildstream.build>`_.

doc/source/install_docker.rst deleted

+-
+-
 -.. _docker:
+-
 -BuildStream inside Docker
 --------------------------
 -If your system cannot provide the base system requirements for BuildStream, then it is possible to run buildstream within a Docker image.
+-
 -The BuildStream project provides
 -`Docker images <https://hub.docker.com/r/buildstream/buildstream-fedora>`_
 -containing BuildStream and its dependencies.
 -This gives you an easy way to get started using BuildStream on any Unix-like
 -platform where Docker is available, including Mac OS X.
+-
 -We recommend using the
 -`bst-here wrapper script <https://gitlab.com/BuildStream/buildstream/blob/master/contrib/bst-here>`_
 -which automates the necessary container setup. You can download it and make
 -it executable like this:
+-
 -.. code:: bash
+-
 -  mkdir -p ~/.local/bin
 -  curl --get https://gitlab.com/BuildStream/buildstream/raw/master/contrib/bst-here > ~/.local/bin/bst-here
 -  chmod +x ~/.local/bin/bst-here
+-
 -Check if ``~/.local/bin`` appears in your PATH environment variable -- if it
 -doesn't, you should
 -`edit your ~/.profile so that it does <https://stackoverflow.com/questions/14637979/>`_.
+-
 -Once the script is available in your PATH, you can run ``bst-here`` to open a
 -shell session inside a new container based off the latest version of the
 -buildstream-fedora Docker image. The current working directory will be mounted
 -inside the container at ``/src``.
+-
 -You can also run individual BuildStream commands as ``bst-here COMMAND``. For
 -example: ``bst-here show systems/my-system.bst``. Note that BuildStream won't
 -be able to integrate with Bash tab-completion if you invoke it in this way.
+-
 -Two Docker volumes are set up by the ``bst-here`` script:
+-
 - * ``buildstream-cache --`` mounted at ``~/.cache/buildstream``
 - * ``buildstream-config --`` mounted at ``~/.config/``
+-
 -These are necessary so that your BuildStream cache and configuration files
 -persist between invocations of ``bst-here``.

doc/source/install_linux_distro.rst deleted

+-
+-
 -.. _install_linux_distro:
+-
 -Installing from distro packages
 -===============================
 -BuildStream is available on some linux distributions, here are
 -some install instructions for the linux distributions which
 -have packaged BuildStream.
+-
+-
 -Arch Linux
 -----------
 -Packages for Arch exist in `AUR <https://wiki.archlinux.org/index.php/Arch_User_Repository#Installing_packages>`_.
 -Two different package versions are available:
+-
 -* Latest release: `buildstream <https://aur.archlinux.org/packages/buildstream>`_
 -* Latest development snapshot: `buildstream-git <https://aur.archlinux.org/packages/buildstream-git>`_
+-
+-
 -Fedora
 -------
 -BuildStream is not yet in the official Fedora repositories, but you can
 -install it from a Copr::
+-
 -  sudo dnf copr enable bochecha/buildstream
 -  sudo dnf install buildstream
+-
 -Optionally, install the ``buildstream-docs`` package to have the BuildStream
 -documentation in Devhelp or GNOME Builder.

doc/source/install_source.rst deleted

+-
+-
 -Installing from source
 -======================
 -Until BuildStream is available in :ref:`your distro <install_linux_distro>`, you will
 -need to install it yourself from source.
+-
+-
 -Installing dependencies
 ------------------------
 -Before installing BuildStream from source, it is necessary to first install
 -the system dependencies. Below are some linux distribution specific instructions
 -for installing these dependencies.
+-
 -BuildStream requires the following base system requirements:
+-
 -* python3 >= 3.5
 -* bubblewrap >= 0.1.2
 -* fuse2
+-
 -BuildStream also depends on the host tools for the :mod:`Source <buildstream.source>` plugins.
 -Refer to the respective :ref:`source plugin <plugins_sources>` documentation for host tool
 -requirements of specific plugins.
+-
 -The default plugins with extra host dependencies are:
+-
 -* bzr
 -* deb
 -* git
 -* ostree
 -* patch
 -* pip
 -* tar
+-
 -If you intend to push built artifacts to a remote artifact server,
 -which requires special permissions, you will also need:
+-
 -* ssh
+-
+-
 -Arch Linux
 -~~~~~~~~~~
 -Install the dependencies with::
+-
 -  sudo pacman -S \
 -      python fuse2 bubblewrap \
 -      python-pip
+-
 -For the default plugins::
+-
 -  sudo pacman -S \
 -      bzr git lzip ostree patch python-gobject
+-
+-
 -The package *python-arpy* is required by the deb source plugin. This is not
 -obtainable via `pacman`, you must get *python-arpy* from AUR:
 -https://aur.archlinux.org/packages/python-arpy/
+-
 -To install::
+-
 -  wget https://aur.archlinux.org/cgit/aur.git/snapshot/python-arpy.tar.gz
 -  tar -xvf python-arpy.tar.gz
 -  cd python-arpy
 -  makepkg -si
+-
+-
 -Debian
 -~~~~~~
 -Install the dependencies with::
+-
 -  sudo apt-get install \
 -      python3 fuse bubblewrap \
 -      python3-pip python3-dev
+-
 -For the default plugins:
+-
+-
 -Stretch
 -+++++++
 -With stretch, you first need to ensure that you have the backports repository
 -setup as described `here <https://backports.debian.org/Instructions/>`_
+-
 -By adding the following line to your sources.list::
+-
 -  deb http://deb.debian.org/debian stretch-backports main
+-
 -And then running::
+-
 -  sudo apt update
+-
 -At this point you should be able to get the system requirements for the default plugins with::
+-
 -  sudo apt install \
 -      bzr git lzip patch python3-arpy python3-gi
 -  sudo apt install -t stretch-backports \
 -      gir1.2-ostree-1.0 ostree
+-
+-
 -Buster or Sid
 -+++++++++++++
 -For debian unstable or testing, only the following line should be enough
 -to get the system requirements for the default plugins installed::
+-
 -  sudo apt-get install \
 -      lzip gir1.2-ostree-1.0 git bzr ostree patch python3-arpy python3-gi
+-
+-
 -Fedora
 -~~~~~~
 -For recent fedora systems, the following line should get you the system
 -requirements you need::
+-
 -  dnf install -y \
 -      python3 fuse bubblewrap \
 -      python3-pip python3-devel
+-
 -For the default plugins::
+-
 -  dnf install -y \
 -      bzr git lzip patch ostree python3-gobject
 -  pip3 install --user arpy
+-
+-
 -Ubuntu
 -~~~~~~
+-
+-
 -Ubuntu 18.04 LTS or later
 -+++++++++++++++++++++++++
 -Install the dependencies with::
+-
 -  sudo apt install \
 -      python3 fuse bubblewrap \
 -      python3-pip python3-dev
+-
 -For the default plugins::
+-
 -  sudo apt install \
 -      bzr gir1.2-ostree-1.0 git lzip ostree patch python3-arpy python3-gi
+-
+-
 -Ubuntu 16.04 LTS
 -++++++++++++++++
 -On Ubuntu 16.04, neither `bubblewrap <https://github.com/projectatomic/bubblewrap/>`_
 -or `ostree <https://github.com/ostreedev/ostree>`_ are available in the official repositories.
 -You will need to install them in whichever way you see fit. Refer the the upstream documentation
 -for advice on this.
+-
+-
 -Installing
 -----------
 -Once you have the base system dependencies, you can install the BuildStream
 -python package as a regular user.
+-
+-
 -Installing from PyPI (recommended)
 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 -Since we only ever publish :ref:`release versions <install_semantic_versioning>` on
 -PyPI, it is currently recommended to use this installation path. This will
 -ensure that you always have the latest recommended version of BuildStream that
 -we recommend.
+-
 -To install from PyPI, you will additionally require:
+-
 -* pip for python3 (only required for setup)
 -* Python 3 development libraries and headers
+-
 -Simply run the following command::
+-
 -  pip3 install --user BuildStream
+-
 -This will install latest stable version of BuildStream and its pure python
 -dependencies into your user's homedir in ``~/.local``.
+-
 -Keep following the instructions below to ensure that the ``bst``
 -command is in your ``PATH`` and to enable bash completions for it.
+-
 -.. note::
+-
 -  If you want a specific version of BuildStream, you can install it using
 -  ``pip install --user BuildStream==<version-number>``
+-
+-
 -Upgrading from PyPI
 -+++++++++++++++++++
 -Once you have already installed BuildStream from PyPI, you can later update
 -to the latest recommended version like so::
+-
 -  pip install --user --upgrade BuildStream
+-
+-
 -.. _install_git_checkout:
+-
 -Installing from a git checkout
 -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 -To install directly from the `git repository <https://gitlab.com/BuildStream/buildstream.git>`_
 -using python's ``pip`` package manager, you will additionally require:
+-
 -* pip for python3 (only required for setup)
 -* Python 3 development libraries and headers
 -* git (to checkout BuildStream)
+-
 -Before installing, please check the existing tags in the git repository
 -and determine which version you want to install, and whether you want
 -to install an official release version (recommended), or a development snapshot
 -to help us out testing the bleeding edge of development. Follow the
 -:ref:`semantic versioning guide <install_semantic_versioning>` to determine
 -which tag you intend to install.
+-
 -Run the following commands::
+-
 -  git clone https://gitlab.com/BuildStream/buildstream.git
 -  cd buildstream
 -  git checkout <desired release tag>
 -  pip3 install --user -e .
+-
 -This will install buildstream's pure python dependencies into
 -your user's homedir in ``~/.local`` and will run BuildStream directly
 -from the git checkout directory.
+-
 -Keep following the instructions below to ensure that the ``bst``
 -command is in your ``PATH`` and to enable bash completions for it.
+-
 -.. note::
+-
 -   We recommend the ``-e`` option because you can upgrade your
 -   installation by simply updating the checked out git repository.
+-
 -   If you want a full installation that is not linked to your
 -   git checkout, just omit the ``-e`` option from the above commands.
+-
+-
 -Upgrading from a git checkout
 -+++++++++++++++++++++++++++++
 -If you installed BuildStream from a local git checkout using ``-e`` option, all
 -you need to do to upgrade BuildStream is to update your local git checkout::
+-
 -  cd /path/to/buildstream
 -  git pull --rebase
+-
 -If you did not specify the ``-e`` option at install time or the dependancies
 -have changed, you will need to cleanly reinstall BuildStream::
+-
 -  pip3 uninstall buildstream
 -  cd /path/to/buildstream
 -  git pull --rebase
 -  pip3 install --user .
+-
 -.. note::
+-
 -   If BuildStream has added any dependencies since the last upgrade,
 -   you will need to uninstall and reinstall to ensure those dependencies
 -   are met, regardless of whether you have used the ``-e`` option at
 -   install time.
+-
+-
 -Post install setup
 -------------------
 -After having installed from source using any of the above methods, some
 -setup will be required to use BuildStream.
+-
+-
 -Adjust PATH
 -~~~~~~~~~~~
 -Since BuildStream is now installed under your local user's install directories,
 -you need to ensure that ``PATH`` is adjusted.
+-
 -A regular way to do this is to add the following line to the end of your ``~/.bashrc``::
+-
 -  export PATH="${PATH}:${HOME}/.local/bin"
+-
 -.. note::
+-
 -   You will have to restart your terminal in order for these changes to take effect.
+-
+-
 -Bash completions
 -~~~~~~~~~~~~~~~~
 -Bash completions are supported by sourcing the ``buildstream/data/bst``
 -script found in the BuildStream repository. On many systems this script
 -can be installed into a completions directory but when installing BuildStream
 -without a package manager this is not an option.
+-
 -To enable completions for an installation of BuildStream you
 -installed yourself from git, just append the script verbatim
 -to your ``~/.bash_completion``:
+-
 -.. literalinclude:: ../../buildstream/data/bst
 -   :language: yaml

doc/source/install_versions.rst deleted

+-
+-
 -.. _install_semantic_versioning:
+-
 -Semantic Versioning
 -===================
 -BuildStream follows the Semantic Versioning Convention `(SemVer) <https://semver.org/>`_,
 -and uses even minor point numbers to denote releases intended for users while
 -odd minor point numbers represent development snapshops.
+-
 -For example, for a given version number ``X.Y.Z``
 - * The ``X.<even number>.*`` versions are releases intended for users.
 - * The ``X.<odd number>.*`` versions are development spanshots intended for testing.
+-
 -If you are :ref:`installing from git <install_git_checkout>`, please look for the latest
 -tag to ensure you're getting the latest release.
+-
 -* Latest release:
+-
 -  .. include:: release-badge.rst
+-
 -* Latest development snapshot:
+-
 -  .. include:: snapshot-badge.rst

doc/source/main_install.rst deleted

+-
+-
 -.. _install:
+-
 -Install
 -=======
+-
 -.. include:: release-badge.rst
+-
 -.. include:: snapshot-badge.rst
+-
 -This section provides instructions for installing BuildStream and its
 -companion artifact server on various platforms, along with any installation
 -related materials.
+-
 -.. note::
+-
 -   BuildStream is currently only supported natively on Linux. Users of Unix-like
 -   systems where Docker is available can still use BuildStream by following the
 -   :ref:`Docker install guide <docker>`
+-
 -.. toctree::
 -   :maxdepth: 1
+-
 -   install_source
 -   install_linux_distro
 -   install_docker
 -   install_artifacts
 -   install_versions

doc/source/main_using.rst

 +.. _using:
++
  Using
  =====
  This section includes user facing documentation including tutorials,
@@ -15,3 +17,4 @@ guides and information on user preferences and configuration.
     using_examples
     using_config
     using_commands
 +   using_configuring_artifact_server

doc/source/release-badge.rst deleted

+-
 -.. Use this file to include the badge in the documentation, but not in
 -   the README.rst or gitlab rendered materials, that doesnt work.
+-
 -   This is partly a workaround for a sphinx issue, we will be able
 -   to avoid the raw html once this is implemented in sphinx:
+-
 -       https://github.com/sphinx-doc/sphinx/issues/2240
+-
 -   Using the <object> tag instead of the <img> tag which sphinx generates
 -   allows the svg to be "interactive", for us this basically means that
 -   the link we encode in the badge svg is used, rather than static urls
 -   which need to be used around the <img> tag.
+-
 -   WARNING: The custom CSS on the style tag will need to change if we
 -            change the theme, so that the <object> tag behaves similar
 -	    to how the <img> tag is themed by the style sheets.
+-
 -.. raw:: html
+-
 -   <a class="reference external image-reference">
 -     <object style="margin-bottom:24px;vertical-align:middle"
 -             data=""
 -	     type="image/svg+xml"/>
 -     </object>
 -   </a>

doc/source/snapshot-badge.rst deleted

+-
 -.. Use this file to include the badge in the documentation, but not in
 -   the README.rst or gitlab rendered materials, that doesnt work.
+-
 -   This is partly a workaround for a sphinx issue, we will be able
 -   to avoid the raw html once this is implemented in sphinx:
+-
 -       https://github.com/sphinx-doc/sphinx/issues/2240
+-
 -   Using the <object> tag instead of the <img> tag which sphinx generates
 -   allows the svg to be "interactive", for us this basically means that
 -   the link we encode in the badge svg is used, rather than static urls
 -   which need to be used around the <img> tag.
+-
 -   WARNING: The custom CSS on the style tag will need to change if we
 -            change the theme, so that the <object> tag behaves similar
 -	    to how the <img> tag is themed by the style sheets.
+-
 -.. raw:: html
+-
 -   <a class="reference external image-reference">
 -     <object style="margin-bottom:24px;vertical-align:middle"
 -             data=""
 -	     type="image/svg+xml"/>
 -     </object>
 -   </a>

doc/source/install_artifacts.rst → doc/source/using_configuring_artifact_server.rst

@@ -2,8 +2,8 @@
  .. _artifacts:
 -Installing an artifact server
 -=============================
 +Configuring Artifact Server
 +===========================
  BuildStream caches the results of builds in a local artifact cache, and will
  avoid building an element if there is a suitable build already present in the
  local artifact cache.
@@ -72,7 +72,7 @@ Installing the server
  ~~~~~~~~~~~~~~~~~~~~~
  You will also need to install BuildStream on the artifact server in order
  to receive uploaded artifacts over ssh. Follow the instructions for installing
 -BuildStream :ref:`here <install>`
 +BuildStream `here <https://buildstream.build/install.html>`_.
  When installing BuildStream on the artifact server, it must be installed
  in a system wide location, with ``pip3 install .`` in the BuildStream

doc/source/using_examples.rst

++
 +.. _examples:
++
  Examples
  ========
  This page contains documentation for real examples of BuildStream projects,

doc/source/using_tutorial.rst

++
 +.. _tutorial:
++
  Tutorial
  ========
  This is a step by step walkthrough meant help the user quickly get

tests/frontend/yamlcache.py

 +import os
 +import pytest
 +import hashlib
 +import tempfile
 +from ruamel import yaml
++
 +from tests.testutils import cli, generate_junction, create_element_size, create_repo
 +from buildstream import _yaml
 +from buildstream._yamlcache import YamlCache
 +from buildstream._project import Project
 +from buildstream._context import Context
 +from contextlib import contextmanager
++
++
 +def generate_project(tmpdir, ref_storage, with_junction, name="test"):
 +    if with_junction == 'junction':
 +        subproject_dir = generate_project(
 +            tmpdir, ref_storage,
 +            'no-junction', name='test-subproject'
 +        )
++
 +    project_dir = os.path.join(tmpdir, name)
 +    os.makedirs(project_dir)
 +    # project.conf
 +    project_conf_path = os.path.join(project_dir, 'project.conf')
 +    elements_path = 'elements'
 +    project_conf = {
 +        'name': name,
 +        'element-path': elements_path,
 +        'ref-storage': ref_storage,
 +    }
 +    _yaml.dump(project_conf, project_conf_path)
++
 +    # elements
 +    if with_junction == 'junction':
 +        junction_name = 'junction.bst'
 +        junction_dir = os.path.join(project_dir, elements_path)
 +        junction_path = os.path.join(project_dir, elements_path, junction_name)
 +        os.makedirs(junction_dir)
 +        generate_junction(tmpdir, subproject_dir, junction_path)
 +        element_depends = [{'junction': junction_name, 'filename': 'test.bst'}]
 +    else:
 +        element_depends = []
++
 +    element_name = 'test.bst'
 +    create_element_size(element_name, project_dir, elements_path, element_depends, 1)
++
 +    return project_dir
++
++
 +@contextmanager
 +def with_yamlcache(project_dir):
 +    context = Context()
 +    project = Project(project_dir, context)
 +    cache_file = YamlCache.get_cache_file(project_dir)
 +    with YamlCache.open(context, cache_file) as yamlcache:
 +        yield yamlcache, project
++
++
 +def yamlcache_key(yamlcache, in_file, copy_tree=False):
 +    with open(in_file) as f:
 +        key = yamlcache._calculate_key(f.read(), copy_tree)
 +    return key
++
++
 +def modified_file(input_file, tmpdir):
 +    with open(input_file) as f:
 +        data = f.read()
 +    assert 'variables' not in data
 +    data += '\nvariables: {modified: True}\n'
 +    _, temppath = tempfile.mkstemp(dir=tmpdir, text=True)
 +    with open(temppath, 'w') as f:
 +        f.write(data)
++
 +    return temppath
++
++
 +@pytest.mark.parametrize('ref_storage', ['inline', 'project.refs'])
 +@pytest.mark.parametrize('with_junction', ['no-junction', 'junction'])
 +@pytest.mark.parametrize('move_project', ['move', 'no-move'])
 +def test_yamlcache_used(cli, tmpdir, ref_storage, with_junction, move_project):
 +    # Generate the project
 +    project = generate_project(str(tmpdir), ref_storage, with_junction)
 +    if with_junction == 'junction':
 +        result = cli.run(project=project, args=['fetch', '--track', 'junction.bst'])
 +        result.assert_success()
++
 +    # bst show to put it in the cache
 +    result = cli.run(project=project, args=['show', 'test.bst'])
 +    result.assert_success()
++
 +    element_path = os.path.join(project, 'elements', 'test.bst')
 +    with with_yamlcache(project) as (yc, prj):
 +        # Check that it's in the cache
 +        assert yc.is_cached(prj, element_path)
++
 +        # *Absolutely* horrible cache corruption to check it's being used
 +        # Modifying the data from the cache is fraught with danger,
 +        # so instead I'll load a modified version of the original file
 +        temppath = modified_file(element_path, str(tmpdir))
 +        contents = _yaml.load(temppath, copy_tree=False, project=prj)
 +        key = yamlcache_key(yc, element_path)
 +        yc.put_from_key(prj, element_path, key, contents)
++
 +    # Show that a variable has been added
 +    result = cli.run(project=project, args=['show', '--format', '%{vars}', 'test.bst'])
 +    result.assert_success()
 +    data = yaml.safe_load(result.output)
 +    assert 'modified' in data
 +    assert data['modified'] == 'True'
++
++
 +@pytest.mark.parametrize('ref_storage', ['inline', 'project.refs'])
 +@pytest.mark.parametrize('with_junction', ['junction', 'no-junction'])
 +def test_yamlcache_changed_file(cli, tmpdir, ref_storage, with_junction):
 +    # i.e. a file is cached, the file is changed, loading the file (with cache) returns new data
 +    # inline and junction can only be changed by opening a workspace
 +    # Generate the project
 +    project = generate_project(str(tmpdir), ref_storage, with_junction)
 +    if with_junction == 'junction':
 +        result = cli.run(project=project, args=['fetch', '--track', 'junction.bst'])
 +        result.assert_success()
++
 +    # bst show to put it in the cache
 +    result = cli.run(project=project, args=['show', 'test.bst'])
 +    result.assert_success()
++
 +    element_path = os.path.join(project, 'elements', 'test.bst')
 +    with with_yamlcache(project) as (yc, prj):
 +        # Check that it's in the cache then modify
 +        assert yc.is_cached(prj, element_path)
 +        with open(element_path, "a") as f:
 +            f.write('\nvariables: {modified: True}\n')
 +        # Load modified yaml cache file into cache
 +        _yaml.load(element_path, copy_tree=False, project=prj, yaml_cache=yc)
++
 +    # Show that a variable has been added
 +    result = cli.run(project=project, args=['show', '--format', '%{vars}', 'test.bst'])
 +    result.assert_success()
 +    data = yaml.safe_load(result.output)
 +    assert 'modified' in data
 +    assert data['modified'] == 'True'

tests/yaml/yaml.py

  import os
  import pytest
 +import tempfile
  from collections import Mapping
  from buildstream import _yaml
  from buildstream._exceptions import LoadError, LoadErrorReason
 +from buildstream._context import Context
 +from buildstream._yamlcache import YamlCache
  DATA_DIR = os.path.join(
      os.path.dirname(os.path.realpath(__file__)),
@@ -150,6 +153,21 @@ def test_composite_preserve_originals(datafiles):
      assert(_yaml.node_get(orig_extra, str, 'old') == 'new')
 +def load_yaml_file(filename, *, cache_path, shortname=None, from_cache='raw'):
++
 +    _, temppath = tempfile.mkstemp(dir=os.path.join(cache_path.dirname, cache_path.basename), text=True)
 +    context = Context()
++
 +    with YamlCache.open(context, temppath) as yc:
 +        if from_cache == 'raw':
 +            return _yaml.load(filename, shortname)
 +        elif from_cache == 'cached':
 +            _yaml.load(filename, shortname, yaml_cache=yc)
 +            return _yaml.load(filename, shortname, yaml_cache=yc)
 +        else:
 +            assert False
++
++
  # Tests for list composition
+ #
  # Each test composits a filename on top of basics.yaml, and tests
@@ -165,6 +183,7 @@ def test_composite_preserve_originals(datafiles):
  #    prov_col: The expected provenance column of "mood"
+ #
  @pytest.mark.datafiles(os.path.join(DATA_DIR))
 +@pytest.mark.parametrize('caching', [('raw'), ('cached')])
  @pytest.mark.parametrize("filename,index,length,mood,prov_file,prov_line,prov_col", [
      # Test results of compositing with the (<) prepend directive
@@ -195,14 +214,15 @@ def test_composite_preserve_originals(datafiles):
      ('implicitoverwrite.yaml', 0, 2, 'overwrite1', 'implicitoverwrite.yaml', 4, 8),
      ('implicitoverwrite.yaml', 1, 2, 'overwrite2', 'implicitoverwrite.yaml', 6, 8),
  ])
 -def test_list_composition(datafiles, filename,
 +def test_list_composition(datafiles, filename, tmpdir,
                            index, length, mood,
 -                          prov_file, prov_line, prov_col):
 -    base = os.path.join(datafiles.dirname, datafiles.basename, 'basics.yaml')
 -    overlay = os.path.join(datafiles.dirname, datafiles.basename, filename)
 +                          prov_file, prov_line, prov_col, caching):
 +    base_file = os.path.join(datafiles.dirname, datafiles.basename, 'basics.yaml')
 +    overlay_file = os.path.join(datafiles.dirname, datafiles.basename, filename)
++
 +    base = load_yaml_file(base_file, cache_path=tmpdir, shortname='basics.yaml', from_cache=caching)
 +    overlay = load_yaml_file(overlay_file, cache_path=tmpdir, shortname=filename, from_cache=caching)
 -    base = _yaml.load(base, shortname='basics.yaml')
 -    overlay = _yaml.load(overlay, shortname=filename)
      _yaml.composite_dict(base, overlay)
      children = _yaml.node_get(base, list, 'children')
@@ -254,6 +274,7 @@ def test_list_deletion(datafiles):
  #    prov_col: The expected provenance column of "mood"
+ #
  @pytest.mark.datafiles(os.path.join(DATA_DIR))
 +@pytest.mark.parametrize('caching', [('raw'), ('cached')])
  @pytest.mark.parametrize("filename1,filename2,index,length,mood,prov_file,prov_line,prov_col", [
      # Test results of compositing literal list with (>) and then (<)
@@ -310,9 +331,9 @@ def test_list_deletion(datafiles):
      ('listoverwrite.yaml', 'listprepend.yaml', 2, 4, 'overwrite1', 'listoverwrite.yaml', 5, 10),
      ('listoverwrite.yaml', 'listprepend.yaml', 3, 4, 'overwrite2', 'listoverwrite.yaml', 7, 10),
  ])
 -def test_list_composition_twice(datafiles, filename1, filename2,
 +def test_list_composition_twice(datafiles, tmpdir, filename1, filename2,
                                  index, length, mood,
 -                                prov_file, prov_line, prov_col):
 +                                prov_file, prov_line, prov_col, caching):
      file_base = os.path.join(datafiles.dirname, datafiles.basename, 'basics.yaml')
      file1 = os.path.join(datafiles.dirname, datafiles.basename, filename1)
      file2 = os.path.join(datafiles.dirname, datafiles.basename, filename2)
@@ -320,9 +341,9 @@ def test_list_composition_twice(datafiles, filename1, filename2,
      #####################
      # Round 1 - Fight !
      #####################
 -    base = _yaml.load(file_base, shortname='basics.yaml')
 -    overlay1 = _yaml.load(file1, shortname=filename1)
 -    overlay2 = _yaml.load(file2, shortname=filename2)
 +    base = load_yaml_file(file_base, cache_path=tmpdir, shortname='basics.yaml', from_cache=caching)
 +    overlay1 = load_yaml_file(file1, cache_path=tmpdir, shortname=filename1, from_cache=caching)
 +    overlay2 = load_yaml_file(file2, cache_path=tmpdir, shortname=filename2, from_cache=caching)
      _yaml.composite_dict(base, overlay1)
      _yaml.composite_dict(base, overlay2)
@@ -337,9 +358,9 @@ def test_list_composition_twice(datafiles, filename1, filename2,
      #####################
      # Round 2 - Fight !
      #####################
 -    base = _yaml.load(file_base, shortname='basics.yaml')
 -    overlay1 = _yaml.load(file1, shortname=filename1)
 -    overlay2 = _yaml.load(file2, shortname=filename2)
 +    base = load_yaml_file(file_base, cache_path=tmpdir, shortname='basics.yaml', from_cache=caching)
 +    overlay1 = load_yaml_file(file1, cache_path=tmpdir, shortname=filename1, from_cache=caching)
 +    overlay2 = load_yaml_file(file2, cache_path=tmpdir, shortname=filename2, from_cache=caching)
      _yaml.composite_dict(overlay1, overlay2)
      _yaml.composite_dict(base, overlay1)

[Notes] [Git][BuildStream/buildstream][lachlan/pickle-yaml-test-list-composite] 11 commits: doc: updates considering website being live now

Lachlan pushed to branch lachlan/pickle-yaml-test-list-composite at BuildStream / buildstream

Commits:

19 changed files:

Changes: