[Notes] [Git][BuildStream/buildstream][tpollard/566] 27 commits: sandbox

Tom Pollard pushed to branch tpollard/566 at BuildStream / buildstream

Commits:

81f7a964

by Tristan Van Berkom at 2019-01-08T16:55:58Z

sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.

When code is faulty, in a plugin or in the core, we should
get a stack trace and a BUG message.

edcc43ed

by Tristan Van Berkom at 2019-01-08T19:19:10Z

Merge branch 'tristan/sandbox-programming-error' into 'master'

sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.

See merge request BuildStream/buildstream!1046

17c5ca2d

by Valentin David at 2019-01-09T10:45:23Z

Use relative path to project directory for remote execution certificates/keys

605f8d11

by Valentin David at 2019-01-09T10:45:23Z

Add support for user remote execution configuration

Fixes #631.

65ea03ab

by Valentin David at 2019-01-09T10:45:23Z

Add support for https channel to remote execution and actions servers

Fixes #780.

ba05c4fa

by Valentin David at 2019-01-09T12:58:59Z

Merge branch 'valentindavid/remote_execution_configuration' into 'master'

Remote execution configuration: HTTPS and user configuration

Closes #780 and #631

See merge request BuildStream/buildstream!1030

eb0dbcfc

by Valentin David at 2019-01-09T13:56:58Z

buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY

ScriptElement does not use Sandbox.get_directory. It works using it
with remote execution.

Fixes #850

f7681925

by Valentin David at 2019-01-09T14:44:22Z

Merge branch 'valentindavid/script_virtual_directory' into 'master'

buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY

Closes #850

See merge request BuildStream/buildstream!1047

f87d1c93

by Jim MacArthur at 2019-01-09T15:01:18Z

sandboxremote.py: Rename 'instance_name' option to 'instance-name'

It was 'instance-name' in the documentation.

ed8bc5bc

by Jim MacArthur at 2019-01-09T15:30:28Z

Merge branch 'jmac/rename_instance_name' into 'master'

Rename 'instance_name' option to 'instance-name'

See merge request BuildStream/buildstream!1048

f29a0995

by Tristan Van Berkom at 2019-01-09T19:51:14Z

Support running test environments in parallel with `detox`

This patch namespaces the test temp directory and the output
coverage report file with the name of the environment under test,
such that separately run tests do not access the same files.

When running tests without tox, directly through setup.py,
then the tmp directory will still be `./tmp`.

  * .gitignore: Added new .coverage-reports/ directory

  * .gitlab-ci.yml: Rely on tox to combine and report coverage, only
    tell tox about the COVERAGE_PREFIX so that results can be namespaced
    by CI job name.

    This change also publishes the sources and final combined `.coverage`
    file in an output gitlab artifact for inspection, and lists some missing
    dependencies to the `coverage` job.

  * tox.ini: Add comments and refactor main [testenv] section so that
    other environments dont inherit too much unrelated cruft.

    Generate the coverate reports in the respective {envtmpdir} so that
    all per-process coverage files are prefixed with a full path, ensuring
    that concurrent runs don't mix reports and addressing concerns
    raised in #844.

    Also implemented new `tox -e coverage` environment to combine
    any found coverage and print a report.

  * .coveragerc: Omit .tox/ directory from coverage stats

Fixes issue #844

d364ad02

by Tristan Van Berkom at 2019-01-09T19:52:22Z

CONTRIBUTING.rst: Updated to mention new `coverage` tox environment.

Also point out that it is possible to run test environments in
parallel using the `detox` tool.

1e352434

by Tristan Van Berkom at 2019-01-09T20:36:06Z

Merge branch 'tristan/detox-tests' into 'master'

Allow using detox, and fix/refactor collection of coverage reports

Closes #844

See merge request BuildStream/buildstream!1051

116c1070

by Tristan Van Berkom at 2019-01-09T20:39:15Z

.coveragerc: Omit some things which are irrelevant to cover.

  * Omit versioneer's _version.py

  * Omit our __main__.py which is used only internally for generating
    documentation

c8f3616d

by Tristan Van Berkom at 2019-01-09T21:12:29Z

Merge branch 'tristan/refine-coverage' into 'master'

.coveragerc: Omit some things which are irrelevant to cover.

See merge request BuildStream/buildstream!1052

c87bb592

by Jürg Billeter at 2019-01-10T09:43:37Z

_platform/platform.py: Add canonicalize_arch() method

26e33346

by Jürg Billeter at 2019-01-10T12:50:15Z

_options/optionarch.py: Accept architecture aliases

Accept common architecture aliases for arch options instead of only
accepting the canonicalized, OS-independent architecture name. This
restores compatibility with existing projects and may simplify option
handling for projects that only target a single OS (and thus, do not
need OS-independent architecture names).

06deb4c4

by Jürg Billeter at 2019-01-10T12:50:15Z

element.py: Accept architecture aliases for sandbox config

Accept common architecture aliases for the sandbox config for
consistency with arch options.

0c2a66b3

by Jürg Billeter at 2019-01-10T13:06:02Z

tests/format/optionarch.py: Add tests for architecture aliases

f86bc760

by Jürg Billeter at 2019-01-10T13:34:56Z

Merge branch 'juerg/arch' into 'master'

Accept architecture aliases

See merge request BuildStream/buildstream!1034

347eb34e

by Angelos Evripiotis at 2019-01-10T14:38:23Z

contributing: fix 'oprtation' and some other typos

6bc27bc1

by Angelos Evripiotis at 2019-01-10T15:07:46Z

Merge branch 'aevri/contrib-typos' into 'master'

contributing: fix 'oprtation' and some other typos

See merge request BuildStream/buildstream!1055

0b83d024

by Tristan Van Berkom at 2019-01-10T20:02:50Z

_pipeline.py: Fix the planner to retain the original order when depth sorting

The algorithm adds elements to a dictionary and then sorts the dictionary
by the discovered depths while recursing - using an OrderedDict is enough
to ensure a stable order.

This fixes `bst show --deps plan ...` reporting different results on
each invocation.

This fixes an aspect of #824

630e26f1

by Tristan Van Berkom at 2019-01-10T20:03:52Z

testutils/runcli.py: Added Result.get_start_order()

Added a function to report the list of elements which appeared in
a given queue in the order of their first appearance.

d1d7de57

by Tristan Van Berkom at 2019-01-11T04:13:16Z

tests/frontend/order.py: Adding tests ensuring stable element processing order

For each sample project and expected result order, this test ensures that:

  * bst show --deps plan: Always shows the expected build order.

  * bst source fetch: Always executes in the expected build order.

  * bst build: Always builds in the expected order (disabled).

The build part of the test is currently disabled as the scheduler
seems to be messing with the order even in a `--builders 1` scenario.

276b8d48

by Tristan Van Berkom at 2019-01-11T04:42:24Z

Merge branch 'tristan/element-processing-order' into 'master'

Make build plan element list stable

See merge request BuildStream/buildstream!1058

7436c06b

by Tom Pollard at 2019-01-11T11:47:25Z

WIP: Make uploading of build trees configurable

27 changed files:

.coveragerc
.gitignore
.gitlab-ci.yml
CONTRIBUTING.rst
buildstream/_artifactcache/artifactcache.py
buildstream/_artifactcache/cascache.py
buildstream/_context.py
buildstream/_options/optionarch.py
buildstream/_pipeline.py
buildstream/_platform/platform.py
buildstream/_project.py
buildstream/element.py
buildstream/plugins/elements/script.py
buildstream/sandbox/_sandboxremote.py
buildstream/sandbox/sandbox.py
doc/source/format_project.rst
doc/source/using_config.rst
tests/artifactcache/config.py
+ tests/format/option-arch-alias/element.bst
+ tests/format/option-arch-alias/project.conf
+ tests/format/option-arch-unknown/element.bst
+ tests/format/option-arch-unknown/project.conf
tests/format/optionarch.py
+ tests/frontend/order.py
+ tests/integration/pushbuildtrees.py
tests/testutils/runcli.py
tox.ini

Changes:

.coveragerc

@@ -4,11 +4,15 @@ include =
    */buildstream/*
  omit =
 -  # Omit profiling helper module
 +  # Omit some internals
    */buildstream/_profile.py
 +  */buildstream/__main__.py
 +  */buildstream/_version.py
    # Omit generated code
    */buildstream/_protos/*
    */.eggs/*
 +  # Omit .tox directory
 +  */.tox/*
  [report]
  show_missing = True

.gitignore

@@ -13,11 +13,12 @@ tests/**/*.pyc
  integration-cache/
  tmp
  .coverage
 +.coverage-reports/
  .coverage.*
  .cache
  .pytest_cache/
  *.bst/
 -.tox
 +.tox/
  # Pycache, in case buildstream is ran directly from within the source
  # tree

.gitlab-ci.yml

@@ -13,6 +13,7 @@ variables:
    PYTEST_ADDOPTS: "--color=yes"
    INTEGRATION_CACHE: "${CI_PROJECT_DIR}/cache/integration-cache"
    TEST_COMMAND: "tox -- --color=yes --integration"
 +  COVERAGE_PREFIX: "${CI_JOB_NAME}."
  #####################################################
@@ -24,9 +25,6 @@ variables:
  .tests-template: &tests
    stage: test
 -  variables:
 -    COVERAGE_DIR: coverage-linux
+-
    before_script:
    # Diagnostics
    - mount
@@ -40,14 +38,11 @@ variables:
    - su buildstream -c "${TEST_COMMAND}"
    after_script:
 -  # Collect our reports
 -  - mkdir -p ${COVERAGE_DIR}
 -  - cp .coverage ${COVERAGE_DIR}/coverage."${CI_JOB_NAME}"
    except:
    - schedules
    artifacts:
      paths:
 -    - ${COVERAGE_DIR}
 +    - .coverage-reports
  tests-debian-9:
    image: buildstream/testsuite-debian:9-5da27168-32c47d1c
@@ -83,7 +78,6 @@ tests-unix:
    <<: *tests
    variables:
      BST_FORCE_BACKEND: "unix"
 -    COVERAGE_DIR: coverage-unix
    script:
@@ -239,22 +233,22 @@ coverage:
    stage: post
    coverage: '/TOTAL +\d+ +\d+ +(\d+\.\d+)%/'
    script:
 -    - pip3 install -r requirements/requirements.txt -r requirements/dev-requirements.txt
 -    - pip3 install --no-index .
 -    - mkdir report
 -    - cd report
 -    - cp ../coverage-unix/coverage.* .
 -    - cp ../coverage-linux/coverage.* .
 -    - ls coverage.*
 -    - coverage combine --rcfile=../.coveragerc -a coverage.*
 -    - coverage report --rcfile=../.coveragerc -m
 +    - cp -a .coverage-reports/ ./coverage-sources
 +    - tox -e coverage
 +    - cp -a .coverage-reports/ ./coverage-report
    dependencies:
    - tests-debian-9
    - tests-fedora-27
    - tests-fedora-28
 +  - tests-fedora-missing-deps
 +  - tests-ubuntu-18.04
    - tests-unix
    except:
    - schedules
 +  artifacts:
 +    paths:
 +    - coverage-sources/
 +    - coverage-report/
  # Deploy, only for merges which land on master branch.
+ #

CONTRIBUTING.rst

@@ -553,7 +553,7 @@ One problem which arises from this is that we end up having symbols
  which are *public* according to the :ref:`rules discussed in the previous section
  <contributing_public_and_private>`, but must be hidden away from the
  *"Public API Surface"*. For example, BuildStream internal classes need
 -to invoke methods on the ``Element`` and ``Source`` classes, wheras these
 +to invoke methods on the ``Element`` and ``Source`` classes, whereas these
  methods need to be hidden from the *"Public API Surface"*.
  This is where BuildStream deviates from the PEP-8 standard for public
@@ -631,7 +631,7 @@ An element plugin will derive from Element by importing::
    from buildstream import Element
 -When importing utilities specifically, dont import function names
 +When importing utilities specifically, don't import function names
  from there, instead import the module itself::
    from . import utils
@@ -737,7 +737,7 @@ Abstract methods
  ~~~~~~~~~~~~~~~~
  In BuildStream, an *"Abstract Method"* is a bit of a misnomer and does
  not match up to how Python defines abstract methods, we need to seek out
 -a new nomanclature to refer to these methods.
 +a new nomenclature to refer to these methods.
  In Python, an *"Abstract Method"* is a method which **must** be
  implemented by a subclass, whereas all methods in Python can be
@@ -960,7 +960,7 @@ possible, and avoid any cyclic relationships in modules.
  For instance, the ``Source`` objects are owned by ``Element``
  objects in the BuildStream data model, and as such the ``Element``
  will delegate some activities to the ``Source`` objects in its
 -possesion. The ``Source`` objects should however never call functions
 +possession. The ``Source`` objects should however never call functions
  on the ``Element`` object, nor should the ``Source`` object itself
  have any understanding of what an ``Element`` is.
@@ -1223,7 +1223,7 @@ For further information about using the reStructuredText with sphinx, please see
  Building Docs
  ~~~~~~~~~~~~~
  Before you can build the docs, you will end to ensure that you have installed
 -the required :ref:`buid dependencies <contributing_build_deps>` as mentioned
 +the required :ref:`build dependencies <contributing_build_deps>` as mentioned
  in the testing section above.
  To build the documentation, just run the following::
@@ -1365,7 +1365,7 @@ Structure of an example
  '''''''''''''''''''''''
  The :ref:`tutorial <tutorial>` and the :ref:`examples <examples>` sections
  of the documentation contain a series of sample projects, each chapter in
 -the tutoral, or standalone example uses a sample project.
 +the tutorial, or standalone example uses a sample project.
  Here is the the structure for adding new examples and tutorial chapters.
@@ -1471,8 +1471,8 @@ Installing build dependencies
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Some of BuildStream's dependencies have non-python build dependencies. When
  running tests with ``tox``, you will first need to install these dependencies.
 -Exact steps to install these will depend on your oprtation systemm. Commands
 -for installing them for some common distributions are lised below.
 +Exact steps to install these will depend on your operating system. Commands
 +for installing them for some common distributions are listed below.
  For Fedora-based systems::
@@ -1498,6 +1498,13 @@ option when running tox::
    tox -e py37
 +If you would like to test and lint at the same time, or if you do have multiple
 +python versions installed and would like to test against multiple versions, then
 +we recommend using `detox <https://github.com/tox-dev/detox>`_, just run it with
 +the same arguments you would give `tox`::
++
 +  detox -e lint,py36,py37
++
  Linting is performed separately from testing. In order to run the linting step which
  consists of running the ``pycodestyle`` and ``pylint`` tools, run the following::
@@ -1533,7 +1540,7 @@ the frontend tests you can do::
    tox -- tests/frontend/
 -Specific tests can be chosen by using the :: delimeter after the test module.
 +Specific tests can be chosen by using the :: delimiter after the test module.
  If you wanted to run the test_build_track test within frontend/buildtrack.py you could do::
    tox -- tests/frontend/buildtrack.py::test_build_track
@@ -1553,7 +1560,7 @@ can run ``tox`` with ``-r`` or  ``--recreate`` option.
  .. note::
     By default, we do not allow use of site packages in our ``tox``
 -   confguration to enable running the tests in an isolated environment.
 +   configuration to enable running the tests in an isolated environment.
     If you need to enable use of site packages for whatever reason, you can
     do so by passing the ``--sitepackages`` option to ``tox``. Also, you will
     not need to install any of the build dependencies mentioned above if you
@@ -1574,10 +1581,23 @@ can run ``tox`` with ``-r`` or  ``--recreate`` option.
       ./setup.py test --addopts 'tests/frontend/buildtrack.py::test_build_track'
 +Observing coverage
 +~~~~~~~~~~~~~~~~~~
 +Once you have run the tests using `tox` (or `detox`), some coverage reports will
 +have been left behind.
++
 +To view the coverage report of the last test run, simply run::
++
 +  tox -e coverage
++
 +This will collate any reports from separate python environments that may be
 +under test before displaying the combined coverage.
++
++
  Adding tests
  ~~~~~~~~~~~~
  Tests are found in the tests subdirectory, inside of which
 -there is a separarate directory for each *domain* of tests.
 +there is a separate directory for each *domain* of tests.
  All tests are collected as::
    tests/*/*.py

buildstream/_artifactcache/artifactcache.py

@@ -74,6 +74,7 @@ class ArtifactCache():
          self._has_fetch_remotes = False
          self._has_push_remotes = False
 +        self._has_partial_push_remotes = False
          os.makedirs(self.extractdir, exist_ok=True)
@@ -398,6 +399,9 @@ class ArtifactCache():
                  self._has_fetch_remotes = True
                  if remote_spec.push:
                      self._has_push_remotes = True
 +                    # Partial push requires generic push option to also be set
 +                    if remote_spec.partial_push:
 +                        self._has_partial_push_remotes = True
                  remotes[remote_spec.url] = CASRemote(remote_spec)
@@ -596,6 +600,32 @@ class ArtifactCache():
              remotes_for_project = self._remotes[element._get_project()]
              return any(remote.spec.push for remote in remotes_for_project)
 +    # has_partial_push_remotes():
 +    #
 +    # Check whether any remote repositories are available for pushing
 +    # non-complete artifacts. This option requires the generic push value
 +    # to also be set.
 +    #
 +    # Args:
 +    #     element (Element): The Element to check
 +    #
 +    # Returns:
 +    #   (bool): True if any remote repository is configured for optional
 +    #            partial pushes, False otherwise
 +    #
 +    def has_partial_push_remotes(self, *, element=None):
 +        # If there's no partial push remotes available, we can't partial push at all
 +        if not self._has_partial_push_remotes:
 +            return False
 +        elif element is None:
 +            # At least one remote is set to allow partial pushes
 +            return True
 +        else:
 +            # Check whether the specified element's project has push remotes configured
 +            # to not accept partial artifact pushes
 +            remotes_for_project = self._remotes[element._get_project()]
 +            return any(remote.spec.partial_push for remote in remotes_for_project)
++
      # push():
+     #
      # Push committed artifact to remote repository.
@@ -603,6 +633,8 @@ class ArtifactCache():
      # Args:
      #     element (Element): The Element whose artifact is to be pushed
      #     keys (list): The cache keys to use
 +    #     partial(bool): If the artifact is cached in a partial state
 +    #     subdir(string): Optional subdir to not push
+     #
      # Returns:
      #   (bool): True if any remote was updated, False if no pushes were required
@@ -610,12 +642,25 @@ class ArtifactCache():
      # Raises:
      #   (ArtifactError): if there was an error
+     #
 -    def push(self, element, keys):
 +    def push(self, element, keys, partial=False, subdir=None):
          refs = [self.get_artifact_fullname(element, key) for key in list(keys)]
          project = element._get_project()
 -        push_remotes = [r for r in self._remotes[project] if r.spec.push]
 +        push_remotes = []
 +        partial_remotes = []
++
 +        # Create list of remotes to push to, given current element and partial push config
 +        if not partial:
 +            push_remotes = [r for r in self._remotes[project] if (r.spec.push and not r.spec.partial_push)]
++
 +        if self._has_partial_push_remotes:
 +            # Create a specific list of the remotes expecting the artifact to be push in a partial
 +            # state. This list needs to be pushed in a partial state, without the optional subdir if
 +            # exists locally. No need to attempt pushing a partial artifact to a remote that is queued to
 +            # to also recieve a full artifact
 +            partial_remotes = [r for r in self._remotes[project] if (r.spec.partial_push and r.spec.push) and
 +                               r not in push_remotes]
          pushed = False
@@ -624,7 +669,9 @@ class ArtifactCache():
              display_key = element._get_brief_display_key()
              element.status("Pushing artifact {} -> {}".format(display_key, remote.spec.url))
 -            if self.cas.push(refs, remote):
 +            # Passing the optional subdir allows for remote artifacts that are cached in a 'partial'
 +            # state to be completed
 +            if self.cas.push(refs, remote, subdir=subdir):
                  element.info("Pushed artifact {} -> {}".format(display_key, remote.spec.url))
                  pushed = True
              else:
@@ -632,6 +679,19 @@ class ArtifactCache():
                      remote.spec.url, element._get_brief_display_key()
                  ))
 +        for remote in partial_remotes:
 +            remote.init()
 +            display_key = element._get_brief_display_key()
 +            element.status("Pushing partial artifact {} -> {}".format(display_key, remote.spec.url))
++
 +            if self.cas.push(refs, remote, excluded_subdirs=subdir):
 +                element.info("Pushed partial artifact {} -> {}".format(display_key, remote.spec.url))
 +                pushed = True
 +            else:
 +                element.info("Remote ({}) already has {} partial cached".format(
 +                    remote.spec.url, element._get_brief_display_key()
 +                ))
++
          return pushed
      # pull():
@@ -659,14 +719,23 @@ class ArtifactCache():
                  element.status("Pulling artifact {} <- {}".format(display_key, remote.spec.url))
                  if self.cas.pull(ref, remote, progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs):
 -                    element.info("Pulled artifact {} <- {}".format(display_key, remote.spec.url))
                      if subdir:
 -                        # Attempt to extract subdir into artifact extract dir if it already exists
 -                        # without containing the subdir. If the respective artifact extract dir does not
 -                        # exist a complete extraction will complete.
 -                        self.extract(element, key, subdir)
 -                    # no need to pull from additional remotes
 -                    return True
 +                        if not self.contains_subdir_artifact(element, key, subdir):
 +                            # The pull was expecting the specific subdir to be present in the remote, attempt
 +                            # to find it in other available remotes
 +                            element.info("Pulled partial artifact {} <- {}. Attempting to retrieve {} from remotes"
 +                                         .format(display_key, remote.spec.url, subdir))
 +                        else:
 +                            element.info("Pulled artifact {} <- {}".format(display_key, remote.spec.url))
 +                            # Attempt to extract subdir into artifact extract dir if it already exists
 +                            # without containing the subdir. If the respective artifact extract dir does not
 +                            # exist a complete extraction will complete.
 +                            self.extract(element, key, subdir)
 +                            # no need to pull from additional remotes
 +                            return True
 +                    else:
 +                        element.info("Pulled artifact {} <- {}".format(display_key, remote.spec.url))
 +                        return True
                  else:
                      element.info("Remote ({}) does not have {} cached".format(
                          remote.spec.url, element._get_brief_display_key()

buildstream/_artifactcache/cascache.py

@@ -45,7 +45,8 @@ from .. import _yaml
  _MAX_PAYLOAD_BYTES = 1024 * 1024
 -class CASRemoteSpec(namedtuple('CASRemoteSpec', 'url push server_cert client_key client_cert instance_name')):
 +class CASRemoteSpec(namedtuple('CASRemoteSpec',
 +                               'url push partial_push server_cert client_key client_cert instance_name')):
      # _new_from_config_node
+     #
@@ -53,15 +54,24 @@ class CASRemoteSpec(namedtuple('CASRemoteSpec', 'url push server_cert client_key
+     #
      @staticmethod
      def _new_from_config_node(spec_node, basedir=None):
 -        _yaml.node_validate(spec_node, ['url', 'push', 'server-cert', 'client-key', 'client-cert', 'instance_name'])
 +        _yaml.node_validate(spec_node,
 +                            ['url', 'push', 'allow-partial-push', 'server-cert', 'client-key',
 +                             'client-cert', 'instance_name'])
          url = _yaml.node_get(spec_node, str, 'url')
          push = _yaml.node_get(spec_node, bool, 'push', default_value=False)
 +        partial_push = _yaml.node_get(spec_node, bool, 'allow-partial-push', default_value=False)
++
 +        # partial_push depends on push, raise error if not configured correctly
 +        if partial_push and not push:
 +            provenance = _yaml.node_get_provenance(spec_node, 'allow-partial-push')
 +            raise LoadError(LoadErrorReason.INVALID_DATA,
 +                            "{}: allow-partial-push also requires push to be set".format(provenance))
          if not url:
              provenance = _yaml.node_get_provenance(spec_node, 'url')
              raise LoadError(LoadErrorReason.INVALID_DATA,
                              "{}: empty artifact cache URL".format(provenance))
 -        instance_name = _yaml.node_get(spec_node, str, 'instance_name', default_value=None)
 +        instance_name = _yaml.node_get(spec_node, str, 'instance-name', default_value=None)
          server_cert = _yaml.node_get(spec_node, str, 'server-cert', default_value=None)
          if server_cert and basedir:
@@ -85,10 +95,10 @@ class CASRemoteSpec(namedtuple('CASRemoteSpec', 'url push server_cert client_key
              raise LoadError(LoadErrorReason.INVALID_DATA,
                              "{}: 'client-cert' was specified without 'client-key'".format(provenance))
 -        return CASRemoteSpec(url, push, server_cert, client_key, client_cert, instance_name)
 +        return CASRemoteSpec(url, push, partial_push, server_cert, client_key, client_cert, instance_name)
 -CASRemoteSpec.__new__.__defaults__ = (None, None, None, None)
 +CASRemoteSpec.__new__.__defaults__ = (False, None, None, None, None)
  class BlobNotFound(CASError):
@@ -283,34 +293,47 @@ class CASCache():
      #   (bool): True if pull was successful, False if ref was not available
+     #
      def pull(self, ref, remote, *, progress=None, subdir=None, excluded_subdirs=None):
 -        try:
 -            remote.init()
 -            request = buildstream_pb2.GetReferenceRequest(instance_name=remote.spec.instance_name)
 -            request.key = ref
 -            response = remote.ref_storage.GetReference(request)
 +        tree_found = False
 -            tree = remote_execution_pb2.Digest()
 -            tree.hash = response.digest.hash
 -            tree.size_bytes = response.digest.size_bytes
 +        while True:
 +            try:
 +                if not tree_found:
 +                    remote.init()
 -            # Check if the element artifact is present, if so just fetch the subdir.
 -            if subdir and os.path.exists(self.objpath(tree)):
 -                self._fetch_subdir(remote, tree, subdir)
 -            else:
 -                # Fetch artifact, excluded_subdirs determined in pullqueue
 -                self._fetch_directory(remote, tree, excluded_subdirs=excluded_subdirs)
 +                    request = buildstream_pb2.GetReferenceRequest(instance_name=remote.spec.instance_name)
 +                    request.key = ref
 +                    response = remote.ref_storage.GetReference(request)
 -            self.set_ref(ref, tree)
 +                    tree = remote_execution_pb2.Digest()
 +                    tree.hash = response.digest.hash
 +                    tree.size_bytes = response.digest.size_bytes
 -            return True
 -        except grpc.RpcError as e:
 -            if e.code() != grpc.StatusCode.NOT_FOUND:
 -                raise CASError("Failed to pull ref {}: {}".format(ref, e)) from e
 -            else:
 -                return False
 -        except BlobNotFound as e:
 -            return False
 +                # Check if the element artifact is present, if so just fetch the subdir.
 +                if subdir and os.path.exists(self.objpath(tree)):
 +                    self._fetch_subdir(remote, tree, subdir)
 +                else:
 +                    # Fetch artifact, excluded_subdirs determined in pullqueue
 +                    self._fetch_directory(remote, tree, excluded_subdirs=excluded_subdirs)
++
 +                self.set_ref(ref, tree)
++
 +                return True
 +            except grpc.RpcError as e:
 +                if e.code() != grpc.StatusCode.NOT_FOUND:
 +                    raise CASError("Failed to pull ref {}: {}".format(ref, e)) from e
 +                else:
 +                    return False
 +            except BlobNotFound as e:
 +                if not excluded_subdirs and subdir:
 +                    # The remote has the top level digest but could not complete a full pull,
 +                    # attempt partial without the need to initialise and check for the artifact
 +                    # digest. This default behaviour of dropping back to partial pulls could
 +                    # be made a configurable warning given at artfictcache level.
 +                    tree_found = True
 +                    excluded_subdirs, subdir = subdir, excluded_subdirs
 +                else:
 +                    return False
      # pull_tree():
+     #
@@ -355,6 +378,8 @@ class CASCache():
      # Args:
      #     refs (list): The refs to push
      #     remote (CASRemote): The remote to push to
 +    #     subdir (string): Optional specific subdir to include in the push
 +    #     excluded_subdirs (list): The optional list of subdirs to not push
+     #
      # Returns:
      #   (bool): True if any remote was updated, False if no pushes were required
@@ -362,7 +387,7 @@ class CASCache():
      # Raises:
      #   (CASError): if there was an error
+     #
 -    def push(self, refs, remote):
 +    def push(self, refs, remote, *, subdir=None, excluded_subdirs=None):
          skipped_remote = True
          try:
              for ref in refs:
@@ -376,15 +401,18 @@ class CASCache():
                      response = remote.ref_storage.GetReference(request)
                      if response.digest.hash == tree.hash and response.digest.size_bytes == tree.size_bytes:
 -                        # ref is already on the server with the same tree
 -                        continue
 +                        # ref is already on the server with the same tree, however it might be partially cached.
 +                        # If artifact is not set to be pushed partially attempt to 'complete' the remote artifact if
 +                        # needed, else continue.
 +                        if excluded_subdirs or self.verify_digest_on_remote(remote, self._get_subdir(tree, subdir)):
 +                            continue
                  except grpc.RpcError as e:
                      if e.code() != grpc.StatusCode.NOT_FOUND:
                          # Intentionally re-raise RpcError for outer except block.
                          raise
 -                self._send_directory(remote, tree)
 +                self._send_directory(remote, tree, excluded_dir=excluded_subdirs)
                  request = buildstream_pb2.UpdateReferenceRequest(instance_name=remote.spec.instance_name)
                  request.keys.append(ref)
@@ -866,10 +894,17 @@ class CASCache():
                  a += 1
                  b += 1
 -    def _reachable_refs_dir(self, reachable, tree, update_mtime=False):
 +    def _reachable_refs_dir(self, reachable, tree, update_mtime=False, subdir=False):
          if tree.hash in reachable:
              return
 +        # If looping through subdir digests, skip processing if
 +        # ref path does not exist, allowing for partial objects
 +        if subdir and not os.path.exists(self.objpath(tree)):
 +            return
++
 +        # Raises FileNotFound exception is path does not exist,
 +        # which should only be thrown on the top level digest
          if update_mtime:
              os.utime(self.objpath(tree))
@@ -886,9 +921,9 @@ class CASCache():
              reachable.add(filenode.digest.hash)
          for dirnode in directory.directories:
 -            self._reachable_refs_dir(reachable, dirnode.digest, update_mtime=update_mtime)
 +            self._reachable_refs_dir(reachable, dirnode.digest, update_mtime=update_mtime, subdir=True)
 -    def _required_blobs(self, directory_digest):
 +    def _required_blobs(self, directory_digest, excluded_dir=None):
          # parse directory, and recursively add blobs
          d = remote_execution_pb2.Digest()
          d.hash = directory_digest.hash
@@ -907,7 +942,8 @@ class CASCache():
              yield d
          for dirnode in directory.directories:
 -            yield from self._required_blobs(dirnode.digest)
 +            if dirnode.name != excluded_dir:
 +                yield from self._required_blobs(dirnode.digest)
      def _fetch_blob(self, remote, digest, stream):
          resource_name_components = ['blobs', digest.hash, str(digest.size_bytes)]
@@ -1029,6 +1065,7 @@ class CASCache():
              objpath = self._ensure_blob(remote, dir_digest)
              directory = remote_execution_pb2.Directory()
++
              with open(objpath, 'rb') as f:
                  directory.ParseFromString(f.read())
@@ -1104,9 +1141,8 @@ class CASCache():
          assert response.committed_size == digest.size_bytes
 -    def _send_directory(self, remote, digest, u_uid=uuid.uuid4()):
 -        required_blobs = self._required_blobs(digest)
+-
 +    def _send_directory(self, remote, digest, u_uid=uuid.uuid4(), excluded_dir=None):
 +        required_blobs = self._required_blobs(digest, excluded_dir=excluded_dir)
          missing_blobs = dict()
          # Limit size of FindMissingBlobs request
          for required_blobs_group in _grouper(required_blobs, 512):

buildstream/_context.py

@@ -34,6 +34,7 @@ from ._artifactcache import ArtifactCache
  from ._artifactcache.cascache import CASCache
  from ._workspaces import Workspaces, WorkspaceProjectCache, WORKSPACE_PROJECT_FILE
  from .plugin import _plugin_lookup
 +from .sandbox import SandboxRemote
  # Context()
@@ -72,6 +73,9 @@ class Context():
          # The locations from which to push and pull prebuilt artifacts
          self.artifact_cache_specs = None
 +        # The global remote execution configuration
 +        self.remote_execution_specs = None
++
          # The directory to store build logs
          self.logdir = None
@@ -187,7 +191,7 @@ class Context():
          _yaml.node_validate(defaults, [
              'sourcedir', 'builddir', 'artifactdir', 'logdir',
              'scheduler', 'artifacts', 'logging', 'projects',
 -            'cache', 'prompt', 'workspacedir',
 +            'cache', 'prompt', 'workspacedir', 'remote-execution'
          ])
          for directory in ['sourcedir', 'builddir', 'artifactdir', 'logdir', 'workspacedir']:
@@ -212,6 +216,8 @@ class Context():
          # Load artifact share configuration
          self.artifact_cache_specs = ArtifactCache.specs_from_config_node(defaults)
 +        self.remote_execution_specs = SandboxRemote.specs_from_config_node(defaults)
++
          # Load pull build trees configuration
          self.pull_buildtrees = _yaml.node_get(cache, bool, 'pull-buildtrees')
@@ -271,7 +277,8 @@ class Context():
          # Shallow validation of overrides, parts of buildstream which rely
          # on the overrides are expected to validate elsewhere.
          for _, overrides in _yaml.node_items(self._project_overrides):
 -            _yaml.node_validate(overrides, ['artifacts', 'options', 'strict', 'default-mirror'])
 +            _yaml.node_validate(overrides, ['artifacts', 'options', 'strict', 'default-mirror',
 +                                            'remote-execution'])
          profile_end(Topics.LOAD_CONTEXT, 'load')

buildstream/_options/optionarch.py

@@ -17,6 +17,8 @@
  #  Authors:
  #        Tristan Van Berkom <tristan vanberkom codethink co uk>
 +from .. import _yaml
 +from .._exceptions import LoadError, LoadErrorReason, PlatformError
  from .._platform import Platform
  from .optionenum import OptionEnum
@@ -41,7 +43,34 @@ class OptionArch(OptionEnum):
          super(OptionArch, self).load(node, allow_default_definition=False)
      def load_default_value(self, node):
 -        return Platform.get_host_arch()
 +        arch = Platform.get_host_arch()
++
 +        default_value = None
++
 +        for index, value in enumerate(self.values):
 +            try:
 +                canonical_value = Platform.canonicalize_arch(value)
 +                if default_value is None and canonical_value == arch:
 +                    default_value = value
 +                    # Do not terminate the loop early to ensure we validate
 +                    # all values in the list.
 +            except PlatformError as e:
 +                provenance = _yaml.node_get_provenance(node, key='values', indices=[index])
 +                prefix = ""
 +                if provenance:
 +                    prefix = "{}: ".format(provenance)
 +                raise LoadError(LoadErrorReason.INVALID_DATA,
 +                                "{}Invalid value for {} option '{}': {}"
 +                                .format(prefix, self.OPTION_TYPE, self.name, e))
++
 +        if default_value is None:
 +            # Host architecture is not supported by the project.
 +            # Do not raise an error here as the user may override it.
 +            # If the user does not override it, an error will be raised
 +            # by resolve()/validate().
 +            default_value = arch
++
 +        return default_value
      def resolve(self):

buildstream/_pipeline.py

@@ -22,6 +22,7 @@
  import os
  import itertools
  from operator import itemgetter
 +from collections import OrderedDict
  from ._exceptions import PipelineError
  from ._message import Message, MessageType
@@ -479,7 +480,7 @@ class Pipeline():
+ #
  class _Planner():
      def __init__(self):
 -        self.depth_map = {}
 +        self.depth_map = OrderedDict()
          self.visiting_elements = set()
      # Here we want to traverse the same element more than once when

buildstream/_platform/platform.py

@@ -77,20 +77,17 @@ class Platform():
      def get_host_os():
          return os.uname()[0]
 -    # get_host_arch():
 +    # canonicalize_arch():
+     #
 -    # This returns the architecture of the host machine. The possible values
 -    # map from uname -m in order to be a OS independent list.
 +    # This returns the canonical, OS-independent architecture name
 +    # or raises a PlatformError if the architecture is unknown.
+     #
 -    # Returns:
 -    #    (string): String representing the architecture
      @staticmethod
 -    def get_host_arch():
 -        # get the hardware identifier from uname
 -        uname_machine = os.uname()[4]
 -        uname_to_arch = {
 +    def canonicalize_arch(arch):
 +        aliases = {
 +            "aarch32": "aarch32",
              "aarch64": "aarch64",
 -            "aarch64_be": "aarch64-be",
 +            "aarch64-be": "aarch64-be",
              "amd64": "x86-64",
              "arm": "aarch32",
              "armv8l": "aarch64",
@@ -99,17 +96,34 @@ class Platform():
              "i486": "x86-32",
              "i586": "x86-32",
              "i686": "x86-32",
 +            "power-isa-be": "power-isa-be",
 +            "power-isa-le": "power-isa-le",
              "ppc64": "power-isa-be",
              "ppc64le": "power-isa-le",
              "sparc": "sparc-v9",
              "sparc64": "sparc-v9",
 -            "x86_64": "x86-64"
 +            "sparc-v9": "sparc-v9",
 +            "x86-32": "x86-32",
 +            "x86-64": "x86-64"
+         }
++
          try:
 -            return uname_to_arch[uname_machine]
 +            return aliases[arch.replace('_', '-')]
          except KeyError:
 -            raise PlatformError("uname gave unsupported machine architecture: {}"
 -                                .format(uname_machine))
 +            raise PlatformError("Unknown architecture: {}".format(arch))
++
 +    # get_host_arch():
 +    #
 +    # This returns the architecture of the host machine. The possible values
 +    # map from uname -m in order to be a OS independent list.
 +    #
 +    # Returns:
 +    #    (string): String representing the architecture
 +    @staticmethod
 +    def get_host_arch():
 +        # get the hardware identifier from uname
 +        uname_machine = os.uname()[4]
 +        return Platform.canonicalize_arch(uname_machine)
      ##################################################################
      #                        Sandbox functions                       #

buildstream/_project.py

@@ -507,7 +507,16 @@ class Project():
          self.artifact_cache_specs = ArtifactCache.specs_from_config_node(config, self.directory)
          # Load remote-execution configuration for this project
 -        self.remote_execution_specs = SandboxRemote.specs_from_config_node(config, self.directory)
 +        project_specs = SandboxRemote.specs_from_config_node(config, self.directory)
 +        override_specs = SandboxRemote.specs_from_config_node(
 +            self._context.get_overrides(self.name), self.directory)
++
 +        if override_specs is not None:
 +            self.remote_execution_specs = override_specs
 +        elif project_specs is not None:
 +            self.remote_execution_specs = project_specs
 +        else:
 +            self.remote_execution_specs = self._context.remote_execution_specs
          # Load sandbox environment variables
          self.base_environment = _yaml.node_get(config, Mapping, 'environment')

buildstream/element.py

@@ -1800,13 +1800,19 @@ class Element(Plugin):
      #   (bool): True if this element does not need a push job to be created
+     #
      def _skip_push(self):
++
          if not self.__artifacts.has_push_remotes(element=self):
              # No push remotes for this element's project
              return True
          # Do not push elements that aren't cached, or that are cached with a dangling buildtree
 -        # artifact unless element type is expected to have an an empty buildtree directory
 -        if not self._cached_buildtree():
 +        # artifact unless element type is expected to have an an empty buildtree directory. Check
 +        # that this default behaviour is not overriden via a remote configured to allow pushing
 +        # artifacts without their corresponding buildtree.
 +        if not self._cached():
 +            return True
++
 +        if not self._cached_buildtree() and not self.__artifacts.has_partial_push_remotes(element=self):
              return True
          # Do not push tainted artifact
@@ -1817,11 +1823,14 @@ class Element(Plugin):
      # _push():
+     #
 -    # Push locally cached artifact to remote artifact repository.
 +    # Push locally cached artifact to remote artifact repository. An attempt
 +    # will be made to push partial artifacts if given current config dictates.
 +    # If a remote set for 'full' artifact pushes is found to be cached partially
 +    # in the remote, an attempt will be made to 'complete' it.
+     #
      # Returns:
      #   (bool): True if the remote was updated, False if it already existed
 -    #           and no updated was required
 +    #           and no update was required
+     #
      def _push(self):
          self.__assert_cached()
@@ -1830,8 +1839,17 @@ class Element(Plugin):
              self.warn("Not pushing tainted artifact.")
              return False
 -        # Push all keys used for local commit
 -        pushed = self.__artifacts.push(self, self.__get_cache_keys_for_commit())
 +        # Push all keys used for local commit, this could be full or partial,
 +        # given previous _skip_push() logic. If buildtree isn't cached, then
 +        # set partial push
++
 +        partial = False
 +        subdir = 'buildtree'
 +        if not self._cached_buildtree():
 +            partial = True
++
 +        pushed = self.__artifacts.push(self, self.__get_cache_keys_for_commit(), partial=partial, subdir=subdir)
++
          if not pushed:
              return False
@@ -2441,11 +2459,17 @@ class Element(Plugin):
          # Sandbox config, unlike others, has fixed members so we should validate them
          _yaml.node_validate(sandbox_config, ['build-uid', 'build-gid', 'build-os', 'build-arch'])
 +        build_arch = self.node_get_member(sandbox_config, str, 'build-arch', default=None)
 +        if build_arch:
 +            build_arch = Platform.canonicalize_arch(build_arch)
 +        else:
 +            build_arch = host_arch
++
          return SandboxConfig(
              self.node_get_member(sandbox_config, int, 'build-uid'),
              self.node_get_member(sandbox_config, int, 'build-gid'),
              self.node_get_member(sandbox_config, str, 'build-os', default=host_os),
 -            self.node_get_member(sandbox_config, str, 'build-arch', default=host_arch))
 +            build_arch)
      # This makes a special exception for the split rules, which
      # elements may extend but whos defaults are defined in the project.

buildstream/plugins/elements/script.py

@@ -42,6 +42,9 @@ import buildstream
  class ScriptElement(buildstream.ScriptElement):
      # pylint: disable=attribute-defined-outside-init
 +    # This plugin has been modified to avoid the use of Sandbox.get_directory
 +    BST_VIRTUAL_DIRECTORY = True
++
      def configure(self, node):
          for n in self.node_get_member(node, list, 'layout', []):
              dst = self.node_subst_member(n, 'destination')

buildstream/sandbox/_sandboxremote.py

@@ -62,10 +62,32 @@ class SandboxRemote(Sandbox):
          self.storage_url = config.storage_service['url']
          self.exec_url = config.exec_service['url']
 +        exec_certs = {}
 +        for key in ['client-cert', 'client-key', 'server-cert']:
 +            if key in config.exec_service:
 +                with open(config.exec_service[key], 'rb') as f:
 +                    exec_certs[key] = f.read()
++
 +        self.exec_credentials = grpc.ssl_channel_credentials(
 +            root_certificates=exec_certs.get('server-cert'),
 +            private_key=exec_certs.get('client-key'),
 +            certificate_chain=exec_certs.get('client-cert'))
++
 +        action_certs = {}
 +        for key in ['client-cert', 'client-key', 'server-cert']:
 +            if key in config.action_service:
 +                with open(config.action_service[key], 'rb') as f:
 +                    action_certs[key] = f.read()
++
          if config.action_service:
              self.action_url = config.action_service['url']
 +            self.action_credentials = grpc.ssl_channel_credentials(
 +                root_certificates=action_certs.get('server-cert'),
 +                private_key=action_certs.get('client-key'),
 +                certificate_chain=action_certs.get('client-cert'))
          else:
              self.action_url = None
 +            self.action_credentials = None
          self.server_instance = config.exec_service.get('instance', None)
          self.storage_instance = config.storage_service.get('instance', None)
@@ -81,7 +103,7 @@ class SandboxRemote(Sandbox):
          self._get_context().message(Message(None, MessageType.INFO, msg))
      @staticmethod
 -    def specs_from_config_node(config_node, basedir):
 +    def specs_from_config_node(config_node, basedir=None):
          def require_node(config, keyname):
              val = config.get(keyname)
@@ -109,10 +131,10 @@ class SandboxRemote(Sandbox):
          remote_exec_storage_config = require_node(remote_config, 'storage-service')
          remote_exec_action_config = remote_config.get('action-cache-service', {})
 -        _yaml.node_validate(remote_exec_service_config, ['url', 'instance'])
 +        _yaml.node_validate(remote_exec_service_config, ['url', 'instance'] + tls_keys)
          _yaml.node_validate(remote_exec_storage_config, ['url', 'instance'] + tls_keys)
          if remote_exec_action_config:
 -            _yaml.node_validate(remote_exec_action_config, ['url'])
 +            _yaml.node_validate(remote_exec_action_config, ['url'] + tls_keys)
          else:
              remote_config['action-service'] = None
@@ -135,6 +157,19 @@ class SandboxRemote(Sandbox):
                                        "remote-execution configuration. Your config is missing '{}'."
                                        .format(str(provenance), tls_keys, key))
 +        def resolve_path(path):
 +            if basedir and path:
 +                return os.path.join(basedir, path)
 +            else:
 +                return path
++
 +        for key in tls_keys:
 +            for d in (remote_config['execution-service'],
 +                      remote_config['storage-service'],
 +                      remote_exec_action_config):
 +                if key in d:
 +                    d[key] = resolve_path(d[key])
++
          spec = RemoteExecutionSpec(remote_config['execution-service'],
                                     remote_config['storage-service'],
                                     remote_exec_action_config)
@@ -295,6 +330,8 @@ class SandboxRemote(Sandbox):
                                 "for example: http://buildservice:50051.")
          if url.scheme == 'http':
              channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
 +        elif url.scheme == 'https':
 +            channel = grpc.secure_channel('{}:{}'.format(url.hostname, url.port), self.exec_credentials)
          else:
              raise SandboxError("Remote execution currently only supports the 'http' protocol "
                                 "and '{}' was supplied.".format(url.scheme))
@@ -352,11 +389,11 @@ class SandboxRemote(Sandbox):
          if not url.port:
              raise SandboxError("You must supply a protocol and port number in the action-cache-service url, "
                                 "for example: http://buildservice:50051.")
 -        if not url.scheme == "http":
 -            raise SandboxError("Currently only support http for the action cache"
 -                               "and {} was supplied".format(url.scheme))
 +        if url.scheme == 'http':
 +            channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
 +        elif url.scheme == 'https':
 +            channel = grpc.secure_channel('{}:{}'.format(url.hostname, url.port), self.action_credentials)
 -        channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
          request = remote_execution_pb2.GetActionResultRequest(action_digest=action_digest)
          stub = remote_execution_pb2_grpc.ActionCacheStub(channel)
          try:

buildstream/sandbox/sandbox.py

@@ -288,8 +288,8 @@ class Sandbox():
              command = [command]
          if self.__batch:
 -            if flags != self.__batch.flags:
 -                raise SandboxError("Inconsistent sandbox flags in single command batch")
 +            assert flags == self.__batch.flags, \
 +                "Inconsistent sandbox flags in single command batch"
              batch_command = _SandboxBatchCommand(command, cwd=cwd, env=env, label=label)
@@ -326,8 +326,8 @@ class Sandbox():
          if self.__batch:
              # Nested batch
 -            if flags != self.__batch.flags:
 -                raise SandboxError("Inconsistent sandbox flags in single command batch")
 +            assert flags == self.__batch.flags, \
 +                "Inconsistent sandbox flags in single command batch"
              parent_group = self.__batch.current_group
              parent_group.append(group)

doc/source/format_project.rst

@@ -218,6 +218,7 @@ The use of ports are required to distinguish between pull only access and
  push/pull access. For information regarding the server/client certificates
  and keys, please see: :ref:`Key pair for the server <server_authentication>`.
 +.. _project_remote_execution:
  Remote execution
  ~~~~~~~~~~~~~~~~
@@ -243,9 +244,6 @@ using the `remote-execution` option:
      action-cache-service:
        url: http://bar.action.com:50052
 -The execution-service part of remote execution does not support encrypted
 -connections yet, so the protocol must always be http.
+-
  storage-service specifies a remote CAS store and the parameters are the
  same as those used to specify an :ref:`artifact server <artifacts>`.
@@ -268,6 +266,9 @@ instance names.
  The Remote Execution API can be found via https://github.com/bazelbuild/remote-apis.
 +Remote execution configuration can be also provided in the `user
 +configuration <user_config_remote_execution>`.
++
  .. _project_essentials_mirrors:
  Mirrors

doc/source/using_config.rst

@@ -59,6 +59,15 @@ configuration:
       # Add another cache to pull from
       - url: https://anothercache.com/artifacts:8080
         server-cert: another_server.crt
 +     # Add a cache to push/pull to/from, specifying
 +       that you wish to push artifacts in a 'partial'
 +       state (this being without the respective buildtree).
 +       Note that allow-partial-push requires push to also
 +       be set.
 +     - url: https://anothercache.com/artifacts:11003
 +       push: true
 +       allow-partial-push: true
++
  .. note::
@@ -86,6 +95,14 @@ configuration:
           # Add another cache to pull from
           - url: https://ourprojectcache.com/artifacts:8080
             server-cert: project_server.crt
 +         # Add a cache to push/pull to/from, specifying
 +           that you wish to push artifacts in a 'partial'
 +           state (this being without the respective buildtree).
 +           Note that allow-partial-push requires push to also
 +           be set.
 +         - url: https://anothercache.com/artifacts:11003
 +           push: true
 +           allow-partial-push: true
  .. note::
@@ -100,6 +117,54 @@ pull only access and push/pull access. For information regarding this and the
  server/client certificates and keys, please see:
  :ref:`Key pair for the server <server_authentication>`.
 +.. _user_config_remote_execution:
++
 +Remote execution
 +~~~~~~~~~~~~~~~~
++
 +The same configuration for :ref:`remote execution <project_remote_execution>`
 +in ``project.conf`` can be provided in the user configuation.
++
 +There is only one remote execution configuration used per project.
++
 +The project overrides will be taken in priority. The global
 +configuration will be used as fallback.
++
 +1. Global remote execution fallback:
++
 +.. code:: yaml
++
 +  remote-execution:
 +    execution-service:
 +      url: http://execution.fallback.example.com:50051
 +      instance-name: main
 +    storage-service:
 +      url: https://storage.fallback.example.com:11002/
 +      server-cert: /keys/server.crt
 +      client-cert: /keys/client.crt
 +      client-key: /keys/client.key
 +      instance-name: main
 +    action-cache-service:
 +      url: http://action.flalback.example.com:50052
++
 +2. Project override:
++
 +.. code:: yaml
++
 +  projects:
 +    some_project:
 +      remote-execution:
 +        execution-service:
 +          url: http://execution.some_project.example.com:50051
 +          instance-name: main
 +        storage-service:
 +          url: https://storage.some_project.example.com:11002/
 +          server-cert: /some_project_keys/server.crt
 +          client-cert: /some_project_keys/client.crt
 +          client-key: /some_project_keys/client.key
 +          instance-name: main
 +        action-cache-service:
 +          url: http://action.some_project.example.com:50052
  Strict build plan

tests/artifactcache/config.py

@@ -140,3 +140,28 @@ def test_missing_certs(cli, datafiles, config_key, config_value):
      # This does not happen for a simple `bst show`.
      result = cli.run(project=project, args=['pull', 'element.bst'])
      result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA)
++
++
 +# Assert that if allow-partial-push is specified as true without push also being
 +# set likewise, we get a comprehensive LoadError instead of an unhandled exception.
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_partial_push_error(cli, datafiles):
 +    project = os.path.join(datafiles.dirname, datafiles.basename, 'project', 'elements')
++
 +    project_conf = {
 +        'name': 'test',
++
 +        'artifacts': {
 +            'url': 'https://cache.example.com:12345',
 +            'allow-partial-push': 'True'
 +        }
 +    }
 +    project_conf_file = os.path.join(project, 'project.conf')
 +    _yaml.dump(project_conf, project_conf_file)
++
 +    # Use `pull` here to ensure we try to initialize the remotes, triggering the error
 +    #
 +    # This does not happen for a simple `bst show`.
 +    result = cli.run(project=project, args=['pull', 'target.bst'])
 +    result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA)
 +    assert "allow-partial-push also requires push to be set" in result.stderr

tests/format/option-arch-alias/element.bst

 +kind: autotools
 +variables:
 +  result: "Nothing"
 +  (?):
 +  - machine_arch == "arm":
 +      result: "Army"
 +  - machine_arch == "x86_64":
 +      result: "X86-64y"

tests/format/option-arch-alias/project.conf

 +name: test
++
 +options:
 +  machine_arch:
 +    type: arch
 +    description: The machine architecture
 +    values:
 +    - arm
 +    - x86_64

tests/format/option-arch-unknown/element.bst

 +kind: autotools
 +variables:
 +  result: "Nothing"
 +  (?):
 +  - machine_arch == "aarch32":
 +      result: "Army"
 +  - machine_arch == "aarch64":
 +      result: "Aarchy"
 +  - machine_arch == "x86-128":
 +      result: "X86-128y"

tests/format/option-arch-unknown/project.conf

 +name: test
++
 +options:
 +  machine_arch:
 +    type: arch
 +    description: The machine architecture
 +    values:
 +    - aarch32
 +    - aarch64
 +    - x86-128

tests/format/optionarch.py

@@ -75,3 +75,47 @@ def test_unsupported_arch(cli, datafiles):
          ])
          result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_alias(cli, datafiles):
++
 +    with override_uname_arch("arm"):
 +        project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch-alias')
 +        result = cli.run(project=project, silent=True, args=[
 +            'show',
 +            '--deps', 'none',
 +            '--format', '%{vars}',
 +            'element.bst'
 +        ])
++
 +        result.assert_success()
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_unknown_host_arch(cli, datafiles):
++
 +    with override_uname_arch("x86_128"):
 +        project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch')
 +        result = cli.run(project=project, silent=True, args=[
 +            'show',
 +            '--deps', 'none',
 +            '--format', '%{vars}',
 +            'element.bst'
 +        ])
++
 +        result.assert_main_error(ErrorDomain.PLATFORM, None)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_unknown_project_arch(cli, datafiles):
++
 +    project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch-unknown')
 +    result = cli.run(project=project, silent=True, args=[
 +        'show',
 +        '--deps', 'none',
 +        '--format', '%{vars}',
 +        'element.bst'
 +    ])
++
 +    result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA)

tests/frontend/order.py

 +import os
++
 +import pytest
 +from tests.testutils import cli, create_repo
++
 +from buildstream import _yaml
++
 +# Project directory
 +DATA_DIR = os.path.join(
 +    os.path.dirname(os.path.realpath(__file__)),
 +    "project",
 +)
++
++
 +def create_element(repo, name, path, dependencies, ref=None):
 +    element = {
 +        'kind': 'import',
 +        'sources': [
 +            repo.source_config(ref=ref)
 +        ],
 +        'depends': dependencies
 +    }
 +    _yaml.dump(element, os.path.join(path, name))
++
++
 +# This tests a variety of scenarios and checks that the order in
 +# which things are processed remains stable.
 +#
 +# This is especially important in order to ensure that our
 +# depth sorting and optimization of which elements should be
 +# processed first is doing it's job right, and that we are
 +# promoting elements to the build queue as soon as possible
 +#
 +# Parameters:
 +#    targets (target elements): The targets to invoke bst with
 +#    template (dict): The project template dictionary, for create_element()
 +#    expected (list): A list of element names in the expected order
 +#
 +@pytest.mark.datafiles(os.path.join(DATA_DIR))
 +@pytest.mark.parametrize("target,template,expected", [
 +    # First simple test
 +    ('3.bst', {
 +        '0.bst': ['1.bst'],
 +        '1.bst': [],
 +        '2.bst': ['0.bst'],
 +        '3.bst': ['0.bst', '1.bst', '2.bst']
 +    }, ['1.bst', '0.bst', '2.bst', '3.bst']),
++
 +    # A more complicated test with build of build dependencies
 +    ('target.bst', {
 +        'a.bst': [],
 +        'base.bst': [],
 +        'timezones.bst': [],
 +        'middleware.bst': [{'filename': 'base.bst', 'type': 'build'}],
 +        'app.bst': [{'filename': 'middleware.bst', 'type': 'build'}],
 +        'target.bst': ['a.bst', 'base.bst', 'middleware.bst', 'app.bst', 'timezones.bst']
 +    }, ['base.bst', 'middleware.bst', 'a.bst', 'app.bst', 'timezones.bst', 'target.bst']),
 +])
 +@pytest.mark.parametrize("operation", [('show'), ('fetch'), ('build')])
 +def test_order(cli, datafiles, tmpdir, operation, target, template, expected):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    dev_files_path = os.path.join(project, 'files', 'dev-files')
 +    element_path = os.path.join(project, 'elements')
++
 +    # FIXME: Remove this when the test passes reliably.
 +    #
 +    #        There is no reason why the order should not
 +    #        be preserved when the builders is set to 1,
 +    #        the scheduler queue processing still seems to
 +    #        be losing the order.
 +    #
 +    if operation == 'build':
 +        pytest.skip("FIXME: This still only sometimes passes")
++
 +    # Configure to only allow one fetcher at a time, make it easy to
 +    # determine what is being planned in what order.
 +    cli.configure({
 +        'scheduler': {
 +            'fetchers': 1,
 +            'builders': 1
 +        }
 +    })
++
 +    # Build the project from the template, make import elements
 +    # all with the same repo
 +    #
 +    repo = create_repo('git', str(tmpdir))
 +    ref = repo.create(dev_files_path)
 +    for element, dependencies in template.items():
 +        create_element(repo, element, element_path, dependencies, ref=ref)
 +        repo.add_commit()
++
 +    # Run test and collect results
 +    if operation == 'show':
 +        result = cli.run(args=['show', '--deps', 'plan', '--format', '%{name}', target], project=project, silent=True)
 +        result.assert_success()
 +        results = result.output.splitlines()
 +    else:
 +        if operation == 'fetch':
 +            result = cli.run(args=['source', 'fetch', target], project=project, silent=True)
 +        else:
 +            result = cli.run(args=[operation, target], project=project, silent=True)
 +        result.assert_success()
 +        results = result.get_start_order(operation)
++
 +    # Assert the order
 +    print("Expected order: {}".format(expected))
 +    print("Observed result order: {}".format(results))
 +    assert results == expected

tests/integration/pushbuildtrees.py

 +import os
 +import shutil
 +import pytest
 +import subprocess
++
 +from buildstream import _yaml
 +from tests.testutils import cli_integration as cli, create_artifact_share
 +from tests.testutils.integration import assert_contains
 +from tests.testutils.site import HAVE_BWRAP, IS_LINUX
 +from buildstream._exceptions import ErrorDomain, LoadErrorReason
++
++
 +DATA_DIR = os.path.join(
 +    os.path.dirname(os.path.realpath(__file__)),
 +    "project"
 +)
++
++
 +# Remove artifact cache & set cli.config value of pull-buildtrees
 +# to false, which is the default user context. The cache has to be
 +# cleared as just forcefully removing the refpath leaves dangling objects.
 +def default_state(cli, tmpdir, share):
 +    shutil.rmtree(os.path.join(str(tmpdir), 'artifacts'))
 +    cli.configure({
 +        'artifacts': {'url': share.repo, 'push': False},
 +        'artifactdir': os.path.join(str(tmpdir), 'artifacts'),
 +        'cache': {'pull-buildtrees': False},
 +    })
++
++
 +# Tests to capture the integration of the optionl push of buildtrees.
 +# The behaviour should encompass pushing artifacts that are already cached
 +# without a buildtree as well as artifacts that are cached with their buildtree.
 +# This option is handled via 'allow-partial-push' on a per artifact remote config
 +# node basis. Multiple remote config nodes can point to the same url and as such can
 +# have different 'allow-partial-push' options, tests need to cover this using project
 +# confs.
 +@pytest.mark.integration
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.skipif(IS_LINUX and not HAVE_BWRAP, reason='Only available with bubblewrap on Linux')
 +def test_pushbuildtrees(cli, tmpdir, datafiles, integration_cache):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    element_name = 'autotools/amhello.bst'
++
 +    # Create artifact shares for pull & push testing
 +    with create_artifact_share(os.path.join(str(tmpdir), 'share1')) as share1,\
 +        create_artifact_share(os.path.join(str(tmpdir), 'share2')) as share2,\
 +        create_artifact_share(os.path.join(str(tmpdir), 'share3')) as share3,\
 +        create_artifact_share(os.path.join(str(tmpdir), 'share4')) as share4:
++
 +        cli.configure({
 +            'artifacts': {'url': share1.repo, 'push': True},
 +            'artifactdir': os.path.join(str(tmpdir), 'artifacts')
 +        })
++
 +        cli.configure({'artifacts': [{'url': share1.repo, 'push': True},
 +                                     {'url': share2.repo, 'push': True, 'allow-partial-push': True}]})
++
 +        # Build autotools element, checked pushed, delete local.
 +        # As share 2 has push & allow-partial-push set a true, it
 +        # should have pushed the artifacts, without the cached buildtrees,
 +        # to it.
 +        result = cli.run(project=project, args=['build', element_name])
 +        assert result.exit_code == 0
 +        assert cli.get_element_state(project, element_name) == 'cached'
 +        elementdigest = share1.has_artifact('test', element_name, cli.get_element_key(project, element_name))
 +        buildtreedir = os.path.join(str(tmpdir), 'artifacts', 'extract', 'test', 'autotools-amhello',
 +                                    elementdigest.hash, 'buildtree')
 +        assert os.path.isdir(buildtreedir)
 +        assert element_name in result.get_partial_pushed_elements()
 +        assert element_name in result.get_pushed_elements()
 +        assert share1.has_artifact('test', element_name, cli.get_element_key(project, element_name))
 +        assert share2.has_artifact('test', element_name, cli.get_element_key(project, element_name))
 +        default_state(cli, tmpdir, share1)
++
 +        # Check that after explictly pulling an artifact without it's buildtree,
 +        # we can push it to another remote that is configured to accept the partial
 +        # artifact
 +        result = cli.run(project=project, args=['pull', element_name])
 +        assert element_name in result.get_pulled_elements()
 +        cli.configure({'artifacts': {'url': share3.repo, 'push': True, 'allow-partial-push': True}})
 +        assert cli.get_element_state(project, element_name) == 'cached'
 +        assert not os.path.isdir(buildtreedir)
 +        result = cli.run(project=project, args=['push', element_name])
 +        assert result.exit_code == 0
 +        assert element_name in result.get_partial_pushed_elements()
 +        assert element_name not in result.get_pushed_elements()
 +        assert share3.has_artifact('test', element_name, cli.get_element_key(project, element_name))
 +        default_state(cli, tmpdir, share3)
++
 +        # Delete the local cache and pull the partial artifact from share 3,
 +        # this should not include the buildtree when extracted locally, even when
 +        # pull-buildtrees is given as a cli parameter as no available remotes will
 +        # contain the buildtree
 +        assert not os.path.isdir(buildtreedir)
 +        assert cli.get_element_state(project, element_name) != 'cached'
 +        result = cli.run(project=project, args=['--pull-buildtrees', 'pull', element_name])
 +        assert element_name in result.get_partial_pulled_elements()
 +        assert not os.path.isdir(buildtreedir)
 +        default_state(cli, tmpdir, share3)
++
 +        # Delete the local cache and attempt to pull a 'full' artifact, including its
 +        # buildtree. As with before share3 being the first listed remote will not have
 +        # the buildtree available and should spawn a partial pull. Having share1 as the
 +        # second available remote should allow the buildtree to be pulled thus 'completing'
 +        # the artifact
 +        cli.configure({'artifacts': [{'url': share3.repo, 'push': True, 'allow-partial-push': True},
 +                                     {'url': share1.repo, 'push': True}]})
 +        assert cli.get_element_state(project, element_name) != 'cached'
 +        result = cli.run(project=project, args=['--pull-buildtrees', 'pull', element_name])
 +        assert element_name in result.get_partial_pulled_elements()
 +        assert element_name in result.get_pulled_elements()
 +        assert "Attempting to retrieve buildtree from remotes" in result.stderr
 +        assert os.path.isdir(buildtreedir)
 +        assert cli.get_element_state(project, element_name) == 'cached'
++
 +        # Test that we are able to 'complete' an artifact on a server which is cached partially,
 +        # but has now been configured for full artifact pushing. This should require only pushing
 +        # the missing blobs, which should be those of just the buildtree. In this case changing
 +        # share3 to full pushes should exercise this
 +        cli.configure({'artifacts': {'url': share3.repo, 'push': True}})
 +        result = cli.run(project=project, args=['push', element_name])
 +        assert element_name in result.get_pushed_elements()
++
 +        # Ensure that the same remote url can be defined multiple times with differing push
 +        # config. Buildstream supports the same remote having different configurations which
 +        # partial pushing could be different for elements defined at a top level project.conf to
 +        # those from a junctioned project. Assert that elements are pushed to the same remote in
 +        # a state defined via their respective project.confs
 +        default_state(cli, tmpdir, share1)
 +        cli.configure({'artifactdir': os.path.join(str(tmpdir), 'artifacts')}, reset=True)
 +        junction = os.path.join(project, 'elements', 'junction')
 +        os.mkdir(junction)
 +        shutil.copy2(os.path.join(project, 'elements', element_name), junction)
++
 +        junction_conf = {}
 +        junction_conf['name'] = 'amhello'
 +        junction_conf['artifacts'] = {'url': share4.repo, 'push': True, 'allow-partial-push': True}
 +        _yaml.dump(junction_conf, os.path.join(junction, 'project.conf'))
 +        with open(os.path.join(project, 'project.conf'), 'a') as f:
 +            f.write("artifacts:\n  url: {}\n  push: True\n".format(share4.repo))
++
 +        # Read project.conf, the junction project.conf and buildstream.conf
 +        # before running bst
 +        with open(os.path.join(project, 'project.conf'), 'r') as f:
 +            print(f.read())
 +        with open(os.path.join(junction, 'project.conf'), 'r') as f:
 +            print(f.read())
 +        with open(os.path.join(project, 'cache', 'buildstream.conf'), 'r') as f:
 +            print(f.read())
++
 +        result = cli.run(project=project, args=['build', 'junction/amhello.bst'])
++
 +        # Read project.conf, the junction project.conf and buildstream.conf
 +        # after running bst
 +        with open(os.path.join(project, 'project.conf'), 'r') as f:
 +            print(f.read())
 +        with open(os.path.join(junction, 'project.conf'), 'r') as f:
 +            print(f.read())
 +        with open(os.path.join(project, 'cache', 'buildstream.conf'), 'r') as f:
 +            print(f.read())
++
 +        assert 'junction/amhello.bst' in result.get_partial_pushed_elements()
 +        assert 'base/base-alpine.bst' in result.get_pushed_elements()

tests/testutils/runcli.py

@@ -167,6 +167,23 @@ class Result():
      def assert_shell_error(self, fail_message=''):
          assert self.exit_code == 1, fail_message
 +    # get_start_order()
 +    #
 +    # Gets the list of elements processed in a given queue, in the
 +    # order of their first appearances in the session.
 +    #
 +    # Args:
 +    #    activity (str): The queue activity name (like 'fetch')
 +    #
 +    # Returns:
 +    #    (list): A list of element names in the order which they first appeared in the result
 +    #
 +    def get_start_order(self, activity):
 +        results = re.findall(r'\[\s*{}:(\S+)\s*\]\s*START\s*.*\.log'.format(activity), self.stderr)
 +        if results is None:
 +            return []
 +        return list(results)
++
      # get_tracked_elements()
+     #
      # Produces a list of element names on which tracking occurred
@@ -191,6 +208,13 @@ class Result():
          return list(pushed)
 +    def get_partial_pushed_elements(self):
 +        pushed = re.findall(r'\[\s*push:(\S+)\s*\]\s*INFO\s*Pushed partial artifact', self.stderr)
 +        if pushed is None:
 +            return []
++
 +        return list(pushed)
++
      def get_pulled_elements(self):
          pulled = re.findall(r'\[\s*pull:(\S+)\s*\]\s*INFO\s*Pulled artifact', self.stderr)
          if pulled is None:
@@ -198,6 +222,13 @@ class Result():
          return list(pulled)
 +    def get_partial_pulled_elements(self):
 +        pulled = re.findall(r'\[\s*pull:(\S+)\s*\]\s*INFO\s*Pulled partial artifact', self.stderr)
 +        if pulled is None:
 +            return []
++
 +        return list(pulled)
++
  class Cli():
@@ -218,11 +249,15 @@ class Cli():
+     #
      # Args:
      #    config (dict): The user configuration to use
 +    #    reset (bool): Optional reset of stored config
+     #
 -    def configure(self, config):
 +    def configure(self, config, reset=False):
          if self.config is None:
              self.config = {}
 +        if reset:
 +            self.config.clear()
++
          for key, val in config.items():
              self.config[key] = val

tox.ini

 +#
 +# Tox global configuration
 +#
  [tox]
  envlist = py35,py36,py37
  skip_missing_interpreters = true
 +#
 +# Defaults for all environments
 +#
 +# Anything specified here is iherited by the sections
 +#
  [testenv]
 -commands = pytest {posargs}
 +commands =
 +    pytest --basetemp {envtmpdir} {posargs}
 +    mkdir -p .coverage-reports
 +    mv {envtmpdir}/.coverage {toxinidir}/.coverage-reports/.coverage.{env:COVERAGE_PREFIX:}{envname}
  deps =
      -rrequirements/requirements.txt
      -rrequirements/dev-requirements.txt
@@ -13,6 +24,32 @@ passenv =
      GI_TYPELIB_PATH
      INTEGRATION_CACHE
 +#
 +# These keys are not inherited by any other sections
 +#
 +setenv =
 +    py{35,36,37}: COVERAGE_FILE = {envtmpdir}/.coverage
 +whitelist_externals =
 +    py{35,36,37}:
 +        mv
 +        mkdir
++
 +#
 +# Coverage reporting
 +#
 +[testenv:coverage]
 +commands =
 +    - coverage combine --rcfile={toxinidir}/.coveragerc {toxinidir}/.coverage-reports/
 +    coverage report --rcfile={toxinidir}/.coveragerc -m
 +deps =
 +    -rrequirements/requirements.txt
 +    -rrequirements/dev-requirements.txt
 +setenv =
 +    COVERAGE_FILE = {toxinidir}/.coverage-reports/.coverage
++
 +#
 +# Running linters
 +#
  [testenv:lint]
  commands =
      pycodestyle
@@ -22,6 +59,9 @@ deps =
      -rrequirements/dev-requirements.txt
      -rrequirements/plugin-requirements.txt
 +#
 +# Building documentation
 +#
  [testenv:docs]
  commands =
      make -C doc

[Notes] [Git][BuildStream/buildstream][tpollard/566] 27 commits: sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.

Tom Pollard pushed to branch tpollard/566 at BuildStream / buildstream

Commits:

27 changed files:

Changes: