[Notes] [Git][BuildStream/buildstream][jennis/add_artifacts

James Ennis pushed to branch jennis/add_artifacts_completion at BuildStream / buildstream

Commits:

17c5ca2d

by Valentin David at 2019-01-09T10:45:23Z

Use relative path to project directory for remote execution certificates/keys

605f8d11

by Valentin David at 2019-01-09T10:45:23Z

Add support for user remote execution configuration

Fixes #631.

65ea03ab

by Valentin David at 2019-01-09T10:45:23Z

Add support for https channel to remote execution and actions servers

Fixes #780.

ba05c4fa

by Valentin David at 2019-01-09T12:58:59Z

Merge branch 'valentindavid/remote_execution_configuration' into 'master'

Remote execution configuration: HTTPS and user configuration

Closes #780 and #631

See merge request BuildStream/buildstream!1030

eb0dbcfc

by Valentin David at 2019-01-09T13:56:58Z

buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY

ScriptElement does not use Sandbox.get_directory. It works using it
with remote execution.

Fixes #850

f7681925

by Valentin David at 2019-01-09T14:44:22Z

Merge branch 'valentindavid/script_virtual_directory' into 'master'

buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY

Closes #850

See merge request BuildStream/buildstream!1047

f87d1c93

by Jim MacArthur at 2019-01-09T15:01:18Z

sandboxremote.py: Rename 'instance_name' option to 'instance-name'

It was 'instance-name' in the documentation.

ed8bc5bc

by Jim MacArthur at 2019-01-09T15:30:28Z

Merge branch 'jmac/rename_instance_name' into 'master'

Rename 'instance_name' option to 'instance-name'

See merge request BuildStream/buildstream!1048

f29a0995

by Tristan Van Berkom at 2019-01-09T19:51:14Z

Support running test environments in parallel with `detox`

This patch namespaces the test temp directory and the output
coverage report file with the name of the environment under test,
such that separately run tests do not access the same files.

When running tests without tox, directly through setup.py,
then the tmp directory will still be `./tmp`.

  * .gitignore: Added new .coverage-reports/ directory

  * .gitlab-ci.yml: Rely on tox to combine and report coverage, only
    tell tox about the COVERAGE_PREFIX so that results can be namespaced
    by CI job name.

    This change also publishes the sources and final combined `.coverage`
    file in an output gitlab artifact for inspection, and lists some missing
    dependencies to the `coverage` job.

  * tox.ini: Add comments and refactor main [testenv] section so that
    other environments dont inherit too much unrelated cruft.

    Generate the coverate reports in the respective {envtmpdir} so that
    all per-process coverage files are prefixed with a full path, ensuring
    that concurrent runs don't mix reports and addressing concerns
    raised in #844.

    Also implemented new `tox -e coverage` environment to combine
    any found coverage and print a report.

  * .coveragerc: Omit .tox/ directory from coverage stats

Fixes issue #844

d364ad02

by Tristan Van Berkom at 2019-01-09T19:52:22Z

CONTRIBUTING.rst: Updated to mention new `coverage` tox environment.

Also point out that it is possible to run test environments in
parallel using the `detox` tool.

1e352434

by Tristan Van Berkom at 2019-01-09T20:36:06Z

Merge branch 'tristan/detox-tests' into 'master'

Allow using detox, and fix/refactor collection of coverage reports

Closes #844

See merge request BuildStream/buildstream!1051

116c1070

by Tristan Van Berkom at 2019-01-09T20:39:15Z

.coveragerc: Omit some things which are irrelevant to cover.

  * Omit versioneer's _version.py

  * Omit our __main__.py which is used only internally for generating
    documentation

c8f3616d

by Tristan Van Berkom at 2019-01-09T21:12:29Z

Merge branch 'tristan/refine-coverage' into 'master'

.coveragerc: Omit some things which are irrelevant to cover.

See merge request BuildStream/buildstream!1052

c87bb592

by Jürg Billeter at 2019-01-10T09:43:37Z

_platform/platform.py: Add canonicalize_arch() method

26e33346

by Jürg Billeter at 2019-01-10T12:50:15Z

_options/optionarch.py: Accept architecture aliases

Accept common architecture aliases for arch options instead of only
accepting the canonicalized, OS-independent architecture name. This
restores compatibility with existing projects and may simplify option
handling for projects that only target a single OS (and thus, do not
need OS-independent architecture names).

06deb4c4

by Jürg Billeter at 2019-01-10T12:50:15Z

element.py: Accept architecture aliases for sandbox config

Accept common architecture aliases for the sandbox config for
consistency with arch options.

0c2a66b3

by Jürg Billeter at 2019-01-10T13:06:02Z

tests/format/optionarch.py: Add tests for architecture aliases

f86bc760

by Jürg Billeter at 2019-01-10T13:34:56Z

Merge branch 'juerg/arch' into 'master'

Accept architecture aliases

See merge request BuildStream/buildstream!1034

347eb34e

by Angelos Evripiotis at 2019-01-10T14:38:23Z

contributing: fix 'oprtation' and some other typos

6bc27bc1

by Angelos Evripiotis at 2019-01-10T15:07:46Z

Merge branch 'aevri/contrib-typos' into 'master'

contributing: fix 'oprtation' and some other typos

See merge request BuildStream/buildstream!1055

9760e0fb

by Jürg Billeter at 2019-01-10T16:03:22Z

_frontend/cli.py: Also check original args for --config in bash completion

The path of the config file generated by testutils for completion tests
is passed as regular argument, not via COMP_WORDS. Use that config file
in complete_artifact() to ensure the test uses the right artifact
directory.

fdffb5f2

by James Ennis at 2019-01-10T16:03:22Z

completions.py: Add a test for our artifact ref autocompletions

Since the artifact subcommand group has been created, we have been able
to tab complete both element names and artifact refs as arguments to
the artifact subcommands (e.g. bst artifact log), this commit adds a
test for this.

22 changed files:

.coveragerc
.gitignore
.gitlab-ci.yml
CONTRIBUTING.rst
buildstream/_artifactcache/cascache.py
buildstream/_context.py
buildstream/_frontend/cli.py
buildstream/_options/optionarch.py
buildstream/_platform/platform.py
buildstream/_project.py
buildstream/element.py
buildstream/plugins/elements/script.py
buildstream/sandbox/_sandboxremote.py
doc/source/format_project.rst
doc/source/using_config.rst
tests/completions/completions.py
+ tests/format/option-arch-alias/element.bst
+ tests/format/option-arch-alias/project.conf
+ tests/format/option-arch-unknown/element.bst
+ tests/format/option-arch-unknown/project.conf
tests/format/optionarch.py
tox.ini

Changes:

.coveragerc

@@ -4,11 +4,15 @@ include =
    */buildstream/*
  omit =
 -  # Omit profiling helper module
 +  # Omit some internals
    */buildstream/_profile.py
 +  */buildstream/__main__.py
 +  */buildstream/_version.py
    # Omit generated code
    */buildstream/_protos/*
    */.eggs/*
 +  # Omit .tox directory
 +  */.tox/*
  [report]
  show_missing = True

.gitignore

@@ -13,11 +13,12 @@ tests/**/*.pyc
  integration-cache/
  tmp
  .coverage
 +.coverage-reports/
  .coverage.*
  .cache
  .pytest_cache/
  *.bst/
 -.tox
 +.tox/
  # Pycache, in case buildstream is ran directly from within the source
  # tree

.gitlab-ci.yml

@@ -13,6 +13,7 @@ variables:
    PYTEST_ADDOPTS: "--color=yes"
    INTEGRATION_CACHE: "${CI_PROJECT_DIR}/cache/integration-cache"
    TEST_COMMAND: "tox -- --color=yes --integration"
 +  COVERAGE_PREFIX: "${CI_JOB_NAME}."
  #####################################################
@@ -24,9 +25,6 @@ variables:
  .tests-template: &tests
    stage: test
 -  variables:
 -    COVERAGE_DIR: coverage-linux
+-
    before_script:
    # Diagnostics
    - mount
@@ -40,14 +38,11 @@ variables:
    - su buildstream -c "${TEST_COMMAND}"
    after_script:
 -  # Collect our reports
 -  - mkdir -p ${COVERAGE_DIR}
 -  - cp .coverage ${COVERAGE_DIR}/coverage."${CI_JOB_NAME}"
    except:
    - schedules
    artifacts:
      paths:
 -    - ${COVERAGE_DIR}
 +    - .coverage-reports
  tests-debian-9:
    image: buildstream/testsuite-debian:9-5da27168-32c47d1c
@@ -83,7 +78,6 @@ tests-unix:
    <<: *tests
    variables:
      BST_FORCE_BACKEND: "unix"
 -    COVERAGE_DIR: coverage-unix
    script:
@@ -239,22 +233,22 @@ coverage:
    stage: post
    coverage: '/TOTAL +\d+ +\d+ +(\d+\.\d+)%/'
    script:
 -    - pip3 install -r requirements/requirements.txt -r requirements/dev-requirements.txt
 -    - pip3 install --no-index .
 -    - mkdir report
 -    - cd report
 -    - cp ../coverage-unix/coverage.* .
 -    - cp ../coverage-linux/coverage.* .
 -    - ls coverage.*
 -    - coverage combine --rcfile=../.coveragerc -a coverage.*
 -    - coverage report --rcfile=../.coveragerc -m
 +    - cp -a .coverage-reports/ ./coverage-sources
 +    - tox -e coverage
 +    - cp -a .coverage-reports/ ./coverage-report
    dependencies:
    - tests-debian-9
    - tests-fedora-27
    - tests-fedora-28
 +  - tests-fedora-missing-deps
 +  - tests-ubuntu-18.04
    - tests-unix
    except:
    - schedules
 +  artifacts:
 +    paths:
 +    - coverage-sources/
 +    - coverage-report/
  # Deploy, only for merges which land on master branch.
+ #

CONTRIBUTING.rst

@@ -553,7 +553,7 @@ One problem which arises from this is that we end up having symbols
  which are *public* according to the :ref:`rules discussed in the previous section
  <contributing_public_and_private>`, but must be hidden away from the
  *"Public API Surface"*. For example, BuildStream internal classes need
 -to invoke methods on the ``Element`` and ``Source`` classes, wheras these
 +to invoke methods on the ``Element`` and ``Source`` classes, whereas these
  methods need to be hidden from the *"Public API Surface"*.
  This is where BuildStream deviates from the PEP-8 standard for public
@@ -631,7 +631,7 @@ An element plugin will derive from Element by importing::
    from buildstream import Element
 -When importing utilities specifically, dont import function names
 +When importing utilities specifically, don't import function names
  from there, instead import the module itself::
    from . import utils
@@ -737,7 +737,7 @@ Abstract methods
  ~~~~~~~~~~~~~~~~
  In BuildStream, an *"Abstract Method"* is a bit of a misnomer and does
  not match up to how Python defines abstract methods, we need to seek out
 -a new nomanclature to refer to these methods.
 +a new nomenclature to refer to these methods.
  In Python, an *"Abstract Method"* is a method which **must** be
  implemented by a subclass, whereas all methods in Python can be
@@ -960,7 +960,7 @@ possible, and avoid any cyclic relationships in modules.
  For instance, the ``Source`` objects are owned by ``Element``
  objects in the BuildStream data model, and as such the ``Element``
  will delegate some activities to the ``Source`` objects in its
 -possesion. The ``Source`` objects should however never call functions
 +possession. The ``Source`` objects should however never call functions
  on the ``Element`` object, nor should the ``Source`` object itself
  have any understanding of what an ``Element`` is.
@@ -1223,7 +1223,7 @@ For further information about using the reStructuredText with sphinx, please see
  Building Docs
  ~~~~~~~~~~~~~
  Before you can build the docs, you will end to ensure that you have installed
 -the required :ref:`buid dependencies <contributing_build_deps>` as mentioned
 +the required :ref:`build dependencies <contributing_build_deps>` as mentioned
  in the testing section above.
  To build the documentation, just run the following::
@@ -1365,7 +1365,7 @@ Structure of an example
  '''''''''''''''''''''''
  The :ref:`tutorial <tutorial>` and the :ref:`examples <examples>` sections
  of the documentation contain a series of sample projects, each chapter in
 -the tutoral, or standalone example uses a sample project.
 +the tutorial, or standalone example uses a sample project.
  Here is the the structure for adding new examples and tutorial chapters.
@@ -1471,8 +1471,8 @@ Installing build dependencies
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Some of BuildStream's dependencies have non-python build dependencies. When
  running tests with ``tox``, you will first need to install these dependencies.
 -Exact steps to install these will depend on your oprtation systemm. Commands
 -for installing them for some common distributions are lised below.
 +Exact steps to install these will depend on your operating system. Commands
 +for installing them for some common distributions are listed below.
  For Fedora-based systems::
@@ -1498,6 +1498,13 @@ option when running tox::
    tox -e py37
 +If you would like to test and lint at the same time, or if you do have multiple
 +python versions installed and would like to test against multiple versions, then
 +we recommend using `detox <https://github.com/tox-dev/detox>`_, just run it with
 +the same arguments you would give `tox`::
++
 +  detox -e lint,py36,py37
++
  Linting is performed separately from testing. In order to run the linting step which
  consists of running the ``pycodestyle`` and ``pylint`` tools, run the following::
@@ -1533,7 +1540,7 @@ the frontend tests you can do::
    tox -- tests/frontend/
 -Specific tests can be chosen by using the :: delimeter after the test module.
 +Specific tests can be chosen by using the :: delimiter after the test module.
  If you wanted to run the test_build_track test within frontend/buildtrack.py you could do::
    tox -- tests/frontend/buildtrack.py::test_build_track
@@ -1553,7 +1560,7 @@ can run ``tox`` with ``-r`` or  ``--recreate`` option.
  .. note::
     By default, we do not allow use of site packages in our ``tox``
 -   confguration to enable running the tests in an isolated environment.
 +   configuration to enable running the tests in an isolated environment.
     If you need to enable use of site packages for whatever reason, you can
     do so by passing the ``--sitepackages`` option to ``tox``. Also, you will
     not need to install any of the build dependencies mentioned above if you
@@ -1574,10 +1581,23 @@ can run ``tox`` with ``-r`` or  ``--recreate`` option.
       ./setup.py test --addopts 'tests/frontend/buildtrack.py::test_build_track'
 +Observing coverage
 +~~~~~~~~~~~~~~~~~~
 +Once you have run the tests using `tox` (or `detox`), some coverage reports will
 +have been left behind.
++
 +To view the coverage report of the last test run, simply run::
++
 +  tox -e coverage
++
 +This will collate any reports from separate python environments that may be
 +under test before displaying the combined coverage.
++
++
  Adding tests
  ~~~~~~~~~~~~
  Tests are found in the tests subdirectory, inside of which
 -there is a separarate directory for each *domain* of tests.
 +there is a separate directory for each *domain* of tests.
  All tests are collected as::
    tests/*/*.py

buildstream/_artifactcache/cascache.py

@@ -53,7 +53,7 @@ class CASRemoteSpec(namedtuple('CASRemoteSpec', 'url push server_cert client_key
+     #
      @staticmethod
      def _new_from_config_node(spec_node, basedir=None):
 -        _yaml.node_validate(spec_node, ['url', 'push', 'server-cert', 'client-key', 'client-cert', 'instance_name'])
 +        _yaml.node_validate(spec_node, ['url', 'push', 'server-cert', 'client-key', 'client-cert', 'instance-name'])
          url = _yaml.node_get(spec_node, str, 'url')
          push = _yaml.node_get(spec_node, bool, 'push', default_value=False)
          if not url:
@@ -61,7 +61,7 @@ class CASRemoteSpec(namedtuple('CASRemoteSpec', 'url push server_cert client_key
              raise LoadError(LoadErrorReason.INVALID_DATA,
                              "{}: empty artifact cache URL".format(provenance))
 -        instance_name = _yaml.node_get(spec_node, str, 'instance_name', default_value=None)
 +        instance_name = _yaml.node_get(spec_node, str, 'instance-name', default_value=None)
          server_cert = _yaml.node_get(spec_node, str, 'server-cert', default_value=None)
          if server_cert and basedir:

buildstream/_context.py

@@ -34,6 +34,7 @@ from ._artifactcache import ArtifactCache
  from ._artifactcache.cascache import CASCache
  from ._workspaces import Workspaces, WorkspaceProjectCache, WORKSPACE_PROJECT_FILE
  from .plugin import _plugin_lookup
 +from .sandbox import SandboxRemote
  # Context()
@@ -72,6 +73,9 @@ class Context():
          # The locations from which to push and pull prebuilt artifacts
          self.artifact_cache_specs = None
 +        # The global remote execution configuration
 +        self.remote_execution_specs = None
++
          # The directory to store build logs
          self.logdir = None
@@ -187,7 +191,7 @@ class Context():
          _yaml.node_validate(defaults, [
              'sourcedir', 'builddir', 'artifactdir', 'logdir',
              'scheduler', 'artifacts', 'logging', 'projects',
 -            'cache', 'prompt', 'workspacedir',
 +            'cache', 'prompt', 'workspacedir', 'remote-execution'
          ])
          for directory in ['sourcedir', 'builddir', 'artifactdir', 'logdir', 'workspacedir']:
@@ -212,6 +216,8 @@ class Context():
          # Load artifact share configuration
          self.artifact_cache_specs = ArtifactCache.specs_from_config_node(defaults)
 +        self.remote_execution_specs = SandboxRemote.specs_from_config_node(defaults)
++
          # Load pull build trees configuration
          self.pull_buildtrees = _yaml.node_get(cache, bool, 'pull-buildtrees')
@@ -271,7 +277,8 @@ class Context():
          # Shallow validation of overrides, parts of buildstream which rely
          # on the overrides are expected to validate elsewhere.
          for _, overrides in _yaml.node_items(self._project_overrides):
 -            _yaml.node_validate(overrides, ['artifacts', 'options', 'strict', 'default-mirror'])
 +            _yaml.node_validate(overrides, ['artifacts', 'options', 'strict', 'default-mirror',
 +                                            'remote-execution'])
          profile_end(Topics.LOAD_CONTEXT, 'load')

buildstream/_frontend/cli.py

@@ -2,6 +2,7 @@ import os
  import sys
  from contextlib import ExitStack
  from fnmatch import fnmatch
 +from functools import partial
  from tempfile import TemporaryDirectory
  import click
@@ -111,11 +112,14 @@ def complete_target(args, incomplete):
      return complete_list
 -def complete_artifact(args, incomplete):
 +def complete_artifact(orig_args, args, incomplete):
      from .._context import Context
      ctx = Context()
      config = None
 +    for i, arg in enumerate(orig_args):
 +        if arg in ('-c', '--config'):
 +            config = orig_args[i + 1]
      for i, arg in enumerate(args):
          if arg in ('-c', '--config'):
              config = args[i + 1]
@@ -128,8 +132,9 @@ def complete_artifact(args, incomplete):
      return complete_list
 -def override_completions(cmd, cmd_param, args, incomplete):
 +def override_completions(orig_args, cmd, cmd_param, args, incomplete):
      """
 +    :param orig_args: original, non-completion args
      :param cmd_param: command definition
      :param args: full list of args typed before the incomplete arg
      :param incomplete: the incomplete text to autocomplete
@@ -150,7 +155,7 @@ def override_completions(cmd, cmd_param, args, incomplete):
                  cmd_param.opts == ['--track-except']):
              return complete_target(args, incomplete)
          if cmd_param.name == 'artifacts':
 -            return complete_artifact(args, incomplete)
 +            return complete_artifact(orig_args, args, incomplete)
      raise CompleteUnhandled()
@@ -161,7 +166,7 @@ def override_main(self, args=None, prog_name=None, complete_var=None,
      # Hook for the Bash completion.  This only activates if the Bash
      # completion is actually enabled, otherwise this is quite a fast
      # noop.
 -    if main_bashcomplete(self, prog_name, override_completions):
 +    if main_bashcomplete(self, prog_name, partial(override_completions, args)):
          # If we're running tests we cant just go calling exit()
          # from the main process.

buildstream/_options/optionarch.py

@@ -17,6 +17,8 @@
  #  Authors:
  #        Tristan Van Berkom <tristan vanberkom codethink co uk>
 +from .. import _yaml
 +from .._exceptions import LoadError, LoadErrorReason, PlatformError
  from .._platform import Platform
  from .optionenum import OptionEnum
@@ -41,7 +43,34 @@ class OptionArch(OptionEnum):
          super(OptionArch, self).load(node, allow_default_definition=False)
      def load_default_value(self, node):
 -        return Platform.get_host_arch()
 +        arch = Platform.get_host_arch()
++
 +        default_value = None
++
 +        for index, value in enumerate(self.values):
 +            try:
 +                canonical_value = Platform.canonicalize_arch(value)
 +                if default_value is None and canonical_value == arch:
 +                    default_value = value
 +                    # Do not terminate the loop early to ensure we validate
 +                    # all values in the list.
 +            except PlatformError as e:
 +                provenance = _yaml.node_get_provenance(node, key='values', indices=[index])
 +                prefix = ""
 +                if provenance:
 +                    prefix = "{}: ".format(provenance)
 +                raise LoadError(LoadErrorReason.INVALID_DATA,
 +                                "{}Invalid value for {} option '{}': {}"
 +                                .format(prefix, self.OPTION_TYPE, self.name, e))
++
 +        if default_value is None:
 +            # Host architecture is not supported by the project.
 +            # Do not raise an error here as the user may override it.
 +            # If the user does not override it, an error will be raised
 +            # by resolve()/validate().
 +            default_value = arch
++
 +        return default_value
      def resolve(self):

buildstream/_platform/platform.py

@@ -77,20 +77,17 @@ class Platform():
      def get_host_os():
          return os.uname()[0]
 -    # get_host_arch():
 +    # canonicalize_arch():
+     #
 -    # This returns the architecture of the host machine. The possible values
 -    # map from uname -m in order to be a OS independent list.
 +    # This returns the canonical, OS-independent architecture name
 +    # or raises a PlatformError if the architecture is unknown.
+     #
 -    # Returns:
 -    #    (string): String representing the architecture
      @staticmethod
 -    def get_host_arch():
 -        # get the hardware identifier from uname
 -        uname_machine = os.uname()[4]
 -        uname_to_arch = {
 +    def canonicalize_arch(arch):
 +        aliases = {
 +            "aarch32": "aarch32",
              "aarch64": "aarch64",
 -            "aarch64_be": "aarch64-be",
 +            "aarch64-be": "aarch64-be",
              "amd64": "x86-64",
              "arm": "aarch32",
              "armv8l": "aarch64",
@@ -99,17 +96,34 @@ class Platform():
              "i486": "x86-32",
              "i586": "x86-32",
              "i686": "x86-32",
 +            "power-isa-be": "power-isa-be",
 +            "power-isa-le": "power-isa-le",
              "ppc64": "power-isa-be",
              "ppc64le": "power-isa-le",
              "sparc": "sparc-v9",
              "sparc64": "sparc-v9",
 -            "x86_64": "x86-64"
 +            "sparc-v9": "sparc-v9",
 +            "x86-32": "x86-32",
 +            "x86-64": "x86-64"
+         }
++
          try:
 -            return uname_to_arch[uname_machine]
 +            return aliases[arch.replace('_', '-')]
          except KeyError:
 -            raise PlatformError("uname gave unsupported machine architecture: {}"
 -                                .format(uname_machine))
 +            raise PlatformError("Unknown architecture: {}".format(arch))
++
 +    # get_host_arch():
 +    #
 +    # This returns the architecture of the host machine. The possible values
 +    # map from uname -m in order to be a OS independent list.
 +    #
 +    # Returns:
 +    #    (string): String representing the architecture
 +    @staticmethod
 +    def get_host_arch():
 +        # get the hardware identifier from uname
 +        uname_machine = os.uname()[4]
 +        return Platform.canonicalize_arch(uname_machine)
      ##################################################################
      #                        Sandbox functions                       #

buildstream/_project.py

@@ -507,7 +507,16 @@ class Project():
          self.artifact_cache_specs = ArtifactCache.specs_from_config_node(config, self.directory)
          # Load remote-execution configuration for this project
 -        self.remote_execution_specs = SandboxRemote.specs_from_config_node(config, self.directory)
 +        project_specs = SandboxRemote.specs_from_config_node(config, self.directory)
 +        override_specs = SandboxRemote.specs_from_config_node(
 +            self._context.get_overrides(self.name), self.directory)
++
 +        if override_specs is not None:
 +            self.remote_execution_specs = override_specs
 +        elif project_specs is not None:
 +            self.remote_execution_specs = project_specs
 +        else:
 +            self.remote_execution_specs = self._context.remote_execution_specs
          # Load sandbox environment variables
          self.base_environment = _yaml.node_get(config, Mapping, 'environment')

buildstream/element.py

@@ -2441,11 +2441,17 @@ class Element(Plugin):
          # Sandbox config, unlike others, has fixed members so we should validate them
          _yaml.node_validate(sandbox_config, ['build-uid', 'build-gid', 'build-os', 'build-arch'])
 +        build_arch = self.node_get_member(sandbox_config, str, 'build-arch', default=None)
 +        if build_arch:
 +            build_arch = Platform.canonicalize_arch(build_arch)
 +        else:
 +            build_arch = host_arch
++
          return SandboxConfig(
              self.node_get_member(sandbox_config, int, 'build-uid'),
              self.node_get_member(sandbox_config, int, 'build-gid'),
              self.node_get_member(sandbox_config, str, 'build-os', default=host_os),
 -            self.node_get_member(sandbox_config, str, 'build-arch', default=host_arch))
 +            build_arch)
      # This makes a special exception for the split rules, which
      # elements may extend but whos defaults are defined in the project.

buildstream/plugins/elements/script.py

@@ -42,6 +42,9 @@ import buildstream
  class ScriptElement(buildstream.ScriptElement):
      # pylint: disable=attribute-defined-outside-init
 +    # This plugin has been modified to avoid the use of Sandbox.get_directory
 +    BST_VIRTUAL_DIRECTORY = True
++
      def configure(self, node):
          for n in self.node_get_member(node, list, 'layout', []):
              dst = self.node_subst_member(n, 'destination')

buildstream/sandbox/_sandboxremote.py

@@ -62,10 +62,32 @@ class SandboxRemote(Sandbox):
          self.storage_url = config.storage_service['url']
          self.exec_url = config.exec_service['url']
 +        exec_certs = {}
 +        for key in ['client-cert', 'client-key', 'server-cert']:
 +            if key in config.exec_service:
 +                with open(config.exec_service[key], 'rb') as f:
 +                    exec_certs[key] = f.read()
++
 +        self.exec_credentials = grpc.ssl_channel_credentials(
 +            root_certificates=exec_certs.get('server-cert'),
 +            private_key=exec_certs.get('client-key'),
 +            certificate_chain=exec_certs.get('client-cert'))
++
 +        action_certs = {}
 +        for key in ['client-cert', 'client-key', 'server-cert']:
 +            if key in config.action_service:
 +                with open(config.action_service[key], 'rb') as f:
 +                    action_certs[key] = f.read()
++
          if config.action_service:
              self.action_url = config.action_service['url']
 +            self.action_credentials = grpc.ssl_channel_credentials(
 +                root_certificates=action_certs.get('server-cert'),
 +                private_key=action_certs.get('client-key'),
 +                certificate_chain=action_certs.get('client-cert'))
          else:
              self.action_url = None
 +            self.action_credentials = None
          self.server_instance = config.exec_service.get('instance', None)
          self.storage_instance = config.storage_service.get('instance', None)
@@ -81,7 +103,7 @@ class SandboxRemote(Sandbox):
          self._get_context().message(Message(None, MessageType.INFO, msg))
      @staticmethod
 -    def specs_from_config_node(config_node, basedir):
 +    def specs_from_config_node(config_node, basedir=None):
          def require_node(config, keyname):
              val = config.get(keyname)
@@ -109,10 +131,10 @@ class SandboxRemote(Sandbox):
          remote_exec_storage_config = require_node(remote_config, 'storage-service')
          remote_exec_action_config = remote_config.get('action-cache-service', {})
 -        _yaml.node_validate(remote_exec_service_config, ['url', 'instance'])
 +        _yaml.node_validate(remote_exec_service_config, ['url', 'instance'] + tls_keys)
          _yaml.node_validate(remote_exec_storage_config, ['url', 'instance'] + tls_keys)
          if remote_exec_action_config:
 -            _yaml.node_validate(remote_exec_action_config, ['url'])
 +            _yaml.node_validate(remote_exec_action_config, ['url'] + tls_keys)
          else:
              remote_config['action-service'] = None
@@ -135,6 +157,19 @@ class SandboxRemote(Sandbox):
                                        "remote-execution configuration. Your config is missing '{}'."
                                        .format(str(provenance), tls_keys, key))
 +        def resolve_path(path):
 +            if basedir and path:
 +                return os.path.join(basedir, path)
 +            else:
 +                return path
++
 +        for key in tls_keys:
 +            for d in (remote_config['execution-service'],
 +                      remote_config['storage-service'],
 +                      remote_exec_action_config):
 +                if key in d:
 +                    d[key] = resolve_path(d[key])
++
          spec = RemoteExecutionSpec(remote_config['execution-service'],
                                     remote_config['storage-service'],
                                     remote_exec_action_config)
@@ -295,6 +330,8 @@ class SandboxRemote(Sandbox):
                                 "for example: http://buildservice:50051.")
          if url.scheme == 'http':
              channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
 +        elif url.scheme == 'https':
 +            channel = grpc.secure_channel('{}:{}'.format(url.hostname, url.port), self.exec_credentials)
          else:
              raise SandboxError("Remote execution currently only supports the 'http' protocol "
                                 "and '{}' was supplied.".format(url.scheme))
@@ -352,11 +389,11 @@ class SandboxRemote(Sandbox):
          if not url.port:
              raise SandboxError("You must supply a protocol and port number in the action-cache-service url, "
                                 "for example: http://buildservice:50051.")
 -        if not url.scheme == "http":
 -            raise SandboxError("Currently only support http for the action cache"
 -                               "and {} was supplied".format(url.scheme))
 +        if url.scheme == 'http':
 +            channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
 +        elif url.scheme == 'https':
 +            channel = grpc.secure_channel('{}:{}'.format(url.hostname, url.port), self.action_credentials)
 -        channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
          request = remote_execution_pb2.GetActionResultRequest(action_digest=action_digest)
          stub = remote_execution_pb2_grpc.ActionCacheStub(channel)
          try:

doc/source/format_project.rst

@@ -218,6 +218,7 @@ The use of ports are required to distinguish between pull only access and
  push/pull access. For information regarding the server/client certificates
  and keys, please see: :ref:`Key pair for the server <server_authentication>`.
 +.. _project_remote_execution:
  Remote execution
  ~~~~~~~~~~~~~~~~
@@ -243,9 +244,6 @@ using the `remote-execution` option:
      action-cache-service:
        url: http://bar.action.com:50052
 -The execution-service part of remote execution does not support encrypted
 -connections yet, so the protocol must always be http.
+-
  storage-service specifies a remote CAS store and the parameters are the
  same as those used to specify an :ref:`artifact server <artifacts>`.
@@ -268,6 +266,9 @@ instance names.
  The Remote Execution API can be found via https://github.com/bazelbuild/remote-apis.
 +Remote execution configuration can be also provided in the `user
 +configuration <user_config_remote_execution>`.
++
  .. _project_essentials_mirrors:
  Mirrors

doc/source/using_config.rst

@@ -100,6 +100,54 @@ pull only access and push/pull access. For information regarding this and the
  server/client certificates and keys, please see:
  :ref:`Key pair for the server <server_authentication>`.
 +.. _user_config_remote_execution:
++
 +Remote execution
 +~~~~~~~~~~~~~~~~
++
 +The same configuration for :ref:`remote execution <project_remote_execution>`
 +in ``project.conf`` can be provided in the user configuation.
++
 +There is only one remote execution configuration used per project.
++
 +The project overrides will be taken in priority. The global
 +configuration will be used as fallback.
++
 +1. Global remote execution fallback:
++
 +.. code:: yaml
++
 +  remote-execution:
 +    execution-service:
 +      url: http://execution.fallback.example.com:50051
 +      instance-name: main
 +    storage-service:
 +      url: https://storage.fallback.example.com:11002/
 +      server-cert: /keys/server.crt
 +      client-cert: /keys/client.crt
 +      client-key: /keys/client.key
 +      instance-name: main
 +    action-cache-service:
 +      url: http://action.flalback.example.com:50052
++
 +2. Project override:
++
 +.. code:: yaml
++
 +  projects:
 +    some_project:
 +      remote-execution:
 +        execution-service:
 +          url: http://execution.some_project.example.com:50051
 +          instance-name: main
 +        storage-service:
 +          url: https://storage.some_project.example.com:11002/
 +          server-cert: /some_project_keys/server.crt
 +          client-cert: /some_project_keys/client.crt
 +          client-key: /some_project_keys/client.key
 +          instance-name: main
 +        action-cache-service:
 +          url: http://action.some_project.example.com:50052
  Strict build plan

tests/completions/completions.py

@@ -281,3 +281,44 @@ def test_argument_element_invalid(datafiles, cli, project, cmd, word_idx, expect
  ])
  def test_help_commands(cli, cmd, word_idx, expected):
      assert_completion(cli, cmd, word_idx, expected)
++
++
 +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'project'))
 +def test_argument_artifact(cli, tmpdir, datafiles):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
++
 +    # Build an import element with no dependencies (as there will only be ONE cache key)
 +    result = cli.run(project=project, args=['build', 'import-bin.bst'])  # Has no dependencies
 +    result.assert_success()
++
 +    # Get the key and the artifact ref ($project/$element_name/$key)
 +    key = cli.get_element_key(project, 'import-bin.bst')
 +    artifact = os.path.join('test', 'import-bin', key)
++
 +    # Test autocompletion of the artifact
 +    cmds = [
 +        'bst artifact log ',
 +        'bst artifact log t',
 +        'bst artifact log test/'
 +    ]
++
 +    for i, cmd in enumerate(cmds):
 +        word_idx = 3
 +        result = cli.run(project=project, cwd=project, env={
 +            '_BST_COMPLETION': 'complete',
 +            'COMP_WORDS': cmd,
 +            'COMP_CWORD': str(word_idx)
 +        })
 +        words = []
 +        if result.output:
 +            words = result.output.splitlines()  # This leaves an extra space on each e.g. 'foo.bst ']
 +            words = [word.strip() for word in words]
++
 +            if i == 0:
 +                expected = PROJECT_ELEMENTS + [artifact]  # We should now be able to see the artifact
 +            elif i == 1:
 +                expected = ['target.bst', artifact]
 +            elif i == 2:
 +                expected = [artifact]
++
 +            assert expected == words

tests/format/option-arch-alias/element.bst

 +kind: autotools
 +variables:
 +  result: "Nothing"
 +  (?):
 +  - machine_arch == "arm":
 +      result: "Army"
 +  - machine_arch == "x86_64":
 +      result: "X86-64y"

tests/format/option-arch-alias/project.conf

 +name: test
++
 +options:
 +  machine_arch:
 +    type: arch
 +    description: The machine architecture
 +    values:
 +    - arm
 +    - x86_64

tests/format/option-arch-unknown/element.bst

 +kind: autotools
 +variables:
 +  result: "Nothing"
 +  (?):
 +  - machine_arch == "aarch32":
 +      result: "Army"
 +  - machine_arch == "aarch64":
 +      result: "Aarchy"
 +  - machine_arch == "x86-128":
 +      result: "X86-128y"

tests/format/option-arch-unknown/project.conf

 +name: test
++
 +options:
 +  machine_arch:
 +    type: arch
 +    description: The machine architecture
 +    values:
 +    - aarch32
 +    - aarch64
 +    - x86-128

tests/format/optionarch.py

@@ -75,3 +75,47 @@ def test_unsupported_arch(cli, datafiles):
          ])
          result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_alias(cli, datafiles):
++
 +    with override_uname_arch("arm"):
 +        project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch-alias')
 +        result = cli.run(project=project, silent=True, args=[
 +            'show',
 +            '--deps', 'none',
 +            '--format', '%{vars}',
 +            'element.bst'
 +        ])
++
 +        result.assert_success()
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_unknown_host_arch(cli, datafiles):
++
 +    with override_uname_arch("x86_128"):
 +        project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch')
 +        result = cli.run(project=project, silent=True, args=[
 +            'show',
 +            '--deps', 'none',
 +            '--format', '%{vars}',
 +            'element.bst'
 +        ])
++
 +        result.assert_main_error(ErrorDomain.PLATFORM, None)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_unknown_project_arch(cli, datafiles):
++
 +    project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch-unknown')
 +    result = cli.run(project=project, silent=True, args=[
 +        'show',
 +        '--deps', 'none',
 +        '--format', '%{vars}',
 +        'element.bst'
 +    ])
++
 +    result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA)

tox.ini

 +#
 +# Tox global configuration
 +#
  [tox]
  envlist = py35,py36,py37
  skip_missing_interpreters = true
 +#
 +# Defaults for all environments
 +#
 +# Anything specified here is iherited by the sections
 +#
  [testenv]
 -commands = pytest {posargs}
 +commands =
 +    pytest --basetemp {envtmpdir} {posargs}
 +    mkdir -p .coverage-reports
 +    mv {envtmpdir}/.coverage {toxinidir}/.coverage-reports/.coverage.{env:COVERAGE_PREFIX:}{envname}
  deps =
      -rrequirements/requirements.txt
      -rrequirements/dev-requirements.txt
@@ -13,6 +24,32 @@ passenv =
      GI_TYPELIB_PATH
      INTEGRATION_CACHE
 +#
 +# These keys are not inherited by any other sections
 +#
 +setenv =
 +    py{35,36,37}: COVERAGE_FILE = {envtmpdir}/.coverage
 +whitelist_externals =
 +    py{35,36,37}:
 +        mv
 +        mkdir
++
 +#
 +# Coverage reporting
 +#
 +[testenv:coverage]
 +commands =
 +    - coverage combine --rcfile={toxinidir}/.coveragerc {toxinidir}/.coverage-reports/
 +    coverage report --rcfile={toxinidir}/.coveragerc -m
 +deps =
 +    -rrequirements/requirements.txt
 +    -rrequirements/dev-requirements.txt
 +setenv =
 +    COVERAGE_FILE = {toxinidir}/.coverage-reports/.coverage
++
 +#
 +# Running linters
 +#
  [testenv:lint]
  commands =
      pycodestyle
@@ -22,6 +59,9 @@ deps =
      -rrequirements/dev-requirements.txt
      -rrequirements/plugin-requirements.txt
 +#
 +# Building documentation
 +#
  [testenv:docs]
  commands =
      make -C doc

[Notes] [Git][BuildStream/buildstream][jennis/add_artifacts_completion] 22 commits: Use relative path to project directory for remote execution certificates/keys

James Ennis pushed to branch jennis/add_artifacts_completion at BuildStream / buildstream

Commits:

22 changed files:

Changes: