[Notes] [Git][BuildStream/buildstream][tristan/detox-tests] 11 commits:

Tristan Van Berkom pushed to branch tristan/detox-tests at BuildStream / buildstream

Commits:

17c5ca2d

by Valentin David at 2019-01-09T10:45:23Z

Use relative path to project directory for remote execution certificates/keys

605f8d11

by Valentin David at 2019-01-09T10:45:23Z

Add support for user remote execution configuration

Fixes #631.

65ea03ab

by Valentin David at 2019-01-09T10:45:23Z

Add support for https channel to remote execution and actions servers

Fixes #780.

ba05c4fa

by Valentin David at 2019-01-09T12:58:59Z

Merge branch 'valentindavid/remote_execution_configuration' into 'master'

Remote execution configuration: HTTPS and user configuration

Closes #780 and #631

See merge request BuildStream/buildstream!1030

eb0dbcfc

by Valentin David at 2019-01-09T13:56:58Z

buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY

ScriptElement does not use Sandbox.get_directory. It works using it
with remote execution.

Fixes #850

f7681925

by Valentin David at 2019-01-09T14:44:22Z

Merge branch 'valentindavid/script_virtual_directory' into 'master'

buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY

Closes #850

See merge request BuildStream/buildstream!1047

f87d1c93

by Jim MacArthur at 2019-01-09T15:01:18Z

sandboxremote.py: Rename 'instance_name' option to 'instance-name'

It was 'instance-name' in the documentation.

ed8bc5bc

by Jim MacArthur at 2019-01-09T15:30:28Z

Merge branch 'jmac/rename_instance_name' into 'master'

Rename 'instance_name' option to 'instance-name'

See merge request BuildStream/buildstream!1048

3a06278e

by Tristan Van Berkom at 2019-01-09T15:38:56Z

Support running test environments in parallel with `detox`

This patch namespaces the test temp directory and the output
coverage report file with the name of the environment under test,
such that separately run tests do not access the same files.

When running tests without tox, directly through setup.py,
then the tmp directory will still be `./tmp`.

  * .gitignore: Added new .coverage-reports/ directory

  * .gitlab-ci.yml: Rely on tox to combine and report coverage, only
    tell tox about the COVERAGE_PREFIX so that results can be namespaced
    by CI job name.

    This change also publishes the sources and final combined `.coverage`
    file in an output gitlab artifact for inspection, and lists some missing
    dependencies to the `coverage` job.

  * tox.ini: Add comments and refactor main [testenv] section so that
    other environments dont inherit too much unrelated cruft.

    Set a separate working directory for each test environment,
    solving the parallelism concerns about coverage raised in #844.

    Also implemented new `tox -e coverage` environment to combine
    any found coverage and print a report.

Fixes issue #844

4274889d

by Tristan Van Berkom at 2019-01-09T15:38:56Z

CONTRIBUTING.rst: Updated to mention new `coverage` tox environment.

Also point out that it is possible to run test environments in
parallel using the `detox` tool.

b57deea9
by Tristan Van Berkom at 2019-01-09T15:38:56Z
```
Testing without the chdir
```

Changes:

.gitignore

@@ -13,11 +13,12 @@ tests/**/*.pyc
  integration-cache/
  tmp
  .coverage
 +.coverage-reports/
  .coverage.*
  .cache
  .pytest_cache/
  *.bst/
 -.tox
 +.tox/
  # Pycache, in case buildstream is ran directly from within the source
  # tree

.gitlab-ci.yml

@@ -13,6 +13,7 @@ variables:
    PYTEST_ADDOPTS: "--color=yes"
    INTEGRATION_CACHE: "${CI_PROJECT_DIR}/cache/integration-cache"
    TEST_COMMAND: "tox -- --color=yes --integration"
 +  COVERAGE_PREFIX: "${CI_JOB_NAME}."
  #####################################################
@@ -24,9 +25,6 @@ variables:
  .tests-template: &tests
    stage: test
 -  variables:
 -    COVERAGE_DIR: coverage-linux
+-
    before_script:
    # Diagnostics
    - mount
@@ -40,14 +38,11 @@ variables:
    - su buildstream -c "${TEST_COMMAND}"
    after_script:
 -  # Collect our reports
 -  - mkdir -p ${COVERAGE_DIR}
 -  - cp .coverage ${COVERAGE_DIR}/coverage."${CI_JOB_NAME}"
    except:
    - schedules
    artifacts:
      paths:
 -    - ${COVERAGE_DIR}
 +    - .coverage-reports
  tests-debian-9:
    image: buildstream/testsuite-debian:9-5da27168-32c47d1c
@@ -83,7 +78,6 @@ tests-unix:
    <<: *tests
    variables:
      BST_FORCE_BACKEND: "unix"
 -    COVERAGE_DIR: coverage-unix
    script:
@@ -239,22 +233,22 @@ coverage:
    stage: post
    coverage: '/TOTAL +\d+ +\d+ +(\d+\.\d+)%/'
    script:
 -    - pip3 install -r requirements/requirements.txt -r requirements/dev-requirements.txt
 -    - pip3 install --no-index .
 -    - mkdir report
 -    - cd report
 -    - cp ../coverage-unix/coverage.* .
 -    - cp ../coverage-linux/coverage.* .
 -    - ls coverage.*
 -    - coverage combine --rcfile=../.coveragerc -a coverage.*
 -    - coverage report --rcfile=../.coveragerc -m
 +    - cp -a .coverage-reports/ ./coverage-sources
 +    - tox -e coverage
 +    - cp -a .coverage-reports/ ./coverage-report
    dependencies:
    - tests-debian-9
    - tests-fedora-27
    - tests-fedora-28
 +  - tests-fedora-missing-deps
 +  - tests-ubuntu-18.04
    - tests-unix
    except:
    - schedules
 +  artifacts:
 +    paths:
 +    - coverage-sources/
 +    - coverage-report/
  # Deploy, only for merges which land on master branch.
+ #

CONTRIBUTING.rst

@@ -1498,6 +1498,10 @@ option when running tox::
    tox -e py37
 +If you do have multiple python versions installed and would like to test against
 +multiple versions, then we recommend using `detox <https://github.com/tox-dev/detox>`_,
 +just run it with the same arguments you would give `tox`.
++
  Linting is performed separately from testing. In order to run the linting step which
  consists of running the ``pycodestyle`` and ``pylint`` tools, run the following::
@@ -1574,6 +1578,19 @@ can run ``tox`` with ``-r`` or  ``--recreate`` option.
       ./setup.py test --addopts 'tests/frontend/buildtrack.py::test_build_track'
 +Observing coverage
 +~~~~~~~~~~~~~~~~~~
 +Once you have run the tests using `tox` (or `detox`), some coverage reports will
 +have been left behind.
++
 +To view the coverage report of the last test run, simply run::
++
 +  tox -e coverage
++
 +This will collate any reports from separate python environments that may be
 +under test before displaying the combined coverage.
++
++
  Adding tests
  ~~~~~~~~~~~~
  Tests are found in the tests subdirectory, inside of which

buildstream/_artifactcache/cascache.py

@@ -53,7 +53,7 @@ class CASRemoteSpec(namedtuple('CASRemoteSpec', 'url push server_cert client_key
+     #
      @staticmethod
      def _new_from_config_node(spec_node, basedir=None):
 -        _yaml.node_validate(spec_node, ['url', 'push', 'server-cert', 'client-key', 'client-cert', 'instance_name'])
 +        _yaml.node_validate(spec_node, ['url', 'push', 'server-cert', 'client-key', 'client-cert', 'instance-name'])
          url = _yaml.node_get(spec_node, str, 'url')
          push = _yaml.node_get(spec_node, bool, 'push', default_value=False)
          if not url:
@@ -61,7 +61,7 @@ class CASRemoteSpec(namedtuple('CASRemoteSpec', 'url push server_cert client_key
              raise LoadError(LoadErrorReason.INVALID_DATA,
                              "{}: empty artifact cache URL".format(provenance))
 -        instance_name = _yaml.node_get(spec_node, str, 'instance_name', default_value=None)
 +        instance_name = _yaml.node_get(spec_node, str, 'instance-name', default_value=None)
          server_cert = _yaml.node_get(spec_node, str, 'server-cert', default_value=None)
          if server_cert and basedir:

buildstream/_context.py

@@ -34,6 +34,7 @@ from ._artifactcache import ArtifactCache
  from ._artifactcache.cascache import CASCache
  from ._workspaces import Workspaces, WorkspaceProjectCache, WORKSPACE_PROJECT_FILE
  from .plugin import _plugin_lookup
 +from .sandbox import SandboxRemote
  # Context()
@@ -72,6 +73,9 @@ class Context():
          # The locations from which to push and pull prebuilt artifacts
          self.artifact_cache_specs = None
 +        # The global remote execution configuration
 +        self.remote_execution_specs = None
++
          # The directory to store build logs
          self.logdir = None
@@ -187,7 +191,7 @@ class Context():
          _yaml.node_validate(defaults, [
              'sourcedir', 'builddir', 'artifactdir', 'logdir',
              'scheduler', 'artifacts', 'logging', 'projects',
 -            'cache', 'prompt', 'workspacedir',
 +            'cache', 'prompt', 'workspacedir', 'remote-execution'
          ])
          for directory in ['sourcedir', 'builddir', 'artifactdir', 'logdir', 'workspacedir']:
@@ -212,6 +216,8 @@ class Context():
          # Load artifact share configuration
          self.artifact_cache_specs = ArtifactCache.specs_from_config_node(defaults)
 +        self.remote_execution_specs = SandboxRemote.specs_from_config_node(defaults)
++
          # Load pull build trees configuration
          self.pull_buildtrees = _yaml.node_get(cache, bool, 'pull-buildtrees')
@@ -271,7 +277,8 @@ class Context():
          # Shallow validation of overrides, parts of buildstream which rely
          # on the overrides are expected to validate elsewhere.
          for _, overrides in _yaml.node_items(self._project_overrides):
 -            _yaml.node_validate(overrides, ['artifacts', 'options', 'strict', 'default-mirror'])
 +            _yaml.node_validate(overrides, ['artifacts', 'options', 'strict', 'default-mirror',
 +                                            'remote-execution'])
          profile_end(Topics.LOAD_CONTEXT, 'load')

buildstream/_project.py

@@ -507,7 +507,16 @@ class Project():
          self.artifact_cache_specs = ArtifactCache.specs_from_config_node(config, self.directory)
          # Load remote-execution configuration for this project
 -        self.remote_execution_specs = SandboxRemote.specs_from_config_node(config, self.directory)
 +        project_specs = SandboxRemote.specs_from_config_node(config, self.directory)
 +        override_specs = SandboxRemote.specs_from_config_node(
 +            self._context.get_overrides(self.name), self.directory)
++
 +        if override_specs is not None:
 +            self.remote_execution_specs = override_specs
 +        elif project_specs is not None:
 +            self.remote_execution_specs = project_specs
 +        else:
 +            self.remote_execution_specs = self._context.remote_execution_specs
          # Load sandbox environment variables
          self.base_environment = _yaml.node_get(config, Mapping, 'environment')

buildstream/plugins/elements/script.py

@@ -42,6 +42,9 @@ import buildstream
  class ScriptElement(buildstream.ScriptElement):
      # pylint: disable=attribute-defined-outside-init
 +    # This plugin has been modified to avoid the use of Sandbox.get_directory
 +    BST_VIRTUAL_DIRECTORY = True
++
      def configure(self, node):
          for n in self.node_get_member(node, list, 'layout', []):
              dst = self.node_subst_member(n, 'destination')

buildstream/sandbox/_sandboxremote.py

@@ -62,10 +62,32 @@ class SandboxRemote(Sandbox):
          self.storage_url = config.storage_service['url']
          self.exec_url = config.exec_service['url']
 +        exec_certs = {}
 +        for key in ['client-cert', 'client-key', 'server-cert']:
 +            if key in config.exec_service:
 +                with open(config.exec_service[key], 'rb') as f:
 +                    exec_certs[key] = f.read()
++
 +        self.exec_credentials = grpc.ssl_channel_credentials(
 +            root_certificates=exec_certs.get('server-cert'),
 +            private_key=exec_certs.get('client-key'),
 +            certificate_chain=exec_certs.get('client-cert'))
++
 +        action_certs = {}
 +        for key in ['client-cert', 'client-key', 'server-cert']:
 +            if key in config.action_service:
 +                with open(config.action_service[key], 'rb') as f:
 +                    action_certs[key] = f.read()
++
          if config.action_service:
              self.action_url = config.action_service['url']
 +            self.action_credentials = grpc.ssl_channel_credentials(
 +                root_certificates=action_certs.get('server-cert'),
 +                private_key=action_certs.get('client-key'),
 +                certificate_chain=action_certs.get('client-cert'))
          else:
              self.action_url = None
 +            self.action_credentials = None
          self.server_instance = config.exec_service.get('instance', None)
          self.storage_instance = config.storage_service.get('instance', None)
@@ -81,7 +103,7 @@ class SandboxRemote(Sandbox):
          self._get_context().message(Message(None, MessageType.INFO, msg))
      @staticmethod
 -    def specs_from_config_node(config_node, basedir):
 +    def specs_from_config_node(config_node, basedir=None):
          def require_node(config, keyname):
              val = config.get(keyname)
@@ -109,10 +131,10 @@ class SandboxRemote(Sandbox):
          remote_exec_storage_config = require_node(remote_config, 'storage-service')
          remote_exec_action_config = remote_config.get('action-cache-service', {})
 -        _yaml.node_validate(remote_exec_service_config, ['url', 'instance'])
 +        _yaml.node_validate(remote_exec_service_config, ['url', 'instance'] + tls_keys)
          _yaml.node_validate(remote_exec_storage_config, ['url', 'instance'] + tls_keys)
          if remote_exec_action_config:
 -            _yaml.node_validate(remote_exec_action_config, ['url'])
 +            _yaml.node_validate(remote_exec_action_config, ['url'] + tls_keys)
          else:
              remote_config['action-service'] = None
@@ -135,6 +157,19 @@ class SandboxRemote(Sandbox):
                                        "remote-execution configuration. Your config is missing '{}'."
                                        .format(str(provenance), tls_keys, key))
 +        def resolve_path(path):
 +            if basedir and path:
 +                return os.path.join(basedir, path)
 +            else:
 +                return path
++
 +        for key in tls_keys:
 +            for d in (remote_config['execution-service'],
 +                      remote_config['storage-service'],
 +                      remote_exec_action_config):
 +                if key in d:
 +                    d[key] = resolve_path(d[key])
++
          spec = RemoteExecutionSpec(remote_config['execution-service'],
                                     remote_config['storage-service'],
                                     remote_exec_action_config)
@@ -295,6 +330,8 @@ class SandboxRemote(Sandbox):
                                 "for example: http://buildservice:50051.")
          if url.scheme == 'http':
              channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
 +        elif url.scheme == 'https':
 +            channel = grpc.secure_channel('{}:{}'.format(url.hostname, url.port), self.exec_credentials)
          else:
              raise SandboxError("Remote execution currently only supports the 'http' protocol "
                                 "and '{}' was supplied.".format(url.scheme))
@@ -352,11 +389,11 @@ class SandboxRemote(Sandbox):
          if not url.port:
              raise SandboxError("You must supply a protocol and port number in the action-cache-service url, "
                                 "for example: http://buildservice:50051.")
 -        if not url.scheme == "http":
 -            raise SandboxError("Currently only support http for the action cache"
 -                               "and {} was supplied".format(url.scheme))
 +        if url.scheme == 'http':
 +            channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
 +        elif url.scheme == 'https':
 +            channel = grpc.secure_channel('{}:{}'.format(url.hostname, url.port), self.action_credentials)
 -        channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
          request = remote_execution_pb2.GetActionResultRequest(action_digest=action_digest)
          stub = remote_execution_pb2_grpc.ActionCacheStub(channel)
          try:

doc/source/format_project.rst

@@ -218,6 +218,7 @@ The use of ports are required to distinguish between pull only access and
  push/pull access. For information regarding the server/client certificates
  and keys, please see: :ref:`Key pair for the server <server_authentication>`.
 +.. _project_remote_execution:
  Remote execution
  ~~~~~~~~~~~~~~~~
@@ -243,9 +244,6 @@ using the `remote-execution` option:
      action-cache-service:
        url: http://bar.action.com:50052
 -The execution-service part of remote execution does not support encrypted
 -connections yet, so the protocol must always be http.
+-
  storage-service specifies a remote CAS store and the parameters are the
  same as those used to specify an :ref:`artifact server <artifacts>`.
@@ -268,6 +266,9 @@ instance names.
  The Remote Execution API can be found via https://github.com/bazelbuild/remote-apis.
 +Remote execution configuration can be also provided in the `user
 +configuration <user_config_remote_execution>`.
++
  .. _project_essentials_mirrors:
  Mirrors

doc/source/using_config.rst

@@ -100,6 +100,54 @@ pull only access and push/pull access. For information regarding this and the
  server/client certificates and keys, please see:
  :ref:`Key pair for the server <server_authentication>`.
 +.. _user_config_remote_execution:
++
 +Remote execution
 +~~~~~~~~~~~~~~~~
++
 +The same configuration for :ref:`remote execution <project_remote_execution>`
 +in ``project.conf`` can be provided in the user configuation.
++
 +There is only one remote execution configuration used per project.
++
 +The project overrides will be taken in priority. The global
 +configuration will be used as fallback.
++
 +1. Global remote execution fallback:
++
 +.. code:: yaml
++
 +  remote-execution:
 +    execution-service:
 +      url: http://execution.fallback.example.com:50051
 +      instance-name: main
 +    storage-service:
 +      url: https://storage.fallback.example.com:11002/
 +      server-cert: /keys/server.crt
 +      client-cert: /keys/client.crt
 +      client-key: /keys/client.key
 +      instance-name: main
 +    action-cache-service:
 +      url: http://action.flalback.example.com:50052
++
 +2. Project override:
++
 +.. code:: yaml
++
 +  projects:
 +    some_project:
 +      remote-execution:
 +        execution-service:
 +          url: http://execution.some_project.example.com:50051
 +          instance-name: main
 +        storage-service:
 +          url: https://storage.some_project.example.com:11002/
 +          server-cert: /some_project_keys/server.crt
 +          client-cert: /some_project_keys/client.crt
 +          client-key: /some_project_keys/client.key
 +          instance-name: main
 +        action-cache-service:
 +          url: http://action.some_project.example.com:50052
  Strict build plan

tox.ini

 +#
 +# Tox global configuration
 +#
  [tox]
  envlist = py35,py36,py37
  skip_missing_interpreters = true
 +#
 +# Defaults for all environments
 +#
 +# Anything specified here is iherited by the sections
 +#
  [testenv]
 -commands = pytest {posargs}
 +commands =
 +    pytest --basetemp {envtmpdir} {posargs} {toxinidir}
 +    mkdir -p {toxinidir}/.coverage-reports
  deps =
      -rrequirements/requirements.txt
      -rrequirements/dev-requirements.txt
@@ -13,6 +23,32 @@ passenv =
      GI_TYPELIB_PATH
      INTEGRATION_CACHE
 +#
 +# These keys are not inherited by any other sections
 +#
 +setenv =
 +    py{35,36,37}: COVERAGE_FILE = .coverage.{env:COVERAGE_PREFIX:}{envname}
 +whitelist_externals =
 +    py{35,36,37}:
 +        cp
 +	mkdir
++
 +#
 +# Coverage reporting
 +#
 +[testenv:coverage]
 +commands =
 +    - coverage combine --rcfile={toxinidir}/.coveragerc {toxinidir}/.coverage-reports/
 +    coverage report --rcfile={toxinidir}/.coveragerc -m
 +deps =
 +    -rrequirements/requirements.txt
 +    -rrequirements/dev-requirements.txt
 +setenv =
 +    COVERAGE_FILE = {toxinidir}/.coverage-reports/.coverage
++
 +#
 +# Running linters
 +#
  [testenv:lint]
  commands =
      pycodestyle
@@ -22,6 +58,9 @@ deps =
      -rrequirements/dev-requirements.txt
      -rrequirements/plugin-requirements.txt
 +#
 +# Building documentation
 +#
  [testenv:docs]
  commands =
      make -C doc

[Notes] [Git][BuildStream/buildstream][tristan/detox-tests] 11 commits: Use relative path to project directory for remote execution certificates/keys

Tristan Van Berkom pushed to branch tristan/detox-tests at BuildStream / buildstream

Commits:

11 changed files:

Changes: