[Notes] [Git][BuildStream/buildstream][jjardon/freedesktop-sdk

Valentin David pushed to branch jjardon/freedesktop-sdk_latest at BuildStream / buildstream

Commits:

eb0dbcfc

by Valentin David at 2019-01-09T13:56:58Z

buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY

ScriptElement does not use Sandbox.get_directory. It works using it
with remote execution.

Fixes #850

f7681925

by Valentin David at 2019-01-09T14:44:22Z

Merge branch 'valentindavid/script_virtual_directory' into 'master'

buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY

Closes #850

See merge request BuildStream/buildstream!1047

f87d1c93

by Jim MacArthur at 2019-01-09T15:01:18Z

sandboxremote.py: Rename 'instance_name' option to 'instance-name'

It was 'instance-name' in the documentation.

ed8bc5bc

by Jim MacArthur at 2019-01-09T15:30:28Z

Merge branch 'jmac/rename_instance_name' into 'master'

Rename 'instance_name' option to 'instance-name'

See merge request BuildStream/buildstream!1048

f29a0995

by Tristan Van Berkom at 2019-01-09T19:51:14Z

Support running test environments in parallel with `detox`

This patch namespaces the test temp directory and the output
coverage report file with the name of the environment under test,
such that separately run tests do not access the same files.

When running tests without tox, directly through setup.py,
then the tmp directory will still be `./tmp`.

  * .gitignore: Added new .coverage-reports/ directory

  * .gitlab-ci.yml: Rely on tox to combine and report coverage, only
    tell tox about the COVERAGE_PREFIX so that results can be namespaced
    by CI job name.

    This change also publishes the sources and final combined `.coverage`
    file in an output gitlab artifact for inspection, and lists some missing
    dependencies to the `coverage` job.

  * tox.ini: Add comments and refactor main [testenv] section so that
    other environments dont inherit too much unrelated cruft.

    Generate the coverate reports in the respective {envtmpdir} so that
    all per-process coverage files are prefixed with a full path, ensuring
    that concurrent runs don't mix reports and addressing concerns
    raised in #844.

    Also implemented new `tox -e coverage` environment to combine
    any found coverage and print a report.

  * .coveragerc: Omit .tox/ directory from coverage stats

Fixes issue #844

d364ad02

by Tristan Van Berkom at 2019-01-09T19:52:22Z

CONTRIBUTING.rst: Updated to mention new `coverage` tox environment.

Also point out that it is possible to run test environments in
parallel using the `detox` tool.

1e352434

by Tristan Van Berkom at 2019-01-09T20:36:06Z

Merge branch 'tristan/detox-tests' into 'master'

Allow using detox, and fix/refactor collection of coverage reports

Closes #844

See merge request BuildStream/buildstream!1051

116c1070

by Tristan Van Berkom at 2019-01-09T20:39:15Z

.coveragerc: Omit some things which are irrelevant to cover.

  * Omit versioneer's _version.py

  * Omit our __main__.py which is used only internally for generating
    documentation

c8f3616d

by Tristan Van Berkom at 2019-01-09T21:12:29Z

Merge branch 'tristan/refine-coverage' into 'master'

.coveragerc: Omit some things which are irrelevant to cover.

See merge request BuildStream/buildstream!1052

c87bb592

by Jürg Billeter at 2019-01-10T09:43:37Z

_platform/platform.py: Add canonicalize_arch() method

26e33346

by Jürg Billeter at 2019-01-10T12:50:15Z

_options/optionarch.py: Accept architecture aliases

Accept common architecture aliases for arch options instead of only
accepting the canonicalized, OS-independent architecture name. This
restores compatibility with existing projects and may simplify option
handling for projects that only target a single OS (and thus, do not
need OS-independent architecture names).

06deb4c4

by Jürg Billeter at 2019-01-10T12:50:15Z

element.py: Accept architecture aliases for sandbox config

Accept common architecture aliases for the sandbox config for
consistency with arch options.

0c2a66b3

by Jürg Billeter at 2019-01-10T13:06:02Z

tests/format/optionarch.py: Add tests for architecture aliases

f86bc760

by Jürg Billeter at 2019-01-10T13:34:56Z

Merge branch 'juerg/arch' into 'master'

Accept architecture aliases

See merge request BuildStream/buildstream!1034

347eb34e

by Angelos Evripiotis at 2019-01-10T14:38:23Z

contributing: fix 'oprtation' and some other typos

6bc27bc1

by Angelos Evripiotis at 2019-01-10T15:07:46Z

Merge branch 'aevri/contrib-typos' into 'master'

contributing: fix 'oprtation' and some other typos

See merge request BuildStream/buildstream!1055

0b83d024

by Tristan Van Berkom at 2019-01-10T20:02:50Z

_pipeline.py: Fix the planner to retain the original order when depth sorting

The algorithm adds elements to a dictionary and then sorts the dictionary
by the discovered depths while recursing - using an OrderedDict is enough
to ensure a stable order.

This fixes `bst show --deps plan ...` reporting different results on
each invocation.

This fixes an aspect of #824

630e26f1

by Tristan Van Berkom at 2019-01-10T20:03:52Z

testutils/runcli.py: Added Result.get_start_order()

Added a function to report the list of elements which appeared in
a given queue in the order of their first appearance.

d1d7de57

by Tristan Van Berkom at 2019-01-11T04:13:16Z

tests/frontend/order.py: Adding tests ensuring stable element processing order

For each sample project and expected result order, this test ensures that:

  * bst show --deps plan: Always shows the expected build order.

  * bst source fetch: Always executes in the expected build order.

  * bst build: Always builds in the expected order (disabled).

The build part of the test is currently disabled as the scheduler
seems to be messing with the order even in a `--builders 1` scenario.

276b8d48

by Tristan Van Berkom at 2019-01-11T04:42:24Z

Merge branch 'tristan/element-processing-order' into 'master'

Make build plan element list stable

See merge request BuildStream/buildstream!1058

fc3dcec8

by Angelos Evripiotis at 2019-01-11T16:45:00Z

dev-requirements: pytest-cov==2.6.1 for '--no-cov'

Bump the version of pytest-cov, so that we pick up the fix for
the '--no-cov' option breaking in pytest 4.0:

    https://github.com/pytest-dev/pytest-cov/pull/230

This was the error you would get prior to this version:

    _pytest.warning_types.RemovedInPytest4Warning: config.warn has been
    deprecated, use warnings.warn instead

Although it says 'deprecated', it did result in
termination with stack trace.

f86b7ff3

by Angelos Evripiotis at 2019-01-11T16:45:00Z

contributing: '--last-failed' and '--no-cov' tips

To help new folks get to grips with pytest, add some tips on other
frequently used features.

d983f231

by Angelos Evripiotis at 2019-01-11T17:19:06Z

Merge branch 'aevri/tox_no_cov' into 'master'

pytest-cov==2.6.1, contributing: '--last-failed' and '--no-cov' tips

See merge request BuildStream/buildstream!1059

829a2b93

by Tristan Van Berkom at 2019-01-12T21:32:13Z

_scheduler/queues/queue.py: Put elements in the skip list, not jobs

Minor correction, looks like we're not observing this queue
otherwise we'd be seeing crashes.

993e30ae

by Tristan Van Berkom at 2019-01-12T22:02:55Z

Merge branch 'tristan/fix-skipped-elements' into 'master'

_scheduler/queues/queue.py: Put elements in the skip list, not jobs

See merge request BuildStream/buildstream!1063

32732e01

by Javier Jardón at 2019-01-14T09:04:01Z

.gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25

20 changed files:

.coveragerc
.gitignore
.gitlab-ci.yml
CONTRIBUTING.rst
buildstream/_artifactcache/cascache.py
buildstream/_options/optionarch.py
buildstream/_pipeline.py
buildstream/_platform/platform.py
buildstream/_scheduler/queues/queue.py
buildstream/element.py
buildstream/plugins/elements/script.py
requirements/dev-requirements.txt
+ tests/format/option-arch-alias/element.bst
+ tests/format/option-arch-alias/project.conf
+ tests/format/option-arch-unknown/element.bst
+ tests/format/option-arch-unknown/project.conf
tests/format/optionarch.py
+ tests/frontend/order.py
tests/testutils/runcli.py
tox.ini

Changes:

.coveragerc

@@ -4,11 +4,15 @@ include =
    */buildstream/*
  omit =
 -  # Omit profiling helper module
 +  # Omit some internals
    */buildstream/_profile.py
 +  */buildstream/__main__.py
 +  */buildstream/_version.py
    # Omit generated code
    */buildstream/_protos/*
    */.eggs/*
 +  # Omit .tox directory
 +  */.tox/*
  [report]
  show_missing = True

.gitignore

@@ -13,11 +13,12 @@ tests/**/*.pyc
  integration-cache/
  tmp
  .coverage
 +.coverage-reports/
  .coverage.*
  .cache
  .pytest_cache/
  *.bst/
 -.tox
 +.tox/
  # Pycache, in case buildstream is ran directly from within the source
  # tree

.gitlab-ci.yml

@@ -13,6 +13,7 @@ variables:
    PYTEST_ADDOPTS: "--color=yes"
    INTEGRATION_CACHE: "${CI_PROJECT_DIR}/cache/integration-cache"
    TEST_COMMAND: "tox -- --color=yes --integration"
 +  COVERAGE_PREFIX: "${CI_JOB_NAME}."
  #####################################################
@@ -24,9 +25,6 @@ variables:
  .tests-template: &tests
    stage: test
 -  variables:
 -    COVERAGE_DIR: coverage-linux
+-
    before_script:
    # Diagnostics
    - mount
@@ -40,14 +38,11 @@ variables:
    - su buildstream -c "${TEST_COMMAND}"
    after_script:
 -  # Collect our reports
 -  - mkdir -p ${COVERAGE_DIR}
 -  - cp .coverage ${COVERAGE_DIR}/coverage."${CI_JOB_NAME}"
    except:
    - schedules
    artifacts:
      paths:
 -    - ${COVERAGE_DIR}
 +    - .coverage-reports
  tests-debian-9:
    image: buildstream/testsuite-debian:9-5da27168-32c47d1c
@@ -83,7 +78,6 @@ tests-unix:
    <<: *tests
    variables:
      BST_FORCE_BACKEND: "unix"
 -    COVERAGE_DIR: coverage-unix
    script:
@@ -147,7 +141,7 @@ docs:
    variables:
      BST_EXT_URL: git+https://gitlab.com/BuildStream/bst-external.git
      BST_EXT_REF: 573843768f4d297f85dc3067465b3c7519a8dcc3 # 0.7.0
 -    FD_SDK_REF: 612f66e218445eee2b1a9d7dd27c9caba571612e # freedesktop-sdk-18.08.19-54-g612f66e2
 +    FD_SDK_REF: freedesktop-sdk-18.08.25-0-g250939d465d6dd7768a215f1fa59c4a3412fc337
    before_script:
    - |
      mkdir -p "${HOME}/.config"
@@ -239,22 +233,22 @@ coverage:
    stage: post
    coverage: '/TOTAL +\d+ +\d+ +(\d+\.\d+)%/'
    script:
 -    - pip3 install -r requirements/requirements.txt -r requirements/dev-requirements.txt
 -    - pip3 install --no-index .
 -    - mkdir report
 -    - cd report
 -    - cp ../coverage-unix/coverage.* .
 -    - cp ../coverage-linux/coverage.* .
 -    - ls coverage.*
 -    - coverage combine --rcfile=../.coveragerc -a coverage.*
 -    - coverage report --rcfile=../.coveragerc -m
 +    - cp -a .coverage-reports/ ./coverage-sources
 +    - tox -e coverage
 +    - cp -a .coverage-reports/ ./coverage-report
    dependencies:
    - tests-debian-9
    - tests-fedora-27
    - tests-fedora-28
 +  - tests-fedora-missing-deps
 +  - tests-ubuntu-18.04
    - tests-unix
    except:
    - schedules
 +  artifacts:
 +    paths:
 +    - coverage-sources/
 +    - coverage-report/
  # Deploy, only for merges which land on master branch.
+ #

CONTRIBUTING.rst

@@ -553,7 +553,7 @@ One problem which arises from this is that we end up having symbols
  which are *public* according to the :ref:`rules discussed in the previous section
  <contributing_public_and_private>`, but must be hidden away from the
  *"Public API Surface"*. For example, BuildStream internal classes need
 -to invoke methods on the ``Element`` and ``Source`` classes, wheras these
 +to invoke methods on the ``Element`` and ``Source`` classes, whereas these
  methods need to be hidden from the *"Public API Surface"*.
  This is where BuildStream deviates from the PEP-8 standard for public
@@ -631,7 +631,7 @@ An element plugin will derive from Element by importing::
    from buildstream import Element
 -When importing utilities specifically, dont import function names
 +When importing utilities specifically, don't import function names
  from there, instead import the module itself::
    from . import utils
@@ -737,7 +737,7 @@ Abstract methods
  ~~~~~~~~~~~~~~~~
  In BuildStream, an *"Abstract Method"* is a bit of a misnomer and does
  not match up to how Python defines abstract methods, we need to seek out
 -a new nomanclature to refer to these methods.
 +a new nomenclature to refer to these methods.
  In Python, an *"Abstract Method"* is a method which **must** be
  implemented by a subclass, whereas all methods in Python can be
@@ -960,7 +960,7 @@ possible, and avoid any cyclic relationships in modules.
  For instance, the ``Source`` objects are owned by ``Element``
  objects in the BuildStream data model, and as such the ``Element``
  will delegate some activities to the ``Source`` objects in its
 -possesion. The ``Source`` objects should however never call functions
 +possession. The ``Source`` objects should however never call functions
  on the ``Element`` object, nor should the ``Source`` object itself
  have any understanding of what an ``Element`` is.
@@ -1223,7 +1223,7 @@ For further information about using the reStructuredText with sphinx, please see
  Building Docs
  ~~~~~~~~~~~~~
  Before you can build the docs, you will end to ensure that you have installed
 -the required :ref:`buid dependencies <contributing_build_deps>` as mentioned
 +the required :ref:`build dependencies <contributing_build_deps>` as mentioned
  in the testing section above.
  To build the documentation, just run the following::
@@ -1365,7 +1365,7 @@ Structure of an example
  '''''''''''''''''''''''
  The :ref:`tutorial <tutorial>` and the :ref:`examples <examples>` sections
  of the documentation contain a series of sample projects, each chapter in
 -the tutoral, or standalone example uses a sample project.
 +the tutorial, or standalone example uses a sample project.
  Here is the the structure for adding new examples and tutorial chapters.
@@ -1471,8 +1471,8 @@ Installing build dependencies
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Some of BuildStream's dependencies have non-python build dependencies. When
  running tests with ``tox``, you will first need to install these dependencies.
 -Exact steps to install these will depend on your oprtation systemm. Commands
 -for installing them for some common distributions are lised below.
 +Exact steps to install these will depend on your operating system. Commands
 +for installing them for some common distributions are listed below.
  For Fedora-based systems::
@@ -1498,6 +1498,13 @@ option when running tox::
    tox -e py37
 +If you would like to test and lint at the same time, or if you do have multiple
 +python versions installed and would like to test against multiple versions, then
 +we recommend using `detox <https://github.com/tox-dev/detox>`_, just run it with
 +the same arguments you would give `tox`::
++
 +  detox -e lint,py36,py37
++
  Linting is performed separately from testing. In order to run the linting step which
  consists of running the ``pycodestyle`` and ``pylint`` tools, run the following::
@@ -1527,17 +1534,27 @@ You can always abort on the first failure by running::
    tox -- -x
 +Similarly, you may also be interested in the ``--last-failed`` and
 +``--failed-first`` options as per the
 +`pytest cache <https://docs.pytest.org/en/latest/cache.html>`_ documentation.
++
  If you want to run a specific test or a group of tests, you
  can specify a prefix to match. E.g. if you want to run all of
  the frontend tests you can do::
    tox -- tests/frontend/
 -Specific tests can be chosen by using the :: delimeter after the test module.
 +Specific tests can be chosen by using the :: delimiter after the test module.
  If you wanted to run the test_build_track test within frontend/buildtrack.py you could do::
    tox -- tests/frontend/buildtrack.py::test_build_track
 +When running only a few tests, you may find the coverage and timing output
 +excessive, there are options to trim them. Note that coverage step will fail.
 +Here is an example::
++
 +  tox -- --no-cov --durations=1 tests/frontend/buildtrack.py::test_build_track
++
  We also have a set of slow integration tests that are disabled by
  default - you will notice most of them marked with SKIP in the pytest
  output. To run them, you can use::
@@ -1553,7 +1570,7 @@ can run ``tox`` with ``-r`` or  ``--recreate`` option.
  .. note::
     By default, we do not allow use of site packages in our ``tox``
 -   confguration to enable running the tests in an isolated environment.
 +   configuration to enable running the tests in an isolated environment.
     If you need to enable use of site packages for whatever reason, you can
     do so by passing the ``--sitepackages`` option to ``tox``. Also, you will
     not need to install any of the build dependencies mentioned above if you
@@ -1574,10 +1591,23 @@ can run ``tox`` with ``-r`` or  ``--recreate`` option.
       ./setup.py test --addopts 'tests/frontend/buildtrack.py::test_build_track'
 +Observing coverage
 +~~~~~~~~~~~~~~~~~~
 +Once you have run the tests using `tox` (or `detox`), some coverage reports will
 +have been left behind.
++
 +To view the coverage report of the last test run, simply run::
++
 +  tox -e coverage
++
 +This will collate any reports from separate python environments that may be
 +under test before displaying the combined coverage.
++
++
  Adding tests
  ~~~~~~~~~~~~
  Tests are found in the tests subdirectory, inside of which
 -there is a separarate directory for each *domain* of tests.
 +there is a separate directory for each *domain* of tests.
  All tests are collected as::
    tests/*/*.py

buildstream/_artifactcache/cascache.py

@@ -53,7 +53,7 @@ class CASRemoteSpec(namedtuple('CASRemoteSpec', 'url push server_cert client_key
+     #
      @staticmethod
      def _new_from_config_node(spec_node, basedir=None):
 -        _yaml.node_validate(spec_node, ['url', 'push', 'server-cert', 'client-key', 'client-cert', 'instance_name'])
 +        _yaml.node_validate(spec_node, ['url', 'push', 'server-cert', 'client-key', 'client-cert', 'instance-name'])
          url = _yaml.node_get(spec_node, str, 'url')
          push = _yaml.node_get(spec_node, bool, 'push', default_value=False)
          if not url:
@@ -61,7 +61,7 @@ class CASRemoteSpec(namedtuple('CASRemoteSpec', 'url push server_cert client_key
              raise LoadError(LoadErrorReason.INVALID_DATA,
                              "{}: empty artifact cache URL".format(provenance))
 -        instance_name = _yaml.node_get(spec_node, str, 'instance_name', default_value=None)
 +        instance_name = _yaml.node_get(spec_node, str, 'instance-name', default_value=None)
          server_cert = _yaml.node_get(spec_node, str, 'server-cert', default_value=None)
          if server_cert and basedir:

buildstream/_options/optionarch.py

@@ -17,6 +17,8 @@
  #  Authors:
  #        Tristan Van Berkom <tristan vanberkom codethink co uk>
 +from .. import _yaml
 +from .._exceptions import LoadError, LoadErrorReason, PlatformError
  from .._platform import Platform
  from .optionenum import OptionEnum
@@ -41,7 +43,34 @@ class OptionArch(OptionEnum):
          super(OptionArch, self).load(node, allow_default_definition=False)
      def load_default_value(self, node):
 -        return Platform.get_host_arch()
 +        arch = Platform.get_host_arch()
++
 +        default_value = None
++
 +        for index, value in enumerate(self.values):
 +            try:
 +                canonical_value = Platform.canonicalize_arch(value)
 +                if default_value is None and canonical_value == arch:
 +                    default_value = value
 +                    # Do not terminate the loop early to ensure we validate
 +                    # all values in the list.
 +            except PlatformError as e:
 +                provenance = _yaml.node_get_provenance(node, key='values', indices=[index])
 +                prefix = ""
 +                if provenance:
 +                    prefix = "{}: ".format(provenance)
 +                raise LoadError(LoadErrorReason.INVALID_DATA,
 +                                "{}Invalid value for {} option '{}': {}"
 +                                .format(prefix, self.OPTION_TYPE, self.name, e))
++
 +        if default_value is None:
 +            # Host architecture is not supported by the project.
 +            # Do not raise an error here as the user may override it.
 +            # If the user does not override it, an error will be raised
 +            # by resolve()/validate().
 +            default_value = arch
++
 +        return default_value
      def resolve(self):

buildstream/_pipeline.py

@@ -22,6 +22,7 @@
  import os
  import itertools
  from operator import itemgetter
 +from collections import OrderedDict
  from ._exceptions import PipelineError
  from ._message import Message, MessageType
@@ -479,7 +480,7 @@ class Pipeline():
+ #
  class _Planner():
      def __init__(self):
 -        self.depth_map = {}
 +        self.depth_map = OrderedDict()
          self.visiting_elements = set()
      # Here we want to traverse the same element more than once when

buildstream/_platform/platform.py

@@ -77,20 +77,17 @@ class Platform():
      def get_host_os():
          return os.uname()[0]
 -    # get_host_arch():
 +    # canonicalize_arch():
+     #
 -    # This returns the architecture of the host machine. The possible values
 -    # map from uname -m in order to be a OS independent list.
 +    # This returns the canonical, OS-independent architecture name
 +    # or raises a PlatformError if the architecture is unknown.
+     #
 -    # Returns:
 -    #    (string): String representing the architecture
      @staticmethod
 -    def get_host_arch():
 -        # get the hardware identifier from uname
 -        uname_machine = os.uname()[4]
 -        uname_to_arch = {
 +    def canonicalize_arch(arch):
 +        aliases = {
 +            "aarch32": "aarch32",
              "aarch64": "aarch64",
 -            "aarch64_be": "aarch64-be",
 +            "aarch64-be": "aarch64-be",
              "amd64": "x86-64",
              "arm": "aarch32",
              "armv8l": "aarch64",
@@ -99,17 +96,34 @@ class Platform():
              "i486": "x86-32",
              "i586": "x86-32",
              "i686": "x86-32",
 +            "power-isa-be": "power-isa-be",
 +            "power-isa-le": "power-isa-le",
              "ppc64": "power-isa-be",
              "ppc64le": "power-isa-le",
              "sparc": "sparc-v9",
              "sparc64": "sparc-v9",
 -            "x86_64": "x86-64"
 +            "sparc-v9": "sparc-v9",
 +            "x86-32": "x86-32",
 +            "x86-64": "x86-64"
+         }
++
          try:
 -            return uname_to_arch[uname_machine]
 +            return aliases[arch.replace('_', '-')]
          except KeyError:
 -            raise PlatformError("uname gave unsupported machine architecture: {}"
 -                                .format(uname_machine))
 +            raise PlatformError("Unknown architecture: {}".format(arch))
++
 +    # get_host_arch():
 +    #
 +    # This returns the architecture of the host machine. The possible values
 +    # map from uname -m in order to be a OS independent list.
 +    #
 +    # Returns:
 +    #    (string): String representing the architecture
 +    @staticmethod
 +    def get_host_arch():
 +        # get the hardware identifier from uname
 +        uname_machine = os.uname()[4]
 +        return Platform.canonicalize_arch(uname_machine)
      ##################################################################
      #                        Sandbox functions                       #

buildstream/_scheduler/queues/queue.py

@@ -170,9 +170,9 @@ class Queue():
          skip = [job for job in jobs if self.status(job.element) == QueueStatus.SKIP]
          wait = [job for job in jobs if job not in skip]
 +        self.skipped_elements.extend([job.element for job in skip])
          self._wait_queue.extend(wait)
          self._done_queue.extend(skip)
 -        self.skipped_elements.extend(skip)
      # dequeue()
+     #

buildstream/element.py

@@ -2441,11 +2441,17 @@ class Element(Plugin):
          # Sandbox config, unlike others, has fixed members so we should validate them
          _yaml.node_validate(sandbox_config, ['build-uid', 'build-gid', 'build-os', 'build-arch'])
 +        build_arch = self.node_get_member(sandbox_config, str, 'build-arch', default=None)
 +        if build_arch:
 +            build_arch = Platform.canonicalize_arch(build_arch)
 +        else:
 +            build_arch = host_arch
++
          return SandboxConfig(
              self.node_get_member(sandbox_config, int, 'build-uid'),
              self.node_get_member(sandbox_config, int, 'build-gid'),
              self.node_get_member(sandbox_config, str, 'build-os', default=host_os),
 -            self.node_get_member(sandbox_config, str, 'build-arch', default=host_arch))
 +            build_arch)
      # This makes a special exception for the split rules, which
      # elements may extend but whos defaults are defined in the project.

buildstream/plugins/elements/script.py

@@ -42,6 +42,9 @@ import buildstream
  class ScriptElement(buildstream.ScriptElement):
      # pylint: disable=attribute-defined-outside-init
 +    # This plugin has been modified to avoid the use of Sandbox.get_directory
 +    BST_VIRTUAL_DIRECTORY = True
++
      def configure(self, node):
          for n in self.node_get_member(node, list, 'layout', []):
              dst = self.node_subst_member(n, 'destination')

requirements/dev-requirements.txt

@@ -2,7 +2,7 @@ coverage==4.4
  pylint==2.2.2
  pycodestyle==2.4.0
  pytest==4.0.2
 -pytest-cov==2.6.0
 +pytest-cov==2.6.1
  pytest-datafiles==2.0
  pytest-env==0.6.2
  pytest-xdist==1.25.0

tests/format/option-arch-alias/element.bst

 +kind: autotools
 +variables:
 +  result: "Nothing"
 +  (?):
 +  - machine_arch == "arm":
 +      result: "Army"
 +  - machine_arch == "x86_64":
 +      result: "X86-64y"

tests/format/option-arch-alias/project.conf

 +name: test
++
 +options:
 +  machine_arch:
 +    type: arch
 +    description: The machine architecture
 +    values:
 +    - arm
 +    - x86_64

tests/format/option-arch-unknown/element.bst

 +kind: autotools
 +variables:
 +  result: "Nothing"
 +  (?):
 +  - machine_arch == "aarch32":
 +      result: "Army"
 +  - machine_arch == "aarch64":
 +      result: "Aarchy"
 +  - machine_arch == "x86-128":
 +      result: "X86-128y"

tests/format/option-arch-unknown/project.conf

 +name: test
++
 +options:
 +  machine_arch:
 +    type: arch
 +    description: The machine architecture
 +    values:
 +    - aarch32
 +    - aarch64
 +    - x86-128

tests/format/optionarch.py

@@ -75,3 +75,47 @@ def test_unsupported_arch(cli, datafiles):
          ])
          result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_alias(cli, datafiles):
++
 +    with override_uname_arch("arm"):
 +        project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch-alias')
 +        result = cli.run(project=project, silent=True, args=[
 +            'show',
 +            '--deps', 'none',
 +            '--format', '%{vars}',
 +            'element.bst'
 +        ])
++
 +        result.assert_success()
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_unknown_host_arch(cli, datafiles):
++
 +    with override_uname_arch("x86_128"):
 +        project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch')
 +        result = cli.run(project=project, silent=True, args=[
 +            'show',
 +            '--deps', 'none',
 +            '--format', '%{vars}',
 +            'element.bst'
 +        ])
++
 +        result.assert_main_error(ErrorDomain.PLATFORM, None)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_unknown_project_arch(cli, datafiles):
++
 +    project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch-unknown')
 +    result = cli.run(project=project, silent=True, args=[
 +        'show',
 +        '--deps', 'none',
 +        '--format', '%{vars}',
 +        'element.bst'
 +    ])
++
 +    result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA)

tests/frontend/order.py

 +import os
++
 +import pytest
 +from tests.testutils import cli, create_repo
++
 +from buildstream import _yaml
++
 +# Project directory
 +DATA_DIR = os.path.join(
 +    os.path.dirname(os.path.realpath(__file__)),
 +    "project",
 +)
++
++
 +def create_element(repo, name, path, dependencies, ref=None):
 +    element = {
 +        'kind': 'import',
 +        'sources': [
 +            repo.source_config(ref=ref)
 +        ],
 +        'depends': dependencies
 +    }
 +    _yaml.dump(element, os.path.join(path, name))
++
++
 +# This tests a variety of scenarios and checks that the order in
 +# which things are processed remains stable.
 +#
 +# This is especially important in order to ensure that our
 +# depth sorting and optimization of which elements should be
 +# processed first is doing it's job right, and that we are
 +# promoting elements to the build queue as soon as possible
 +#
 +# Parameters:
 +#    targets (target elements): The targets to invoke bst with
 +#    template (dict): The project template dictionary, for create_element()
 +#    expected (list): A list of element names in the expected order
 +#
 +@pytest.mark.datafiles(os.path.join(DATA_DIR))
 +@pytest.mark.parametrize("target,template,expected", [
 +    # First simple test
 +    ('3.bst', {
 +        '0.bst': ['1.bst'],
 +        '1.bst': [],
 +        '2.bst': ['0.bst'],
 +        '3.bst': ['0.bst', '1.bst', '2.bst']
 +    }, ['1.bst', '0.bst', '2.bst', '3.bst']),
++
 +    # A more complicated test with build of build dependencies
 +    ('target.bst', {
 +        'a.bst': [],
 +        'base.bst': [],
 +        'timezones.bst': [],
 +        'middleware.bst': [{'filename': 'base.bst', 'type': 'build'}],
 +        'app.bst': [{'filename': 'middleware.bst', 'type': 'build'}],
 +        'target.bst': ['a.bst', 'base.bst', 'middleware.bst', 'app.bst', 'timezones.bst']
 +    }, ['base.bst', 'middleware.bst', 'a.bst', 'app.bst', 'timezones.bst', 'target.bst']),
 +])
 +@pytest.mark.parametrize("operation", [('show'), ('fetch'), ('build')])
 +def test_order(cli, datafiles, tmpdir, operation, target, template, expected):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    dev_files_path = os.path.join(project, 'files', 'dev-files')
 +    element_path = os.path.join(project, 'elements')
++
 +    # FIXME: Remove this when the test passes reliably.
 +    #
 +    #        There is no reason why the order should not
 +    #        be preserved when the builders is set to 1,
 +    #        the scheduler queue processing still seems to
 +    #        be losing the order.
 +    #
 +    if operation == 'build':
 +        pytest.skip("FIXME: This still only sometimes passes")
++
 +    # Configure to only allow one fetcher at a time, make it easy to
 +    # determine what is being planned in what order.
 +    cli.configure({
 +        'scheduler': {
 +            'fetchers': 1,
 +            'builders': 1
 +        }
 +    })
++
 +    # Build the project from the template, make import elements
 +    # all with the same repo
 +    #
 +    repo = create_repo('git', str(tmpdir))
 +    ref = repo.create(dev_files_path)
 +    for element, dependencies in template.items():
 +        create_element(repo, element, element_path, dependencies, ref=ref)
 +        repo.add_commit()
++
 +    # Run test and collect results
 +    if operation == 'show':
 +        result = cli.run(args=['show', '--deps', 'plan', '--format', '%{name}', target], project=project, silent=True)
 +        result.assert_success()
 +        results = result.output.splitlines()
 +    else:
 +        if operation == 'fetch':
 +            result = cli.run(args=['source', 'fetch', target], project=project, silent=True)
 +        else:
 +            result = cli.run(args=[operation, target], project=project, silent=True)
 +        result.assert_success()
 +        results = result.get_start_order(operation)
++
 +    # Assert the order
 +    print("Expected order: {}".format(expected))
 +    print("Observed result order: {}".format(results))
 +    assert results == expected

tests/testutils/runcli.py

@@ -167,6 +167,23 @@ class Result():
      def assert_shell_error(self, fail_message=''):
          assert self.exit_code == 1, fail_message
 +    # get_start_order()
 +    #
 +    # Gets the list of elements processed in a given queue, in the
 +    # order of their first appearances in the session.
 +    #
 +    # Args:
 +    #    activity (str): The queue activity name (like 'fetch')
 +    #
 +    # Returns:
 +    #    (list): A list of element names in the order which they first appeared in the result
 +    #
 +    def get_start_order(self, activity):
 +        results = re.findall(r'\[\s*{}:(\S+)\s*\]\s*START\s*.*\.log'.format(activity), self.stderr)
 +        if results is None:
 +            return []
 +        return list(results)
++
      # get_tracked_elements()
+     #
      # Produces a list of element names on which tracking occurred

tox.ini

 +#
 +# Tox global configuration
 +#
  [tox]
  envlist = py35,py36,py37
  skip_missing_interpreters = true
 +#
 +# Defaults for all environments
 +#
 +# Anything specified here is iherited by the sections
 +#
  [testenv]
 -commands = pytest {posargs}
 +commands =
 +    pytest --basetemp {envtmpdir} {posargs}
 +    mkdir -p .coverage-reports
 +    mv {envtmpdir}/.coverage {toxinidir}/.coverage-reports/.coverage.{env:COVERAGE_PREFIX:}{envname}
  deps =
      -rrequirements/requirements.txt
      -rrequirements/dev-requirements.txt
@@ -13,6 +24,32 @@ passenv =
      GI_TYPELIB_PATH
      INTEGRATION_CACHE
 +#
 +# These keys are not inherited by any other sections
 +#
 +setenv =
 +    py{35,36,37}: COVERAGE_FILE = {envtmpdir}/.coverage
 +whitelist_externals =
 +    py{35,36,37}:
 +        mv
 +        mkdir
++
 +#
 +# Coverage reporting
 +#
 +[testenv:coverage]
 +commands =
 +    - coverage combine --rcfile={toxinidir}/.coveragerc {toxinidir}/.coverage-reports/
 +    coverage report --rcfile={toxinidir}/.coveragerc -m
 +deps =
 +    -rrequirements/requirements.txt
 +    -rrequirements/dev-requirements.txt
 +setenv =
 +    COVERAGE_FILE = {toxinidir}/.coverage-reports/.coverage
++
 +#
 +# Running linters
 +#
  [testenv:lint]
  commands =
      pycodestyle
@@ -22,6 +59,9 @@ deps =
      -rrequirements/dev-requirements.txt
      -rrequirements/plugin-requirements.txt
 +#
 +# Building documentation
 +#
  [testenv:docs]
  commands =
      make -C doc

[Notes] [Git][BuildStream/buildstream][jjardon/freedesktop-sdk_latest] 26 commits: buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY

Valentin David pushed to branch jjardon/freedesktop-sdk_latest at BuildStream / buildstream

Commits:

20 changed files:

Changes: