[Notes] [Git][BuildStream/buildstream][aevri/tox_no_cov] 13 commits:

Angelos Evripiotis pushed to branch aevri/tox_no_cov at BuildStream / buildstream

Commits:

c87bb592

by Jürg Billeter at 2019-01-10T09:43:37Z

_platform/platform.py: Add canonicalize_arch() method

26e33346

by Jürg Billeter at 2019-01-10T12:50:15Z

_options/optionarch.py: Accept architecture aliases

Accept common architecture aliases for arch options instead of only
accepting the canonicalized, OS-independent architecture name. This
restores compatibility with existing projects and may simplify option
handling for projects that only target a single OS (and thus, do not
need OS-independent architecture names).

06deb4c4

by Jürg Billeter at 2019-01-10T12:50:15Z

element.py: Accept architecture aliases for sandbox config

Accept common architecture aliases for the sandbox config for
consistency with arch options.

0c2a66b3

by Jürg Billeter at 2019-01-10T13:06:02Z

tests/format/optionarch.py: Add tests for architecture aliases

f86bc760

by Jürg Billeter at 2019-01-10T13:34:56Z

Merge branch 'juerg/arch' into 'master'

Accept architecture aliases

See merge request BuildStream/buildstream!1034

347eb34e

by Angelos Evripiotis at 2019-01-10T14:38:23Z

contributing: fix 'oprtation' and some other typos

6bc27bc1

by Angelos Evripiotis at 2019-01-10T15:07:46Z

Merge branch 'aevri/contrib-typos' into 'master'

contributing: fix 'oprtation' and some other typos

See merge request BuildStream/buildstream!1055

0b83d024

by Tristan Van Berkom at 2019-01-10T20:02:50Z

_pipeline.py: Fix the planner to retain the original order when depth sorting

The algorithm adds elements to a dictionary and then sorts the dictionary
by the discovered depths while recursing - using an OrderedDict is enough
to ensure a stable order.

This fixes `bst show --deps plan ...` reporting different results on
each invocation.

This fixes an aspect of #824

630e26f1

by Tristan Van Berkom at 2019-01-10T20:03:52Z

testutils/runcli.py: Added Result.get_start_order()

Added a function to report the list of elements which appeared in
a given queue in the order of their first appearance.

d1d7de57

by Tristan Van Berkom at 2019-01-11T04:13:16Z

tests/frontend/order.py: Adding tests ensuring stable element processing order

For each sample project and expected result order, this test ensures that:

  * bst show --deps plan: Always shows the expected build order.

  * bst source fetch: Always executes in the expected build order.

  * bst build: Always builds in the expected order (disabled).

The build part of the test is currently disabled as the scheduler
seems to be messing with the order even in a `--builders 1` scenario.

276b8d48

by Tristan Van Berkom at 2019-01-11T04:42:24Z

Merge branch 'tristan/element-processing-order' into 'master'

Make build plan element list stable

See merge request BuildStream/buildstream!1058

fc3dcec8

by Angelos Evripiotis at 2019-01-11T16:45:00Z

dev-requirements: pytest-cov==2.6.1 for '--no-cov'

Bump the version of pytest-cov, so that we pick up the fix for
the '--no-cov' option breaking in pytest 4.0:

    https://github.com/pytest-dev/pytest-cov/pull/230

This was the error you would get prior to this version:

    _pytest.warning_types.RemovedInPytest4Warning: config.warn has been
    deprecated, use warnings.warn instead

Although it says 'deprecated', it did result in
termination with stack trace.

f86b7ff3

by Angelos Evripiotis at 2019-01-11T16:45:00Z

contributing: '--last-failed' and '--no-cov' tips

To help new folks get to grips with pytest, add some tips on other
frequently used features.

13 changed files:

CONTRIBUTING.rst
buildstream/_options/optionarch.py
buildstream/_pipeline.py
buildstream/_platform/platform.py
buildstream/element.py
requirements/dev-requirements.txt
+ tests/format/option-arch-alias/element.bst
+ tests/format/option-arch-alias/project.conf
+ tests/format/option-arch-unknown/element.bst
+ tests/format/option-arch-unknown/project.conf
tests/format/optionarch.py
+ tests/frontend/order.py
tests/testutils/runcli.py

Changes:

CONTRIBUTING.rst

@@ -553,7 +553,7 @@ One problem which arises from this is that we end up having symbols
  which are *public* according to the :ref:`rules discussed in the previous section
  <contributing_public_and_private>`, but must be hidden away from the
  *"Public API Surface"*. For example, BuildStream internal classes need
 -to invoke methods on the ``Element`` and ``Source`` classes, wheras these
 +to invoke methods on the ``Element`` and ``Source`` classes, whereas these
  methods need to be hidden from the *"Public API Surface"*.
  This is where BuildStream deviates from the PEP-8 standard for public
@@ -631,7 +631,7 @@ An element plugin will derive from Element by importing::
    from buildstream import Element
 -When importing utilities specifically, dont import function names
 +When importing utilities specifically, don't import function names
  from there, instead import the module itself::
    from . import utils
@@ -737,7 +737,7 @@ Abstract methods
  ~~~~~~~~~~~~~~~~
  In BuildStream, an *"Abstract Method"* is a bit of a misnomer and does
  not match up to how Python defines abstract methods, we need to seek out
 -a new nomanclature to refer to these methods.
 +a new nomenclature to refer to these methods.
  In Python, an *"Abstract Method"* is a method which **must** be
  implemented by a subclass, whereas all methods in Python can be
@@ -960,7 +960,7 @@ possible, and avoid any cyclic relationships in modules.
  For instance, the ``Source`` objects are owned by ``Element``
  objects in the BuildStream data model, and as such the ``Element``
  will delegate some activities to the ``Source`` objects in its
 -possesion. The ``Source`` objects should however never call functions
 +possession. The ``Source`` objects should however never call functions
  on the ``Element`` object, nor should the ``Source`` object itself
  have any understanding of what an ``Element`` is.
@@ -1223,7 +1223,7 @@ For further information about using the reStructuredText with sphinx, please see
  Building Docs
  ~~~~~~~~~~~~~
  Before you can build the docs, you will end to ensure that you have installed
 -the required :ref:`buid dependencies <contributing_build_deps>` as mentioned
 +the required :ref:`build dependencies <contributing_build_deps>` as mentioned
  in the testing section above.
  To build the documentation, just run the following::
@@ -1365,7 +1365,7 @@ Structure of an example
  '''''''''''''''''''''''
  The :ref:`tutorial <tutorial>` and the :ref:`examples <examples>` sections
  of the documentation contain a series of sample projects, each chapter in
 -the tutoral, or standalone example uses a sample project.
 +the tutorial, or standalone example uses a sample project.
  Here is the the structure for adding new examples and tutorial chapters.
@@ -1471,8 +1471,8 @@ Installing build dependencies
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Some of BuildStream's dependencies have non-python build dependencies. When
  running tests with ``tox``, you will first need to install these dependencies.
 -Exact steps to install these will depend on your oprtation systemm. Commands
 -for installing them for some common distributions are lised below.
 +Exact steps to install these will depend on your operating system. Commands
 +for installing them for some common distributions are listed below.
  For Fedora-based systems::
@@ -1534,17 +1534,27 @@ You can always abort on the first failure by running::
    tox -- -x
 +Similarly, you may also be interested in the ``--last-failed`` and
 +``--failed-first`` options as per the
 +`pytest cache <https://docs.pytest.org/en/latest/cache.html>`_ documentation.
++
  If you want to run a specific test or a group of tests, you
  can specify a prefix to match. E.g. if you want to run all of
  the frontend tests you can do::
    tox -- tests/frontend/
 -Specific tests can be chosen by using the :: delimeter after the test module.
 +Specific tests can be chosen by using the :: delimiter after the test module.
  If you wanted to run the test_build_track test within frontend/buildtrack.py you could do::
    tox -- tests/frontend/buildtrack.py::test_build_track
 +When running only a few tests, you may find the coverage and timing output
 +excessive, there are options to trim them. Note that coverage step will fail.
 +Here is an example::
++
 +  tox -- --no-cov --durations=1 tests/frontend/buildtrack.py::test_build_track
++
  We also have a set of slow integration tests that are disabled by
  default - you will notice most of them marked with SKIP in the pytest
  output. To run them, you can use::
@@ -1560,7 +1570,7 @@ can run ``tox`` with ``-r`` or  ``--recreate`` option.
  .. note::
     By default, we do not allow use of site packages in our ``tox``
 -   confguration to enable running the tests in an isolated environment.
 +   configuration to enable running the tests in an isolated environment.
     If you need to enable use of site packages for whatever reason, you can
     do so by passing the ``--sitepackages`` option to ``tox``. Also, you will
     not need to install any of the build dependencies mentioned above if you
@@ -1597,7 +1607,7 @@ under test before displaying the combined coverage.
  Adding tests
  ~~~~~~~~~~~~
  Tests are found in the tests subdirectory, inside of which
 -there is a separarate directory for each *domain* of tests.
 +there is a separate directory for each *domain* of tests.
  All tests are collected as::
    tests/*/*.py

buildstream/_options/optionarch.py

@@ -17,6 +17,8 @@
  #  Authors:
  #        Tristan Van Berkom <tristan vanberkom codethink co uk>
 +from .. import _yaml
 +from .._exceptions import LoadError, LoadErrorReason, PlatformError
  from .._platform import Platform
  from .optionenum import OptionEnum
@@ -41,7 +43,34 @@ class OptionArch(OptionEnum):
          super(OptionArch, self).load(node, allow_default_definition=False)
      def load_default_value(self, node):
 -        return Platform.get_host_arch()
 +        arch = Platform.get_host_arch()
++
 +        default_value = None
++
 +        for index, value in enumerate(self.values):
 +            try:
 +                canonical_value = Platform.canonicalize_arch(value)
 +                if default_value is None and canonical_value == arch:
 +                    default_value = value
 +                    # Do not terminate the loop early to ensure we validate
 +                    # all values in the list.
 +            except PlatformError as e:
 +                provenance = _yaml.node_get_provenance(node, key='values', indices=[index])
 +                prefix = ""
 +                if provenance:
 +                    prefix = "{}: ".format(provenance)
 +                raise LoadError(LoadErrorReason.INVALID_DATA,
 +                                "{}Invalid value for {} option '{}': {}"
 +                                .format(prefix, self.OPTION_TYPE, self.name, e))
++
 +        if default_value is None:
 +            # Host architecture is not supported by the project.
 +            # Do not raise an error here as the user may override it.
 +            # If the user does not override it, an error will be raised
 +            # by resolve()/validate().
 +            default_value = arch
++
 +        return default_value
      def resolve(self):

buildstream/_pipeline.py

@@ -22,6 +22,7 @@
  import os
  import itertools
  from operator import itemgetter
 +from collections import OrderedDict
  from ._exceptions import PipelineError
  from ._message import Message, MessageType
@@ -479,7 +480,7 @@ class Pipeline():
+ #
  class _Planner():
      def __init__(self):
 -        self.depth_map = {}
 +        self.depth_map = OrderedDict()
          self.visiting_elements = set()
      # Here we want to traverse the same element more than once when

buildstream/_platform/platform.py

@@ -77,20 +77,17 @@ class Platform():
      def get_host_os():
          return os.uname()[0]
 -    # get_host_arch():
 +    # canonicalize_arch():
+     #
 -    # This returns the architecture of the host machine. The possible values
 -    # map from uname -m in order to be a OS independent list.
 +    # This returns the canonical, OS-independent architecture name
 +    # or raises a PlatformError if the architecture is unknown.
+     #
 -    # Returns:
 -    #    (string): String representing the architecture
      @staticmethod
 -    def get_host_arch():
 -        # get the hardware identifier from uname
 -        uname_machine = os.uname()[4]
 -        uname_to_arch = {
 +    def canonicalize_arch(arch):
 +        aliases = {
 +            "aarch32": "aarch32",
              "aarch64": "aarch64",
 -            "aarch64_be": "aarch64-be",
 +            "aarch64-be": "aarch64-be",
              "amd64": "x86-64",
              "arm": "aarch32",
              "armv8l": "aarch64",
@@ -99,17 +96,34 @@ class Platform():
              "i486": "x86-32",
              "i586": "x86-32",
              "i686": "x86-32",
 +            "power-isa-be": "power-isa-be",
 +            "power-isa-le": "power-isa-le",
              "ppc64": "power-isa-be",
              "ppc64le": "power-isa-le",
              "sparc": "sparc-v9",
              "sparc64": "sparc-v9",
 -            "x86_64": "x86-64"
 +            "sparc-v9": "sparc-v9",
 +            "x86-32": "x86-32",
 +            "x86-64": "x86-64"
+         }
++
          try:
 -            return uname_to_arch[uname_machine]
 +            return aliases[arch.replace('_', '-')]
          except KeyError:
 -            raise PlatformError("uname gave unsupported machine architecture: {}"
 -                                .format(uname_machine))
 +            raise PlatformError("Unknown architecture: {}".format(arch))
++
 +    # get_host_arch():
 +    #
 +    # This returns the architecture of the host machine. The possible values
 +    # map from uname -m in order to be a OS independent list.
 +    #
 +    # Returns:
 +    #    (string): String representing the architecture
 +    @staticmethod
 +    def get_host_arch():
 +        # get the hardware identifier from uname
 +        uname_machine = os.uname()[4]
 +        return Platform.canonicalize_arch(uname_machine)
      ##################################################################
      #                        Sandbox functions                       #

buildstream/element.py

@@ -2441,11 +2441,17 @@ class Element(Plugin):
          # Sandbox config, unlike others, has fixed members so we should validate them
          _yaml.node_validate(sandbox_config, ['build-uid', 'build-gid', 'build-os', 'build-arch'])
 +        build_arch = self.node_get_member(sandbox_config, str, 'build-arch', default=None)
 +        if build_arch:
 +            build_arch = Platform.canonicalize_arch(build_arch)
 +        else:
 +            build_arch = host_arch
++
          return SandboxConfig(
              self.node_get_member(sandbox_config, int, 'build-uid'),
              self.node_get_member(sandbox_config, int, 'build-gid'),
              self.node_get_member(sandbox_config, str, 'build-os', default=host_os),
 -            self.node_get_member(sandbox_config, str, 'build-arch', default=host_arch))
 +            build_arch)
      # This makes a special exception for the split rules, which
      # elements may extend but whos defaults are defined in the project.

requirements/dev-requirements.txt

@@ -2,7 +2,7 @@ coverage==4.4
  pylint==2.2.2
  pycodestyle==2.4.0
  pytest==4.0.2
 -pytest-cov==2.6.0
 +pytest-cov==2.6.1
  pytest-datafiles==2.0
  pytest-env==0.6.2
  pytest-xdist==1.25.0

tests/format/option-arch-alias/element.bst

 +kind: autotools
 +variables:
 +  result: "Nothing"
 +  (?):
 +  - machine_arch == "arm":
 +      result: "Army"
 +  - machine_arch == "x86_64":
 +      result: "X86-64y"

tests/format/option-arch-alias/project.conf

 +name: test
++
 +options:
 +  machine_arch:
 +    type: arch
 +    description: The machine architecture
 +    values:
 +    - arm
 +    - x86_64

tests/format/option-arch-unknown/element.bst

 +kind: autotools
 +variables:
 +  result: "Nothing"
 +  (?):
 +  - machine_arch == "aarch32":
 +      result: "Army"
 +  - machine_arch == "aarch64":
 +      result: "Aarchy"
 +  - machine_arch == "x86-128":
 +      result: "X86-128y"

tests/format/option-arch-unknown/project.conf

 +name: test
++
 +options:
 +  machine_arch:
 +    type: arch
 +    description: The machine architecture
 +    values:
 +    - aarch32
 +    - aarch64
 +    - x86-128

tests/format/optionarch.py

@@ -75,3 +75,47 @@ def test_unsupported_arch(cli, datafiles):
          ])
          result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_alias(cli, datafiles):
++
 +    with override_uname_arch("arm"):
 +        project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch-alias')
 +        result = cli.run(project=project, silent=True, args=[
 +            'show',
 +            '--deps', 'none',
 +            '--format', '%{vars}',
 +            'element.bst'
 +        ])
++
 +        result.assert_success()
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_unknown_host_arch(cli, datafiles):
++
 +    with override_uname_arch("x86_128"):
 +        project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch')
 +        result = cli.run(project=project, silent=True, args=[
 +            'show',
 +            '--deps', 'none',
 +            '--format', '%{vars}',
 +            'element.bst'
 +        ])
++
 +        result.assert_main_error(ErrorDomain.PLATFORM, None)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_unknown_project_arch(cli, datafiles):
++
 +    project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch-unknown')
 +    result = cli.run(project=project, silent=True, args=[
 +        'show',
 +        '--deps', 'none',
 +        '--format', '%{vars}',
 +        'element.bst'
 +    ])
++
 +    result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA)

tests/frontend/order.py

 +import os
++
 +import pytest
 +from tests.testutils import cli, create_repo
++
 +from buildstream import _yaml
++
 +# Project directory
 +DATA_DIR = os.path.join(
 +    os.path.dirname(os.path.realpath(__file__)),
 +    "project",
 +)
++
++
 +def create_element(repo, name, path, dependencies, ref=None):
 +    element = {
 +        'kind': 'import',
 +        'sources': [
 +            repo.source_config(ref=ref)
 +        ],
 +        'depends': dependencies
 +    }
 +    _yaml.dump(element, os.path.join(path, name))
++
++
 +# This tests a variety of scenarios and checks that the order in
 +# which things are processed remains stable.
 +#
 +# This is especially important in order to ensure that our
 +# depth sorting and optimization of which elements should be
 +# processed first is doing it's job right, and that we are
 +# promoting elements to the build queue as soon as possible
 +#
 +# Parameters:
 +#    targets (target elements): The targets to invoke bst with
 +#    template (dict): The project template dictionary, for create_element()
 +#    expected (list): A list of element names in the expected order
 +#
 +@pytest.mark.datafiles(os.path.join(DATA_DIR))
 +@pytest.mark.parametrize("target,template,expected", [
 +    # First simple test
 +    ('3.bst', {
 +        '0.bst': ['1.bst'],
 +        '1.bst': [],
 +        '2.bst': ['0.bst'],
 +        '3.bst': ['0.bst', '1.bst', '2.bst']
 +    }, ['1.bst', '0.bst', '2.bst', '3.bst']),
++
 +    # A more complicated test with build of build dependencies
 +    ('target.bst', {
 +        'a.bst': [],
 +        'base.bst': [],
 +        'timezones.bst': [],
 +        'middleware.bst': [{'filename': 'base.bst', 'type': 'build'}],
 +        'app.bst': [{'filename': 'middleware.bst', 'type': 'build'}],
 +        'target.bst': ['a.bst', 'base.bst', 'middleware.bst', 'app.bst', 'timezones.bst']
 +    }, ['base.bst', 'middleware.bst', 'a.bst', 'app.bst', 'timezones.bst', 'target.bst']),
 +])
 +@pytest.mark.parametrize("operation", [('show'), ('fetch'), ('build')])
 +def test_order(cli, datafiles, tmpdir, operation, target, template, expected):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    dev_files_path = os.path.join(project, 'files', 'dev-files')
 +    element_path = os.path.join(project, 'elements')
++
 +    # FIXME: Remove this when the test passes reliably.
 +    #
 +    #        There is no reason why the order should not
 +    #        be preserved when the builders is set to 1,
 +    #        the scheduler queue processing still seems to
 +    #        be losing the order.
 +    #
 +    if operation == 'build':
 +        pytest.skip("FIXME: This still only sometimes passes")
++
 +    # Configure to only allow one fetcher at a time, make it easy to
 +    # determine what is being planned in what order.
 +    cli.configure({
 +        'scheduler': {
 +            'fetchers': 1,
 +            'builders': 1
 +        }
 +    })
++
 +    # Build the project from the template, make import elements
 +    # all with the same repo
 +    #
 +    repo = create_repo('git', str(tmpdir))
 +    ref = repo.create(dev_files_path)
 +    for element, dependencies in template.items():
 +        create_element(repo, element, element_path, dependencies, ref=ref)
 +        repo.add_commit()
++
 +    # Run test and collect results
 +    if operation == 'show':
 +        result = cli.run(args=['show', '--deps', 'plan', '--format', '%{name}', target], project=project, silent=True)
 +        result.assert_success()
 +        results = result.output.splitlines()
 +    else:
 +        if operation == 'fetch':
 +            result = cli.run(args=['source', 'fetch', target], project=project, silent=True)
 +        else:
 +            result = cli.run(args=[operation, target], project=project, silent=True)
 +        result.assert_success()
 +        results = result.get_start_order(operation)
++
 +    # Assert the order
 +    print("Expected order: {}".format(expected))
 +    print("Observed result order: {}".format(results))
 +    assert results == expected

tests/testutils/runcli.py

@@ -167,6 +167,23 @@ class Result():
      def assert_shell_error(self, fail_message=''):
          assert self.exit_code == 1, fail_message
 +    # get_start_order()
 +    #
 +    # Gets the list of elements processed in a given queue, in the
 +    # order of their first appearances in the session.
 +    #
 +    # Args:
 +    #    activity (str): The queue activity name (like 'fetch')
 +    #
 +    # Returns:
 +    #    (list): A list of element names in the order which they first appeared in the result
 +    #
 +    def get_start_order(self, activity):
 +        results = re.findall(r'\[\s*{}:(\S+)\s*\]\s*START\s*.*\.log'.format(activity), self.stderr)
 +        if results is None:
 +            return []
 +        return list(results)
++
      # get_tracked_elements()
+     #
      # Produces a list of element names on which tracking occurred

[Notes] [Git][BuildStream/buildstream][aevri/tox_no_cov] 13 commits: _platform/platform.py: Add canonicalize_arch() method

Angelos Evripiotis pushed to branch aevri/tox_no_cov at BuildStream / buildstream

Commits:

13 changed files:

Changes: