[Notes] [Git][BuildStream/buildstream][tristan/element-processing-order] 10 commits: _platform/platform.py: Add canonicalize_arch() method

[Tristan Van Berkom <gitlab mg gitlab com>]

Title: GitLab

Tristan Van Berkom pushed to branch tristan/element-processing-order at BuildStream / buildstream

Commits:

• c87bb592
  by Jürg Billeter at 2019-01-10T09:43:37Z

  _platform/platform.py: Add canonicalize_arch() method
  

• 26e33346
  by Jürg Billeter at 2019-01-10T12:50:15Z

  _options/optionarch.py: Accept architecture aliases
  
  Accept common architecture aliases for arch options instead of only
  accepting the canonicalized, OS-independent architecture name. This
  restores compatibility with existing projects and may simplify option
  handling for projects that only target a single OS (and thus, do not
  need OS-independent architecture names).
  

• 06deb4c4
  by Jürg Billeter at 2019-01-10T12:50:15Z

  element.py: Accept architecture aliases for sandbox config
  
  Accept common architecture aliases for the sandbox config for
  consistency with arch options.
  

• 0c2a66b3
  by Jürg Billeter at 2019-01-10T13:06:02Z

  tests/format/optionarch.py: Add tests for architecture aliases
  

• f86bc760
  by Jürg Billeter at 2019-01-10T13:34:56Z

  Merge branch 'juerg/arch' into 'master'
  
  Accept architecture aliases
  
  See merge request BuildStream/buildstream!1034

• 347eb34e
  by Angelos Evripiotis at 2019-01-10T14:38:23Z

  contributing: fix 'oprtation' and some other typos
  

• 6bc27bc1
  by Angelos Evripiotis at 2019-01-10T15:07:46Z

  Merge branch 'aevri/contrib-typos' into 'master'
  
  contributing: fix 'oprtation' and some other typos
  
  See merge request BuildStream/buildstream!1055

• 0b83d024
  by Tristan Van Berkom at 2019-01-10T20:02:50Z

  _pipeline.py: Fix the planner to retain the original order when depth sorting
  
  The algorithm adds elements to a dictionary and then sorts the dictionary
  by the discovered depths while recursing - using an OrderedDict is enough
  to ensure a stable order.
  
  This fixes `bst show --deps plan ...` reporting different results on
  each invocation.
  
  This fixes an aspect of #824
  

• 630e26f1
  by Tristan Van Berkom at 2019-01-10T20:03:52Z

  testutils/runcli.py: Added Result.get_start_order()
  
  Added a function to report the list of elements which appeared in
  a given queue in the order of their first appearance.
  

• 5419216d
  by Tristan Van Berkom at 2019-01-10T20:03:52Z

  tests/frontend/buildorder.py: Adding tests ensuring expected build order
  
  For each sample project and expected result order, this test ensures that:
  
    * bst show --deps plan: Always shows the expected build order.
  
    * bst build: Always builds in the expected order.
  

12 changed files:

• CONTRIBUTING.rst
• buildstream/_options/optionarch.py
• buildstream/_pipeline.py
• buildstream/_platform/platform.py
• buildstream/element.py
• + tests/format/option-arch-alias/element.bst
• + tests/format/option-arch-alias/project.conf
• + tests/format/option-arch-unknown/element.bst
• + tests/format/option-arch-unknown/project.conf
• tests/format/optionarch.py
• + tests/frontend/buildorder.py
• tests/testutils/runcli.py

Changes:

CONTRIBUTING.rst

---

...	...	@@ -553,7 +553,7 @@ One problem which arises from this is that we end up having symbols
553	553	which are *public* according to the :ref:`rules discussed in the previous section
554	554	<contributing_public_and_private>`, but must be hidden away from the
555	555	*"Public API Surface"*. For example, BuildStream internal classes need
556		-to invoke methods on the ``Element`` and ``Source`` classes, wheras these
	556	+to invoke methods on the ``Element`` and ``Source`` classes, whereas these
557	557	methods need to be hidden from the *"Public API Surface"*.
558	558
559	559	This is where BuildStream deviates from the PEP-8 standard for public
...	...	@@ -631,7 +631,7 @@ An element plugin will derive from Element by importing::
631	631
632	632	from buildstream import Element
633	633
634		-When importing utilities specifically, dont import function names
	634	+When importing utilities specifically, don't import function names
635	635	from there, instead import the module itself::
636	636
637	637	from . import utils
...	...	@@ -737,7 +737,7 @@ Abstract methods
737	737	~~~~~~~~~~~~~~~~
738	738	In BuildStream, an *"Abstract Method"* is a bit of a misnomer and does
739	739	not match up to how Python defines abstract methods, we need to seek out
740		-a new nomanclature to refer to these methods.
	740	+a new nomenclature to refer to these methods.
741	741
742	742	In Python, an *"Abstract Method"* is a method which **must** be
743	743	implemented by a subclass, whereas all methods in Python can be
...	...	@@ -960,7 +960,7 @@ possible, and avoid any cyclic relationships in modules.
960	960	For instance, the ``Source`` objects are owned by ``Element``
961	961	objects in the BuildStream data model, and as such the ``Element``
962	962	will delegate some activities to the ``Source`` objects in its
963		-possesion. The ``Source`` objects should however never call functions
	963	+possession. The ``Source`` objects should however never call functions
964	964	on the ``Element`` object, nor should the ``Source`` object itself
965	965	have any understanding of what an ``Element`` is.
966	966
...	...	@@ -1223,7 +1223,7 @@ For further information about using the reStructuredText with sphinx, please see
1223	1223	Building Docs
1224	1224	~~~~~~~~~~~~~
1225	1225	Before you can build the docs, you will end to ensure that you have installed
1226		-the required :ref:`buid dependencies <contributing_build_deps>` as mentioned
	1226	+the required :ref:`build dependencies <contributing_build_deps>` as mentioned
1227	1227	in the testing section above.
1228	1228
1229	1229	To build the documentation, just run the following::
...	...	@@ -1365,7 +1365,7 @@ Structure of an example
1365	1365	'''''''''''''''''''''''
1366	1366	The :ref:`tutorial <tutorial>` and the :ref:`examples <examples>` sections
1367	1367	of the documentation contain a series of sample projects, each chapter in
1368		-the tutoral, or standalone example uses a sample project.
	1368	+the tutorial, or standalone example uses a sample project.
1369	1369
1370	1370	Here is the the structure for adding new examples and tutorial chapters.
1371	1371
...	...	@@ -1471,8 +1471,8 @@ Installing build dependencies
1471	1471	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
1472	1472	Some of BuildStream's dependencies have non-python build dependencies. When
1473	1473	running tests with ``tox``, you will first need to install these dependencies.
1474		-Exact steps to install these will depend on your oprtation systemm. Commands
1475		-for installing them for some common distributions are lised below.
	1474	+Exact steps to install these will depend on your operating system. Commands
	1475	+for installing them for some common distributions are listed below.
1476	1476
1477	1477	For Fedora-based systems::
1478	1478
...	...	@@ -1540,7 +1540,7 @@ the frontend tests you can do::
1540	1540
1541	1541	tox -- tests/frontend/
1542	1542
1543		-Specific tests can be chosen by using the :: delimeter after the test module.
	1543	+Specific tests can be chosen by using the :: delimiter after the test module.
1544	1544	If you wanted to run the test_build_track test within frontend/buildtrack.py you could do::
1545	1545
1546	1546	tox -- tests/frontend/buildtrack.py::test_build_track
...	...	@@ -1560,7 +1560,7 @@ can run ``tox`` with ``-r`` or ``--recreate`` option.
1560	1560	.. note::
1561	1561
1562	1562	By default, we do not allow use of site packages in our ``tox``
1563		- confguration to enable running the tests in an isolated environment.
	1563	+ configuration to enable running the tests in an isolated environment.
1564	1564	If you need to enable use of site packages for whatever reason, you can
1565	1565	do so by passing the ``--sitepackages`` option to ``tox``. Also, you will
1566	1566	not need to install any of the build dependencies mentioned above if you
...	...	@@ -1597,7 +1597,7 @@ under test before displaying the combined coverage.
1597	1597	Adding tests
1598	1598	~~~~~~~~~~~~
1599	1599	Tests are found in the tests subdirectory, inside of which
1600		-there is a separarate directory for each *domain* of tests.
	1600	+there is a separate directory for each *domain* of tests.
1601	1601	All tests are collected as::
1602	1602
1603	1603	tests/*/*.py

buildstream/_options/optionarch.py

---

...	...	@@ -17,6 +17,8 @@
17	17	# Authors:
18	18	# Tristan Van Berkom <tristan vanberkom codethink co uk>
19	19
	20	+from .. import _yaml
	21	+from .._exceptions import LoadError, LoadErrorReason, PlatformError
20	22	from .._platform import Platform
21	23	from .optionenum import OptionEnum
22	24
...	...	@@ -41,7 +43,34 @@ class OptionArch(OptionEnum):
41	43	super(OptionArch, self).load(node, allow_default_definition=False)
42	44
43	45	def load_default_value(self, node):
44		- return Platform.get_host_arch()
	46	+ arch = Platform.get_host_arch()
	47	+
	48	+ default_value = None
	49	+
	50	+ for index, value in enumerate(self.values):
	51	+ try:
	52	+ canonical_value = Platform.canonicalize_arch(value)
	53	+ if default_value is None and canonical_value == arch:
	54	+ default_value = value
	55	+ # Do not terminate the loop early to ensure we validate
	56	+ # all values in the list.
	57	+ except PlatformError as e:
	58	+ provenance = _yaml.node_get_provenance(node, key='values', indices=[index])
	59	+ prefix = ""
	60	+ if provenance:
	61	+ prefix = "{}: ".format(provenance)
	62	+ raise LoadError(LoadErrorReason.INVALID_DATA,
	63	+ "{}Invalid value for {} option '{}': {}"
	64	+ .format(prefix, self.OPTION_TYPE, self.name, e))
	65	+
	66	+ if default_value is None:
	67	+ # Host architecture is not supported by the project.
	68	+ # Do not raise an error here as the user may override it.
	69	+ # If the user does not override it, an error will be raised
	70	+ # by resolve()/validate().
	71	+ default_value = arch
	72	+
	73	+ return default_value
45	74
46	75	def resolve(self):
47	76

buildstream/_pipeline.py

---

...	...	@@ -22,6 +22,7 @@
22	22	import os
23	23	import itertools
24	24	from operator import itemgetter
	25	+from collections import OrderedDict
25	26
26	27	from ._exceptions import PipelineError
27	28	from ._message import Message, MessageType
...	...	@@ -479,7 +480,7 @@ class Pipeline():
479	480	#
480	481	class _Planner():
481	482	def __init__(self):
482		- self.depth_map = {}
	483	+ self.depth_map = OrderedDict()
483	484	self.visiting_elements = set()
484	485
485	486	# Here we want to traverse the same element more than once when

buildstream/_platform/platform.py

---

...	...	@@ -77,20 +77,17 @@ class Platform():
77	77	def get_host_os():
78	78	return os.uname()[0]
79	79
80		- # get_host_arch():
	80	+ # canonicalize_arch():
81	81	#
82		- # This returns the architecture of the host machine. The possible values
83		- # map from uname -m in order to be a OS independent list.
	82	+ # This returns the canonical, OS-independent architecture name
	83	+ # or raises a PlatformError if the architecture is unknown.
84	84	#
85		- # Returns:
86		- # (string): String representing the architecture
87	85	@staticmethod
88		- def get_host_arch():
89		- # get the hardware identifier from uname
90		- uname_machine = os.uname()[4]
91		- uname_to_arch = {
	86	+ def canonicalize_arch(arch):
	87	+ aliases = {
	88	+ "aarch32": "aarch32",
92	89	"aarch64": "aarch64",
93		- "aarch64_be": "aarch64-be",
	90	+ "aarch64-be": "aarch64-be",
94	91	"amd64": "x86-64",
95	92	"arm": "aarch32",
96	93	"armv8l": "aarch64",
...	...	@@ -99,17 +96,34 @@ class Platform():
99	96	"i486": "x86-32",
100	97	"i586": "x86-32",
101	98	"i686": "x86-32",
	99	+ "power-isa-be": "power-isa-be",
	100	+ "power-isa-le": "power-isa-le",
102	101	"ppc64": "power-isa-be",
103	102	"ppc64le": "power-isa-le",
104	103	"sparc": "sparc-v9",
105	104	"sparc64": "sparc-v9",
106		- "x86_64": "x86-64"
	105	+ "sparc-v9": "sparc-v9",
	106	+ "x86-32": "x86-32",
	107	+ "x86-64": "x86-64"
107	108	}
	109	+
108	110	try:
109		- return uname_to_arch[uname_machine]
	111	+ return aliases[arch.replace('_', '-')]
110	112	except KeyError:
111		- raise PlatformError("uname gave unsupported machine architecture: {}"
112		- .format(uname_machine))
	113	+ raise PlatformError("Unknown architecture: {}".format(arch))
	114	+
	115	+ # get_host_arch():
	116	+ #
	117	+ # This returns the architecture of the host machine. The possible values
	118	+ # map from uname -m in order to be a OS independent list.
	119	+ #
	120	+ # Returns:
	121	+ # (string): String representing the architecture
	122	+ @staticmethod
	123	+ def get_host_arch():
	124	+ # get the hardware identifier from uname
	125	+ uname_machine = os.uname()[4]
	126	+ return Platform.canonicalize_arch(uname_machine)
113	127
114	128	##################################################################
115	129	# Sandbox functions #

buildstream/element.py

---

...	...	@@ -2441,11 +2441,17 @@ class Element(Plugin):
2441	2441	# Sandbox config, unlike others, has fixed members so we should validate them
2442	2442	_yaml.node_validate(sandbox_config, ['build-uid', 'build-gid', 'build-os', 'build-arch'])
2443	2443
	2444	+ build_arch = self.node_get_member(sandbox_config, str, 'build-arch', default=None)
	2445	+ if build_arch:
	2446	+ build_arch = Platform.canonicalize_arch(build_arch)
	2447	+ else:
	2448	+ build_arch = host_arch
	2449	+
2444	2450	return SandboxConfig(
2445	2451	self.node_get_member(sandbox_config, int, 'build-uid'),
2446	2452	self.node_get_member(sandbox_config, int, 'build-gid'),
2447	2453	self.node_get_member(sandbox_config, str, 'build-os', default=host_os),
2448		- self.node_get_member(sandbox_config, str, 'build-arch', default=host_arch))
	2454	+ build_arch)
2449	2455
2450	2456	# This makes a special exception for the split rules, which
2451	2457	# elements may extend but whos defaults are defined in the project.

tests/format/option-arch-alias/element.bst

---

	1	+kind: autotools
	2	+variables:
	3	+ result: "Nothing"
	4	+ (?):
	5	+ - machine_arch == "arm":
	6	+ result: "Army"
	7	+ - machine_arch == "x86_64":
	8	+ result: "X86-64y"

tests/format/option-arch-alias/project.conf

---

	1	+name: test
	2	+
	3	+options:
	4	+ machine_arch:
	5	+ type: arch
	6	+ description: The machine architecture
	7	+ values:
	8	+ - arm
	9	+ - x86_64

tests/format/option-arch-unknown/element.bst

---

	1	+kind: autotools
	2	+variables:
	3	+ result: "Nothing"
	4	+ (?):
	5	+ - machine_arch == "aarch32":
	6	+ result: "Army"
	7	+ - machine_arch == "aarch64":
	8	+ result: "Aarchy"
	9	+ - machine_arch == "x86-128":
	10	+ result: "X86-128y"

tests/format/option-arch-unknown/project.conf

---

	1	+name: test
	2	+
	3	+options:
	4	+ machine_arch:
	5	+ type: arch
	6	+ description: The machine architecture
	7	+ values:
	8	+ - aarch32
	9	+ - aarch64
	10	+ - x86-128

tests/format/optionarch.py

---

...	...	@@ -75,3 +75,47 @@ def test_unsupported_arch(cli, datafiles):
75	75	])
76	76
77	77	result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA)
	78	+
	79	+
	80	+@pytest.mark.datafiles(DATA_DIR)
	81	+def test_alias(cli, datafiles):
	82	+
	83	+ with override_uname_arch("arm"):
	84	+ project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch-alias')
	85	+ result = cli.run(project=project, silent=True, args=[
	86	+ 'show',
	87	+ '--deps', 'none',
	88	+ '--format', '%{vars}',
	89	+ 'element.bst'
	90	+ ])
	91	+
	92	+ result.assert_success()
	93	+
	94	+
	95	+@pytest.mark.datafiles(DATA_DIR)
	96	+def test_unknown_host_arch(cli, datafiles):
	97	+
	98	+ with override_uname_arch("x86_128"):
	99	+ project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch')
	100	+ result = cli.run(project=project, silent=True, args=[
	101	+ 'show',
	102	+ '--deps', 'none',
	103	+ '--format', '%{vars}',
	104	+ 'element.bst'
	105	+ ])
	106	+
	107	+ result.assert_main_error(ErrorDomain.PLATFORM, None)
	108	+
	109	+
	110	+@pytest.mark.datafiles(DATA_DIR)
	111	+def test_unknown_project_arch(cli, datafiles):
	112	+
	113	+ project = os.path.join(datafiles.dirname, datafiles.basename, 'option-arch-unknown')
	114	+ result = cli.run(project=project, silent=True, args=[
	115	+ 'show',
	116	+ '--deps', 'none',
	117	+ '--format', '%{vars}',
	118	+ 'element.bst'
	119	+ ])
	120	+
	121	+ result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA)

tests/frontend/buildorder.py

---

	1	+import os
	2	+
	3	+import pytest
	4	+from tests.testutils import cli, create_repo
	5	+
	6	+from buildstream import _yaml
	7	+
	8	+# Project directory
	9	+DATA_DIR = os.path.join(
	10	+ os.path.dirname(os.path.realpath(__file__)),
	11	+ "project",
	12	+)
	13	+
	14	+
	15	+def create_element(repo, name, path, dependencies, ref=None):
	16	+ element = {
	17	+ 'kind': 'import',
	18	+ 'sources': [
	19	+ repo.source_config(ref=ref)
	20	+ ],
	21	+ 'depends': dependencies
	22	+ }
	23	+ _yaml.dump(element, os.path.join(path, name))
	24	+
	25	+
	26	+# This tests a variety of scenarios and checks that the order in
	27	+# which things are processed remains stable.
	28	+#
	29	+# This is especially important in order to ensure that our
	30	+# depth sorting and optimization of which elements should be
	31	+# processed first is doing it's job right, and that we are
	32	+# promoting elements to the build queue as soon as possible
	33	+#
	34	+# Parameters:
	35	+# targets (target elements): The targets to invoke bst with
	36	+# template (dict): The project template dictionary, for create_element()
	37	+# expected (list): A list of element names in the expected order
	38	+#
	39	+@pytest.mark.datafiles(os.path.join(DATA_DIR))
	40	+@pytest.mark.parametrize("target,template,expected", [
	41	+ # First simple test
	42	+ ('3.bst', {
	43	+ '0.bst': ['1.bst'],
	44	+ '1.bst': [],
	45	+ '2.bst': ['0.bst'],
	46	+ '3.bst': ['0.bst', '1.bst', '2.bst']
	47	+ },
	48	+ ['1.bst', '0.bst', '2.bst', '3.bst']),
	49	+
	50	+ # A more complicated test with build of build dependencies
	51	+ ('target.bst', {
	52	+ 'a.bst': [],
	53	+ 'base.bst': [],
	54	+ 'timezones.bst': [],
	55	+ 'middleware.bst': [{'filename': 'base.bst', 'type': 'build'}],
	56	+ 'app.bst': [{'filename': 'middleware.bst', 'type': 'build'}],
	57	+ 'target.bst': ['a.bst', 'base.bst', 'middleware.bst', 'app.bst', 'timezones.bst']
	58	+ },
	59	+ ['base.bst', 'middleware.bst', 'a.bst', 'app.bst', 'timezones.bst', 'target.bst']),
	60	+])
	61	+def test_build_order(cli, datafiles, tmpdir, target, template, expected):
	62	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	63	+ dev_files_path = os.path.join(project, 'files', 'dev-files')
	64	+ element_path = os.path.join(project, 'elements')
	65	+
	66	+ # Configure to only allow one fetcher at a time, make it easy to
	67	+ # determine what is being planned in what order.
	68	+ cli.configure({
	69	+ 'scheduler': {
	70	+ 'fetchers': 1,
	71	+ 'builders': 1
	72	+ }
	73	+ })
	74	+
	75	+ # Build the project from the template, make import elements
	76	+ # all with the same repo
	77	+ #
	78	+ repo = create_repo('git', str(tmpdir))
	79	+ ref = repo.create(dev_files_path)
	80	+ for element, dependencies in template.items():
	81	+ create_element(repo, element, element_path, dependencies, ref=ref)
	82	+ repo.add_commit()
	83	+
	84	+ # Show it for debugging purposes
	85	+ result = cli.run(args=['show', '--deps', 'plan', '--format', '%{name}', target], project=project, silent=True)
	86	+ elements = result.output.splitlines()
	87	+ print("Show reported the following order: {}".format(elements))
	88	+
	89	+ # Run the build
	90	+ result = cli.run(args=['build', target], project=project, silent=True)
	91	+ result.assert_success()
	92	+
	93	+ # Check the order of elements which passed through the build queue
	94	+ builds = result.get_start_order('build')
	95	+ print("Builds started in the following order: {}".format(builds))
	96	+
	97	+ # First check that we are building in the order we planned to build,
	98	+ # by asserting that the `bst show` results are in the same order
	99	+ # as the observed build order.
	100	+ assert elements == builds
	101	+
	102	+ # Now assert that this order is the same as that we expected
	103	+ assert elements == expected

tests/testutils/runcli.py

---

...	...	@@ -167,6 +167,23 @@ class Result():
167	167	def assert_shell_error(self, fail_message=''):
168	168	assert self.exit_code == 1, fail_message
169	169
	170	+ # get_start_order()
	171	+ #
	172	+ # Gets the list of elements processed in a given queue, in the
	173	+ # order of their first appearances in the session.
	174	+ #
	175	+ # Args:
	176	+ # activity (str): The queue activity name (like 'fetch')
	177	+ #
	178	+ # Returns:
	179	+ # (list): A list of element names in the order which they first appeared in the result
	180	+ #
	181	+ def get_start_order(self, activity):
	182	+ results = re.findall(r'\[\s*{}:(\S+)\s*\]\s*START\s*.*\.log'.format(activity), self.stderr)
	183	+ if results is None:
	184	+ return []
	185	+ return list(results)
	186	+
170	187	# get_tracked_elements()
171	188	#
172	189	# Produces a list of element names on which tracking occurred

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.