[Notes] [Git][BuildStream/buildstream][bschubert/rework-sort] 14 commits: tox.ini: Specify minimum version of click-man

[Benjamin Schubert <gitlab mg gitlab com>]

Title: GitLab

Benjamin Schubert pushed to branch bschubert/rework-sort at BuildStream / buildstream

Commits:

• 36746730
  by Chandan Singh at 2019-01-31T10:50:05Z

  tox.ini: Specify minimum version of click-man
  
  `click-man` versions < 0.3.0 do not properly support multiple
  entrypoints. Since this was added to `tox` after `0.3.0` was released,
  `tox` should never be pulling older versions. But, let's add it here for
  documentation purposes.
  
  See
  https://gitlab.com/BuildStream/buildstream/merge_requests/1107#note_135187046
  for some background on this.
  

• fa4a21ce
  by Chandan Singh at 2019-01-31T12:15:43Z

  Merge branch 'chandan/min-version-click-man' into 'master'
  
  tox.ini: Specify minimum version of click-man
  
  See merge request BuildStream/buildstream!1120

• dd791373
  by Chandan Singh at 2019-01-31T14:32:44Z

  testutils/site.py: Support parsing more exotic git versions
  
  We use output of `git --version` to determine if we can run some tests
  that rely on features from newer git versions. Usually, we expect the
  output to be like:
  
      git version 2.17.2
  
  On some platforms, like MacOS, there could be a suffix after the version
  string, so that it looks something like:
  
      git version 2.17.2 (Apple Git-113)
  
  This causes things to fail like so:
  
      ValueError: invalid literal for int() with base 10: '2 (Apple Git-113)\n'
  
  Fix logic around `HAVE_OLD_GIT` such that we split the output of
  `git --version` without limit on how many times we split. Previously we
  used to split only twice so the suffixes like `(Apple Git-113)` are not
  part of the parsed version.
  

• 96c0fbd6
  by Chandan Singh at 2019-01-31T15:39:19Z

  Merge branch 'chandan/fix-git-version-mac' into 'master'
  
  testutils/site.py: Support parsing more exotic git versions
  
  See merge request BuildStream/buildstream!1118

• d25e2795
  by Benjamin Schubert at 2019-01-31T17:06:23Z

  Add LoadElement in dependency list for LoadElement idrectly
  
  This removes the need for the 'Dependency' list to then be matched
  with the corresponding LoadElement and will allow iterating the
  graph more easily
  

• 2d0eebbf
  by Benjamin Schubert at 2019-01-31T17:06:23Z

  Pass element directly to collect_element
  
  This simplifies the loading
  

• 583bd97d
  by Benjamin Schubert at 2019-02-01T10:26:37Z

  Merge branch 'bschubert/loader' into 'master'
  
  Cleanup loader by linking LoadElements sooner
  
  See merge request BuildStream/buildstream!1122

• 51cec3da
  by Phil Dawson at 2019-02-01T14:25:44Z

  tests/cachekey: Test cache keys are independent of target elements
  

• 2b38aabe
  by Phil Dawson at 2019-02-01T15:33:00Z

  Merge branch 'phil/cache-key-stability-test' into 'master'
  
  tests/cachekey: Test cache keys are independent of target elements
  
  See merge request BuildStream/buildstream!1123

• dbb3d232
  by James Ennis at 2019-02-01T15:51:32Z

  filter.py/filter.yaml: Documentation improvements
  

• 7e4205cb
  by James Ennis at 2019-02-01T15:51:32Z

  filter.py: Add an example to the documentation
  

• 4109a34a
  by James Ennis at 2019-02-01T17:11:29Z

  Merge branch 'jennis/filter-docs' into 'master'
  
  Improve our filter documentation
  
  Closes #278
  
  See merge request BuildStream/buildstream!1112

• b7ccc471
  by Benjamin Schubert at 2019-02-02T09:31:44Z

  Add reverse dependencies in the LoadElement
  
  This will allow for easier navigation in the graph and will
  unblock a refactoring for the scheduler
  

• b9d164ef
  by Benjamin Schubert at 2019-02-02T09:31:44Z

  fixup! Add reverse dependencies in the LoadElement
  

14 changed files:

• buildstream/_loader/loadelement.py
• buildstream/_loader/loader.py
• buildstream/_loader/metaelement.py
• buildstream/_profile.py
• buildstream/plugins/elements/filter.py
• buildstream/plugins/elements/filter.yaml
• tests/cachekey/cachekey.py
• + tests/cachekey/project/elements/key-stability/aaa.bst
• + tests/cachekey/project/elements/key-stability/t1.bst
• + tests/cachekey/project/elements/key-stability/t2.bst
• + tests/cachekey/project/elements/key-stability/top-level.bst
• + tests/cachekey/project/elements/key-stability/zzz.bst
• tests/testutils/site.py
• tox.ini

Changes:

buildstream/_loader/loadelement.py

---

...	...	@@ -19,6 +19,7 @@
19	19
20	20	# System imports
21	21	from collections.abc import Mapping
	22	+import weakref
22	23
23	24	# BuildStream toplevel imports
24	25	from .._exceptions import LoadError, LoadErrorReason
...	...	@@ -39,6 +40,20 @@ from .types import Symbol, Dependency
39	40	# loader (Loader): The Loader object for this element
40	41	#
41	42	class LoadElement():
	43	+ # Dependency():
	44	+ #
	45	+ # A link from a LoadElement to its dependencies.
	46	+ #
	47	+ # Keeps a link to one of the current Element's dependencies, together with
	48	+ # its dependency type.
	49	+ #
	50	+ # Args:
	51	+ # element (LoadElement): a LoadElement on which there is a dependency
	52	+ # dep_type (str): the type of dependency this dependency link is
	53	+ class Dependency:
	54	+ def __init__(self, element, dep_type):
	55	+ self.element = element
	56	+ self.dep_type = dep_type
42	57
43	58	def __init__(self, node, filename, loader):
44	59
...	...	@@ -48,7 +63,7 @@ class LoadElement():
48	63	self.node = node # The YAML node
49	64	self.name = filename # The element name
50	65	self.full_name = None # The element full name (with associated junction)
51		- self.deps = None # The list of Dependency objects
	66	+ self.visited = False
52	67
53	68	#
54	69	# Private members
...	...	@@ -74,8 +89,32 @@ class LoadElement():
74	89	'build-depends', 'runtime-depends',
75	90	])
76	91
77		- # Extract the Dependencies
78		- self.deps = _extract_depends_from_node(self.node)
	92	+ self.dependencies = []
	93	+ self.__reverse_dependencies = []
	94	+
	95	+ @property
	96	+ def junction(self):
	97	+ return self._loader.project.junction
	98	+
	99	+ # reverse_dependencies()
	100	+ #
	101	+ # Iterable over the Element's reverse dependencies
	102	+ #
	103	+ @property
	104	+ def reverse_dependencies(self):
	105	+ yield from self.__reverse_dependencies
	106	+
	107	+ # add_reverse_dependency()
	108	+ #
	109	+ # Add an element as a reverse dependency of the current element
	110	+ #
	111	+ # Args:
	112	+ # element (LoadElement): LoadElement to add as a reverse dependency
	113	+ #
	114	+ def add_reverse_dependency(self, element):
	115	+ # Adding a weakref here is important as we would end up with
	116	+ # unbreakable circles of dependencies
	117	+ self.__reverse_dependencies.append(weakref.proxy(element))
79	118
80	119	# depends():
81	120	#
...	...	@@ -101,8 +140,8 @@ class LoadElement():
101	140	return
102	141
103	142	self._dep_cache = {}
104		- for dep in self.deps:
105		- elt = self._loader.get_element_for_dep(dep)
	143	+ for dep in self.dependencies:
	144	+ elt = dep.element
106	145
107	146	# Ensure the cache of the element we depend on
108	147	elt._ensure_depends_cache()

buildstream/_loader/loader.py

---

...	...	@@ -19,7 +19,7 @@
19	19
20	20	import os
21	21	from functools import cmp_to_key
22		-from collections import namedtuple
	22	+from collections import defaultdict, deque
23	23	from collections.abc import Mapping
24	24	import tempfile
25	25	import shutil
...	...	@@ -32,8 +32,8 @@ from .._profile import Topics, profile_start, profile_end
32	32	from .._includes import Includes
33	33	from .._yamlcache import YamlCache
34	34
35		-from .types import Symbol, Dependency
36		-from .loadelement import LoadElement
	35	+from .types import Symbol
	36	+from .loadelement import LoadElement, _extract_depends_from_node
37	37	from . import MetaElement
38	38	from . import MetaSource
39	39	from ..types import CoreWarnings
...	...	@@ -112,7 +112,7 @@ class Loader():
112	112
113	113	# First pass, recursively load files and populate our table of LoadElements
114	114	#
115		- deps = []
	115	+ target_elements = []
116	116
117	117	# XXX This will need to be changed to the context's top-level project if this method
118	118	# is ever used for subprojects
...	...	@@ -122,10 +122,10 @@ class Loader():
122	122	with YamlCache.open(self._context, cache_file) as yaml_cache:
123	123	for target in targets:
124	124	profile_start(Topics.LOAD_PROJECT, target)
125		- junction, name, loader = self._parse_name(target, rewritable, ticker,
126		- fetch_subprojects=fetch_subprojects)
127		- loader._load_file(name, rewritable, ticker, fetch_subprojects, yaml_cache)
128		- deps.append(Dependency(name, junction=junction))
	125	+ _junction, name, loader = self._parse_name(target, rewritable, ticker,
	126	+ fetch_subprojects=fetch_subprojects)
	127	+ element = loader._load_file(name, rewritable, ticker, fetch_subprojects, yaml_cache)
	128	+ target_elements.append(element)
129	129	profile_end(Topics.LOAD_PROJECT, target)
130	130
131	131	#
...	...	@@ -134,29 +134,34 @@ class Loader():
134	134
135	135	# Set up a dummy element that depends on all top-level targets
136	136	# to resolve potential circular dependencies between them
137		- DummyTarget = namedtuple('DummyTarget', ['name', 'full_name', 'deps'])
138		-
139		- dummy = DummyTarget(name='', full_name='', deps=deps)
140		- self._elements[''] = dummy
	137	+ dummy_target = LoadElement("", "", self)
	138	+ dummy_target.dependencies.extend(
	139	+ LoadElement.Dependency(element, Symbol.RUNTIME)
	140	+ for element in target_elements
	141	+ )
141	142
142	143	profile_key = "_".join(t for t in targets)
143	144	profile_start(Topics.CIRCULAR_CHECK, profile_key)
144		- self._check_circular_deps('')
	145	+ self._check_circular_deps(dummy_target)
145	146	profile_end(Topics.CIRCULAR_CHECK, profile_key)
146	147
147		- ret = []
148	148	#
149		- # Sort direct dependencies of elements by their dependency ordering
	149	+ # Load and sort direct dependencies of elements by their dependency ordering
150	150	#
151		- for target in targets:
152		- profile_start(Topics.SORT_DEPENDENCIES, target)
153		- junction, name, loader = self._parse_name(target, rewritable, ticker,
154		- fetch_subprojects=fetch_subprojects)
155		- loader._sort_dependencies(name)
156		- profile_end(Topics.SORT_DEPENDENCIES, target)
157		- # Finally, wrap what we have into LoadElements and return the target
158		- #
159		- ret.append(loader._collect_element(name))
	151	+ key = "-".join([e.name for e in target_elements])
	152	+ profile_start(Topics.COLLECT_ELEMENTS, key)
	153	+ ret = loader._collect_elements(target_elements)
	154	+ profile_end(Topics.COLLECT_ELEMENTS, key)
	155	+
	156	+ print("#" * 60)
	157	+ for e in ret:
	158	+ print(e.name)
	159	+ print("dependencies:")
	160	+ for d in e.dependencies:
	161	+ print("\t", d.name)
	162	+ print("build deps:")
	163	+ for d in e.build_dependencies:
	164	+ print("\t", d.name)
160	165
161	166	return ret
162	167
...	...	@@ -184,22 +189,6 @@ class Loader():
184	189	if os.path.exists(self._tempdir):
185	190	shutil.rmtree(self._tempdir)
186	191
187		- # get_element_for_dep():
188		- #
189		- # Gets a cached LoadElement by Dependency object
190		- #
191		- # This is used by LoadElement
192		- #
193		- # Args:
194		- # dep (Dependency): The dependency to search for
195		- #
196		- # Returns:
197		- # (LoadElement): The cached LoadElement
198		- #
199		- def get_element_for_dep(self, dep):
200		- loader = self._get_loader_for_dep(dep)
201		- return loader._elements[dep.name]
202		-
203	192	###########################################
204	193	# Private Methods #
205	194	###########################################
...	...	@@ -272,8 +261,10 @@ class Loader():
272	261
273	262	self._elements[filename] = element
274	263
	264	+ dependencies = _extract_depends_from_node(node)
	265	+
275	266	# Load all dependency files for the new LoadElement
276		- for dep in element.deps:
	267	+ for dep in dependencies:
277	268	if dep.junction:
278	269	self._load_file(dep.junction, rewritable, ticker, fetch_subprojects, yaml_cache)
279	270	loader = self._get_loader(dep.junction, rewritable=rewritable, ticker=ticker,
...	...	@@ -288,7 +279,10 @@ class Loader():
288	279	"{}: Cannot depend on junction"
289	280	.format(dep.provenance))
290	281
291		- deps_names = [dep.name for dep in element.deps]
	282	+ element.dependencies.append(LoadElement.Dependency(dep_element, dep.dep_type))
	283	+ dep_element.add_reverse_dependency(element)
	284	+
	285	+ deps_names = [dep.name for dep in dependencies]
292	286	self._warn_invalid_elements(deps_names)
293	287
294	288	return element
...	...	@@ -299,12 +293,12 @@ class Loader():
299	293	# dependencies already resolved.
300	294	#
301	295	# Args:
302		- # element_name (str): The element-path relative element name to check
	296	+ # element (str): The element to check
303	297	#
304	298	# Raises:
305	299	# (LoadError): In case there was a circular dependency error
306	300	#
307		- def _check_circular_deps(self, element_name, check_elements=None, validated=None, sequence=None):
	301	+ def _check_circular_deps(self, element, check_elements=None, validated=None, sequence=None):
308	302
309	303	if check_elements is None:
310	304	check_elements = {}
...	...	@@ -313,38 +307,31 @@ class Loader():
313	307	if sequence is None:
314	308	sequence = []
315	309
316		- element = self._elements[element_name]
317		-
318		- # element name must be unique across projects
319		- # to be usable as key for the check_elements and validated dicts
320		- element_name = element.full_name
321		-
322	310	# Skip already validated branches
323		- if validated.get(element_name) is not None:
	311	+ if validated.get(element) is not None:
324	312	return
325	313
326		- if check_elements.get(element_name) is not None:
	314	+ if check_elements.get(element) is not None:
327	315	# Create `chain`, the loop of element dependencies from this
328	316	# element back to itself, by trimming everything before this
329	317	# element from the sequence under consideration.
330		- chain = sequence[sequence.index(element_name):]
331		- chain.append(element_name)
	318	+ chain = sequence[sequence.index(element.full_name):]
	319	+ chain.append(element.full_name)
332	320	raise LoadError(LoadErrorReason.CIRCULAR_DEPENDENCY,
333	321	("Circular dependency detected at element: {}\n" +
334	322	"Dependency chain: {}")
335		- .format(element.name, " -> ".join(chain)))
	323	+ .format(element.full_name, " -> ".join(chain)))
336	324
337	325	# Push / Check each dependency / Pop
338		- check_elements[element_name] = True
339		- sequence.append(element_name)
340		- for dep in element.deps:
341		- loader = self._get_loader_for_dep(dep)
342		- loader._check_circular_deps(dep.name, check_elements, validated, sequence)
343		- del check_elements[element_name]
	326	+ check_elements[element] = True
	327	+ sequence.append(element.full_name)
	328	+ for dep in element.dependencies:
	329	+ dep.element._loader._check_circular_deps(dep.element, check_elements, validated, sequence)
	330	+ del check_elements[element]
344	331	sequence.pop()
345	332
346	333	# Eliminate duplicate paths
347		- validated[element_name] = True
	334	+ validated[element] = True
348	335
349	336	# _sort_dependencies():
350	337	#
...	...	@@ -357,28 +344,21 @@ class Loader():
357	344	# sorts throughout the build process.
358	345	#
359	346	# Args:
360		- # element_name (str): The element-path relative element name to sort
	347	+ # element (LoadElement): The element to sort
361	348	#
362		- def _sort_dependencies(self, element_name, visited=None):
	349	+ def _sort_dependencies(self, element, visited=None):
363	350	if visited is None:
364		- visited = {}
365		-
366		- element = self._elements[element_name]
367		-
368		- # element name must be unique across projects
369		- # to be usable as key for the visited dict
370		- element_name = element.full_name
	351	+ visited = set()
371	352
372		- if visited.get(element_name) is not None:
	353	+ if element in visited:
373	354	return
374	355
375		- for dep in element.deps:
376		- loader = self._get_loader_for_dep(dep)
377		- loader._sort_dependencies(dep.name, visited=visited)
	356	+ for dep in element.dependencies:
	357	+ dep.element._loader._sort_dependencies(dep.element, visited=visited)
378	358
379	359	def dependency_cmp(dep_a, dep_b):
380		- element_a = self.get_element_for_dep(dep_a)
381		- element_b = self.get_element_for_dep(dep_b)
	360	+ element_a = dep_a.element
	361	+ element_b = dep_b.element
382	362
383	363	# Sort on inter element dependency first
384	364	if element_a.depends(element_b):
...	...	@@ -395,21 +375,21 @@ class Loader():
395	375	return -1
396	376
397	377	# All things being equal, string comparison.
398		- if dep_a.name > dep_b.name:
	378	+ if element_a.name > element_b.name:
399	379	return 1
400		- elif dep_a.name < dep_b.name:
	380	+ elif element_a.name < element_b.name:
401	381	return -1
402	382
403	383	# Sort local elements before junction elements
404	384	# and use string comparison between junction elements
405		- if dep_a.junction and dep_b.junction:
406		- if dep_a.junction > dep_b.junction:
	385	+ if element_a.junction and element_b.junction:
	386	+ if element_a.junction > element_b.junction:
407	387	return 1
408		- elif dep_a.junction < dep_b.junction:
	388	+ elif element_a.junction < element_b.junction:
409	389	return -1
410		- elif dep_a.junction:
	390	+ elif element_a.junction:
411	391	return -1
412		- elif dep_b.junction:
	392	+ elif element_b.junction:
413	393	return 1
414	394
415	395	# This wont ever happen
...	...	@@ -418,26 +398,23 @@ class Loader():
418	398	# Now dependency sort, we ensure that if any direct dependency
419	399	# directly or indirectly depends on another direct dependency,
420	400	# it is found later in the list.
421		- element.deps.sort(key=cmp_to_key(dependency_cmp))
	401	+ element.dependencies.sort(key=cmp_to_key(dependency_cmp))
422	402
423		- visited[element_name] = True
	403	+ visited.add(element)
424	404
425	405	# _collect_element()
426	406	#
427	407	# Collect the toplevel elements we have
428	408	#
429	409	# Args:
430		- # element_name (str): The element-path relative element name to sort
	410	+ # element (LoadElement): The element for which to load a MetaElement
431	411	#
432	412	# Returns:
433	413	# (MetaElement): A recursively loaded MetaElement
434	414	#
435		- def _collect_element(self, element_name):
436		-
437		- element = self._elements[element_name]
438		-
	415	+ def _collect_element(self, element):
439	416	# Return the already built one, if we already built it
440		- meta_element = self._meta_elements.get(element_name)
	417	+ meta_element = self._meta_elements.get(element.name)
441	418	if meta_element:
442	419	return meta_element
443	420
...	...	@@ -461,10 +438,10 @@ class Loader():
461	438	del source[Symbol.DIRECTORY]
462	439
463	440	index = sources.index(source)
464		- meta_source = MetaSource(element_name, index, element_kind, kind, source, directory)
	441	+ meta_source = MetaSource(element.name, index, element_kind, kind, source, directory)
465	442	meta_sources.append(meta_source)
466	443
467		- meta_element = MetaElement(self.project, element_name, element_kind,
	444	+ meta_element = MetaElement(self.project, element.name, element_kind,
468	445	elt_provenance, meta_sources,
469	446	_yaml.node_get(node, Mapping, Symbol.CONFIG, default_value={}),
470	447	_yaml.node_get(node, Mapping, Symbol.VARIABLES, default_value={}),
...	...	@@ -475,19 +452,196 @@ class Loader():
475	452	element_kind == 'junction')
476	453
477	454	# Cache it now, make sure it's already there before recursing
478		- self._meta_elements[element_name] = meta_element
	455	+ self._meta_elements[element.name] = meta_element
479	456
480		- # Descend
481		- for dep in element.deps:
482		- loader = self._get_loader_for_dep(dep)
483		- meta_dep = loader._collect_element(dep.name)
484		- if dep.dep_type != 'runtime':
485		- meta_element.build_dependencies.append(meta_dep)
486		- if dep.dep_type != 'build':
487		- meta_element.dependencies.append(meta_dep)
	457	+ # # Descend
	458	+ # for dep in element.dependencies:
	459	+ # loader = dep.element._loader
	460	+ # meta_dep = loader._collect_element(dep.element)
	461	+ # if dep.dep_type != 'runtime':
	462	+ # meta_element.build_dependencies.append(meta_dep)
	463	+ # if dep.dep_type != 'build':
	464	+ # meta_element.dependencies.append(meta_dep)
488	465
489	466	return meta_element
490	467
	468	+ def _collect_elements(self, elements):
	469	+ elements_to_load = deque(elements)
	470	+
	471	+ def compare_unprocessed(dep_a, dep_b):
	472	+ if dep_a.dep_type != dep_b.dep_type:
	473	+ if dep_a.dep_type == Symbol.RUNTIME:
	474	+ return 1
	475	+ if dep_b.dep_type == Symbol.RUNTIME:
	476	+ return -1
	477	+
	478	+ element_a = dep_a.element
	479	+ element_b = dep_b.element
	480	+
	481	+ # All things being equal, string comparison.
	482	+ if element_a.name > element_b.name:
	483	+ return 1
	484	+ elif element_a.name < element_b.name:
	485	+ return -1
	486	+
	487	+ # Sort local elements before junction elements
	488	+ # and use string comparison between junction elements
	489	+ if element_a.junction and element_b.junction:
	490	+ if element_a.junction > element_b.junction:
	491	+ return 1
	492	+ elif element_a.junction < element_b.junction:
	493	+ return -1
	494	+ elif element_a.junction:
	495	+ return -1
	496	+ elif element_b.junction:
	497	+ return 1
	498	+
	499	+ return 0
	500	+
	501	+ print("#" * 60)
	502	+
	503	+ while elements_to_load:
	504	+ print([e.name for e in elements_to_load])
	505	+ element = elements_to_load.popleft()
	506	+
	507	+ print("Treating", element.name)
	508	+
	509	+ if element.visited:
	510	+ print("\tAlready visited")
	511	+ continue
	512	+
	513	+ unprocessed_dependencies = [
	514	+ dep
	515	+ for dep in element.dependencies
	516	+ if dep.element.visited is False
	517	+ ]
	518	+
	519	+ if unprocessed_dependencies:
	520	+ print("\tUnprocessed deps:", [e.element.name for e in unprocessed_dependencies])
	521	+ elements_to_load.appendleft(element)
	522	+ elements_to_load.extendleft(
	523	+ dep.element
	524	+ for dep in sorted(unprocessed_dependencies, key=cmp_to_key(compare_unprocessed), reverse=True)
	525	+ )
	526	+ continue
	527	+
	528	+ element.visited = True
	529	+ meta_element = element._loader._collect_element(element)
	530	+
	531	+ for dep in element.dependencies:
	532	+ print("\tElement has dependency:", dep.element.name)
	533	+ meta_dep = dep.element._loader._collect_element(dep.element)
	534	+
	535	+ if dep.dep_type != Symbol.RUNTIME:
	536	+ meta_element.build_dependencies.append(meta_dep)
	537	+ if dep.dep_type != Symbol.BUILD:
	538	+ meta_element.dependencies.append(meta_dep)
	539	+
	540	+ from operator import attrgetter
	541	+ meta_element.build_dependencies.sort(key=attrgetter("index"))
	542	+ meta_element.dependencies.sort(key=attrgetter("index"))
	543	+
	544	+ ret = [self._collect_element(e) for e in elements]
	545	+
	546	+ print("RETURN:", [e.name for e in ret])
	547	+ return ret
	548	+
	549	+ # levels = defaultdict(list)
	550	+
	551	+ # while elements_to_load:
	552	+ # print("#" * 60)
	553	+ # print([e.name for e in elements_to_load])
	554	+ # element = elements_to_load.popleft()
	555	+ # print("Treating element", element.name)
	556	+
	557	+ # if element.level is not None: # This node has already been treated
	558	+ # print("Element", element.name, "already treated with level", element.level)
	559	+ # continue
	560	+
	561	+ # unprocessed_reverse_dependencies = [
	562	+ # rdep
	563	+ # for rdep in element.reverse_dependencies
	564	+ # if rdep.level is None
	565	+ # ]
	566	+
	567	+ # if unprocessed_reverse_dependencies:
	568	+ # print("Element has unprocessed reverse dependencies:", [e.name for e in unprocessed_reverse_dependencies])
	569	+ # elements_to_load.appendleft(element)
	570	+ # elements_to_load.extendleft(unprocessed_reverse_dependencies)
	571	+ # continue
	572	+
	573	+ # element.level = 1 + max((e.level for e in element.reverse_dependencies), default=0)
	574	+ # levels[element.level].append(element)
	575	+ # print("Element level is", element.level)
	576	+ # print("Levels: ", {
	577	+ # key: [e.name for e in value]
	578	+ # for key, value in levels.items()
	579	+ # })
	580	+
	581	+ # def sort_by_dep_type(dep_a, dep_b):
	582	+ # if dep_a.dep_type != dep_b.dep_type:
	583	+ # if dep_a.dep_type == Symbol.RUNTIME:
	584	+ # return 1
	585	+ # if dep_b.dep_type == Symbol.RUNTIME:
	586	+ # return -1
	587	+
	588	+ # element_a = dep_a.element
	589	+ # element_b = dep_b.element
	590	+
	591	+ # # All things being equal, string comparison.
	592	+ # if element_a.name > element_b.name:
	593	+ # return 1
	594	+ # elif element_a.name < element_b.name:
	595	+ # return -1
	596	+
	597	+ # # Sort local elements before junction elements
	598	+ # # and use string comparison between junction elements
	599	+ # if element_a.junction and element_b.junction:
	600	+ # if element_a.junction > element_b.junction:
	601	+ # return 1
	602	+ # elif element_a.junction < element_b.junction:
	603	+ # return -1
	604	+ # elif element_a.junction:
	605	+ # return -1
	606	+ # elif element_b.junction:
	607	+ # return 1
	608	+
	609	+ # return 0
	610	+
	611	+ # elements_to_load.extend(
	612	+ # dep.element
	613	+ # for dep in sorted(element.dependencies, key=cmp_to_key(sort_by_dep_type))
	614	+ # )
	615	+
	616	+ # ret = []
	617	+
	618	+ # from operator import attrgetter
	619	+
	620	+ # print("Sorting levels")
	621	+
	622	+ # for _level in sorted(levels.keys(), reverse=True):
	623	+ # print("## Treating level", _level)
	624	+ # for element in levels[_level]:
	625	+ # print("\t## Element", element.name)
	626	+ # meta_element = self._collect_element(element)
	627	+
	628	+ # for dep in element.dependencies:
	629	+ # meta_dep = self._collect_element(dep.element)
	630	+
	631	+ # if dep.dep_type != Symbol.RUNTIME:
	632	+ # meta_element.build_dependencies.append(meta_dep)
	633	+ # if dep.dep_type != Symbol.BUILD:
	634	+ # meta_element.dependencies.append(meta_dep)
	635	+
	636	+ # meta_element.build_dependencies.sort(key=attrgetter("index"))
	637	+ # meta_element.dependencies.sort(key=attrgetter("index"))
	638	+
	639	+ # if _level == 1:
	640	+ # ret.append(meta_element)
	641	+
	642	+ # return ret
	643	+
	644	+
491	645	# _get_loader():
492	646	#
493	647	# Return loader for specified junction
...	...	@@ -539,7 +693,7 @@ class Loader():
539	693	return None
540	694
541	695	# meta junction element
542		- meta_element = self._collect_element(filename)
	696	+ meta_element = self._collect_element(self._elements[filename])
543	697	if meta_element.kind != 'junction':
544	698	raise LoadError(LoadErrorReason.INVALID_DATA,
545	699	"{}: Expected junction but element kind is {}".format(filename, meta_element.kind))
...	...	@@ -601,23 +755,6 @@ class Loader():
601	755
602	756	return loader
603	757
604		- # _get_loader_for_dep():
605		- #
606		- # Gets the appropriate Loader for a Dependency object
607		- #
608		- # Args:
609		- # dep (Dependency): A Dependency object
610		- #
611		- # Returns:
612		- # (Loader): The Loader object to use for this Dependency
613		- #
614		- def _get_loader_for_dep(self, dep):
615		- if dep.junction:
616		- # junction dependency, delegate to appropriate loader
617		- return self._loaders[dep.junction]
618		- else:
619		- return self
620		-
621	758	# _parse_name():
622	759	#
623	760	# Get junction and base name of element along with loader for the sub-project

buildstream/_loader/metaelement.py

---

...	...	@@ -17,9 +17,13 @@
17	17	# Authors:
18	18	# Tristan Van Berkom <tristan vanberkom codethink co uk>
19	19
	20	+import itertools
	21	+
20	22
21	23	class MetaElement():
22	24
	25	+ index_generator = itertools.count()
	26	+
23	27	# MetaElement()
24	28	#
25	29	# An abstract object holding data suitable for constructing an Element
...	...	@@ -55,3 +59,4 @@ class MetaElement():
55	59	self.build_dependencies = []
56	60	self.dependencies = []
57	61	self.first_pass = first_pass
	62	+ self.index = next(self.index_generator)

buildstream/_profile.py

---

...	...	@@ -43,7 +43,7 @@ initialized = False
43	43	# The special 'all' value will enable all profiles.
44	44	class Topics():
45	45	CIRCULAR_CHECK = 'circ-dep-check'
46		- SORT_DEPENDENCIES = 'sort-deps'
	46	+ COLLECT_ELEMENTS = 'collect-elements'
47	47	LOAD_LOADER = 'load-loader'
48	48	LOAD_CONTEXT = 'load-context'
49	49	LOAD_PROJECT = 'load-project'

buildstream/plugins/elements/filter.py

---

...	...	@@ -20,25 +20,127 @@
20	20	"""
21	21	filter - Extract a subset of files from another element
22	22	=======================================================
23		-This filters another element by producing an output that is a subset of
24		-the filtered element.
	23	+Filter another element by producing an output that is a subset of
	24	+the parent element's output. Subsets are defined by the parent element's
	25	+:ref:`split rules <public_split_rules>`.
25	26
26		-To specify the element to filter, specify it as the one and only build
27		-dependency to filter. See :ref:`Dependencies <format_dependencies>`
28		-for what dependencies are and how to specify them.
	27	+Overview
	28	+--------
	29	+A filter element must have exactly one *build* dependency, where said
	30	+dependency is the 'parent' element which we would like to filter.
	31	+Runtime dependencies may also be specified, which can be useful to propagate
	32	+forward from this filter element onto its reverse dependencies.
	33	+See :ref:`Dependencies <format_dependencies>` to see how we specify dependencies.
29	34
30		-Dependencies aside from the filtered element may be specified, but
31		-they must be runtime dependencies only. This can be useful to propagate
32		-runtime dependencies forward from this filter element onto its reverse
33		-dependencies.
	35	+When workspaces are opened, closed or reset on a filter element, or this
	36	+element is tracked, the filter element will transparently pass on the command
	37	+to its parent element (the sole build-dependency).
34	38
35		-When workspaces are opened, closed or reset on this element, or this
36		-element is tracked, instead of erroring due to a lack of sources, this
37		-element will transparently pass on the command to its sole build-dependency.
	39	+Example
	40	+-------
	41	+Consider a simple import element, ``import.bst`` which imports the local files
	42	+'foo', 'bar' and 'baz' (each stored in ``files/``, relative to the project's root):
38	43
39		-The default configuration and possible options are as such:
40		- .. literalinclude:: ../../../buildstream/plugins/elements/filter.yaml
41		- :language: yaml
	44	+.. code:: yaml
	45	+
	46	+ kind: import
	47	+
	48	+ # Specify sources to import
	49	+ sources:
	50	+ - kind: local
	51	+ path: files
	52	+
	53	+ # Specify public domain data, visible to other elements
	54	+ public:
	55	+ bst:
	56	+ split-rules:
	57	+ foo:
	58	+ - /foo
	59	+ bar:
	60	+ - /bar
	61	+
	62	+.. note::
	63	+
	64	+ We can make an element's metadata visible to all reverse dependencies by making use
	65	+ of the ``public:`` field. See the :ref:`public data documentation <format_public>`
	66	+ for more information.
	67	+
	68	+In this example, ``import.bst`` will serve as the 'parent' of the filter element, thus
	69	+its output will be filtered. It is important to understand that the artifact of the
	70	+above element will contain the files: 'foo', 'bar' and 'baz'.
	71	+
	72	+Now, to produce an element whose artifact contains the file 'foo', and exlusively 'foo',
	73	+we can define the following filter, ``filter-foo.bst``:
	74	+
	75	+.. code:: yaml
	76	+
	77	+ kind: filter
	78	+
	79	+ # Declare the sole build-dependency of the filter element
	80	+ depends:
	81	+ - filename: import.bst
	82	+ type: build
	83	+
	84	+ # Declare a list of domains to include in the filter's artifact
	85	+ config:
	86	+ include:
	87	+ - foo
	88	+
	89	+.. note::
	90	+
	91	+ We can also specify build-dependencies with a 'build-depends' field which has been
	92	+ available since :ref:`format version 14 <project_format_version>`. See the
	93	+ :ref:`Build-Depends documentation <format_build_depends>` for more detail.
	94	+
	95	+It should be noted that an 'empty' ``include:`` list would, by default, include all
	96	+split-rules specified in the parent element, which, in this example, would be the
	97	+files 'foo' and 'bar' (the file 'baz' was not covered by any split rules).
	98	+
	99	+Equally, we can use the ``exclude:`` statement to create the same artifact (which
	100	+only contains the file 'foo') by declaring the following element, ``exclude-bar.bst``:
	101	+
	102	+.. code:: yaml
	103	+
	104	+ kind: filter
	105	+
	106	+ # Declare the sole build-dependency of the filter element
	107	+ depends:
	108	+ - filename: import.bst
	109	+ type: build
	110	+
	111	+ # Declare a list of domains to exclude in the filter's artifact
	112	+ config:
	113	+ exclude:
	114	+ - bar
	115	+
	116	+In addition to the ``include:`` and ``exclude:`` fields, there exists an ``include-orphans:``
	117	+(Boolean) field, which defaults to ``False``. This will determine whether to include files
	118	+which are not present in the 'split-rules'. For example, if we wanted to filter out all files
	119	+which are not included as split rules we can define the following element, ``filter-misc.bst``:
	120	+
	121	+.. code:: yaml
	122	+
	123	+ kind: filter
	124	+
	125	+ # Declare the sole build-dependency of the filter element
	126	+ depends:
	127	+ - filename: import.bst
	128	+ type: build
	129	+
	130	+ # Filter out all files which are not declared as split rules
	131	+ config:
	132	+ exclude:
	133	+ - foo
	134	+ - bar
	135	+ include-orphans: True
	136	+
	137	+The artifact of ``filter-misc.bst`` will only contain the file 'baz'.
	138	+
	139	+Below is more information regarding the the default configurations and possible options
	140	+of the filter element:
	141	+
	142	+.. literalinclude:: ../../../buildstream/plugins/elements/filter.yaml
	143	+ :language: yaml
42	144	"""
43	145
44	146	from buildstream import Element, ElementError, Scope

buildstream/plugins/elements/filter.yaml

---

...	...	@@ -2,20 +2,21 @@
2	2	# Filter element configuration
3	3	config:
4	4
5		- # A list of domains to include from each artifact, as
6		- # they were defined in the element's 'split-rules'.
	5	+ # A list of domains to include in each artifact, as
	6	+ # they were defined as public data in the parent
	7	+ # element's 'split-rules'.
7	8	#
8	9	# Since domains can be added, it is not an error to
9	10	# specify domains which may not exist for all of the
10	11	# elements in this composition.
11	12	#
12	13	# The default empty list indicates that all domains
13		- # from each dependency should be included.
	14	+ # of the parent's artifact should be included.
14	15	#
15	16	include: []
16	17
17	18	# A list of domains to exclude from each artifact, as
18		- # they were defined in the element's 'split-rules'.
	19	+ # they were defined in the parent element's 'split-rules'.
19	20	#
20	21	# In the case that a file is spoken for by a domain
21	22	# in the 'include' list and another in the 'exclude'
...	...	@@ -23,7 +24,7 @@ config:
23	24	exclude: []
24	25
25	26	# Whether to include orphan files which are not
26		- # included by any of the 'split-rules' present on
27		- # a given element.
	27	+ # included by any of the 'split-rules' present in
	28	+ # the parent element.
28	29	#
29	30	include-orphans: False

tests/cachekey/cachekey.py

---

...	...	@@ -214,3 +214,41 @@ def test_cache_key_fatal_warnings(cli, tmpdir, first_warnings, second_warnings,
214	214	second_keys = run_get_cache_key("second", second_warnings)
215	215
216	216	assert compare_cache_keys(first_keys, second_keys) == identical_keys
	217	+
	218	+
	219	+@pytest.mark.datafiles(DATA_DIR)
	220	+def test_keys_stable_over_targets(cli, datafiles):
	221	+ root_element = 'elements/key-stability/top-level.bst'
	222	+ target1 = 'elements/key-stability/t1.bst'
	223	+ target2 = 'elements/key-stability/t2.bst'
	224	+
	225	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	226	+ full_graph_result = cli.run(project=project, args=[
	227	+ 'show',
	228	+ '--format', '%{name}::%{full-key}',
	229	+ root_element
	230	+ ])
	231	+ full_graph_result.assert_success()
	232	+ all_cache_keys = parse_output_keys(full_graph_result.output)
	233	+
	234	+ ordering1_result = cli.run(project=project, args=[
	235	+ 'show',
	236	+ '--format', '%{name}::%{full-key}',
	237	+ target1,
	238	+ target2
	239	+ ])
	240	+ ordering1_result.assert_success()
	241	+ ordering1_cache_keys = parse_output_keys(ordering1_result.output)
	242	+
	243	+ ordering2_result = cli.run(project=project, args=[
	244	+ 'show',
	245	+ '--format', '%{name}::%{full-key}',
	246	+ target2,
	247	+ target1
	248	+ ])
	249	+ ordering2_result.assert_success()
	250	+ ordering2_cache_keys = parse_output_keys(ordering2_result.output)
	251	+
	252	+ for element in ordering1_cache_keys:
	253	+ assert ordering1_cache_keys[element] == ordering2_cache_keys[element]
	254	+ assert ordering1_cache_keys[element] == all_cache_keys[element]

tests/cachekey/project/elements/key-stability/aaa.bst

---

	1	+kind: import
	2	+sources:
	3	+- kind: local
	4	+ path: elements/key-stability/aaa.bst

tests/cachekey/project/elements/key-stability/t1.bst

---

	1	+kind: import
	2	+sources:
	3	+- kind: local
	4	+ path: elements/key-stability/t1.bst
	5	+depends:
	6	+- elements/key-stability/zzz.bst

tests/cachekey/project/elements/key-stability/t2.bst

---

	1	+kind: import
	2	+sources:
	3	+- kind: local
	4	+ path: elements/key-stability/t2.bst
	5	+depends:
	6	+- elements/key-stability/aaa.bst
	7	+- elements/key-stability/zzz.bst

tests/cachekey/project/elements/key-stability/top-level.bst

---

	1	+kind: import
	2	+sources:
	3	+- kind: local
	4	+ path: elements/key-stability/top-level.bst
	5	+depends:
	6	+- elements/key-stability/t1.bst
	7	+- elements/key-stability/t2.bst

tests/cachekey/project/elements/key-stability/zzz.bst

---

	1	+kind: import
	2	+sources:
	3	+- kind: local
	4	+ path: elements/key-stability/zzz.bst

tests/testutils/site.py

---

...	...	@@ -18,7 +18,7 @@ try:
18	18	utils.get_host_tool('git')
19	19	HAVE_GIT = True
20	20	out = str(subprocess.check_output(['git', '--version']), "utf-8")
21		- version = tuple(int(x) for x in out.split(' ', 2)[2].split('.'))
	21	+ version = tuple(int(x) for x in out.split(' ')[2].split('.'))
22	22	HAVE_OLD_GIT = version < (1, 8, 5)
23	23	except ProgramNotFoundError:
24	24	HAVE_GIT = False

tox.ini

---

...	...	@@ -88,5 +88,5 @@ whitelist_externals =
88	88	commands =
89	89	python3 setup.py --command-packages=click_man.commands man_pages
90	90	deps =
91		- click-man
	91	+ click-man >= 0.3.0
92	92	-rrequirements/requirements.txt

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.