[Notes] [Git][BuildStream/buildstream][tpollard/494] 7 commits: Fix python warnings: Use collections.abc instead collections

[Tom Pollard <gitlab mg gitlab com>]

Title: GitLab

Tom Pollard pushed to branch tpollard/494 at BuildStream / buildstream

Commits:

• f1aa0974
  by Javier Jardón at 2018-10-18T14:13:35Z

  Fix python warnings: Use collections.abc instead collections
  
  Since python 3.3, collections has been moved to collections.abc module.
  For backwards compatibility, they continue to be visible in this module through Python 3.7.
  Subsequently, they will be removed entirely.
  
  See https://docs.python.org/3/library/collections.html
  

• f1550c8e
  by Valentin David at 2018-10-18T16:02:45Z

  Merge branch 'jjardon/warnings_collections' into 'master'
  
  Fix python warnings: Use collections.abc instead collections
  
  See merge request BuildStream/buildstream!883

• 1c24979a
  by Angelos Evripiotis at 2018-10-19T21:17:01Z

  Spelling fixes: 'affect', not 'effect' as verb
  
  Fix most instances of 'effect' being used as a verb instead of a noun.
  Skipped generated instance in doc/source/conf.py.
  

• 43a2eee3
  by Angelos Evripiotis at 2018-10-19T21:48:59Z

  Merge branch 'aevri/affect' into 'master'
  
  Spelling fixes: 'affect', not 'effect' as verb
  
  See merge request BuildStream/buildstream!885

• 802bb13f
  by Tom Pollard at 2018-10-22T10:16:24Z

  _frontend/cli.py: Add --pull-buildtrees to bst build & pull
  
  The addition of the pull-buildtrees flag will be used to override
  the default behaviour of remote artifact cache pulling. Included
  is the required changes to completions tests.
  

• fdeabdd0
  by Tom Pollard at 2018-10-22T10:16:24Z

  buildstream/_context.py: Add pullbuildtrees global user option
  
  The addition of the pullbuildtrees option to the users global
  buildstream.conf will be used to override the default behaviour
  of remote artifact cache pulling.
  

• efcfb1a7
  by Tom Pollard at 2018-10-22T10:16:24Z

  Don't pull artifact buildtrees by default.
  
  The addition of cached buildtrees being included in element
  artifacts has led to mostly redundant download overheads when
  pulling from a remote artifact server. As such the default behaviour
  of pull shouldn't fetch the buildtree object if available.
  
  _stream.py: propagate if pullbuildtrees has been set via cli or
  user config to the PullQueue.
  
  pullqueue.py: add extensible attributes to determine required/
  excluded directories for element pull jobs.
  
  element.py: extend relevant pull logic for specified subdir
  consideration and ensure push logic does not lead to partial
  artifact pushes.
  
  _artifactcache/: artifactcache.py & cascache.py inclusion of
  helper functions for subdir artifact checking & fetching, fetch
  logic extended to only pull required artifact directories.
  extract & checkout updated to handle full/partial operation
  
  tests/: integration test pullbuildtrees.py & fix for testutils/
  artifactshare.py pathing.
  

30 changed files:

• NEWS
• buildstream/_artifactcache/artifactcache.py
• buildstream/_artifactcache/cascache.py
• buildstream/_context.py
• buildstream/_frontend/cli.py
• buildstream/_includes.py
• buildstream/_loader/loadelement.py
• buildstream/_loader/loader.py
• buildstream/_options/optionpool.py
• buildstream/_project.py
• buildstream/_scheduler/queues/pullqueue.py
• buildstream/_scheduler/queues/queue.py
• buildstream/_stream.py
• buildstream/_yaml.py
• buildstream/buildelement.py
• buildstream/element.py
• buildstream/plugin.py
• buildstream/plugins/elements/autotools.yaml
• buildstream/plugins/elements/cmake.yaml
• buildstream/plugins/elements/junction.py
• buildstream/plugins/elements/make.yaml
• buildstream/plugins/elements/manual.yaml
• buildstream/plugins/elements/meson.yaml
• buildstream/plugins/elements/qmake.yaml
• buildstream/plugins/sources/git.py
• buildstream/source.py
• tests/completions/completions.py
• + tests/integration/pullbuildtrees.py
• tests/testutils/artifactshare.py
• tests/yaml/yaml.py

Changes:

NEWS

---

...	...	@@ -31,6 +31,17 @@ buildstream 1.3.1
31	31	new the `conf-root` variable to make the process easier. And there has been
32	32	a bug fix to workspaces so they can be build in workspaces too.
33	33
	34	+ o Due to the element `buildtree` being cached in the respective artifact their
	35	+ size in some cases has significantly increased. In *most* cases the buildtree
	36	+ is not utilised when building targets, as such by default bst 'pull' & 'build'
	37	+ will not fetch buildtrees from remotes. This behaviour can be overriden with
	38	+ the cli option '--pull-buildtrees', or the user configuration option
	39	+ 'pullbuildtrees = True'. The override will also add the buildtree to already
	40	+ cached artifacts. When attempting to populate an artifactcache server with
	41	+ cached artifacts, only 'complete' elements can be pushed. If the element is
	42	+ expected to have a populated buildtree then it must be cached before pushing.
	43	+
	44	+
34	45	=================
35	46	buildstream 1.1.5
36	47	=================

buildstream/_artifactcache/artifactcache.py

---

...	...	@@ -19,7 +19,8 @@
19	19
20	20	import os
21	21	import string
22		-from collections import Mapping, namedtuple
	22	+from collections import namedtuple
	23	+from collections.abc import Mapping
23	24
24	25	from ..types import _KeyStrength
25	26	from .._exceptions import ArtifactError, ImplError, LoadError, LoadErrorReason
...	...	@@ -426,6 +427,22 @@ class ArtifactCache():
426	427	raise ImplError("Cache '{kind}' does not implement contains()"
427	428	.format(kind=type(self).__name__))
428	429
	430	+ # contains_subdir_artifact():
	431	+ #
	432	+ # Check whether an artifact element contains a digest for a subdir
	433	+ # which is populated in the cache, i.e non dangling.
	434	+ #
	435	+ # Args:
	436	+ # element (Element): The Element to check
	437	+ # key (str): The cache key to use
	438	+ # subdir (str): The subdir to check
	439	+ #
	440	+ # Returns: True if the subdir exists & is populated in the cache, False otherwise
	441	+ #
	442	+ def contains_subdir_artifact(self, element, key, subdir):
	443	+ raise ImplError("Cache '{kind}' does not implement contains_subdir_artifact()"
	444	+ .format(kind=type(self).__name__))
	445	+
429	446	# list_artifacts():
430	447	#
431	448	# List artifacts in this cache in LRU order.
...	...	@@ -462,6 +479,7 @@ class ArtifactCache():
462	479	# Args:
463	480	# element (Element): The Element to extract
464	481	# key (str): The cache key to use
	482	+ # subdir (str): The optional subdir to check exists
465	483	#
466	484	# Raises:
467	485	# ArtifactError: In cases there was an OSError, or if the artifact
...	...	@@ -469,7 +487,7 @@ class ArtifactCache():
469	487	#
470	488	# Returns: path to extracted artifact
471	489	#
472		- def extract(self, element, key):
	490	+ def extract(self, element, key, subdir=None):
473	491	raise ImplError("Cache '{kind}' does not implement extract()"
474	492	.format(kind=type(self).__name__))
475	493
...	...	@@ -551,11 +569,13 @@ class ArtifactCache():
551	569	# element (Element): The Element whose artifact is to be fetched
552	570	# key (str): The cache key to use
553	571	# progress (callable): The progress callback, if any
	572	+ # subdir (str): The optional specific subdir to pull
	573	+ # excluded_subdirs (list): The optional list of subdirs to not pull
554	574	#
555	575	# Returns:
556	576	# (bool): True if pull was successful, False if artifact was not available
557	577	#
558		- def pull(self, element, key, *, progress=None):
	578	+ def pull(self, element, key, *, progress=None, subdir=None, excluded_subdirs=None):
559	579	raise ImplError("Cache '{kind}' does not implement pull()"
560	580	.format(kind=type(self).__name__))
561	581

buildstream/_artifactcache/cascache.py

---

...	...	@@ -92,16 +92,36 @@ class CASCache(ArtifactCache):
92	92	# This assumes that the repository doesn't have any dangling pointers
93	93	return os.path.exists(refpath)
94	94
95		- def extract(self, element, key):
	95	+ def contains_subdir_artifact(self, element, key, subdir):
	96	+ tree = self.resolve_ref(self.get_artifact_fullname(element, key))
	97	+
	98	+ # This assumes that the subdir digest is present in the element tree
	99	+ subdirdigest = self._get_subdir(tree, subdir)
	100	+ objpath = self.objpath(subdirdigest)
	101	+
	102	+ # True if subdir content is cached or if empty as expected
	103	+ return os.path.exists(objpath)
	104	+
	105	+ def extract(self, element, key, subdir=None):
96	106	ref = self.get_artifact_fullname(element, key)
97	107
98	108	tree = self.resolve_ref(ref, update_mtime=True)
99	109
100		- dest = os.path.join(self.extractdir, element._get_project().name,
101		- element.normal_name, tree.hash)
	110	+ dest = elementdest = os.path.join(self.extractdir, element._get_project().name,
	111	+ element.normal_name, tree.hash)
	112	+
102	113	if os.path.isdir(dest):
103		- # artifact has already been extracted
104		- return dest
	114	+ if subdir:
	115	+ # Check if we have optional subdir in the local cache and not already extracted
	116	+ subdircached = self.contains_subdir_artifact(element, key, subdir)
	117	+ if subdircached and not os.path.isdir(os.path.join(dest, subdir)):
	118	+ # Artifact has already been extracted without subdir content, only need to checkout the subdir
	119	+ tree = self._get_subdir(tree, subdir)
	120	+ dest = os.path.join(dest, subdir)
	121	+ else:
	122	+ return dest
	123	+ else:
	124	+ return dest
105	125
106	126	with tempfile.TemporaryDirectory(prefix='tmp', dir=self.extractdir) as tmpdir:
107	127	checkoutdir = os.path.join(tmpdir, ref)
...	...	@@ -120,7 +140,7 @@ class CASCache(ArtifactCache):
120	140	raise ArtifactError("Failed to extract artifact for ref '{}': {}"
121	141	.format(ref, e)) from e
122	142
123		- return dest
	143	+ return elementdest
124	144
125	145	def commit(self, element, content, keys):
126	146	refs = [self.get_artifact_fullname(element, key) for key in keys]
...	...	@@ -228,7 +248,7 @@ class CASCache(ArtifactCache):
228	248	remotes_for_project = self._remotes[element._get_project()]
229	249	return any(remote.spec.push for remote in remotes_for_project)
230	250
231		- def pull(self, element, key, *, progress=None):
	251	+ def pull(self, element, key, *, progress=None, subdir=None, excluded_subdirs=None):
232	252	ref = self.get_artifact_fullname(element, key)
233	253
234	254	project = element._get_project()
...	...	@@ -247,8 +267,14 @@ class CASCache(ArtifactCache):
247	267	tree.hash = response.digest.hash
248	268	tree.size_bytes = response.digest.size_bytes
249	269
250		- self._fetch_directory(remote, tree)
	270	+ # Check if the element artifact is present, if so just fetch subdir
	271	+ if subdir and os.path.exists(self.objpath(tree)):
	272	+ self._fetch_subdir(remote, tree, subdir)
	273	+ else:
	274	+ # Fetch artifact, excluded_subdirs determined in pullqueue
	275	+ self._fetch_directory(remote, tree, excluded_subdirs=excluded_subdirs)
251	276
	277	+ # tree is the remote value, so is the same without or without dangling ref locally
252	278	self.set_ref(ref, tree)
253	279
254	280	element.info("Pulled artifact {} <- {}".format(display_key, remote.spec.url))
...	...	@@ -671,8 +697,10 @@ class CASCache(ArtifactCache):
671	697	stat.S_IRGRP | stat.S_IXGRP | stat.S_IROTH | stat.S_IXOTH)
672	698
673	699	for dirnode in directory.directories:
674		- fullpath = os.path.join(dest, dirnode.name)
675		- self._checkout(fullpath, dirnode.digest)
	700	+ # Don't try to checkout a dangling ref
	701	+ if os.path.exists(self.objpath(dirnode.digest)):
	702	+ fullpath = os.path.join(dest, dirnode.name)
	703	+ self._checkout(fullpath, dirnode.digest)
676	704
677	705	for symlinknode in directory.symlinks:
678	706	# symlink
...	...	@@ -950,11 +978,14 @@ class CASCache(ArtifactCache):
950	978	# Args:
951	979	# remote (Remote): The remote to use.
952	980	# dir_digest (Digest): Digest object for the directory to fetch.
	981	+ # excluded_subdirs (list): The optional list of subdirs to not fetch
953	982	#
954		- def _fetch_directory(self, remote, dir_digest):
	983	+ def _fetch_directory(self, remote, dir_digest, *, excluded_subdirs=None):
955	984	fetch_queue = [dir_digest]
956	985	fetch_next_queue = []
957	986	batch = _CASBatchRead(remote)
	987	+ if not excluded_subdirs:
	988	+ excluded_subdirs = []
958	989
959	990	while len(fetch_queue) + len(fetch_next_queue) > 0:
960	991	if len(fetch_queue) == 0:
...	...	@@ -969,8 +1000,9 @@ class CASCache(ArtifactCache):
969	1000	directory.ParseFromString(f.read())
970	1001
971	1002	for dirnode in directory.directories:
972		- batch = self._fetch_directory_node(remote, dirnode.digest, batch,
973		- fetch_queue, fetch_next_queue, recursive=True)
	1003	+ if dirnode.name not in excluded_subdirs:
	1004	+ batch = self._fetch_directory_node(remote, dirnode.digest, batch,
	1005	+ fetch_queue, fetch_next_queue, recursive=True)
974	1006
975	1007	for filenode in directory.files:
976	1008	batch = self._fetch_directory_node(remote, filenode.digest, batch,
...	...	@@ -979,6 +1011,10 @@ class CASCache(ArtifactCache):
979	1011	# Fetch final batch
980	1012	self._fetch_directory_batch(remote, batch, fetch_queue, fetch_next_queue)
981	1013
	1014	+ def _fetch_subdir(self, remote, tree, subdir):
	1015	+ subdirdigest = self._get_subdir(tree, subdir)
	1016	+ self._fetch_directory(remote, subdirdigest)
	1017	+
982	1018	def _fetch_tree(self, remote, digest):
983	1019	# download but do not store the Tree object
984	1020	with tempfile.NamedTemporaryFile(dir=self.tmpdir) as out:

buildstream/_context.py

---

...	...	@@ -19,7 +19,8 @@
19	19
20	20	import os
21	21	import datetime
22		-from collections import deque, Mapping
	22	+from collections import deque
	23	+from collections.abc import Mapping
23	24	from contextlib import contextmanager
24	25	from . import utils
25	26	from . import _cachekey
...	...	@@ -110,6 +111,9 @@ class Context():
110	111	# Make sure the XDG vars are set in the environment before loading anything
111	112	self._init_xdg()
112	113
	114	+ # Whether or not to attempt to pull buildtrees globally
	115	+ self.pullbuildtrees = False
	116	+
113	117	# Private variables
114	118	self._cache_key = None
115	119	self._message_handler = None
...	...	@@ -160,7 +164,7 @@ class Context():
160	164	_yaml.node_validate(defaults, [
161	165	'sourcedir', 'builddir', 'artifactdir', 'logdir',
162	166	'scheduler', 'artifacts', 'logging', 'projects',
163		- 'cache'
	167	+ 'cache', 'pullbuildtrees'
164	168	])
165	169
166	170	for directory in ['sourcedir', 'builddir', 'artifactdir', 'logdir']:
...	...	@@ -185,6 +189,9 @@ class Context():
185	189	# Load artifact share configuration
186	190	self.artifact_cache_specs = ArtifactCache.specs_from_config_node(defaults)
187	191
	192	+ # Load pull buildtrees configuration
	193	+ self.pullbuildtrees = _yaml.node_get(defaults, bool, 'pullbuildtrees', default_value='False')
	194	+
188	195	# Load logging config
189	196	logging = _yaml.node_get(defaults, Mapping, 'logging')
190	197	_yaml.node_validate(logging, [

buildstream/_frontend/cli.py

---

...	...	@@ -305,10 +305,12 @@ def init(app, project_name, format_version, element_path, force):
305	305	help="Allow tracking to cross junction boundaries")
306	306	@click.option('--track-save', default=False, is_flag=True,
307	307	help="Deprecated: This is ignored")
	308	+@click.option('--pull-buildtrees', default=False, is_flag=True,
	309	+ help="Pull buildtrees from a remote cache server")
308	310	@click.argument('elements', nargs=-1,
309	311	type=click.Path(readable=False))
310	312	@click.pass_obj
311		-def build(app, elements, all_, track_, track_save, track_all, track_except, track_cross_junctions):
	313	+def build(app, elements, all_, track_, track_save, track_all, track_except, track_cross_junctions, pull_buildtrees):
312	314	"""Build elements in a pipeline"""
313	315
314	316	if (track_except or track_cross_junctions) and not (track_ or track_all):
...	...	@@ -327,7 +329,8 @@ def build(app, elements, all_, track_, track_save, track_all, track_except, trac
327	329	track_targets=track_,
328	330	track_except=track_except,
329	331	track_cross_junctions=track_cross_junctions,
330		- build_all=all_)
	332	+ build_all=all_,
	333	+ pull_buildtrees=pull_buildtrees)
331	334
332	335
333	336	##################################################################
...	...	@@ -429,10 +432,12 @@ def track(app, elements, deps, except_, cross_junctions):
429	432	help='The dependency artifacts to pull (default: none)')
430	433	@click.option('--remote', '-r',
431	434	help="The URL of the remote cache (defaults to the first configured cache)")
	435	+@click.option('--pull-buildtrees', default=False, is_flag=True,
	436	+ help="Pull buildtrees from a remote cache server")
432	437	@click.argument('elements', nargs=-1,
433	438	type=click.Path(readable=False))
434	439	@click.pass_obj
435		-def pull(app, elements, deps, remote):
	440	+def pull(app, elements, deps, remote, pull_buildtrees):
436	441	"""Pull a built artifact from the configured remote artifact cache.
437	442
438	443	By default the artifact will be pulled one of the configured caches
...	...	@@ -446,7 +451,7 @@ def pull(app, elements, deps, remote):
446	451	all: All dependencies
447	452	"""
448	453	with app.initialized(session_name="Pull"):
449		- app.stream.pull(elements, selection=deps, remote=remote)
	454	+ app.stream.pull(elements, selection=deps, remote=remote, pull_buildtrees=pull_buildtrees)
450	455
451	456
452	457	##################################################################

buildstream/_includes.py

---

1	1	import os
2		-from collections import Mapping
	2	+from collections.abc import Mapping
3	3	from . import _yaml
4	4	from ._exceptions import LoadError, LoadErrorReason
5	5

buildstream/_loader/loadelement.py

---

...	...	@@ -18,7 +18,7 @@
18	18	# Tristan Van Berkom <tristan vanberkom codethink co uk>
19	19
20	20	# System imports
21		-from collections import Mapping
	21	+from collections.abc import Mapping
22	22
23	23	# BuildStream toplevel imports
24	24	from .._exceptions import LoadError, LoadErrorReason

buildstream/_loader/loader.py

---

...	...	@@ -19,7 +19,8 @@
19	19
20	20	import os
21	21	from functools import cmp_to_key
22		-from collections import Mapping, namedtuple
	22	+from collections import namedtuple
	23	+from collections.abc import Mapping
23	24	import tempfile
24	25	import shutil
25	26

buildstream/_options/optionpool.py

---

...	...	@@ -18,7 +18,7 @@
18	18	# Tristan Van Berkom <tristan vanberkom codethink co uk>
19	19	#
20	20
21		-from collections import Mapping
	21	+from collections.abc import Mapping
22	22	import jinja2
23	23
24	24	from .. import _yaml

buildstream/_project.py

---

...	...	@@ -19,7 +19,8 @@
19	19	# Tiago Gomes <tiago gomes codethink co uk>
20	20
21	21	import os
22		-from collections import Mapping, OrderedDict
	22	+from collections import OrderedDict
	23	+from collections.abc import Mapping
23	24	from pluginbase import PluginBase
24	25	from . import utils
25	26	from . import _cachekey

buildstream/_scheduler/queues/pullqueue.py

---

...	...	@@ -32,9 +32,20 @@ class PullQueue(Queue):
32	32	complete_name = "Pulled"
33	33	resources = [ResourceType.DOWNLOAD, ResourceType.CACHE]
34	34
	35	+ def __init__(self, scheduler, buildtrees=False):
	36	+ super().__init__(scheduler)
	37	+
	38	+ # Current default exclusions on pull
	39	+ self._excluded_subdirs = ["buildtree"]
	40	+ self._subdir = None
	41	+ # If buildtrees are to be pulled, remove the value from exclusion list
	42	+ if buildtrees:
	43	+ self._subdir = "buildtree"
	44	+ self._excluded_subdirs.remove(self._subdir)
	45	+
35	46	def process(self, element):
36	47	# returns whether an artifact was downloaded or not
37		- if not element._pull():
	48	+ if not element._pull(subdir=self._subdir, excluded_subdirs=self._excluded_subdirs):
38	49	raise SkipJob(self.action_name)
39	50
40	51	def status(self, element):
...	...	@@ -49,7 +60,7 @@ class PullQueue(Queue):
49	60	if not element._can_query_cache():
50	61	return QueueStatus.WAIT
51	62
52		- if element._pull_pending():
	63	+ if element._pull_pending(subdir=self._subdir):
53	64	return QueueStatus.READY
54	65	else:
55	66	return QueueStatus.SKIP

buildstream/_scheduler/queues/queue.py

---

...	...	@@ -208,7 +208,7 @@ class Queue():
208	208	# This will have different results for elements depending
209	209	# on the Queue.status() implementation.
210	210	#
211		- # o Elements which are QueueStatus.WAIT will not be effected
	211	+ # o Elements which are QueueStatus.WAIT will not be affected
212	212	#
213	213	# o Elements which are QueueStatus.SKIP will move directly
214	214	# to the dequeue pool

buildstream/_stream.py

---

...	...	@@ -160,12 +160,14 @@ class Stream():
160	160	# track_cross_junctions (bool): Whether tracking should cross junction boundaries
161	161	# build_all (bool): Whether to build all elements, or only those
162	162	# which are required to build the target.
	163	+ # pull_buildtrees (bool): Whether to pull buildtrees from a remote cache server
163	164	#
164	165	def build(self, targets, *,
165	166	track_targets=None,
166	167	track_except=None,
167	168	track_cross_junctions=False,
168		- build_all=False):
	169	+ build_all=False,
	170	+ pull_buildtrees=False):
169	171
170	172	if build_all:
171	173	selection = PipelineSelection.ALL
...	...	@@ -195,7 +197,10 @@ class Stream():
195	197	self._add_queue(track_queue, track=True)
196	198
197	199	if self._artifacts.has_fetch_remotes():
198		- self._add_queue(PullQueue(self._scheduler))
	200	+ # Query if pullbuildtrees has been set globally in user config
	201	+ if self._context.pullbuildtrees:
	202	+ pull_buildtrees = True
	203	+ self._add_queue(PullQueue(self._scheduler, buildtrees=pull_buildtrees))
199	204
200	205	self._add_queue(FetchQueue(self._scheduler, skip_cached=True))
201	206	self._add_queue(BuildQueue(self._scheduler))
...	...	@@ -295,7 +300,8 @@ class Stream():
295	300	#
296	301	def pull(self, targets, *,
297	302	selection=PipelineSelection.NONE,
298		- remote=None):
	303	+ remote=None,
	304	+ pull_buildtrees=False):
299	305
300	306	use_config = True
301	307	if remote:
...	...	@@ -310,8 +316,12 @@ class Stream():
310	316	if not self._artifacts.has_fetch_remotes():
311	317	raise StreamError("No artifact caches available for pulling artifacts")
312	318
	319	+ # Query if pullbuildtrees has been set globally in user config
	320	+ if self._context.pullbuildtrees:
	321	+ pull_buildtrees = True
	322	+
313	323	self._pipeline.assert_consistent(elements)
314		- self._add_queue(PullQueue(self._scheduler))
	324	+ self._add_queue(PullQueue(self._scheduler, buildtrees=pull_buildtrees))
315	325	self._enqueue_plan(elements)
316	326	self._run()
317	327

buildstream/_yaml.py

---

...	...	@@ -972,7 +972,7 @@ def node_validate(node, valid_keys):
972	972	#
973	973	# The purpose of this is to create a virtual copy-on-write
974	974	# copy of a dictionary, so that mutating it in any way does
975		-# not effect the underlying dictionaries.
	975	+# not affect the underlying dictionaries.
976	976	#
977	977	# collections.ChainMap covers this already mostly, but fails
978	978	# to record internal state so as to hide keys which have been

buildstream/buildelement.py

---

...	...	@@ -176,7 +176,7 @@ class BuildElement(Element):
176	176
177	177	# Specifying notparallel for a given element effects the
178	178	# cache key, while having the side effect of setting max-jobs to 1,
179		- # which is normally automatically resolved and does not effect
	179	+ # which is normally automatically resolved and does not affect
180	180	# the cache key.
181	181	if self.get_variable('notparallel'):
182	182	dictionary['notparallel'] = True

buildstream/element.py

---

...	...	@@ -76,7 +76,8 @@ import os
76	76	import re
77	77	import stat
78	78	import copy
79		-from collections import Mapping, OrderedDict
	79	+from collections import OrderedDict
	80	+from collections.abc import Mapping
80	81	from contextlib import contextmanager
81	82	import tempfile
82	83	import shutil
...	...	@@ -1692,18 +1693,26 @@ class Element(Plugin):
1692	1693
1693	1694	# _pull_pending()
1694	1695	#
1695		- # Check whether the artifact will be pulled.
	1696	+ # Check whether the artifact will be pulled. If the pull operation is to
	1697	+ # include a specific subdir of the element artifact (from cli or user conf)
	1698	+ # then the local cache is queried for the subdirs existence.
	1699	+ #
	1700	+ # Args:
	1701	+ # subdir (str): Whether the pull has been invoked with a specific subdir set
1696	1702	#
1697	1703	# Returns:
1698	1704	# (bool): Whether a pull operation is pending
1699	1705	#
1700		- def _pull_pending(self):
	1706	+ def _pull_pending(self, subdir=None):
1701	1707	if self._get_workspace():
1702	1708	# Workspace builds are never pushed to artifact servers
1703	1709	return False
1704	1710
1705		- if self.__strong_cached:
1706		- # Artifact already in local cache
	1711	+ if self.__strong_cached and subdir:
	1712	+ # If we've specified a subdir, check if the subdir is cached locally
	1713	+ if self.__artifacts.contains_subdir_artifact(self, self.__strict_cache_key, subdir):
	1714	+ return False
	1715	+ elif self.__strong_cached:
1707	1716	return False
1708	1717
1709	1718	# Pull is pending if artifact remote server available
...	...	@@ -1725,11 +1734,10 @@ class Element(Plugin):
1725	1734
1726	1735	self._update_state()
1727	1736
1728		- def _pull_strong(self, *, progress=None):
	1737	+ def _pull_strong(self, *, progress=None, subdir=None, excluded_subdirs=None):
1729	1738	weak_key = self._get_cache_key(strength=_KeyStrength.WEAK)
1730		-
1731	1739	key = self.__strict_cache_key
1732		- if not self.__artifacts.pull(self, key, progress=progress):
	1740	+ if not self.__artifacts.pull(self, key, progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs):
1733	1741	return False
1734	1742
1735	1743	# update weak ref by pointing it to this newly fetched artifact
...	...	@@ -1737,10 +1745,10 @@ class Element(Plugin):
1737	1745
1738	1746	return True
1739	1747
1740		- def _pull_weak(self, *, progress=None):
	1748	+ def _pull_weak(self, *, progress=None, subdir=None, excluded_subdirs=None):
1741	1749	weak_key = self._get_cache_key(strength=_KeyStrength.WEAK)
1742		-
1743		- if not self.__artifacts.pull(self, weak_key, progress=progress):
	1750	+ if not self.__artifacts.pull(self, weak_key, progress=progress, subdir=subdir,
	1751	+ excluded_subdirs=excluded_subdirs):
1744	1752	return False
1745	1753
1746	1754	# extract strong cache key from this newly fetched artifact
...	...	@@ -1758,17 +1766,17 @@ class Element(Plugin):
1758	1766	#
1759	1767	# Returns: True if the artifact has been downloaded, False otherwise
1760	1768	#
1761		- def _pull(self):
	1769	+ def _pull(self, subdir=None, excluded_subdirs=None):
1762	1770	context = self._get_context()
1763	1771
1764	1772	def progress(percent, message):
1765	1773	self.status(message)
1766	1774
1767	1775	# Attempt to pull artifact without knowing whether it's available
1768		- pulled = self._pull_strong(progress=progress)
	1776	+ pulled = self._pull_strong(progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs)
1769	1777
1770	1778	if not pulled and not self._cached() and not context.get_strict():
1771		- pulled = self._pull_weak(progress=progress)
	1779	+ pulled = self._pull_weak(progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs)
1772	1780
1773	1781	if not pulled:
1774	1782	return False
...	...	@@ -1791,10 +1799,21 @@ class Element(Plugin):
1791	1799	if not self._cached():
1792	1800	return True
1793	1801
1794		- # Do not push tained artifact
	1802	+ # Do not push tainted artifact
1795	1803	if self.__get_tainted():
1796	1804	return True
1797	1805
	1806	+ # Do not push elements that have a dangling buildtree artifact unless element type is
	1807	+ # expected to have an empty buildtree directory
	1808	+ if not self.__artifacts.contains_subdir_artifact(self, self.__strict_cache_key, 'buildtree'):
	1809	+ return True
	1810	+
	1811	+ # strict_cache_key can't be relied on to be available when running in non strict mode
	1812	+ context = self._get_context()
	1813	+ if not context.get_strict():
	1814	+ if not self.__artifacts.contains_subdir_artifact(self, self.__weak_cache_key, 'buildtree'):
	1815	+ return True
	1816	+
1798	1817	return False
1799	1818
1800	1819	# _push():
...	...	@@ -2491,7 +2510,7 @@ class Element(Plugin):
2491	2510	if not context.get_strict() and not self.__artifacts.contains(self, key):
2492	2511	key = self._get_cache_key(strength=_KeyStrength.WEAK)
2493	2512
2494		- return (self.__artifacts.extract(self, key), key)
	2513	+ return (self.__artifacts.extract(self, key, subdir='buildtree'), key)
2495	2514
2496	2515	# __get_artifact_metadata_keys():
2497	2516	#

buildstream/plugin.py

---

...	...	@@ -266,7 +266,7 @@ class Plugin():
266	266	such as an sha256 sum of a tarball content.
267	267
268	268	Elements and Sources should implement this by collecting any configurations
269		- which could possibly effect the output and return a dictionary of these settings.
	269	+ which could possibly affect the output and return a dictionary of these settings.
270	270
271	271	For Sources, this is guaranteed to only be called if
272	272	:func:`Source.get_consistency() <buildstream.source.Source.get_consistency>`

buildstream/plugins/elements/autotools.yaml

---

...	...	@@ -123,7 +123,7 @@ environment:
123	123	V: 1
124	124
125	125	# And dont consider MAKEFLAGS or V as something which may
126		-# effect build output.
	126	+# affect build output.
127	127	environment-nocache:
128	128	- MAKEFLAGS
129	129	- V

buildstream/plugins/elements/cmake.yaml

---

...	...	@@ -66,7 +66,7 @@ environment:
66	66	V: 1
67	67
68	68	# And dont consider JOBS or V as something which may
69		-# effect build output.
	69	+# affect build output.
70	70	environment-nocache:
71	71	- JOBS
72	72	- V

buildstream/plugins/elements/junction.py

---

...	...	@@ -124,7 +124,7 @@ the user to resolve possibly conflicting nested junctions by creating a junction
124	124	with the same name in the top-level project, which then takes precedence.
125	125	"""
126	126
127		-from collections import Mapping
	127	+from collections.abc import Mapping
128	128	from buildstream import Element
129	129	from buildstream._pipeline import PipelineError
130	130

buildstream/plugins/elements/make.yaml

---

...	...	@@ -36,7 +36,7 @@ environment:
36	36	V: 1
37	37
38	38	# And dont consider MAKEFLAGS or V as something which may
39		-# effect build output.
	39	+# affect build output.
40	40	environment-nocache:
41	41	- MAKEFLAGS
42	42	- V

buildstream/plugins/elements/manual.yaml

---

...	...	@@ -35,7 +35,7 @@ environment:
35	35	V: 1
36	36
37	37	# And dont consider MAKEFLAGS or V as something which may
38		-# effect build output.
	38	+# affect build output.
39	39	environment-nocache:
40	40	- MAKEFLAGS
41	41	- V

buildstream/plugins/elements/meson.yaml

---

...	...	@@ -74,6 +74,6 @@ environment:
74	74	%{max-jobs}
75	75
76	76	# And dont consider NINJAJOBS as something which may
77		-# effect build output.
	77	+# affect build output.
78	78	environment-nocache:
79	79	- NINJAJOBS

buildstream/plugins/elements/qmake.yaml

---

...	...	@@ -44,7 +44,7 @@ environment:
44	44	V: 1
45	45
46	46	# And dont consider MAKEFLAGS or V as something which may
47		-# effect build output.
	47	+# affect build output.
48	48	environment-nocache:
49	49	- MAKEFLAGS
50	50	- V

buildstream/plugins/sources/git.py

---

...	...	@@ -89,7 +89,7 @@ import os
89	89	import errno
90	90	import re
91	91	import shutil
92		-from collections import Mapping
	92	+from collections.abc import Mapping
93	93	from io import StringIO
94	94
95	95	from configparser import RawConfigParser
...	...	@@ -415,7 +415,7 @@ class GitSource(Source):
415	415	def get_unique_key(self):
416	416	# Here we want to encode the local name of the repository and
417	417	# the ref, if the user changes the alias to fetch the same sources
418		- # from another location, it should not effect the cache key.
	418	+ # from another location, it should not affect the cache key.
419	419	key = [self.original_url, self.mirror.ref]
420	420
421	421	# Only modify the cache key with checkout_submodules if it's something

buildstream/source.py

---

...	...	@@ -155,7 +155,7 @@ Class Reference
155	155	"""
156	156
157	157	import os
158		-from collections import Mapping
	158	+from collections.abc import Mapping
159	159	from contextlib import contextmanager
160	160
161	161	from . import Plugin, Consistency

tests/completions/completions.py

---

...	...	@@ -103,7 +103,7 @@ def test_commands(cli, cmd, word_idx, expected):
103	103	('bst --no-colors build -', 3, ['--all ', '--track ', '--track-all ',
104	104	'--track-except ',
105	105	'--track-cross-junctions ', '-J ',
106		- '--track-save ']),
	106	+ '--track-save ', '--pull-buildtrees ']),
107	107
108	108	# Test the behavior of completing after an option that has a
109	109	# parameter that cannot be completed, vs an option that has

tests/integration/pullbuildtrees.py

---

	1	+import os
	2	+import shutil
	3	+import pytest
	4	+
	5	+from tests.testutils import cli_integration as cli, create_artifact_share
	6	+from tests.testutils.integration import assert_contains
	7	+
	8	+
	9	+DATA_DIR = os.path.join(
	10	+ os.path.dirname(os.path.realpath(__file__)),
	11	+ "project"
	12	+)
	13	+
	14	+
	15	+# Remove artifact cache & set cli.config value of pullbuildtrees
	16	+# to false, which is the default user context
	17	+def default_state(cli, integration_cache, share):
	18	+ shutil.rmtree(os.path.join(integration_cache, 'artifacts2'))
	19	+ cli.configure({
	20	+ 'pullbuildtrees': False,
	21	+ 'artifacts': {'url': share.repo, 'push': False},
	22	+ 'artifactdir': os.path.join(integration_cache, 'artifacts2')
	23	+ })
	24	+
	25	+
	26	+# A test to capture the integration of the pullbuildtrees
	27	+# behaviour, which by default is to not include the buildtree
	28	+# directory of an element
	29	+@pytest.mark.integration
	30	+@pytest.mark.datafiles(DATA_DIR)
	31	+def test_pullbuildtrees(cli, tmpdir, datafiles, integration_cache):
	32	+
	33	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	34	+ element_name = 'autotools/amhello.bst'
	35	+
	36	+ # Create artifact shares for pull & push testing
	37	+ with create_artifact_share(os.path.join(str(tmpdir), 'share1')) as share1,\
	38	+ create_artifact_share(os.path.join(str(tmpdir), 'share2')) as share2:
	39	+ cli.configure({
	40	+ 'artifacts': {'url': share1.repo, 'push': True},
	41	+ 'artifactdir': os.path.join(integration_cache, 'artifacts2')
	42	+ })
	43	+
	44	+ # Build autotools element, checked pushed, delete local
	45	+ result = cli.run(project=project, args=['build', element_name])
	46	+ assert result.exit_code == 0
	47	+ assert cli.get_element_state(project, element_name) == 'cached'
	48	+ assert share1.has_artifact('test', element_name, cli.get_element_key(project, element_name))
	49	+ default_state(cli, integration_cache, share1)
	50	+
	51	+ # Pull artifact with default config, assert that pulling again
	52	+ # doesn't create a pull job, then assert with buildtrees user
	53	+ # config set creates a pull job.
	54	+ result = cli.run(project=project, args=['pull', element_name])
	55	+ assert element_name in result.get_pulled_elements()
	56	+ result = cli.run(project=project, args=['pull', element_name])
	57	+ assert element_name not in result.get_pulled_elements()
	58	+ cli.configure({'pullbuildtrees': True})
	59	+ result = cli.run(project=project, args=['pull', element_name])
	60	+ assert element_name in result.get_pulled_elements()
	61	+ default_state(cli, integration_cache, share1)
	62	+
	63	+ # Pull artifact with default config, then assert that pulling
	64	+ # with buildtrees cli flag set creates a pull job.
	65	+ result = cli.run(project=project, args=['pull', element_name])
	66	+ assert element_name in result.get_pulled_elements()
	67	+ result = cli.run(project=project, args=['pull', '--pull-buildtrees', element_name])
	68	+ assert element_name in result.get_pulled_elements()
	69	+ default_state(cli, integration_cache, share1)
	70	+
	71	+ # Pull artifact with pullbuildtrees set in user config, then assert
	72	+ # that pulling with the same user config doesn't creates a pull job,
	73	+ # or when buildtrees cli flag is set.
	74	+ cli.configure({'pullbuildtrees': True})
	75	+ result = cli.run(project=project, args=['pull', element_name])
	76	+ assert element_name in result.get_pulled_elements()
	77	+ result = cli.run(project=project, args=['pull', element_name])
	78	+ assert element_name not in result.get_pulled_elements()
	79	+ result = cli.run(project=project, args=['pull', '--pull-buildtrees', element_name])
	80	+ assert element_name not in result.get_pulled_elements()
	81	+ default_state(cli, integration_cache, share1)
	82	+
	83	+ # Pull artifact with default config and buildtrees cli flag set, then assert
	84	+ # that pulling with pullbuildtrees set in user config doesn't create a pull
	85	+ # job.
	86	+ result = cli.run(project=project, args=['pull', '--pull-buildtrees', element_name])
	87	+ assert element_name in result.get_pulled_elements()
	88	+ cli.configure({'pullbuildtrees': True})
	89	+ result = cli.run(project=project, args=['pull', element_name])
	90	+ assert element_name not in result.get_pulled_elements()
	91	+ default_state(cli, integration_cache, share1)
	92	+
	93	+ # Assert that a partial build element (not containing a populated buildtree dir)
	94	+ # can't be pushed to an artifact share, then assert that a complete build element
	95	+ # can be. This will attempt a partial pull from share1 and then a partial push
	96	+ # to share2
	97	+ result = cli.run(project=project, args=['pull', element_name])
	98	+ assert element_name in result.get_pulled_elements()
	99	+ cli.configure({'artifacts': {'url': share2.repo, 'push': True}})
	100	+ result = cli.run(project=project, args=['push', element_name])
	101	+ assert element_name not in result.get_pushed_elements()
	102	+ assert not share2.has_artifact('test', element_name, cli.get_element_key(project, element_name))
	103	+
	104	+ # Assert that after pulling the missing buildtree the element artifact can be
	105	+ # successfully pushed to the remote. This will attempt to pull the buildtree
	106	+ # from share1 and then a 'complete' push to share2
	107	+ cli.configure({'artifacts': {'url': share1.repo, 'push': False}})
	108	+ result = cli.run(project=project, args=['pull', '--pull-buildtrees', element_name])
	109	+ assert element_name in result.get_pulled_elements()
	110	+ cli.configure({'artifacts': {'url': share2.repo, 'push': True}})
	111	+ result = cli.run(project=project, args=['push', element_name])
	112	+ assert element_name in result.get_pushed_elements()
	113	+ assert share2.has_artifact('test', element_name, cli.get_element_key(project, element_name))
	114	+ default_state(cli, integration_cache, share1)

tests/testutils/artifactshare.py

---

...	...	@@ -128,7 +128,7 @@ class ArtifactShare():
128	128
129	129	valid_chars = string.digits + string.ascii_letters + '-._'
130	130	element_name = ''.join([
131		- x if x in valid_chars else '_'
	131	+ x if x in valid_chars else '-'
132	132	for x in element_name
133	133	])
134	134	artifact_key = '{0}/{1}/{2}'.format(project_name, element_name, cache_key)

tests/yaml/yaml.py

---

1	1	import os
2	2	import pytest
3	3	import tempfile
4		-from collections import Mapping
	4	+from collections.abc import Mapping
5	5
6	6	from buildstream import _yaml
7	7	from buildstream._exceptions import LoadError, LoadErrorReason

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.