[Notes] [Git][BuildStream/buildstream][tpollard/494] 8 commits: _casbaseddirectory.py: Enable direct CAS-to-CAS import.

[Tom Pollard <gitlab mg gitlab com>]

Title: GitLab

Tom Pollard pushed to branch tpollard/494 at BuildStream / buildstream

Commits:

• 35ec6b6a
  by Jim MacArthur at 2018-11-09T14:58:22Z

  _casbaseddirectory.py: Enable direct CAS-to-CAS import.
  
  Fixes issue #574.
  

• b93100ec
  by Jim MacArthur at 2018-11-09T14:58:22Z

  _casbaseddirectory: Fix order of list_modified_paths
  
  This will now return paths in the same order as list_relative_paths.
  

• 78691fa0
  by Jim MacArthur at 2018-11-09T14:58:22Z

  virtual_directory_import.py: New test
  

• 7ce6581b
  by Jim MacArthur at 2018-11-09T15:25:46Z

  Merge branch 'jmac/cas_to_cas_v2' into 'master'
  
  Direct CAS-to-CAS import
  
  Closes #574
  
  See merge request BuildStream/buildstream!911

• 6f4351ec
  by Benjamin Schubert at 2018-11-09T16:53:42Z

  source.py: don't let StopIteration propagate to silence() contextmanager
  
  As per PEP 0479 (https://www.python.org/dev/peps/pep-0479/),
  StopIteration thrown in context managers are not valid starting from
  Python 3.7.
  

• e6e03451
  by richardmaw-codethink at 2018-11-12T11:05:15Z

  Merge branch 'bschubert/fix-silence-stopiteration-pep-0479' into 'master'
  
  source.py: don't let StopIteration propagate to silence() contextmanager
  
  See merge request BuildStream/buildstream!945

• 3a4ddf71
  by Tom Pollard at 2018-11-12T11:48:23Z

  _context.py: Add pull_build_trees global user context, the default
  of which is set to False via the addition of pullbuildtree to
  userconfig.yaml
  
  _frontend/app.py & cli.py: Add --pull-build-trees as a bst main
  option, which when passed will override the default or user defined
  context for pull_buildtrees
  
  tests/completions/completions.py: Update for the added flag
  

• e7fa8867
  by Tom Pollard at 2018-11-12T11:48:23Z

  Don't pull artifact build trees by default.
  
  The addition of cached build trees being included in element
  artifacts has led to mostly redundant download overheads when
  pulling from a remote artifact server. As such the default behaviour
  of pull shouldn't fetch the build tree object if available.
  
  pullqueue.py: add extensible attributes to determine required/
  excluded directories for element pull jobs. scheduler context
  is used to determine pull_build_trees context.
  
  element.py: extend relevant pull logic for specified subdir
  consideration and ensure push logic does not lead to partial
  artifact pushes. Change assumption that buildtree can be
  extracted if artifact is cached.
  
  _artifactcache/: artifactcache.py & cascache.py inclusion of
  helper functions for subdir artifact checking & fetching, fetch
  logic extended to only pull required artifact directories.
  extract & checkout updated to handle full/partial operation.
  
  tests/: addition of integration test pullbuildtrees.py,
  buildtrees.py adapted cli options, testutils/artifactshare.py
  has_artifact changed to return artifact digest on true condition.
  

16 changed files:

• NEWS
• buildstream/_artifactcache/artifactcache.py
• buildstream/_artifactcache/cascache.py
• buildstream/_context.py
• buildstream/_frontend/app.py
• buildstream/_frontend/cli.py
• buildstream/_scheduler/queues/pullqueue.py
• buildstream/data/userconfig.yaml
• buildstream/element.py
• buildstream/source.py
• buildstream/storage/_casbaseddirectory.py
• tests/completions/completions.py
• tests/integration/build-tree.py
• + tests/integration/pullbuildtrees.py
• + tests/storage/virtual_directory_import.py
• tests/testutils/artifactshare.py

Changes:

NEWS

---

...	...	@@ -38,13 +38,23 @@ buildstream 1.3.1
38	38	a bug fix to workspaces so they can be build in workspaces too.
39	39
40	40	o Creating a build shell through the interactive mode or `bst shell --build`
41		- will now use the cached build tree. It is now easier to debug local build
42		- failures.
	41	+ will now use the cached buildtree if available locally. It is now easier to
	42	+ debug local build failures.
43	43
44	44	o `bst shell --sysroot` now takes any directory that contains a sysroot,
45	45	instead of just a specially-formatted build-root with a `root` and `scratch`
46	46	subdirectory.
47	47
	48	+ o Due to the element `build tree` being cached in the respective artifact their
	49	+ size in some cases has significantly increased. In *most* cases the build tree
	50	+ is not utilised when building targets, as such by default bst 'pull' & 'build'
	51	+ will not fetch buildtrees from remotes. This behaviour can be overriden with
	52	+ the cli main option '--pull-build-trees', or the user configuration option
	53	+ 'pullbuildtrees = True'. The override will also add the build tree to already
	54	+ cached artifacts. When attempting to populate an artifactcache server with
	55	+ cached artifacts, only 'complete' elements can be pushed. If the element is
	56	+ expected to have a populated build tree then it must be cached before pushing.
	57	+
48	58
49	59	=================
50	60	buildstream 1.1.5

buildstream/_artifactcache/artifactcache.py

---

...	...	@@ -476,6 +476,22 @@ class ArtifactCache():
476	476
477	477	return self.cas.contains(ref)
478	478
	479	+ # contains_subdir_artifact():
	480	+ #
	481	+ # Check whether an artifact element contains a digest for a subdir
	482	+ # which is populated in the cache, i.e non dangling.
	483	+ #
	484	+ # Args:
	485	+ # element (Element): The Element to check
	486	+ # key (str): The cache key to use
	487	+ # subdir (str): The subdir to check
	488	+ #
	489	+ # Returns: True if the subdir exists & is populated in the cache, False otherwise
	490	+ #
	491	+ def contains_subdir_artifact(self, element, key, subdir):
	492	+ ref = self.get_artifact_fullname(element, key)
	493	+ return self.cas.contains_subdir_artifact(ref, subdir)
	494	+
479	495	# list_artifacts():
480	496	#
481	497	# List artifacts in this cache in LRU order.
...	...	@@ -533,6 +549,7 @@ class ArtifactCache():
533	549	# Args:
534	550	# element (Element): The Element to extract
535	551	# key (str): The cache key to use
	552	+ # subdir (str): Optional specific subdir to extract
536	553	#
537	554	# Raises:
538	555	# ArtifactError: In cases there was an OSError, or if the artifact
...	...	@@ -540,12 +557,12 @@ class ArtifactCache():
540	557	#
541	558	# Returns: path to extracted artifact
542	559	#
543		- def extract(self, element, key):
	560	+ def extract(self, element, key, subdir=None):
544	561	ref = self.get_artifact_fullname(element, key)
545	562
546	563	path = os.path.join(self.extractdir, element._get_project().name, element.normal_name)
547	564
548		- return self.cas.extract(ref, path)
	565	+ return self.cas.extract(ref, path, subdir=subdir)
549	566
550	567	# commit():
551	568	#
...	...	@@ -666,11 +683,13 @@ class ArtifactCache():
666	683	# element (Element): The Element whose artifact is to be fetched
667	684	# key (str): The cache key to use
668	685	# progress (callable): The progress callback, if any
	686	+ # subdir (str): The optional specific subdir to pull
	687	+ # excluded_subdirs (list): The optional list of subdirs to not pull
669	688	#
670	689	# Returns:
671	690	# (bool): True if pull was successful, False if artifact was not available
672	691	#
673		- def pull(self, element, key, *, progress=None):
	692	+ def pull(self, element, key, *, progress=None, subdir=None, excluded_subdirs=None):
674	693	ref = self.get_artifact_fullname(element, key)
675	694
676	695	project = element._get_project()
...	...	@@ -680,8 +699,13 @@ class ArtifactCache():
680	699	display_key = element._get_brief_display_key()
681	700	element.status("Pulling artifact {} <- {}".format(display_key, remote.spec.url))
682	701
683		- if self.cas.pull(ref, remote, progress=progress):
	702	+ if self.cas.pull(ref, remote, progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs):
684	703	element.info("Pulled artifact {} <- {}".format(display_key, remote.spec.url))
	704	+ if subdir:
	705	+ # Attempt to extract subdir into artifact extract dir if it already exists
	706	+ # without containing the subdir. If the respective artifact extract dir does not
	707	+ # exist a complete extraction will complete.
	708	+ self.extract(element, key, subdir)
685	709	# no need to pull from additional remotes
686	710	return True
687	711	else:

buildstream/_artifactcache/cascache.py

---

...	...	@@ -82,6 +82,27 @@ class CASCache():
82	82	# This assumes that the repository doesn't have any dangling pointers
83	83	return os.path.exists(refpath)
84	84
	85	+ # contains_subdir_artifact():
	86	+ #
	87	+ # Check whether the specified artifact element tree has a digest for a subdir
	88	+ # which is populated in the cache, i.e non dangling.
	89	+ #
	90	+ # Args:
	91	+ # ref (str): The ref to check
	92	+ # subdir (str): The subdir to check
	93	+ #
	94	+ # Returns: True if the subdir exists & is populated in the cache, False otherwise
	95	+ #
	96	+ def contains_subdir_artifact(self, ref, subdir):
	97	+ tree = self.resolve_ref(ref)
	98	+
	99	+ # This assumes that the subdir digest is present in the element tree
	100	+ subdirdigest = self._get_subdir(tree, subdir)
	101	+ objpath = self.objpath(subdirdigest)
	102	+
	103	+ # True if subdir content is cached or if empty as expected
	104	+ return os.path.exists(objpath)
	105	+
85	106	# extract():
86	107	#
87	108	# Extract cached directory for the specified ref if it hasn't
...	...	@@ -90,19 +111,30 @@ class CASCache():
90	111	# Args:
91	112	# ref (str): The ref whose directory to extract
92	113	# path (str): The destination path
	114	+ # subdir (str): Optional specific dir to extract
93	115	#
94	116	# Raises:
95	117	# CASError: In cases there was an OSError, or if the ref did not exist.
96	118	#
97	119	# Returns: path to extracted directory
98	120	#
99		- def extract(self, ref, path):
	121	+ def extract(self, ref, path, subdir=None):
100	122	tree = self.resolve_ref(ref, update_mtime=True)
101	123
102		- dest = os.path.join(path, tree.hash)
	124	+ originaldest = dest = os.path.join(path, tree.hash)
	125	+
	126	+ # If artifact is already extracted, check if the optional subdir
	127	+ # has also been extracted. If the artifact has not been extracted
	128	+ # a full extraction would include the optional subdir
103	129	if os.path.isdir(dest):
104		- # directory has already been extracted
105		- return dest
	130	+ if subdir:
	131	+ if not os.path.isdir(os.path.join(dest, subdir)):
	132	+ dest = os.path.join(dest, subdir)
	133	+ tree = self._get_subdir(tree, subdir)
	134	+ else:
	135	+ return dest
	136	+ else:
	137	+ return dest
106	138
107	139	with tempfile.TemporaryDirectory(prefix='tmp', dir=self.tmpdir) as tmpdir:
108	140	checkoutdir = os.path.join(tmpdir, ref)
...	...	@@ -120,7 +152,7 @@ class CASCache():
120	152	if e.errno not in [errno.ENOTEMPTY, errno.EEXIST]:
121	153	raise CASError("Failed to extract directory for ref '{}': {}".format(ref, e)) from e
122	154
123		- return dest
	155	+ return originaldest
124	156
125	157	# commit():
126	158	#
...	...	@@ -193,11 +225,13 @@ class CASCache():
193	225	# ref (str): The ref to pull
194	226	# remote (CASRemote): The remote repository to pull from
195	227	# progress (callable): The progress callback, if any
	228	+ # subdir (str): The optional specific subdir to pull
	229	+ # excluded_subdirs (list): The optional list of subdirs to not pull
196	230	#
197	231	# Returns:
198	232	# (bool): True if pull was successful, False if ref was not available
199	233	#
200		- def pull(self, ref, remote, *, progress=None):
	234	+ def pull(self, ref, remote, *, progress=None, subdir=None, excluded_subdirs=None):
201	235	try:
202	236	remote.init()
203	237
...	...	@@ -209,7 +243,12 @@ class CASCache():
209	243	tree.hash = response.digest.hash
210	244	tree.size_bytes = response.digest.size_bytes
211	245
212		- self._fetch_directory(remote, tree)
	246	+ # Check if the element artifact is present, if so just fetch the subdir.
	247	+ if subdir and os.path.exists(self.objpath(tree)):
	248	+ self._fetch_subdir(remote, tree, subdir)
	249	+ else:
	250	+ # Fetch artifact, excluded_subdirs determined in pullqueue
	251	+ self._fetch_directory(remote, tree, excluded_subdirs=excluded_subdirs)
213	252
214	253	self.set_ref(ref, tree)
215	254
...	...	@@ -607,8 +646,10 @@ class CASCache():
607	646	stat.S_IRGRP | stat.S_IXGRP | stat.S_IROTH | stat.S_IXOTH)
608	647
609	648	for dirnode in directory.directories:
610		- fullpath = os.path.join(dest, dirnode.name)
611		- self._checkout(fullpath, dirnode.digest)
	649	+ # Don't try to checkout a dangling ref
	650	+ if os.path.exists(self.objpath(dirnode.digest)):
	651	+ fullpath = os.path.join(dest, dirnode.name)
	652	+ self._checkout(fullpath, dirnode.digest)
612	653
613	654	for symlinknode in directory.symlinks:
614	655	# symlink
...	...	@@ -863,11 +904,14 @@ class CASCache():
863	904	# Args:
864	905	# remote (Remote): The remote to use.
865	906	# dir_digest (Digest): Digest object for the directory to fetch.
	907	+ # excluded_subdirs (list): The optional list of subdirs to not fetch
866	908	#
867		- def _fetch_directory(self, remote, dir_digest):
	909	+ def _fetch_directory(self, remote, dir_digest, *, excluded_subdirs=None):
868	910	fetch_queue = [dir_digest]
869	911	fetch_next_queue = []
870	912	batch = _CASBatchRead(remote)
	913	+ if not excluded_subdirs:
	914	+ excluded_subdirs = []
871	915
872	916	while len(fetch_queue) + len(fetch_next_queue) > 0:
873	917	if not fetch_queue:
...	...	@@ -882,8 +926,9 @@ class CASCache():
882	926	directory.ParseFromString(f.read())
883	927
884	928	for dirnode in directory.directories:
885		- batch = self._fetch_directory_node(remote, dirnode.digest, batch,
886		- fetch_queue, fetch_next_queue, recursive=True)
	929	+ if dirnode.name not in excluded_subdirs:
	930	+ batch = self._fetch_directory_node(remote, dirnode.digest, batch,
	931	+ fetch_queue, fetch_next_queue, recursive=True)
887	932
888	933	for filenode in directory.files:
889	934	batch = self._fetch_directory_node(remote, filenode.digest, batch,
...	...	@@ -892,6 +937,10 @@ class CASCache():
892	937	# Fetch final batch
893	938	self._fetch_directory_batch(remote, batch, fetch_queue, fetch_next_queue)
894	939
	940	+ def _fetch_subdir(self, remote, tree, subdir):
	941	+ subdirdigest = self._get_subdir(tree, subdir)
	942	+ self._fetch_directory(remote, subdirdigest)
	943	+
895	944	def _fetch_tree(self, remote, digest):
896	945	# download but do not store the Tree object
897	946	with tempfile.NamedTemporaryFile(dir=self.tmpdir) as out:

buildstream/_context.py

---

...	...	@@ -104,6 +104,9 @@ class Context():
104	104	# What to do when a build fails in non interactive mode
105	105	self.sched_error_action = 'continue'
106	106
	107	+ # Whether or not to attempt to pull build trees globally
	108	+ self.pull_build_trees = False
	109	+
107	110	# Whether elements must be rebuilt when their dependencies have changed
108	111	self._strict_build_plan = None
109	112
...	...	@@ -160,7 +163,7 @@ class Context():
160	163	_yaml.node_validate(defaults, [
161	164	'sourcedir', 'builddir', 'artifactdir', 'logdir',
162	165	'scheduler', 'artifacts', 'logging', 'projects',
163		- 'cache'
	166	+ 'cache', 'pullbuildtrees'
164	167	])
165	168
166	169	for directory in ['sourcedir', 'builddir', 'artifactdir', 'logdir']:
...	...	@@ -185,6 +188,9 @@ class Context():
185	188	# Load artifact share configuration
186	189	self.artifact_cache_specs = ArtifactCache.specs_from_config_node(defaults)
187	190
	191	+ # Load pull build trees configuration
	192	+ self.pull_build_trees = _yaml.node_get(defaults, bool, 'pullbuildtrees', default_value='False')
	193	+
188	194	# Load logging config
189	195	logging = _yaml.node_get(defaults, Mapping, 'logging')
190	196	_yaml.node_validate(logging, [

buildstream/_frontend/app.py

---

...	...	@@ -182,7 +182,8 @@ class App():
182	182	'fetchers': 'sched_fetchers',
183	183	'builders': 'sched_builders',
184	184	'pushers': 'sched_pushers',
185		- 'network_retries': 'sched_network_retries'
	185	+ 'network_retries': 'sched_network_retries',
	186	+ 'pull_build_trees': 'pull_build_trees'
186	187	}
187	188	for cli_option, context_attr in override_map.items():
188	189	option_value = self._main_options.get(cli_option)

buildstream/_frontend/cli.py

---

...	...	@@ -219,6 +219,8 @@ def print_version(ctx, param, value):
219	219	help="Specify a project option")
220	220	@click.option('--default-mirror', default=None,
221	221	help="The mirror to fetch from first, before attempting other mirrors")
	222	+@click.option('--pull-build-trees', is_flag=True, default=None,
	223	+ help="Include an element's build trees when pulling remote element artifacts")
222	224	@click.pass_context
223	225	def cli(context, **kwargs):
224	226	"""Build and manipulate BuildStream projects

buildstream/_scheduler/queues/pullqueue.py

---

...	...	@@ -32,9 +32,20 @@ class PullQueue(Queue):
32	32	complete_name = "Pulled"
33	33	resources = [ResourceType.DOWNLOAD, ResourceType.CACHE]
34	34
	35	+ def __init__(self, scheduler):
	36	+ super().__init__(scheduler)
	37	+
	38	+ # Current default exclusions on pull
	39	+ self._excluded_subdirs = ["buildtree"]
	40	+ self._subdir = None
	41	+ # If build trees are to be pulled, remove the value from exclusion list
	42	+ if self._scheduler.context.pull_build_trees:
	43	+ self._subdir = "buildtree"
	44	+ self._excluded_subdirs.remove(self._subdir)
	45	+
35	46	def process(self, element):
36	47	# returns whether an artifact was downloaded or not
37		- if not element._pull():
	48	+ if not element._pull(subdir=self._subdir, excluded_subdirs=self._excluded_subdirs):
38	49	raise SkipJob(self.action_name)
39	50
40	51	def status(self, element):
...	...	@@ -49,7 +60,7 @@ class PullQueue(Queue):
49	60	if not element._can_query_cache():
50	61	return QueueStatus.WAIT
51	62
52		- if element._pull_pending():
	63	+ if element._pull_pending(subdir=self._subdir):
53	64	return QueueStatus.READY
54	65	else:
55	66	return QueueStatus.SKIP

buildstream/data/userconfig.yaml

---

...	...	@@ -97,3 +97,5 @@ logging:
97	97
98	98	[%{elapsed}][%{key}][%{element}] %{action} %{message}
99	99
	100	+# Whether to pull buildtrees when downloading element artifacts
	101	+pullbuildtrees: False

buildstream/element.py

---

...	...	@@ -1398,9 +1398,18 @@ class Element(Plugin):
1398	1398	.format(workspace.get_absolute_path())):
1399	1399	workspace.stage(temp_staging_directory)
1400	1400	elif self._cached():
1401		- # We have a cached buildtree to use, instead
1402		- artifact_base, _ = self.__extract()
1403		- import_dir = os.path.join(artifact_base, 'buildtree')
	1401	+ # Check if we have a cached buildtree to use
	1402	+ context = self._get_context()
	1403	+ if context.get_strict():
	1404	+ if self.__artifacts.contains_subdir_artifact(self, self.__strict_cache_key, 'buildtree'):
	1405	+ artifact_base, _ = self.__extract()
	1406	+ import_dir = os.path.join(artifact_base, 'buildtree')
	1407	+ elif self.__artifacts.contains_subdir_artifact(self, self.__weak_cache_key, 'buildtree'):
	1408	+ artifact_base, _ = self.__extract()
	1409	+ import_dir = os.path.join(artifact_base, 'buildtree')
	1410	+ else:
	1411	+ self.warn("{} is cached without a buildtree, the source will be staged instead"
	1412	+ .format(self.name))
1404	1413	else:
1405	1414	# No workspace, stage directly
1406	1415	for source in self.sources():
...	...	@@ -1691,18 +1700,26 @@ class Element(Plugin):
1691	1700
1692	1701	# _pull_pending()
1693	1702	#
1694		- # Check whether the artifact will be pulled.
	1703	+ # Check whether the artifact will be pulled. If the pull operation is to
	1704	+ # include a specific subdir of the element artifact (from cli or user conf)
	1705	+ # then the local cache is queried for the subdirs existence.
	1706	+ #
	1707	+ # Args:
	1708	+ # subdir (str): Whether the pull has been invoked with a specific subdir set
1695	1709	#
1696	1710	# Returns:
1697	1711	# (bool): Whether a pull operation is pending
1698	1712	#
1699		- def _pull_pending(self):
	1713	+ def _pull_pending(self, subdir=None):
1700	1714	if self._get_workspace():
1701	1715	# Workspace builds are never pushed to artifact servers
1702	1716	return False
1703	1717
1704		- if self.__strong_cached:
1705		- # Artifact already in local cache
	1718	+ if self.__strong_cached and subdir:
	1719	+ # If we've specified a subdir, check if the subdir is cached locally
	1720	+ if self.__artifacts.contains_subdir_artifact(self, self.__strict_cache_key, subdir):
	1721	+ return False
	1722	+ elif self.__strong_cached:
1706	1723	return False
1707	1724
1708	1725	# Pull is pending if artifact remote server available
...	...	@@ -1724,50 +1741,27 @@ class Element(Plugin):
1724	1741
1725	1742	self._update_state()
1726	1743
1727		- def _pull_strong(self, *, progress=None):
1728		- weak_key = self._get_cache_key(strength=_KeyStrength.WEAK)
1729		-
1730		- key = self.__strict_cache_key
1731		- if not self.__artifacts.pull(self, key, progress=progress):
1732		- return False
1733		-
1734		- # update weak ref by pointing it to this newly fetched artifact
1735		- self.__artifacts.link_key(self, key, weak_key)
1736		-
1737		- return True
1738		-
1739		- def _pull_weak(self, *, progress=None):
1740		- weak_key = self._get_cache_key(strength=_KeyStrength.WEAK)
1741		-
1742		- if not self.__artifacts.pull(self, weak_key, progress=progress):
1743		- return False
1744		-
1745		- # extract strong cache key from this newly fetched artifact
1746		- self._pull_done()
1747		-
1748		- # create tag for strong cache key
1749		- key = self._get_cache_key(strength=_KeyStrength.STRONG)
1750		- self.__artifacts.link_key(self, weak_key, key)
1751		-
1752		- return True
1753		-
1754	1744	# _pull():
1755	1745	#
1756	1746	# Pull artifact from remote artifact repository into local artifact cache.
1757	1747	#
	1748	+ # Args:
	1749	+ # subdir (str): The optional specific subdir to pull
	1750	+ # excluded_subdirs (list): The optional list of subdirs to not pull
	1751	+ #
1758	1752	# Returns: True if the artifact has been downloaded, False otherwise
1759	1753	#
1760		- def _pull(self):
	1754	+ def _pull(self, subdir=None, excluded_subdirs=None):
1761	1755	context = self._get_context()
1762	1756
1763	1757	def progress(percent, message):
1764	1758	self.status(message)
1765	1759
1766	1760	# Attempt to pull artifact without knowing whether it's available
1767		- pulled = self._pull_strong(progress=progress)
	1761	+ pulled = self.__pull_strong(progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs)
1768	1762
1769	1763	if not pulled and not self._cached() and not context.get_strict():
1770		- pulled = self._pull_weak(progress=progress)
	1764	+ pulled = self.__pull_weak(progress=progress, subdir=subdir, excluded_subdirs=excluded_subdirs)
1771	1765
1772	1766	if not pulled:
1773	1767	return False
...	...	@@ -1790,10 +1784,21 @@ class Element(Plugin):
1790	1784	if not self._cached():
1791	1785	return True
1792	1786
1793		- # Do not push tained artifact
	1787	+ # Do not push tainted artifact
1794	1788	if self.__get_tainted():
1795	1789	return True
1796	1790
	1791	+ # strict_cache_key can't be relied on to be available when running in non strict mode
	1792	+ context = self._get_context()
	1793	+
	1794	+ # Do not push elements that have a dangling buildtree artifact unless element type is
	1795	+ # expected to have an empty buildtree directory
	1796	+ if context.get_strict():
	1797	+ if not self.__artifacts.contains_subdir_artifact(self, self.__strict_cache_key, 'buildtree'):
	1798	+ return True
	1799	+ elif not self.__artifacts.contains_subdir_artifact(self, self.__weak_cache_key, 'buildtree'):
	1800	+ return True
	1801	+
1797	1802	return False
1798	1803
1799	1804	# _push():
...	...	@@ -2674,6 +2679,59 @@ class Element(Plugin):
2674	2679
2675	2680	return utils._deduplicate(keys)
2676	2681
	2682	+ # __pull_strong():
	2683	+ #
	2684	+ # Attempt pulling given element from configured artifact caches with
	2685	+ # the strict cache key
	2686	+ #
	2687	+ # Args:
	2688	+ # progress (callable): The progress callback, if any
	2689	+ # subdir (str): The optional specific subdir to pull
	2690	+ # excluded_subdirs (list): The optional list of subdirs to not pull
	2691	+ #
	2692	+ # Returns:
	2693	+ # (bool): Whether or not the pull was successful
	2694	+ #
	2695	+ def __pull_strong(self, *, progress=None, subdir=None, excluded_subdirs=None):
	2696	+ weak_key = self._get_cache_key(strength=_KeyStrength.WEAK)
	2697	+ key = self.__strict_cache_key
	2698	+ if not self.__artifacts.pull(self, key, progress=progress, subdir=subdir,
	2699	+ excluded_subdirs=excluded_subdirs):
	2700	+ return False
	2701	+
	2702	+ # update weak ref by pointing it to this newly fetched artifact
	2703	+ self.__artifacts.link_key(self, key, weak_key)
	2704	+
	2705	+ return True
	2706	+
	2707	+ # __pull_weak():
	2708	+ #
	2709	+ # Attempt pulling given element from configured artifact caches with
	2710	+ # the weak cache key
	2711	+ #
	2712	+ # Args:
	2713	+ # progress (callable): The progress callback, if any
	2714	+ # subdir (str): The optional specific subdir to pull
	2715	+ # excluded_subdirs (list): The optional list of subdirs to not pull
	2716	+ #
	2717	+ # Returns:
	2718	+ # (bool): Whether or not the pull was successful
	2719	+ #
	2720	+ def __pull_weak(self, *, progress=None, subdir=None, excluded_subdirs=None):
	2721	+ weak_key = self._get_cache_key(strength=_KeyStrength.WEAK)
	2722	+ if not self.__artifacts.pull(self, weak_key, progress=progress, subdir=subdir,
	2723	+ excluded_subdirs=excluded_subdirs):
	2724	+ return False
	2725	+
	2726	+ # extract strong cache key from this newly fetched artifact
	2727	+ self._pull_done()
	2728	+
	2729	+ # create tag for strong cache key
	2730	+ key = self._get_cache_key(strength=_KeyStrength.STRONG)
	2731	+ self.__artifacts.link_key(self, weak_key, key)
	2732	+
	2733	+ return True
	2734	+
2677	2735
2678	2736	def _overlap_error_detail(f, forbidden_overlap_elements, elements):
2679	2737	if forbidden_overlap_elements:

buildstream/source.py

---

...	...	@@ -973,32 +973,34 @@ class Source(Plugin):
973	973	# the items of source_fetchers, if it happens to be a generator.
974	974	#
975	975	source_fetchers = iter(source_fetchers)
976		- try:
977	976
978		- while True:
	977	+ while True:
979	978
980		- with context.silence():
	979	+ with context.silence():
	980	+ try:
981	981	fetcher = next(source_fetchers)
982		-
983		- alias = fetcher._get_alias()
984		- for uri in project.get_alias_uris(alias, first_pass=self.__first_pass):
985		- try:
986		- fetcher.fetch(uri)
987		- # FIXME: Need to consider temporary vs. permanent failures,
988		- # and how this works with retries.
989		- except BstError as e:
990		- last_error = e
991		- continue
992		-
993		- # No error, we're done with this fetcher
	982	+ except StopIteration:
	983	+ # as per PEP479, we are not allowed to let StopIteration
	984	+ # thrown from a context manager.
	985	+ # Catching it here and breaking instead.
994	986	break
995	987
996		- else:
997		- # No break occurred, raise the last detected error
998		- raise last_error
	988	+ alias = fetcher._get_alias()
	989	+ for uri in project.get_alias_uris(alias, first_pass=self.__first_pass):
	990	+ try:
	991	+ fetcher.fetch(uri)
	992	+ # FIXME: Need to consider temporary vs. permanent failures,
	993	+ # and how this works with retries.
	994	+ except BstError as e:
	995	+ last_error = e
	996	+ continue
999	997
1000		- except StopIteration:
1001		- pass
	998	+ # No error, we're done with this fetcher
	999	+ break
	1000	+
	1001	+ else:
	1002	+ # No break occurred, raise the last detected error
	1003	+ raise last_error
1002	1004
1003	1005	# Default codepath is to reinstantiate the Source
1004	1006	#

buildstream/storage/_casbaseddirectory.py

---

...	...	@@ -30,7 +30,6 @@ See also: :ref:`sandboxing`.
30	30	from collections import OrderedDict
31	31
32	32	import os
33		-import tempfile
34	33	import stat
35	34
36	35	from .._protos.build.bazel.remote.execution.v2 import remote_execution_pb2
...	...	@@ -51,6 +50,183 @@ class IndexEntry():
51	50	self.modified = modified
52	51
53	52
	53	+class ResolutionException(VirtualDirectoryError):
	54	+ """ Superclass of all exceptions that can be raised by
	55	+ CasBasedDirectory._resolve. Should not be used outside this module. """
	56	+ pass
	57	+
	58	+
	59	+class InfiniteSymlinkException(ResolutionException):
	60	+ """ Raised when an infinite symlink loop is found. """
	61	+ pass
	62	+
	63	+
	64	+class AbsoluteSymlinkException(ResolutionException):
	65	+ """Raised if we try to follow an absolute symlink (i.e. one whose
	66	+ target starts with the path separator) and we have disallowed
	67	+ following such symlinks.
	68	+ """
	69	+ pass
	70	+
	71	+
	72	+class UnexpectedFileException(ResolutionException):
	73	+ """Raised if we were found a file where a directory or symlink was
	74	+ expected, for example we try to resolve a symlink pointing to
	75	+ /a/b/c but /a/b is a file.
	76	+ """
	77	+ def __init__(self, message=""):
	78	+ """Allow constructor with no arguments, since this can be raised in
	79	+ places where there isn't sufficient information to write the
	80	+ message.
	81	+ """
	82	+ super().__init__(message)
	83	+
	84	+
	85	+class _Resolver():
	86	+ """A class for resolving symlinks inside CAS-based directories. As
	87	+ well as providing a namespace for some functions, this also
	88	+ contains two flags which are constant throughout one resolution
	89	+ operation and the 'seen_objects' list used to detect infinite
	90	+ symlink loops.
	91	+
	92	+ """
	93	+
	94	+ def __init__(self, absolute_symlinks_resolve=True, force_create=False):
	95	+ self.absolute_symlinks_resolve = absolute_symlinks_resolve
	96	+ self.force_create = force_create
	97	+ self.seen_objects = []
	98	+
	99	+ def resolve(self, name, directory):
	100	+ """Resolves any name to an object. If the name points to a symlink in
	101	+ the directory, it returns the thing it points to,
	102	+ recursively.
	103	+
	104	+ Returns a CasBasedDirectory, FileNode or None. None indicates
	105	+ either that 'target' does not exist in this directory, or is a
	106	+ symlink chain which points to a nonexistent name (broken
	107	+ symlink).
	108	+
	109	+ Raises:
	110	+
	111	+ - InfiniteSymlinkException if 'name' points to an infinite
	112	+ symlink loop.
	113	+ - AbsoluteSymlinkException if 'name' points to an absolute
	114	+ symlink and absolute_symlinks_resolve is False.
	115	+ - UnexpectedFileException if at any point during resolution we
	116	+ find a file which we expected to be a directory or symlink.
	117	+
	118	+ If force_create is set, this will attempt to create
	119	+ directories to make symlinks and directories resolve. Files
	120	+ present in symlink target paths will also be removed and
	121	+ replaced with directories. If force_create is off, this will
	122	+ never alter 'directory'.
	123	+
	124	+ """
	125	+
	126	+ # First check for nonexistent things or 'normal' objects and return them
	127	+ if name not in directory.index:
	128	+ return None
	129	+ index_entry = directory.index[name]
	130	+ if isinstance(index_entry.buildstream_object, Directory):
	131	+ return index_entry.buildstream_object
	132	+ elif isinstance(index_entry.pb_object, remote_execution_pb2.FileNode):
	133	+ return index_entry.pb_object
	134	+
	135	+ # Now we must be dealing with a symlink.
	136	+ assert isinstance(index_entry.pb_object, remote_execution_pb2.SymlinkNode)
	137	+
	138	+ symlink_object = index_entry.pb_object
	139	+ if symlink_object in self.seen_objects:
	140	+ # Infinite symlink loop detected
	141	+ message = ("Infinite symlink loop found during resolution. " +
	142	+ "First repeated element is {}".format(name))
	143	+ raise InfiniteSymlinkException(message=message)
	144	+
	145	+ self.seen_objects.append(symlink_object)
	146	+
	147	+ components = symlink_object.target.split(CasBasedDirectory._pb2_path_sep)
	148	+ absolute = symlink_object.target.startswith(CasBasedDirectory._pb2_absolute_path_prefix)
	149	+
	150	+ if absolute:
	151	+ if self.absolute_symlinks_resolve:
	152	+ directory = directory.find_root()
	153	+ # Discard the first empty element
	154	+ components.pop(0)
	155	+ else:
	156	+ # Unresolvable absolute symlink
	157	+ message = "{} is an absolute symlink, which was disallowed during resolution".format(name)
	158	+ raise AbsoluteSymlinkException(message=message)
	159	+
	160	+ resolution = directory
	161	+ while components and isinstance(resolution, CasBasedDirectory):
	162	+ c = components.pop(0)
	163	+ directory = resolution
	164	+
	165	+ try:
	166	+ resolution = self._resolve_path_component(c, directory, components)
	167	+ except UnexpectedFileException as original:
	168	+ errormsg = ("Reached a file called {} while trying to resolve a symlink; " +
	169	+ "cannot proceed. The remaining path components are {}.")
	170	+ raise UnexpectedFileException(errormsg.format(c, components)) from original
	171	+
	172	+ return resolution
	173	+
	174	+ def _resolve_path_component(self, c, directory, components_remaining):
	175	+ if c == ".":
	176	+ resolution = directory
	177	+ elif c == "..":
	178	+ if directory.parent is not None:
	179	+ resolution = directory.parent
	180	+ else:
	181	+ # If directory.parent *is* None, this is an attempt to
	182	+ # access '..' from the root, which is valid under
	183	+ # POSIX; it just returns the root.
	184	+ resolution = directory
	185	+ elif c in directory.index:
	186	+ try:
	187	+ resolution = self._resolve_through_files(c, directory, components_remaining)
	188	+ except UnexpectedFileException as original:
	189	+ errormsg = ("Reached a file called {} while trying to resolve a symlink; " +
	190	+ "cannot proceed. The remaining path components are {}.")
	191	+ raise UnexpectedFileException(errormsg.format(c, components_remaining)) from original
	192	+ else:
	193	+ # c is not in our index
	194	+ if self.force_create:
	195	+ resolution = directory.descend(c, create=True)
	196	+ else:
	197	+ resolution = None
	198	+ return resolution
	199	+
	200	+ def _resolve_through_files(self, c, directory, require_traversable):
	201	+ """A wrapper to resolve() which deals with files being found
	202	+ in the middle of paths, for example trying to resolve a symlink
	203	+ which points to /usr/lib64/libfoo when 'lib64' is a file.
	204	+
	205	+ require_traversable: If this is True, never return a file
	206	+ node. Instead, if force_create is set, destroy the file node,
	207	+ then create and return a normal directory in its place. If
	208	+ force_create is off, throws ResolutionException.
	209	+
	210	+ """
	211	+ resolved_thing = self.resolve(c, directory)
	212	+
	213	+ if isinstance(resolved_thing, remote_execution_pb2.FileNode):
	214	+ if require_traversable:
	215	+ # We have components still to resolve, but one of the path components
	216	+ # is a file.
	217	+ if self.force_create:
	218	+ directory.delete_entry(c)
	219	+ resolved_thing = directory.descend(c, create=True)
	220	+ else:
	221	+ # This is a signal that we hit a file, but don't
	222	+ # have the data to give a proper message, so the
	223	+ # caller should reraise this with a proper
	224	+ # description.
	225	+ raise UnexpectedFileException()
	226	+
	227	+ return resolved_thing
	228	+
	229	+
54	230	# CasBasedDirectory intentionally doesn't call its superclass constuctor,
55	231	# which is meant to be unimplemented.
56	232	# pylint: disable=super-init-not-called
...	...	@@ -168,29 +344,34 @@ class CasBasedDirectory(Directory):
168	344	self.index[name] = IndexEntry(dirnode, buildstream_object=newdir)
169	345	return newdir
170	346
171		- def _add_new_file(self, basename, filename):
	347	+ def _add_file(self, basename, filename, modified=False):
172	348	filenode = self.pb2_directory.files.add()
173	349	filenode.name = filename
174	350	self.cas_cache.add_object(digest=filenode.digest, path=os.path.join(basename, filename))
175	351	is_executable = os.access(os.path.join(basename, filename), os.X_OK)
176	352	filenode.is_executable = is_executable
177		- self.index[filename] = IndexEntry(filenode, modified=(filename in self.index))
	353	+ self.index[filename] = IndexEntry(filenode, modified=modified or filename in self.index)
178	354
179		- def _add_new_link(self, basename, filename):
180		- existing_link = self._find_pb2_entry(filename)
	355	+ def _copy_link_from_filesystem(self, basename, filename):
	356	+ self._add_new_link_direct(filename, os.readlink(os.path.join(basename, filename)))
	357	+
	358	+ def _add_new_link_direct(self, name, target):
	359	+ existing_link = self._find_pb2_entry(name)
181	360	if existing_link:
182	361	symlinknode = existing_link
183	362	else:
184	363	symlinknode = self.pb2_directory.symlinks.add()
185		- symlinknode.name = filename
	364	+ assert isinstance(symlinknode, remote_execution_pb2.SymlinkNode)
	365	+ symlinknode.name = name
186	366	# A symlink node has no digest.
187		- symlinknode.target = os.readlink(os.path.join(basename, filename))
188		- self.index[filename] = IndexEntry(symlinknode, modified=(existing_link is not None))
	367	+ symlinknode.target = target
	368	+ self.index[name] = IndexEntry(symlinknode, modified=(existing_link is not None))
189	369
190	370	def delete_entry(self, name):
191	371	for collection in [self.pb2_directory.files, self.pb2_directory.symlinks, self.pb2_directory.directories]:
192		- if name in collection:
193		- collection.remove(name)
	372	+ for thing in collection:
	373	+ if thing.name == name:
	374	+ collection.remove(thing)
194	375	if name in self.index:
195	376	del self.index[name]
196	377
...	...	@@ -231,9 +412,13 @@ class CasBasedDirectory(Directory):
231	412	if isinstance(entry, CasBasedDirectory):
232	413	return entry.descend(subdirectory_spec[1:], create)
233	414	else:
	415	+ # May be a symlink
	416	+ target = self._resolve(subdirectory_spec[0], force_create=create)
	417	+ if isinstance(target, CasBasedDirectory):
	418	+ return target
234	419	error = "Cannot descend into {}, which is a '{}' in the directory {}"
235	420	raise VirtualDirectoryError(error.format(subdirectory_spec[0],
236		- type(entry).__name__,
	421	+ type(self.index[subdirectory_spec[0]].pb_object).__name__,
237	422	self))
238	423	else:
239	424	if create:
...	...	@@ -254,36 +439,9 @@ class CasBasedDirectory(Directory):
254	439	else:
255	440	return self
256	441
257		- def _resolve_symlink_or_directory(self, name):
258		- """Used only by _import_files_from_directory. Tries to resolve a
259		- directory name or symlink name. 'name' must be an entry in this
260		- directory. It must be a single symlink or directory name, not a path
261		- separated by path separators. If it's an existing directory name, it
262		- just returns the Directory object for that. If it's a symlink, it will
263		- attempt to find the target of the symlink and return that as a
264		- Directory object.
265		-
266		- If a symlink target doesn't exist, it will attempt to create it
267		- as a directory as long as it's within this directory tree.
268		- """
269		-
270		- if isinstance(self.index[name].buildstream_object, Directory):
271		- return self.index[name].buildstream_object
272		- # OK then, it's a symlink
273		- symlink = self._find_pb2_entry(name)
274		- absolute = symlink.target.startswith(CasBasedDirectory._pb2_absolute_path_prefix)
275		- if absolute:
276		- root = self.find_root()
277		- else:
278		- root = self
279		- directory = root
280		- components = symlink.target.split(CasBasedDirectory._pb2_path_sep)
281		- for c in components:
282		- if c == "..":
283		- directory = directory.parent
284		- else:
285		- directory = directory.descend(c, create=True)
286		- return directory
	442	+ def _resolve(self, name, absolute_symlinks_resolve=True, force_create=False):
	443	+ resolver = _Resolver(absolute_symlinks_resolve, force_create)
	444	+ return resolver.resolve(name, self)
287	445
288	446	def _check_replacement(self, name, path_prefix, fileListResult):
289	447	""" Checks whether 'name' exists, and if so, whether we can overwrite it.
...	...	@@ -297,6 +455,7 @@ class CasBasedDirectory(Directory):
297	455	return True
298	456	if (isinstance(existing_entry,
299	457	(remote_execution_pb2.FileNode, remote_execution_pb2.SymlinkNode))):
	458	+ self.delete_entry(name)
300	459	fileListResult.overwritten.append(relative_pathname)
301	460	return True
302	461	elif isinstance(existing_entry, remote_execution_pb2.DirectoryNode):
...	...	@@ -314,23 +473,44 @@ class CasBasedDirectory(Directory):
314	473	.format(name, type(existing_entry)))
315	474	return False # In case asserts are disabled
316	475
317		- def _import_directory_recursively(self, directory_name, source_directory, remaining_path, path_prefix):
318		- """ _import_directory_recursively and _import_files_from_directory will be called alternately
319		- as a directory tree is descended. """
320		- if directory_name in self.index:
321		- subdir = self._resolve_symlink_or_directory(directory_name)
322		- else:
323		- subdir = self._add_directory(directory_name)
324		- new_path_prefix = os.path.join(path_prefix, directory_name)
325		- subdir_result = subdir._import_files_from_directory(os.path.join(source_directory, directory_name),
326		- [os.path.sep.join(remaining_path)],
327		- path_prefix=new_path_prefix)
328		- return subdir_result
	476	+ def _replace_anything_with_dir(self, name, path_prefix, overwritten_files_list):
	477	+ self.delete_entry(name)
	478	+ subdir = self._add_directory(name)
	479	+ overwritten_files_list.append(os.path.join(path_prefix, name))
	480	+ return subdir
329	481
330	482	def _import_files_from_directory(self, source_directory, files, path_prefix=""):
331		- """ Imports files from a traditional directory """
	483	+ """ Imports files from a traditional directory. """
	484	+
	485	+ def _ensure_followable(name, path_prefix):
	486	+ """ Makes sure 'name' is a directory or symlink to a directory which can be descended into. """
	487	+ if isinstance(self.index[name].buildstream_object, Directory):
	488	+ return self.descend(name)
	489	+ try:
	490	+ target = self._resolve(name, force_create=True)
	491	+ except InfiniteSymlinkException:
	492	+ return self._replace_anything_with_dir(name, path_prefix, result.overwritten)
	493	+ if isinstance(target, CasBasedDirectory):
	494	+ return target
	495	+ elif isinstance(target, remote_execution_pb2.FileNode):
	496	+ return self._replace_anything_with_dir(name, path_prefix, result.overwritten)
	497	+ return target
	498	+
	499	+ def _import_directory_recursively(directory_name, source_directory, remaining_path, path_prefix):
	500	+ """ _import_directory_recursively and _import_files_from_directory will be called alternately
	501	+ as a directory tree is descended. """
	502	+ if directory_name in self.index:
	503	+ subdir = _ensure_followable(directory_name, path_prefix)
	504	+ else:
	505	+ subdir = self._add_directory(directory_name)
	506	+ new_path_prefix = os.path.join(path_prefix, directory_name)
	507	+ subdir_result = subdir._import_files_from_directory(os.path.join(source_directory, directory_name),
	508	+ [os.path.sep.join(remaining_path)],
	509	+ path_prefix=new_path_prefix)
	510	+ return subdir_result
	511	+
332	512	result = FileListResult()
333		- for entry in sorted(files):
	513	+ for entry in files:
334	514	split_path = entry.split(os.path.sep)
335	515	# The actual file on the FS we're importing
336	516	import_file = os.path.join(source_directory, entry)
...	...	@@ -338,14 +518,18 @@ class CasBasedDirectory(Directory):
338	518	relative_pathname = os.path.join(path_prefix, entry)
339	519	if len(split_path) > 1:
340	520	directory_name = split_path[0]
341		- # Hand this off to the importer for that subdir. This will only do one file -
342		- # a better way would be to hand off all the files in this subdir at once.
343		- subdir_result = self._import_directory_recursively(directory_name, source_directory,
344		- split_path[1:], path_prefix)
	521	+ # Hand this off to the importer for that subdir.
	522	+
	523	+ # It would be advantageous to batch these together by
	524	+ # directory_name. However, we can't do it out of
	525	+ # order, since importing symlinks affects the results
	526	+ # of other imports.
	527	+ subdir_result = _import_directory_recursively(directory_name, source_directory,
	528	+ split_path[1:], path_prefix)
345	529	result.combine(subdir_result)
346	530	elif os.path.islink(import_file):
347	531	if self._check_replacement(entry, path_prefix, result):
348		- self._add_new_link(source_directory, entry)
	532	+ self._copy_link_from_filesystem(source_directory, entry)
349	533	result.files_written.append(relative_pathname)
350	534	elif os.path.isdir(import_file):
351	535	# A plain directory which already exists isn't a problem; just ignore it.
...	...	@@ -353,10 +537,78 @@ class CasBasedDirectory(Directory):
353	537	self._add_directory(entry)
354	538	elif os.path.isfile(import_file):
355	539	if self._check_replacement(entry, path_prefix, result):
356		- self._add_new_file(source_directory, entry)
	540	+ self._add_file(source_directory, entry, modified=relative_pathname in result.overwritten)
357	541	result.files_written.append(relative_pathname)
358	542	return result
359	543
	544	+ @staticmethod
	545	+ def _files_in_subdir(sorted_files, dirname):
	546	+ """Filters sorted_files and returns only the ones which have
	547	+ 'dirname' as a prefix, with that prefix removed.
	548	+
	549	+ """
	550	+ if not dirname.endswith(os.path.sep):
	551	+ dirname += os.path.sep
	552	+ return [f[len(dirname):] for f in sorted_files if f.startswith(dirname)]
	553	+
	554	+ def _partial_import_cas_into_cas(self, source_directory, files, path_prefix="", file_list_required=True):
	555	+ """ Import only the files and symlinks listed in 'files' from source_directory to this one.
	556	+ Args:
	557	+ source_directory (:class:`.CasBasedDirectory`): The directory to import from
	558	+ files ([str]): List of pathnames to import. Must be a list, not a generator.
	559	+ path_prefix (str): Prefix used to add entries to the file list result.
	560	+ file_list_required: Whether to update the file list while processing.
	561	+ """
	562	+ result = FileListResult()
	563	+ processed_directories = set()
	564	+ for f in files:
	565	+ fullname = os.path.join(path_prefix, f)
	566	+ components = f.split(os.path.sep)
	567	+ if len(components) > 1:
	568	+ # We are importing a thing which is in a subdirectory. We may have already seen this dirname
	569	+ # for a previous file.
	570	+ dirname = components[0]
	571	+ if dirname not in processed_directories:
	572	+ # Now strip off the first directory name and import files recursively.
	573	+ subcomponents = CasBasedDirectory._files_in_subdir(files, dirname)
	574	+ # We will fail at this point if there is a file or symlink to file called 'dirname'.
	575	+ if dirname in self.index:
	576	+ resolved_component = self._resolve(dirname, force_create=True)
	577	+ if isinstance(resolved_component, remote_execution_pb2.FileNode):
	578	+ dest_subdir = self._replace_anything_with_dir(dirname, path_prefix, result.overwritten)
	579	+ else:
	580	+ dest_subdir = resolved_component
	581	+ else:
	582	+ dest_subdir = self.descend(dirname, create=True)
	583	+ src_subdir = source_directory.descend(dirname)
	584	+ import_result = dest_subdir._partial_import_cas_into_cas(src_subdir, subcomponents,
	585	+ path_prefix=fullname,
	586	+ file_list_required=file_list_required)
	587	+ result.combine(import_result)
	588	+ processed_directories.add(dirname)
	589	+ elif isinstance(source_directory.index[f].buildstream_object, CasBasedDirectory):
	590	+ # The thing in the input file list is a directory on
	591	+ # its own. We don't need to do anything other than create it if it doesn't exist.
	592	+ # If we already have an entry with the same name that isn't a directory, that
	593	+ # will be dealt with when importing files in this directory.
	594	+ if f not in self.index:
	595	+ self.descend(f, create=True)
	596	+ else:
	597	+ # We're importing a file or symlink - replace anything with the same name.
	598	+ importable = self._check_replacement(f, path_prefix, result)
	599	+ if importable:
	600	+ item = source_directory.index[f].pb_object
	601	+ if isinstance(item, remote_execution_pb2.FileNode):
	602	+ filenode = self.pb2_directory.files.add(digest=item.digest, name=f,
	603	+ is_executable=item.is_executable)
	604	+ self.index[f] = IndexEntry(filenode, modified=True)
	605	+ else:
	606	+ assert isinstance(item, remote_execution_pb2.SymlinkNode)
	607	+ self._add_new_link_direct(name=f, target=item.target)
	608	+ else:
	609	+ result.ignored.append(os.path.join(path_prefix, f))
	610	+ return result
	611	+
360	612	def import_files(self, external_pathspec, *, files=None,
361	613	report_written=True, update_utimes=False,
362	614	can_link=False):
...	...	@@ -378,28 +630,27 @@ class CasBasedDirectory(Directory):
378	630
379	631	can_link (bool): Ignored, since hard links do not have any meaning within CAS.
380	632	"""
381		- if isinstance(external_pathspec, FileBasedDirectory):
382		- source_directory = external_pathspec._get_underlying_directory()
383		- elif isinstance(external_pathspec, CasBasedDirectory):
384		- # TODO: This transfers from one CAS to another via the
385		- # filesystem, which is very inefficient. Alter this so it
386		- # transfers refs across directly.
387		- with tempfile.TemporaryDirectory(prefix="roundtrip") as tmpdir:
388		- external_pathspec.export_files(tmpdir)
389		- if files is None:
390		- files = list_relative_paths(tmpdir)
391		- result = self._import_files_from_directory(tmpdir, files=files)
392		- return result
393		- else:
394		- source_directory = external_pathspec
395	633
396	634	if files is None:
397		- files = list_relative_paths(source_directory)
	635	+ if isinstance(external_pathspec, str):
	636	+ files = list_relative_paths(external_pathspec)
	637	+ else:
	638	+ assert isinstance(external_pathspec, Directory)
	639	+ files = external_pathspec.list_relative_paths()
	640	+
	641	+ if isinstance(external_pathspec, FileBasedDirectory):
	642	+ source_directory = external_pathspec.get_underlying_directory()
	643	+ result = self._import_files_from_directory(source_directory, files=files)
	644	+ elif isinstance(external_pathspec, str):
	645	+ source_directory = external_pathspec
	646	+ result = self._import_files_from_directory(source_directory, files=files)
	647	+ else:
	648	+ assert isinstance(external_pathspec, CasBasedDirectory)
	649	+ result = self._partial_import_cas_into_cas(external_pathspec, files=list(files))
398	650
399	651	# TODO: No notice is taken of report_written, update_utimes or can_link.
400	652	# Current behaviour is to fully populate the report, which is inefficient,
401	653	# but still correct.
402		- result = self._import_files_from_directory(source_directory, files=files)
403	654
404	655	# We need to recalculate and store the hashes of all directories both
405	656	# up and down the tree; we have changed our directory by importing files
...	...	@@ -511,6 +762,28 @@ class CasBasedDirectory(Directory):
511	762	else:
512	763	self._mark_directory_unmodified()
513	764
	765	+ def _lightweight_resolve_to_index(self, path):
	766	+ """A lightweight function for transforming paths into IndexEntry
	767	+ objects. This does not follow symlinks.
	768	+
	769	+ path: The string to resolve. This should be a series of path
	770	+ components separated by the protocol buffer path separator
	771	+ _pb2_path_sep.
	772	+
	773	+ Returns: the IndexEntry found, or None if any of the path components were not present.
	774	+
	775	+ """
	776	+ directory = self
	777	+ path_components = path.split(CasBasedDirectory._pb2_path_sep)
	778	+ for component in path_components[:-1]:
	779	+ if component not in directory.index:
	780	+ return None
	781	+ if isinstance(directory.index[component].buildstream_object, CasBasedDirectory):
	782	+ directory = directory.index[component].buildstream_object
	783	+ else:
	784	+ return None
	785	+ return directory.index.get(path_components[-1], None)
	786	+
514	787	def list_modified_paths(self):
515	788	"""Provide a list of relative paths which have been modified since the
516	789	last call to mark_unmodified.
...	...	@@ -518,29 +791,43 @@ class CasBasedDirectory(Directory):
518	791	Return value: List(str) - list of modified paths
519	792	"""
520	793
521		- filelist = []
522		- for (k, v) in self.index.items():
523		- if isinstance(v.buildstream_object, CasBasedDirectory):
524		- filelist.extend([k + os.path.sep + x for x in v.buildstream_object.list_modified_paths()])
525		- elif isinstance(v.pb_object, remote_execution_pb2.FileNode) and v.modified:
526		- filelist.append(k)
527		- return filelist
	794	+ for p in self.list_relative_paths():
	795	+ i = self._lightweight_resolve_to_index(p)
	796	+ if i and i.modified:
	797	+ yield p
528	798
529		- def list_relative_paths(self):
	799	+ def list_relative_paths(self, relpath=""):
530	800	"""Provide a list of all relative paths.
531	801
532		- NOTE: This list is not in the same order as utils.list_relative_paths.
533		-
534	802	Return value: List(str) - list of all paths
535	803	"""
536	804
537		- filelist = []
538		- for (k, v) in self.index.items():
539		- if isinstance(v.buildstream_object, CasBasedDirectory):
540		- filelist.extend([k + os.path.sep + x for x in v.buildstream_object.list_relative_paths()])
541		- elif isinstance(v.pb_object, remote_execution_pb2.FileNode):
542		- filelist.append(k)
543		- return filelist
	805	+ symlink_list = filter(lambda i: isinstance(i[1].pb_object, remote_execution_pb2.SymlinkNode),
	806	+ self.index.items())
	807	+ file_list = list(filter(lambda i: isinstance(i[1].pb_object, remote_execution_pb2.FileNode),
	808	+ self.index.items()))
	809	+ directory_list = filter(lambda i: isinstance(i[1].buildstream_object, CasBasedDirectory),
	810	+ self.index.items())
	811	+
	812	+ # We need to mimic the behaviour of os.walk, in which symlinks
	813	+ # to directories count as directories and symlinks to file or
	814	+ # broken symlinks count as files. os.walk doesn't follow
	815	+ # symlinks, so we don't recurse.
	816	+ for (k, v) in sorted(symlink_list):
	817	+ target = self._resolve(k, absolute_symlinks_resolve=True)
	818	+ if isinstance(target, CasBasedDirectory):
	819	+ yield os.path.join(relpath, k)
	820	+ else:
	821	+ file_list.append((k, v))
	822	+
	823	+ if file_list == [] and relpath != "":
	824	+ yield relpath
	825	+ else:
	826	+ for (k, v) in sorted(file_list):
	827	+ yield os.path.join(relpath, k)
	828	+
	829	+ for (k, v) in sorted(directory_list):
	830	+ yield from v.buildstream_object.list_relative_paths(relpath=os.path.join(relpath, k))
544	831
545	832	def recalculate_hash(self):
546	833	""" Recalcuates the hash for this directory and store the results in

tests/completions/completions.py

---

...	...	@@ -42,6 +42,7 @@ MAIN_OPTIONS = [
42	42	"-o ",
43	43	"--option ",
44	44	"--on-error ",
	45	+ "--pull-build-trees ",
45	46	"--pushers ",
46	47	"--strict ",
47	48	"--verbose ",

tests/integration/build-tree.py

---

...	...	@@ -70,8 +70,8 @@ def test_buildtree_pulled(cli, tmpdir, datafiles):
70	70	})
71	71	assert cli.get_element_state(project, element_name) != 'cached'
72	72
73		- # Pull from cache
74		- result = cli.run(project=project, args=['pull', '--deps', 'all', element_name])
	73	+ # Pull from cache, ensuring cli options is set to pull the buildtree
	74	+ result = cli.run(project=project, args=['--pull-build-trees', 'pull', '--deps', 'all', element_name])
75	75	result.assert_success()
76	76
77	77	# Check it's using the cached build tree

tests/integration/pullbuildtrees.py

---

	1	+import os
	2	+import shutil
	3	+import pytest
	4	+
	5	+from tests.testutils import cli_integration as cli, create_artifact_share
	6	+from tests.testutils.integration import assert_contains
	7	+
	8	+
	9	+DATA_DIR = os.path.join(
	10	+ os.path.dirname(os.path.realpath(__file__)),
	11	+ "project"
	12	+)
	13	+
	14	+
	15	+# Remove artifact cache & set cli.config value of pullbuildtrees
	16	+# to false, which is the default user context. The cache has to be
	17	+# cleared as just forcefully removing the refpath leaves dangling objects.
	18	+def default_state(cli, tmpdir, share):
	19	+ shutil.rmtree(os.path.join(str(tmpdir), 'artifacts'))
	20	+ cli.configure({
	21	+ 'pullbuildtrees': False,
	22	+ 'artifacts': {'url': share.repo, 'push': False},
	23	+ 'artifactdir': os.path.join(str(tmpdir), 'artifacts')
	24	+ })
	25	+
	26	+
	27	+# A test to capture the integration of the pullbuildtrees
	28	+# behaviour, which by default is to not include the buildtree
	29	+# directory of an element.
	30	+@pytest.mark.integration
	31	+@pytest.mark.datafiles(DATA_DIR)
	32	+def test_pullbuildtrees(cli, tmpdir, datafiles, integration_cache):
	33	+
	34	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	35	+ element_name = 'autotools/amhello.bst'
	36	+
	37	+ # Create artifact shares for pull & push testing
	38	+ with create_artifact_share(os.path.join(str(tmpdir), 'share1')) as share1,\
	39	+ create_artifact_share(os.path.join(str(tmpdir), 'share2')) as share2:
	40	+ cli.configure({
	41	+ 'artifacts': {'url': share1.repo, 'push': True},
	42	+ 'artifactdir': os.path.join(str(tmpdir), 'artifacts')
	43	+ })
	44	+
	45	+ # Build autotools element, checked pushed, delete local
	46	+ result = cli.run(project=project, args=['build', element_name])
	47	+ assert result.exit_code == 0
	48	+ assert cli.get_element_state(project, element_name) == 'cached'
	49	+ assert share1.has_artifact('test', element_name, cli.get_element_key(project, element_name))
	50	+ default_state(cli, tmpdir, share1)
	51	+
	52	+ # Pull artifact with default config, assert that pulling again
	53	+ # doesn't create a pull job, then assert with buildtrees user
	54	+ # config set creates a pull job.
	55	+ result = cli.run(project=project, args=['pull', element_name])
	56	+ assert element_name in result.get_pulled_elements()
	57	+ result = cli.run(project=project, args=['pull', element_name])
	58	+ assert element_name not in result.get_pulled_elements()
	59	+ cli.configure({'pullbuildtrees': True})
	60	+ result = cli.run(project=project, args=['pull', element_name])
	61	+ assert element_name in result.get_pulled_elements()
	62	+ default_state(cli, tmpdir, share1)
	63	+
	64	+ # Pull artifact with default config, then assert that pulling
	65	+ # with buildtrees cli flag set creates a pull job.
	66	+ # Also assert that the buildtree is added to the artifact's
	67	+ # extract dir
	68	+ result = cli.run(project=project, args=['pull', element_name])
	69	+ assert element_name in result.get_pulled_elements()
	70	+ elementdigest = share1.has_artifact('test', element_name, cli.get_element_key(project, element_name))
	71	+ buildtreedir = os.path.join(str(tmpdir), 'artifacts', 'extract', 'test', 'autotools-amhello',
	72	+ elementdigest.hash, 'buildtree')
	73	+ assert not os.path.isdir(buildtreedir)
	74	+ result = cli.run(project=project, args=['--pull-build-trees', 'pull', element_name])
	75	+ assert element_name in result.get_pulled_elements()
	76	+ assert os.path.isdir(buildtreedir)
	77	+ default_state(cli, tmpdir, share1)
	78	+
	79	+ # Pull artifact with pullbuildtrees set in user config, then assert
	80	+ # that pulling with the same user config doesn't creates a pull job,
	81	+ # or when buildtrees cli flag is set.
	82	+ cli.configure({'pullbuildtrees': True})
	83	+ result = cli.run(project=project, args=['pull', element_name])
	84	+ assert element_name in result.get_pulled_elements()
	85	+ result = cli.run(project=project, args=['pull', element_name])
	86	+ assert element_name not in result.get_pulled_elements()
	87	+ result = cli.run(project=project, args=['--pull-build-trees', 'pull', element_name])
	88	+ assert element_name not in result.get_pulled_elements()
	89	+ default_state(cli, tmpdir, share1)
	90	+
	91	+ # Pull artifact with default config and buildtrees cli flag set, then assert
	92	+ # that pulling with pullbuildtrees set in user config doesn't create a pull
	93	+ # job.
	94	+ result = cli.run(project=project, args=['--pull-build-trees', 'pull', element_name])
	95	+ assert element_name in result.get_pulled_elements()
	96	+ cli.configure({'pullbuildtrees': True})
	97	+ result = cli.run(project=project, args=['pull', element_name])
	98	+ assert element_name not in result.get_pulled_elements()
	99	+ default_state(cli, tmpdir, share1)
	100	+
	101	+ # Assert that a partial build element (not containing a populated buildtree dir)
	102	+ # can't be pushed to an artifact share, then assert that a complete build element
	103	+ # can be. This will attempt a partial pull from share1 and then a partial push
	104	+ # to share2
	105	+ result = cli.run(project=project, args=['pull', element_name])
	106	+ assert element_name in result.get_pulled_elements()
	107	+ cli.configure({'artifacts': {'url': share2.repo, 'push': True}})
	108	+ result = cli.run(project=project, args=['push', element_name])
	109	+ assert element_name not in result.get_pushed_elements()
	110	+ assert not share2.has_artifact('test', element_name, cli.get_element_key(project, element_name))
	111	+
	112	+ # Assert that after pulling the missing buildtree the element artifact can be
	113	+ # successfully pushed to the remote. This will attempt to pull the buildtree
	114	+ # from share1 and then a 'complete' push to share2
	115	+ cli.configure({'artifacts': {'url': share1.repo, 'push': False}})
	116	+ result = cli.run(project=project, args=['--pull-build-trees', 'pull', element_name])
	117	+ assert element_name in result.get_pulled_elements()
	118	+ cli.configure({'artifacts': {'url': share2.repo, 'push': True}})
	119	+ result = cli.run(project=project, args=['push', element_name])
	120	+ assert element_name in result.get_pushed_elements()
	121	+ assert share2.has_artifact('test', element_name, cli.get_element_key(project, element_name))
	122	+ default_state(cli, tmpdir, share1)

tests/storage/virtual_directory_import.py

---

	1	+from hashlib import sha256
	2	+import os
	3	+import pytest
	4	+import random
	5	+import tempfile
	6	+from tests.testutils import cli
	7	+
	8	+from buildstream.storage._casbaseddirectory import CasBasedDirectory
	9	+from buildstream.storage._filebaseddirectory import FileBasedDirectory
	10	+from buildstream._artifactcache import ArtifactCache
	11	+from buildstream._artifactcache.cascache import CASCache
	12	+from buildstream import utils
	13	+
	14	+
	15	+# These are comparitive tests that check that FileBasedDirectory and
	16	+# CasBasedDirectory act identically.
	17	+
	18	+
	19	+class FakeArtifactCache():
	20	+ def __init__(self):
	21	+ self.cas = None
	22	+
	23	+
	24	+class FakeContext():
	25	+ def __init__(self):
	26	+ self.artifactdir = ''
	27	+ self.artifactcache = FakeArtifactCache()
	28	+
	29	+
	30	+# This is a set of example file system contents. It's a set of trees
	31	+# which are either expected to be problematic or were found to be
	32	+# problematic during random testing.
	33	+
	34	+# The test attempts to import each on top of each other to test
	35	+# importing works consistently. Each tuple is defined as (<filename>,
	36	+# <type>, <content>). Type can be 'F' (file), 'S' (symlink) or 'D'
	37	+# (directory) with content being the contents for a file or the
	38	+# destination for a symlink.
	39	+root_filesets = [
	40	+ [('a/b/c/textfile1', 'F', 'This is textfile 1\n')],
	41	+ [('a/b/c/textfile1', 'F', 'This is the replacement textfile 1\n')],
	42	+ [('a/b/d', 'D', '')],
	43	+ [('a/b/c', 'S', '/a/b/d')],
	44	+ [('a/b/d', 'S', '/a/b/c')],
	45	+ [('a/b/d', 'D', ''), ('a/b/c', 'S', '/a/b/d')],
	46	+ [('a/b/c', 'D', ''), ('a/b/d', 'S', '/a/b/c')],
	47	+ [('a/b', 'F', 'This is textfile 1\n')],
	48	+ [('a/b/c', 'F', 'This is textfile 1\n')],
	49	+ [('a/b/c', 'D', '')]
	50	+]
	51	+
	52	+empty_hash_ref = sha256().hexdigest()
	53	+RANDOM_SEED = 69105
	54	+NUM_RANDOM_TESTS = 10
	55	+
	56	+
	57	+def generate_import_roots(rootno, directory):
	58	+ rootname = "root{}".format(rootno)
	59	+ rootdir = os.path.join(directory, "content", rootname)
	60	+ if os.path.exists(rootdir):
	61	+ return
	62	+ for (path, typesymbol, content) in root_filesets[rootno - 1]:
	63	+ if typesymbol == 'F':
	64	+ (dirnames, filename) = os.path.split(path)
	65	+ os.makedirs(os.path.join(rootdir, dirnames), exist_ok=True)
	66	+ with open(os.path.join(rootdir, dirnames, filename), "wt") as f:
	67	+ f.write(content)
	68	+ elif typesymbol == 'D':
	69	+ os.makedirs(os.path.join(rootdir, path), exist_ok=True)
	70	+ elif typesymbol == 'S':
	71	+ (dirnames, filename) = os.path.split(path)
	72	+ os.makedirs(os.path.join(rootdir, dirnames), exist_ok=True)
	73	+ os.symlink(content, os.path.join(rootdir, path))
	74	+
	75	+
	76	+def generate_random_root(rootno, directory):
	77	+ # By seeding the random number generator, we ensure these tests
	78	+ # will be repeatable, at least until Python changes the random
	79	+ # number algorithm.
	80	+ random.seed(RANDOM_SEED + rootno)
	81	+ rootname = "root{}".format(rootno)
	82	+ rootdir = os.path.join(directory, "content", rootname)
	83	+ if os.path.exists(rootdir):
	84	+ return
	85	+ things = []
	86	+ locations = ['.']
	87	+ os.makedirs(rootdir)
	88	+ for i in range(0, 100):
	89	+ location = random.choice(locations)
	90	+ thingname = "node{}".format(i)
	91	+ thing = random.choice(['dir', 'link', 'file'])
	92	+ target = os.path.join(rootdir, location, thingname)
	93	+ if thing == 'dir':
	94	+ os.makedirs(target)
	95	+ locations.append(os.path.join(location, thingname))
	96	+ elif thing == 'file':
	97	+ with open(target, "wt") as f:
	98	+ f.write("This is node {}\n".format(i))
	99	+ elif thing == 'link':
	100	+ symlink_type = random.choice(['absolute', 'relative', 'broken'])
	101	+ if symlink_type == 'broken' or not things:
	102	+ os.symlink("/broken", target)
	103	+ elif symlink_type == 'absolute':
	104	+ symlink_destination = random.choice(things)
	105	+ os.symlink(symlink_destination, target)
	106	+ else:
	107	+ symlink_destination = random.choice(things)
	108	+ relative_link = os.path.relpath(symlink_destination, start=location)
	109	+ os.symlink(relative_link, target)
	110	+ things.append(os.path.join(location, thingname))
	111	+
	112	+
	113	+def file_contents(path):
	114	+ with open(path, "r") as f:
	115	+ result = f.read()
	116	+ return result
	117	+
	118	+
	119	+def file_contents_are(path, contents):
	120	+ return file_contents(path) == contents
	121	+
	122	+
	123	+def create_new_casdir(root_number, fake_context, tmpdir):
	124	+ d = CasBasedDirectory(fake_context)
	125	+ d.import_files(os.path.join(tmpdir, "content", "root{}".format(root_number)))
	126	+ assert d.ref.hash != empty_hash_ref
	127	+ return d
	128	+
	129	+
	130	+def create_new_filedir(root_number, tmpdir):
	131	+ root = os.path.join(tmpdir, "vdir")
	132	+ os.makedirs(root)
	133	+ d = FileBasedDirectory(root)
	134	+ d.import_files(os.path.join(tmpdir, "content", "root{}".format(root_number)))
	135	+ return d
	136	+
	137	+
	138	+def combinations(integer_range):
	139	+ for x in integer_range:
	140	+ for y in integer_range:
	141	+ yield (x, y)
	142	+
	143	+
	144	+def resolve_symlinks(path, root):
	145	+ """ A function to resolve symlinks inside 'path' components apart from the last one.
	146	+ For example, resolve_symlinks('/a/b/c/d', '/a/b')
	147	+ will return '/a/b/f/d' if /a/b/c is a symlink to /a/b/f. The final component of
	148	+ 'path' is not resolved, because we typically want to inspect the symlink found
	149	+ at that path, not its target.
	150	+
	151	+ """
	152	+ components = path.split(os.path.sep)
	153	+ location = root
	154	+ for i in range(0, len(components) - 1):
	155	+ location = os.path.join(location, components[i])
	156	+ if os.path.islink(location):
	157	+ # Resolve the link, add on all the remaining components
	158	+ target = os.path.join(os.readlink(location))
	159	+ tail = os.path.sep.join(components[i + 1:])
	160	+
	161	+ if target.startswith(os.path.sep):
	162	+ # Absolute link - relative to root
	163	+ location = os.path.join(root, target, tail)
	164	+ else:
	165	+ # Relative link - relative to symlink location
	166	+ location = os.path.join(location, target)
	167	+ return resolve_symlinks(location, root)
	168	+ # If we got here, no symlinks were found. Add on the final component and return.
	169	+ location = os.path.join(location, components[-1])
	170	+ return location
	171	+
	172	+
	173	+def directory_not_empty(path):
	174	+ return os.listdir(path)
	175	+
	176	+
	177	+def _import_test(tmpdir, original, overlay, generator_function, verify_contents=False):
	178	+ fake_context = FakeContext()
	179	+ fake_context.artifactcache.cas = CASCache(tmpdir)
	180	+ # Create some fake content
	181	+ generator_function(original, tmpdir)
	182	+ if original != overlay:
	183	+ generator_function(overlay, tmpdir)
	184	+
	185	+ d = create_new_casdir(original, fake_context, tmpdir)
	186	+
	187	+ duplicate_cas = create_new_casdir(original, fake_context, tmpdir)
	188	+
	189	+ assert duplicate_cas.ref.hash == d.ref.hash
	190	+
	191	+ d2 = create_new_casdir(overlay, fake_context, tmpdir)
	192	+ d.import_files(d2)
	193	+ export_dir = os.path.join(tmpdir, "output-{}-{}".format(original, overlay))
	194	+ roundtrip_dir = os.path.join(tmpdir, "roundtrip-{}-{}".format(original, overlay))
	195	+ d2.export_files(roundtrip_dir)
	196	+ d.export_files(export_dir)
	197	+
	198	+ if verify_contents:
	199	+ for item in root_filesets[overlay - 1]:
	200	+ (path, typename, content) = item
	201	+ realpath = resolve_symlinks(path, export_dir)
	202	+ if typename == 'F':
	203	+ if os.path.isdir(realpath) and directory_not_empty(realpath):
	204	+ # The file should not have overwritten the directory in this case.
	205	+ pass
	206	+ else:
	207	+ assert os.path.isfile(realpath), "{} did not exist in the combined virtual directory".format(path)
	208	+ assert file_contents_are(realpath, content)
	209	+ elif typename == 'S':
	210	+ if os.path.isdir(realpath) and directory_not_empty(realpath):
	211	+ # The symlink should not have overwritten the directory in this case.
	212	+ pass
	213	+ else:
	214	+ assert os.path.islink(realpath)
	215	+ assert os.readlink(realpath) == content
	216	+ elif typename == 'D':
	217	+ # We can't do any more tests than this because it
	218	+ # depends on things present in the original. Blank
	219	+ # directories here will be ignored and the original
	220	+ # left in place.
	221	+ assert os.path.lexists(realpath)
	222	+
	223	+ # Now do the same thing with filebaseddirectories and check the contents match
	224	+
	225	+ files = list(utils.list_relative_paths(roundtrip_dir))
	226	+ duplicate_cas._import_files_from_directory(roundtrip_dir, files=files)
	227	+ duplicate_cas._recalculate_recursing_down()
	228	+ if duplicate_cas.parent:
	229	+ duplicate_cas.parent._recalculate_recursing_up(duplicate_cas)
	230	+
	231	+ assert duplicate_cas.ref.hash == d.ref.hash
	232	+
	233	+
	234	+# It's possible to parameterize on both original and overlay values,
	235	+# but this leads to more tests being listed in the output than are
	236	+# comfortable.
	237	+@pytest.mark.parametrize("original", range(1, len(root_filesets) + 1))
	238	+def test_fixed_cas_import(cli, tmpdir, original):
	239	+ for overlay in range(1, len(root_filesets) + 1):
	240	+ _import_test(str(tmpdir), original, overlay, generate_import_roots, verify_contents=True)
	241	+
	242	+
	243	+@pytest.mark.parametrize("original", range(1, NUM_RANDOM_TESTS + 1))
	244	+def test_random_cas_import(cli, tmpdir, original):
	245	+ for overlay in range(1, NUM_RANDOM_TESTS + 1):
	246	+ _import_test(str(tmpdir), original, overlay, generate_random_root, verify_contents=False)
	247	+
	248	+
	249	+def _listing_test(tmpdir, root, generator_function):
	250	+ fake_context = FakeContext()
	251	+ fake_context.artifactcache.cas = CASCache(tmpdir)
	252	+ # Create some fake content
	253	+ generator_function(root, tmpdir)
	254	+
	255	+ d = create_new_filedir(root, tmpdir)
	256	+ filelist = list(d.list_relative_paths())
	257	+
	258	+ d2 = create_new_casdir(root, fake_context, tmpdir)
	259	+ filelist2 = list(d2.list_relative_paths())
	260	+
	261	+ assert filelist == filelist2
	262	+
	263	+
	264	+@pytest.mark.parametrize("root", range(1, 11))
	265	+def test_random_directory_listing(cli, tmpdir, root):
	266	+ _listing_test(str(tmpdir), root, generate_random_root)
	267	+
	268	+
	269	+@pytest.mark.parametrize("root", [1, 2, 3, 4, 5])
	270	+def test_fixed_directory_listing(cli, tmpdir, root):
	271	+ _listing_test(str(tmpdir), root, generate_import_roots)

tests/testutils/artifactshare.py

---

...	...	@@ -114,7 +114,7 @@ class ArtifactShare():
114	114	# cache_key (str): The cache key
115	115	#
116	116	# Returns:
117		- # (bool): True if the artifact exists in the share, otherwise false.
	117	+ # (str): artifact digest if the artifact exists in the share, otherwise None.
118	118	def has_artifact(self, project_name, element_name, cache_key):
119	119
120	120	# NOTE: This should be kept in line with our
...	...	@@ -134,9 +134,9 @@ class ArtifactShare():
134	134
135	135	try:
136	136	tree = self.cas.resolve_ref(artifact_key)
137		- return True
	137	+ return tree
138	138	except CASError:
139		- return False
	139	+ return None
140	140
141	141	# close():
142	142	#

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.