[Notes] [Git][BuildStream/buildstream][jmac/virtual_directories] 12 commits: Add BST_VIRTUAL_DIRECTORY flag for element plugins

[Jim MacArthur <gitlab mg gitlab com>]

Title: GitLab

Jim MacArthur pushed to branch jmac/virtual_directories at BuildStream / buildstream

Commits:

• 480c0184
  by Jim MacArthur at 2018-07-23T14:20:02Z

  Add BST_VIRTUAL_DIRECTORY flag for element plugins
  

• 59fb9152
  by Jim MacArthur at 2018-07-23T14:20:02Z

  Sandbox.py: Rename __root to _root to allow its use by subclasses.
  
  Since access to get_directories is now blocked for some plugins,
  and the subclasses of Sandbox do not have configuration defined
  by YAML files, they need another way to get at the root directory.
  
  NB Could this be done just with get_virtual_directory and .external_directory?
  

• 227bc204
  by Jim MacArthur at 2018-07-24T11:32:59Z

  Add the virtual directory class 'Directory' and one implementation.
  
  buildstream/storage/Directory.py: New file.
  buildstream/storage/_filebaseddirectory.py: New file.
  buildstream/_exceptions.py: New VIRTUAL_FS exception source.
  

• 9aa7db21
  by Jim MacArthur at 2018-07-24T11:32:59Z

  sandbox.py: Two functions to get virtual directories
  
  Add get_virtual_directory and get_virtual_toplevel_directory().
  

• 1a2e4cca
  by Jim MacArthur at 2018-07-24T11:32:59Z

  _stream.py: Convert to virtual directories.
  
  This removes _add_directory_to_tarfile since it is now implemented in
  _filebaseddirectory.py.
  

• 753566f3
  by Jim MacArthur at 2018-07-24T11:32:59Z

  element.py: Conversion to use virtual directories
  

• 0325fd9d
  by Jim MacArthur at 2018-07-24T11:32:59Z

  plugins/elements/compose.py: Convert to virtual directories
  

• 5d277366
  by Jim MacArthur at 2018-07-24T11:32:59Z

  plugins/elements/import.py: Convert to virtual directories
  

• 8d5f8727
  by Jim MacArthur at 2018-07-24T11:32:59Z

  sandbox: Remove instances of get_directory
  
  sandbox/_mount.py, sandbox/_sandboxbwrap.py:
    Remove instances of get_directory
  

• e265478e
  by Jim MacArthur at 2018-07-24T11:32:59Z

  plugins/elements/stack.py: Convert to virtual directories
  

• d5d8d58a
  by Jim MacArthur at 2018-07-24T11:32:59Z

  scriptelement.py: Convert to virtual directories
  

• 7f99207a
  by Jim MacArthur at 2018-07-24T11:32:59Z

  storage: Add __init__.py
  

14 changed files:

• buildstream/_exceptions.py
• buildstream/_stream.py
• buildstream/element.py
• buildstream/plugins/elements/compose.py
• buildstream/plugins/elements/import.py
• buildstream/plugins/elements/stack.py
• buildstream/sandbox/_mount.py
• buildstream/sandbox/_sandboxbwrap.py
• buildstream/sandbox/_sandboxchroot.py
• buildstream/sandbox/sandbox.py
• buildstream/scriptelement.py
• + buildstream/storage/__init__.py
• + buildstream/storage/_filebaseddirectory.py
• + buildstream/storage/directory.py

Changes:

buildstream/_exceptions.py

---

...	...	@@ -88,6 +88,7 @@ class ErrorDomain(Enum):
88	88	ELEMENT = 11
89	89	APP = 12
90	90	STREAM = 13
	91	+ VIRTUAL_FS = 14
91	92
92	93
93	94	# BstError is an internal base exception class for BuildSream

buildstream/_stream.py

---

...	...	@@ -407,15 +407,16 @@ class Stream():
407	407	integrate=integrate) as sandbox:
408	408
409	409	# Copy or move the sandbox to the target directory
410		- sandbox_root = sandbox.get_directory()
	410	+ sandbox_vroot = sandbox.get_virtual_directory()
	411	+
411	412	if not tar:
412	413	with target.timed_activity("Checking out files in '{}'"
413	414	.format(location)):
414	415	try:
415	416	if hardlinks:
416		- self._checkout_hardlinks(sandbox_root, location)
	417	+ self._checkout_hardlinks(sandbox_vroot, location)
417	418	else:
418		- utils.copy_files(sandbox_root, location)
	419	+ sandbox_vroot.export_files(location)
419	420	except OSError as e:
420	421	raise StreamError("Failed to checkout files: '{}'"
421	422	.format(e)) from e
...	...	@@ -424,14 +425,12 @@ class Stream():
424	425	with target.timed_activity("Creating tarball"):
425	426	with os.fdopen(sys.stdout.fileno(), 'wb') as fo:
426	427	with tarfile.open(fileobj=fo, mode="w|") as tf:
427		- Stream._add_directory_to_tarfile(
428		- tf, sandbox_root, '.')
	428	+ sandbox_vroot.export_to_tar(tf, '.')
429	429	else:
430	430	with target.timed_activity("Creating tarball '{}'"
431	431	.format(location)):
432	432	with tarfile.open(location, "w:") as tf:
433		- Stream._add_directory_to_tarfile(
434		- tf, sandbox_root, '.')
	433	+ sandbox_vroot.export_to_tar(tf, '.')
435	434
436	435	except BstError as e:
437	436	raise StreamError("Error while staging dependencies into a sandbox"
...	...	@@ -1046,46 +1045,13 @@ class Stream():
1046	1045
1047	1046	# Helper function for checkout()
1048	1047	#
1049		- def _checkout_hardlinks(self, sandbox_root, directory):
	1048	+ def _checkout_hardlinks(self, sandbox_vroot, directory):
1050	1049	try:
1051		- removed = utils.safe_remove(directory)
	1050	+ utils.safe_remove(directory)
1052	1051	except OSError as e:
1053	1052	raise StreamError("Failed to remove checkout directory: {}".format(e)) from e
1054	1053
1055		- if removed:
1056		- # Try a simple rename of the sandbox root; if that
1057		- # doesnt cut it, then do the regular link files code path
1058		- try:
1059		- os.rename(sandbox_root, directory)
1060		- except OSError:
1061		- os.makedirs(directory, exist_ok=True)
1062		- utils.link_files(sandbox_root, directory)
1063		- else:
1064		- utils.link_files(sandbox_root, directory)
1065		-
1066		- # Add a directory entry deterministically to a tar file
1067		- #
1068		- # This function takes extra steps to ensure the output is deterministic.
1069		- # First, it sorts the results of os.listdir() to ensure the ordering of
1070		- # the files in the archive is the same. Second, it sets a fixed
1071		- # timestamp for each entry. See also https://bugs.python.org/issue24465.
1072		- @staticmethod
1073		- def _add_directory_to_tarfile(tf, dir_name, dir_arcname, mtime=0):
1074		- for filename in sorted(os.listdir(dir_name)):
1075		- name = os.path.join(dir_name, filename)
1076		- arcname = os.path.join(dir_arcname, filename)
1077		-
1078		- tarinfo = tf.gettarinfo(name, arcname)
1079		- tarinfo.mtime = mtime
1080		-
1081		- if tarinfo.isreg():
1082		- with open(name, "rb") as f:
1083		- tf.addfile(tarinfo, f)
1084		- elif tarinfo.isdir():
1085		- tf.addfile(tarinfo)
1086		- Stream._add_directory_to_tarfile(tf, name, arcname, mtime)
1087		- else:
1088		- tf.addfile(tarinfo)
	1054	+ sandbox_vroot.export_files(directory, can_link=True, can_destroy=True)
1089	1055
1090	1056	# Write the element build script to the given directory
1091	1057	def _write_element_script(self, directory, element):

buildstream/element.py

---

...	...	@@ -80,7 +80,6 @@ from collections import Mapping, OrderedDict
80	80	from contextlib import contextmanager
81	81	from enum import Enum
82	82	import tempfile
83		-import time
84	83	import shutil
85	84
86	85	from . import _yaml
...	...	@@ -97,6 +96,9 @@ from . import _site
97	96	from ._platform import Platform
98	97	from .sandbox._config import SandboxConfig
99	98
	99	+from .storage.directory import Directory
	100	+from .storage._filebaseddirectory import FileBasedDirectory, VirtualDirectoryError
	101	+
100	102
101	103	# _KeyStrength():
102	104	#
...	...	@@ -191,6 +193,13 @@ class Element(Plugin):
191	193	*Since: 1.2*
192	194	"""
193	195
	196	+ BST_VIRTUAL_DIRECTORY = False
	197	+ """Whether to raise exceptions if an element uses Sandbox.get_directory
	198	+ instead of Sandbox.get_virtual_directory.
	199	+
	200	+ *Since: 1.4*
	201	+ """
	202	+
194	203	def __init__(self, context, project, artifacts, meta, plugin_conf):
195	204
196	205	self.__cache_key_dict = None # Dict for cache key calculation
...	...	@@ -620,10 +629,10 @@ class Element(Plugin):
620	629
621	630	# Hard link it into the staging area
622	631	#
623		- basedir = sandbox.get_directory()
624		- stagedir = basedir \
	632	+ vbasedir = sandbox.get_virtual_directory()
	633	+ vstagedir = vbasedir \
625	634	if path is None \
626		- else os.path.join(basedir, path.lstrip(os.sep))
	635	+ else vbasedir.descend(path.lstrip(os.sep).split(os.sep))
627	636
628	637	files = list(self.__compute_splits(include, exclude, orphans))
629	638
...	...	@@ -635,15 +644,8 @@ class Element(Plugin):
635	644	link_files = files
636	645	copy_files = []
637	646
638		- link_result = utils.link_files(artifact, stagedir, files=link_files,
639		- report_written=True)
640		- copy_result = utils.copy_files(artifact, stagedir, files=copy_files,
641		- report_written=True)
642		-
643		- cur_time = time.time()
644		-
645		- for f in copy_result.files_written:
646		- os.utime(os.path.join(stagedir, f), times=(cur_time, cur_time))
	647	+ link_result = vstagedir.import_files(artifact, files=link_files, report_written=True, can_link=True)
	648	+ copy_result = vstagedir.import_files(artifact, files=copy_files, report_written=True, update_utimes=True)
647	649
648	650	return link_result.combine(copy_result)
649	651
...	...	@@ -1300,8 +1302,8 @@ class Element(Plugin):
1300	1302	sandbox._set_mount_source(directory, workspace.get_absolute_path())
1301	1303
1302	1304	# Stage all sources that need to be copied
1303		- sandbox_root = sandbox.get_directory()
1304		- host_directory = os.path.join(sandbox_root, directory.lstrip(os.sep))
	1305	+ sandbox_vroot = sandbox.get_virtual_directory()
	1306	+ host_directory = sandbox_vroot.descend(directory.lstrip(os.sep).split(os.sep), create=True)
1305	1307	self._stage_sources_at(host_directory, mount_workspaces=mount_workspaces)
1306	1308
1307	1309	# _stage_sources_at():
...	...	@@ -1309,31 +1311,36 @@ class Element(Plugin):
1309	1311	# Stage this element's sources to a directory
1310	1312	#
1311	1313	# Args:
1312		- # directory (str): An absolute path to stage the sources at
	1314	+ # vdirectory (:class:`.storage.Directory`): A virtual directory object to stage sources into.
1313	1315	# mount_workspaces (bool): mount workspaces if True, copy otherwise
1314	1316	#
1315		- def _stage_sources_at(self, directory, mount_workspaces=True):
	1317	+ def _stage_sources_at(self, vdirectory, mount_workspaces=True):
1316	1318	with self.timed_activity("Staging sources", silent_nested=True):
1317	1319
1318		- if os.path.isdir(directory) and os.listdir(directory):
1319		- raise ElementError("Staging directory '{}' is not empty".format(directory))
1320		-
1321		- workspace = self._get_workspace()
1322		- if workspace:
1323		- # If mount_workspaces is set and we're doing incremental builds,
1324		- # the workspace is already mounted into the sandbox.
1325		- if not (mount_workspaces and self.__can_build_incrementally()):
1326		- with self.timed_activity("Staging local files at {}".format(workspace.path)):
1327		- workspace.stage(directory)
1328		- else:
1329		- # No workspace, stage directly
1330		- for source in self.sources():
1331		- source._stage(directory)
1332		-
	1320	+ if not isinstance(vdirectory, Directory):
	1321	+ vdirectory = FileBasedDirectory(vdirectory)
	1322	+ if not vdirectory.is_empty():
	1323	+ raise ElementError("Staging directory '{}' is not empty".format(vdirectory))
	1324	+
	1325	+ with tempfile.TemporaryDirectory() as temp_staging_directory:
	1326	+
	1327	+ workspace = self._get_workspace()
	1328	+ if workspace:
	1329	+ # If mount_workspaces is set and we're doing incremental builds,
	1330	+ # the workspace is already mounted into the sandbox.
	1331	+ if not (mount_workspaces and self.__can_build_incrementally()):
	1332	+ with self.timed_activity("Staging local files at {}".format(workspace.path)):
	1333	+ workspace.stage(temp_staging_directory)
	1334	+ else:
	1335	+ # No workspace, stage directly
	1336	+ for source in self.sources():
	1337	+ source._stage(temp_staging_directory)
	1338	+
	1339	+ vdirectory.import_files(temp_staging_directory)
1333	1340	# Ensure deterministic mtime of sources at build time
1334		- utils._set_deterministic_mtime(directory)
	1341	+ vdirectory.set_deterministic_mtime()
1335	1342	# Ensure deterministic owners of sources at build time
1336		- utils._set_deterministic_user(directory)
	1343	+ vdirectory.set_deterministic_user()
1337	1344
1338	1345	# _set_required():
1339	1346	#
...	...	@@ -1449,7 +1456,7 @@ class Element(Plugin):
1449	1456	with _signals.terminator(cleanup_rootdir), \
1450	1457	self.__sandbox(rootdir, output_file, output_file, self.__sandbox_config) as sandbox: # nopep8
1451	1458
1452		- sandbox_root = sandbox.get_directory()
	1459	+ sandbox_vroot = sandbox.get_virtual_directory()
1453	1460
1454	1461	# By default, the dynamic public data is the same as the static public data.
1455	1462	# The plugin's assemble() method may modify this, though.
...	...	@@ -1479,23 +1486,24 @@ class Element(Plugin):
1479	1486	#
1480	1487	workspace = self._get_workspace()
1481	1488	if workspace and self.__staged_sources_directory:
1482		- sandbox_root = sandbox.get_directory()
1483		- sandbox_path = os.path.join(sandbox_root,
1484		- self.__staged_sources_directory.lstrip(os.sep))
	1489	+ sandbox_vroot = sandbox.get_virtual_directory()
	1490	+ path_components = self.__staged_sources_directory.lstrip(os.sep).split(os.sep)
	1491	+ sandbox_vpath = sandbox_vroot.descend(path_components)
1485	1492	try:
1486		- utils.copy_files(workspace.path, sandbox_path)
	1493	+ sandbox_vpath.import_files(workspace.path)
1487	1494	except UtilError as e:
1488	1495	self.warn("Failed to preserve workspace state for failed build sysroot: {}"
1489	1496	.format(e))
1490	1497
1491	1498	raise
1492	1499
1493		- collectdir = os.path.join(sandbox_root, collect.lstrip(os.sep))
1494		- if not os.path.exists(collectdir):
	1500	+ try:
	1501	+ collectvdir = sandbox_vroot.descend(collect.lstrip(os.sep).split(os.sep))
	1502	+ except VirtualDirectoryError:
1495	1503	raise ElementError(
1496		- "Directory '{}' was not found inside the sandbox, "
	1504	+ "Subdirectory '{}' of '{}' does not exist following assembly, "
1497	1505	"unable to collect artifact contents"
1498		- .format(collect))
	1506	+ .format(collect, sandbox_vroot))
1499	1507
1500	1508	# At this point, we expect an exception was raised leading to
1501	1509	# an error message, or we have good output to collect.
...	...	@@ -1513,12 +1521,17 @@ class Element(Plugin):
1513	1521	os.mkdir(buildtreedir)
1514	1522
1515	1523	# Hard link files from collect dir to files directory
1516		- utils.link_files(collectdir, filesdir)
	1524	+ collectvdir.export_files(filesdir, can_link=True)
1517	1525
1518		- sandbox_build_dir = os.path.join(sandbox_root, self.get_variable('build-root').lstrip(os.sep))
1519		- # Hard link files from build-root dir to buildtreedir directory
1520		- if os.path.isdir(sandbox_build_dir):
1521		- utils.link_files(sandbox_build_dir, buildtreedir)
	1526	+ try:
	1527	+ # Attempt to hard link files from build-root dir to buildtreedir directory
	1528	+ build_root = self.get_variable('build-root').lstrip(os.sep)
	1529	+ sandbox_build_dir = sandbox_vroot.descend(build_root.split(os.sep))
	1530	+ sandbox_build_dir.export_files(buildtreedir, can_link=True, can_destroy=True)
	1531	+ except VirtualDirectoryError:
	1532	+ # This replaces code which previously did nothing if
	1533	+ # buildtreedir was not a directory, so we do the same.
	1534	+ pass
1522	1535
1523	1536	# Copy build log
1524	1537	log_filename = context.get_log_filename()
...	...	@@ -2043,7 +2056,8 @@ class Element(Plugin):
2043	2056	directory,
2044	2057	stdout=stdout,
2045	2058	stderr=stderr,
2046		- config=config)
	2059	+ config=config,
	2060	+ allow_real_directory=not self.BST_VIRTUAL_DIRECTORY)
2047	2061	yield sandbox
2048	2062
2049	2063	else:

buildstream/plugins/elements/compose.py

---

...	...	@@ -34,7 +34,6 @@ The default configuration and possible options are as such:
34	34	"""
35	35
36	36	import os
37		-from buildstream import utils
38	37	from buildstream import Element, Scope
39	38
40	39
...	...	@@ -56,6 +55,9 @@ class ComposeElement(Element):
56	55	# added, to reduce the potential for confusion
57	56	BST_FORBID_SOURCES = True
58	57
	58	+ # This plugin has been modified to avoid the use of Sandbox.get_directory
	59	+ BST_VIRTUAL_DIRECTORY = True
	60	+
59	61	def configure(self, node):
60	62	self.node_validate(node, [
61	63	'integrate', 'include', 'exclude', 'include-orphans'
...	...	@@ -104,7 +106,8 @@ class ComposeElement(Element):
104	106	orphans=self.include_orphans)
105	107	manifest.update(files)
106	108
107		- basedir = sandbox.get_directory()
	109	+ # Make a snapshot of all the files.
	110	+ vbasedir = sandbox.get_virtual_directory()
108	111	modified_files = set()
109	112	removed_files = set()
110	113	added_files = set()
...	...	@@ -116,38 +119,24 @@ class ComposeElement(Element):
116	119	if require_split:
117	120
118	121	# Make a snapshot of all the files before integration-commands are run.
119		- snapshot = {
120		- f: getmtime(os.path.join(basedir, f))
121		- for f in utils.list_relative_paths(basedir)
122		- }
	122	+ snapshot = set(vbasedir.list_relative_paths())
	123	+ vbasedir.mark_unmodified()
123	124
124	125	for dep in self.dependencies(Scope.BUILD):
125	126	dep.integrate(sandbox)
126	127
127	128	if require_split:
128		-
129	129	# Calculate added, modified and removed files
130		- basedir_contents = set(utils.list_relative_paths(basedir))
	130	+ post_integration_snapshot = vbasedir.list_relative_paths()
	131	+ modified_files = set(vbasedir.list_modified_paths())
	132	+ basedir_contents = set(post_integration_snapshot)
131	133	for path in manifest:
132		- if path in basedir_contents:
133		- if path in snapshot:
134		- preintegration_mtime = snapshot[path]
135		- if preintegration_mtime != getmtime(os.path.join(basedir, path)):
136		- modified_files.add(path)
137		- else:
138		- # If the path appears in the manifest but not the initial snapshot,
139		- # it may be a file staged inside a directory symlink. In this case
140		- # the path we got from the manifest won't show up in the snapshot
141		- # because utils.list_relative_paths() doesn't recurse into symlink
142		- # directories.
143		- pass
144		- elif path in snapshot:
	134	+ if path in snapshot and path not in basedir_contents:
145	135	removed_files.add(path)
146	136
147	137	for path in basedir_contents:
148	138	if path not in snapshot:
149	139	added_files.add(path)
150		-
151	140	self.info("Integration modified {}, added {} and removed {} files"
152	141	.format(len(modified_files), len(added_files), len(removed_files)))
153	142
...	...	@@ -166,8 +155,7 @@ class ComposeElement(Element):
166	155	# instead of into a subdir. The element assemble() method should
167	156	# support this in some way.
168	157	#
169		- installdir = os.path.join(basedir, 'buildstream', 'install')
170		- os.makedirs(installdir, exist_ok=True)
	158	+ installdir = vbasedir.descend(['buildstream', 'install'], create=True)
171	159
172	160	# We already saved the manifest for created files in the integration phase,
173	161	# now collect the rest of the manifest.
...	...	@@ -191,7 +179,7 @@ class ComposeElement(Element):
191	179
192	180	with self.timed_activity("Creating composition", detail=detail, silent_nested=True):
193	181	self.info("Composing {} files".format(len(manifest)))
194		- utils.link_files(basedir, installdir, files=manifest)
	182	+ installdir.import_files(vbasedir, files=manifest, can_link=True)
195	183
196	184	# And we're done
197	185	return os.path.join(os.sep, 'buildstream', 'install')

buildstream/plugins/elements/import.py

---

...	...	@@ -31,7 +31,6 @@ The empty configuration is as such:
31	31	"""
32	32
33	33	import os
34		-import shutil
35	34	from buildstream import Element, BuildElement, ElementError
36	35
37	36
...	...	@@ -68,27 +67,22 @@ class ImportElement(BuildElement):
68	67	# Do not mount workspaces as the files are copied from outside the sandbox
69	68	self._stage_sources_in_sandbox(sandbox, 'input', mount_workspaces=False)
70	69
71		- rootdir = sandbox.get_directory()
72		- inputdir = os.path.join(rootdir, 'input')
73		- outputdir = os.path.join(rootdir, 'output')
	70	+ rootdir = sandbox.get_virtual_directory()
	71	+ inputdir = rootdir.descend(['input'])
	72	+ outputdir = rootdir.descend(['output'], create=True)
74	73
75	74	# The directory to grab
76		- inputdir = os.path.join(inputdir, self.source.lstrip(os.sep))
77		- inputdir = inputdir.rstrip(os.sep)
	75	+ inputdir = inputdir.descend(self.source.strip(os.sep).split(os.sep))
78	76
79	77	# The output target directory
80		- outputdir = os.path.join(outputdir, self.target.lstrip(os.sep))
81		- outputdir = outputdir.rstrip(os.sep)
82		-
83		- # Ensure target directory parent
84		- os.makedirs(os.path.dirname(outputdir), exist_ok=True)
	78	+ outputdir = outputdir.descend(self.target.strip(os.sep).split(os.sep), create=True)
85	79
86		- if not os.path.exists(inputdir):
	80	+ if inputdir.is_empty():
87	81	raise ElementError("{}: No files were found inside directory '{}'"
88	82	.format(self, self.source))
89	83
90	84	# Move it over
91		- shutil.move(inputdir, outputdir)
	85	+ outputdir.import_files(inputdir)
92	86
93	87	# And we're done
94	88	return '/output'

buildstream/plugins/elements/stack.py

---

...	...	@@ -24,7 +24,6 @@ Stack elements are simply a symbolic element used for representing
24	24	a logical group of elements.
25	25	"""
26	26
27		-import os
28	27	from buildstream import Element
29	28
30	29
...	...	@@ -52,7 +51,7 @@ class StackElement(Element):
52	51
53	52	# Just create a dummy empty artifact, its existence is a statement
54	53	# that all this stack's dependencies are built.
55		- rootdir = sandbox.get_directory()
	54	+ vrootdir = sandbox.get_virtual_directory()
56	55
57	56	# XXX FIXME: This is currently needed because the artifact
58	57	# cache wont let us commit an empty artifact.
...	...	@@ -61,10 +60,7 @@ class StackElement(Element):
61	60	# the actual artifact data in a subdirectory, then we
62	61	# will be able to store some additional state in the
63	62	# artifact cache, and we can also remove this hack.
64		- outputdir = os.path.join(rootdir, 'output', 'bst')
65		-
66		- # Ensure target directory parent
67		- os.makedirs(os.path.dirname(outputdir), exist_ok=True)
	63	+ vrootdir.descend(['output', 'bst'], create=True)
68	64
69	65	# And we're done
70	66	return '/output'

buildstream/sandbox/_mount.py

---

...	...	@@ -32,7 +32,8 @@ from .._fuse import SafeHardlinks
32	32	class Mount():
33	33	def __init__(self, sandbox, mount_point, safe_hardlinks):
34	34	scratch_directory = sandbox._get_scratch_directory()
35		- root_directory = sandbox.get_directory()
	35	+ # Getting external_directory here is acceptable as we're part of the sandbox code.
	36	+ root_directory = sandbox.get_virtual_directory().external_directory
36	37
37	38	self.mount_point = mount_point
38	39	self.safe_hardlinks = safe_hardlinks

buildstream/sandbox/_sandboxbwrap.py

---

...	...	@@ -56,7 +56,9 @@ class SandboxBwrap(Sandbox):
56	56
57	57	def run(self, command, flags, *, cwd=None, env=None):
58	58	stdout, stderr = self._get_output()
59		- root_directory = self.get_directory()
	59	+
	60	+ # Allowable access to underlying storage as we're part of the sandbox
	61	+ root_directory = self.get_virtual_directory().external_directory
60	62
61	63	# Fallback to the sandbox default settings for
62	64	# the cwd and env.

buildstream/sandbox/_sandboxchroot.py

---

...	...	@@ -90,7 +90,7 @@ class SandboxChroot(Sandbox):
90	90	# Nonetheless a better solution could perhaps be found.
91	91
92	92	rootfs = stack.enter_context(utils._tempdir(dir='/var/run/buildstream'))
93		- stack.enter_context(self.create_devices(self.get_directory(), flags))
	93	+ stack.enter_context(self.create_devices(self._root, flags))
94	94	stack.enter_context(self.mount_dirs(rootfs, flags, stdout, stderr))
95	95
96	96	if flags & SandboxFlags.INTERACTIVE:

buildstream/sandbox/sandbox.py

---

...	...	@@ -29,7 +29,8 @@ See also: :ref:`sandboxing`.
29	29	"""
30	30
31	31	import os
32		-from .._exceptions import ImplError
	32	+from .._exceptions import ImplError, BstError
	33	+from ..storage._filebaseddirectory import FileBasedDirectory
33	34
34	35
35	36	class SandboxFlags():
...	...	@@ -90,28 +91,63 @@ class Sandbox():
90	91	self.__cwd = None
91	92	self.__env = None
92	93	self.__mount_sources = {}
	94	+ self.__allow_real_directory = kwargs['allow_real_directory']
	95	+
93	96	# Configuration from kwargs common to all subclasses
94	97	self.__config = kwargs['config']
95	98	self.__stdout = kwargs['stdout']
96	99	self.__stderr = kwargs['stderr']
97	100
98		- # Setup the directories
	101	+ # Setup the directories. Root should be available to subclasses, hence
	102	+ # being single-underscore. The others are private to this class.
	103	+ self._root = os.path.join(directory, 'root')
99	104	self.__directory = directory
100		- self.__root = os.path.join(self.__directory, 'root')
101	105	self.__scratch = os.path.join(self.__directory, 'scratch')
102		- for directory_ in [self.__root, self.__scratch]:
	106	+ for directory_ in [self._root, self.__scratch]:
103	107	os.makedirs(directory_, exist_ok=True)
104	108
105	109	def get_directory(self):
106	110	"""Fetches the sandbox root directory
107	111
108	112	The root directory is where artifacts for the base
109		- runtime environment should be staged.
	113	+ runtime environment should be staged. Only works if
	114	+ BST_VIRTUAL_DIRECTORY is not set.
110	115
111	116	Returns:
112	117	(str): The sandbox root directory
	118	+
	119	+ """
	120	+ if self.__allow_real_directory:
	121	+ return self._root
	122	+ else:
	123	+ raise BstError("You can't use get_directory")
	124	+
	125	+ def get_virtual_directory(self):
	126	+ """Fetches the sandbox root directory
	127	+
	128	+ The root directory is where artifacts for the base
	129	+ runtime environment should be staged. Only works if
	130	+ BST_VIRTUAL_DIRECTORY is not set.
	131	+
	132	+ Returns:
	133	+ (str): The sandbox root directory
	134	+
	135	+ """
	136	+ # For now, just create a new Directory every time we're asked
	137	+ return FileBasedDirectory(self._root)
	138	+
	139	+ def get_virtual_toplevel_directory(self):
	140	+ """Fetches the sandbox's toplevel directory
	141	+
	142	+ The toplevel directory contains 'root', 'scratch' and later
	143	+ 'artifact' where output is copied to.
	144	+
	145	+ Returns:
	146	+ (str): The sandbox toplevel directory
	147	+
113	148	"""
114		- return self.__root
	149	+ # For now, just create a new Directory every time we're asked
	150	+ return FileBasedDirectory(self.__directory)
115	151
116	152	def set_environment(self, environment):
117	153	"""Sets the environment variables for the sandbox

buildstream/scriptelement.py

---

...	...	@@ -243,9 +243,8 @@ class ScriptElement(Element):
243	243	with self.timed_activity("Staging {} at {}"
244	244	.format(element.name, item['destination']),
245	245	silent_nested=True):
246		- real_dstdir = os.path.join(sandbox.get_directory(),
247		- item['destination'].lstrip(os.sep))
248		- os.makedirs(os.path.dirname(real_dstdir), exist_ok=True)
	246	+ virtual_dstdir = sandbox.get_virtual_directory()
	247	+ virtual_dstdir.descend(item['destination'].lstrip(os.sep).split(os.sep), create=True)
249	248	element.stage_dependency_artifacts(sandbox, Scope.RUN, path=item['destination'])
250	249
251	250	for item in self.__layout:
...	...	@@ -263,8 +262,8 @@ class ScriptElement(Element):
263	262	for dep in element.dependencies(Scope.RUN):
264	263	dep.integrate(sandbox)
265	264
266		- os.makedirs(os.path.join(sandbox.get_directory(), self.__install_root.lstrip(os.sep)),
267		- exist_ok=True)
	265	+ install_root_path_components = self.__install_root.lstrip(os.sep).split(os.sep)
	266	+ sandbox.get_virtual_directory().descend(install_root_path_components, create=True)
268	267
269	268	def assemble(self, sandbox):
270	269

buildstream/storage/__init__.py

---

	1	+#!/usr/bin/env python3
	2	+#
	3	+# Copyright (C) 2017 Codethink Limited
	4	+#
	5	+# This program is free software; you can redistribute it and/or
	6	+# modify it under the terms of the GNU Lesser General Public
	7	+# License as published by the Free Software Foundation; either
	8	+# version 2 of the License, or (at your option) any later version.
	9	+#
	10	+# This library is distributed in the hope that it will be useful,
	11	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	12	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	13	+# Lesser General Public License for more details.
	14	+#
	15	+# You should have received a copy of the GNU Lesser General Public
	16	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	17	+#
	18	+# Authors:
	19	+# Jim MacArthur <jim macarthur codethink co uk>
	20	+
	21	+from ._filebaseddirectory import FileBasedDirectory
	22	+from .directory import Directory

buildstream/storage/_filebaseddirectory.py

---

	1	+#!/usr/bin/env python3
	2	+#
	3	+# Copyright (C) 2018 Codethink Limited
	4	+#
	5	+# This program is free software; you can redistribute it and/or
	6	+# modify it under the terms of the GNU Lesser General Public
	7	+# License as published by the Free Software Foundation; either
	8	+# version 2 of the License, or (at your option) any later version.
	9	+#
	10	+# This library is distributed in the hope that it will be useful,
	11	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	12	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	13	+# Lesser General Public License for more details.
	14	+#
	15	+# You should have received a copy of the GNU Lesser General Public
	16	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	17	+#
	18	+# Authors:
	19	+# Jim MacArthur <jim macarthur codethink co uk>
	20	+
	21	+"""
	22	+FileBasedDirectory
	23	+=========
	24	+
	25	+Implementation of the Directory class which backs onto a normal POSIX filing system.
	26	+
	27	+See also: :ref:`sandboxing`.
	28	+"""
	29	+
	30	+from collections import OrderedDict
	31	+
	32	+import calendar
	33	+import os
	34	+import time
	35	+from .._exceptions import BstError, ErrorDomain
	36	+from .directory import Directory
	37	+from ..utils import link_files, copy_files, list_relative_paths
	38	+from ..utils import _set_deterministic_user, _set_deterministic_mtime
	39	+
	40	+
	41	+class VirtualDirectoryError(BstError):
	42	+ """Raised by Directory functions when system calls fail.
	43	+ This will be handled internally by the BuildStream core,
	44	+ if you need to handle this error, then it should be reraised,
	45	+ or either of the :class:`.ElementError` or :class:`.SourceError`
	46	+ exceptions should be raised from this error.
	47	+ """
	48	+ def __init__(self, message, reason=None):
	49	+ super().__init__(message, domain=ErrorDomain.VIRTUAL_FS, reason=reason)
	50	+
	51	+
	52	+# Like os.path.getmtime(), but doesnt explode on symlinks
	53	+# Copy/pasted from compose.py
	54	+def getmtime(path):
	55	+ stat = os.lstat(path)
	56	+ return stat.st_mtime
	57	+
	58	+# FileBasedDirectory intentionally doesn't call its superclass constuctor,
	59	+# which is mean to be unimplemented.
	60	+# pylint: disable=super-init-not-called
	61	+
	62	+
	63	+class _FileObject():
	64	+ """A description of a file in a virtual directory. The contents of
	65	+ this class are never used, but there needs to be something present
	66	+ for files so is_empty() works correctly.
	67	+
	68	+ """
	69	+ def __init__(self, virtual_directory: Directory, filename: str):
	70	+ self.directory = virtual_directory
	71	+ self.filename = filename
	72	+
	73	+
	74	+class FileBasedDirectory(Directory):
	75	+ def __init__(self, external_directory=None):
	76	+ self.external_directory = external_directory
	77	+ self.index = OrderedDict()
	78	+ self._directory_read = False
	79	+
	80	+ def _populate_index(self):
	81	+ if self._directory_read:
	82	+ return
	83	+ for entry in os.listdir(self.external_directory):
	84	+ if os.path.isdir(os.path.join(self.external_directory, entry)):
	85	+ self.index[entry] = FileBasedDirectory(os.path.join(self.external_directory, entry))
	86	+ else:
	87	+ self.index[entry] = _FileObject(self, entry)
	88	+ self._directory_read = True
	89	+
	90	+ def descend(self, subdirectory_spec, create=False):
	91	+ """ See superclass Directory for arguments """
	92	+ # It's very common to send a directory name instead of a list and this causes
	93	+ # bizarre errors, so check for it here
	94	+ if not isinstance(subdirectory_spec, list):
	95	+ subdirectory_spec = [subdirectory_spec]
	96	+ if not subdirectory_spec:
	97	+ return self
	98	+
	99	+ # Because of the way split works, it's common to get a list which begins with
	100	+ # an empty string. Detect these and remove them, then start again.
	101	+ if subdirectory_spec[0] == "":
	102	+ return self.descend(subdirectory_spec[1:], create)
	103	+
	104	+ self._populate_index()
	105	+ if subdirectory_spec[0] in self.index:
	106	+ entry = self.index[subdirectory_spec[0]]
	107	+ if isinstance(entry, FileBasedDirectory):
	108	+ new_path = os.path.join(self.external_directory, subdirectory_spec[0])
	109	+ return FileBasedDirectory(new_path).descend(subdirectory_spec[1:], create)
	110	+ else:
	111	+ error = "Cannot descend into {}, which is a '{}' in the directory {}"
	112	+ raise VirtualDirectoryError(error.format(subdirectory_spec[0],
	113	+ type(entry).__name__,
	114	+ self.external_directory))
	115	+ else:
	116	+ if create:
	117	+ new_path = os.path.join(self.external_directory, subdirectory_spec[0])
	118	+ os.makedirs(new_path, exist_ok=True)
	119	+ return FileBasedDirectory(new_path).descend(subdirectory_spec[1:], create)
	120	+ else:
	121	+ error = "No entry called '{}' found in the directory rooted at {}"
	122	+ raise VirtualDirectoryError(error.format(subdirectory_spec[0], self.external_directory))
	123	+
	124	+ def import_files(self, external_pathspec, *, files=None,
	125	+ report_written=True, update_utimes=False,
	126	+ can_link=False):
	127	+ """ See superclass Directory for arguments """
	128	+
	129	+ if isinstance(external_pathspec, Directory):
	130	+ source_directory = external_pathspec.external_directory
	131	+ else:
	132	+ source_directory = external_pathspec
	133	+
	134	+ if can_link and not update_utimes:
	135	+ import_result = link_files(source_directory, self.external_directory, files=files,
	136	+ ignore_missing=False, report_written=report_written)
	137	+ else:
	138	+ import_result = copy_files(source_directory, self.external_directory, files=files,
	139	+ ignore_missing=False, report_written=report_written)
	140	+ if update_utimes:
	141	+ cur_time = time.time()
	142	+
	143	+ for f in import_result.files_written:
	144	+ os.utime(os.path.join(self.external_directory, f), times=(cur_time, cur_time))
	145	+ return import_result
	146	+
	147	+ def set_deterministic_mtime(self):
	148	+ _set_deterministic_mtime(self.external_directory)
	149	+
	150	+ def set_deterministic_user(self):
	151	+ _set_deterministic_user(self.external_directory)
	152	+
	153	+ def export_files(self, to_directory, *, can_link=False, can_destroy=False):
	154	+ if can_destroy:
	155	+ # Try a simple rename of the sandbox root; if that
	156	+ # doesnt cut it, then do the regular link files code path
	157	+ try:
	158	+ os.rename(self.external_directory, to_directory)
	159	+ return
	160	+ except OSError:
	161	+ # Proceed using normal link/copy
	162	+ pass
	163	+
	164	+ os.makedirs(to_directory, exist_ok=True)
	165	+ if can_link:
	166	+ link_files(self.external_directory, to_directory)
	167	+ else:
	168	+ copy_files(self.external_directory, to_directory)
	169	+
	170	+ # Add a directory entry deterministically to a tar file
	171	+ #
	172	+ # This function takes extra steps to ensure the output is deterministic.
	173	+ # First, it sorts the results of os.listdir() to ensure the ordering of
	174	+ # the files in the archive is the same. Second, it sets a fixed
	175	+ # timestamp for each entry. See also https://bugs.python.org/issue24465.
	176	+ def export_to_tar(self, tf, dir_arcname, mtime=0):
	177	+ # We need directories here, including non-empty ones,
	178	+ # so list_relative_paths is not used.
	179	+ for filename in sorted(os.listdir(self.external_directory)):
	180	+ source_name = os.path.join(self.external_directory, filename)
	181	+ arcname = os.path.join(dir_arcname, filename)
	182	+ tarinfo = tf.gettarinfo(source_name, arcname)
	183	+ tarinfo.mtime = mtime
	184	+
	185	+ if tarinfo.isreg():
	186	+ with open(source_name, "rb") as f:
	187	+ tf.addfile(tarinfo, f)
	188	+ elif tarinfo.isdir():
	189	+ tf.addfile(tarinfo)
	190	+ self.descend(filename.split(os.path.sep)).export_to_tar(tf, arcname, mtime)
	191	+ else:
	192	+ tf.addfile(tarinfo)
	193	+
	194	+ def is_empty(self):
	195	+ self._populate_index()
	196	+ return len(self.index) == 0
	197	+
	198	+ def mark_unmodified(self):
	199	+ """ Marks all files in this directory (recursively) as unmodified.
	200	+ """
	201	+ _set_deterministic_mtime(self.external_directory)
	202	+
	203	+ def list_modified_paths(self):
	204	+ """Provide a list of relative paths which have been modified since the
	205	+ last call to mark_unmodified.
	206	+
	207	+ Return value: List(str) - list of modified paths
	208	+ """
	209	+ magic_timestamp = calendar.timegm([2011, 11, 11, 11, 11, 11])
	210	+
	211	+ return [f for f in list_relative_paths(self.external_directory)
	212	+ if getmtime(os.path.join(self.external_directory, f)) != magic_timestamp]
	213	+
	214	+ def list_relative_paths(self):
	215	+ """Provide a list of all relative paths.
	216	+
	217	+ Return value: List(str) - list of all paths
	218	+ """
	219	+
	220	+ return list_relative_paths(self.external_directory)
	221	+
	222	+ def __str__(self):
	223	+ # This returns the whole path (since we don't know where the directory started)
	224	+ # which exposes the sandbox directory; we will have to assume for the time being
	225	+ # that people will not abuse __str__.
	226	+ return self.external_directory

buildstream/storage/directory.py

---

	1	+#!/usr/bin/env python3
	2	+#
	3	+# Copyright (C) 2018 Codethink Limited
	4	+#
	5	+# This program is free software; you can redistribute it and/or
	6	+# modify it under the terms of the GNU Lesser General Public
	7	+# License as published by the Free Software Foundation; either
	8	+# version 2 of the License, or (at your option) any later version.
	9	+#
	10	+# This library is distributed in the hope that it will be useful,
	11	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	12	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	13	+# Lesser General Public License for more details.
	14	+#
	15	+# You should have received a copy of the GNU Lesser General Public
	16	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	17	+#
	18	+# Authors:
	19	+# Jim MacArthur <jim macarthur codethink co uk>
	20	+
	21	+"""
	22	+Directory
	23	+=========
	24	+
	25	+This is a virtual Directory class to isolate the rest of BuildStream
	26	+from the backing store implementation. Sandboxes are allowed to read
	27	+from and write to the underlying storage, but all others must use this
	28	+Directory class to access files and directories in the sandbox.
	29	+
	30	+See also: :ref:`sandboxing`.
	31	+
	32	+"""
	33	+
	34	+
	35	+class Directory():
	36	+ def __init__(self, external_directory=None):
	37	+ raise NotImplementedError()
	38	+
	39	+ def descend(self, subdirectory_spec, create=False):
	40	+ """Descend one or more levels of directory hierarchy and return a new
	41	+ Directory object for that directory.
	42	+
	43	+ Args:
	44	+ subdirectory_spec (list of str): A list of strings which are all directory
	45	+ names.
	46	+ create (boolean): If this is true, the directories will be created if
	47	+ they don't already exist.
	48	+
	49	+ Yields:
	50	+ A Directory object representing the found directory.
	51	+
	52	+ Raises:
	53	+ VirtualDirectoryError: if any of the components in subdirectory_spec
	54	+ cannot be found, or are files, or symlinks to files.
	55	+
	56	+ """
	57	+ raise NotImplementedError()
	58	+
	59	+ # Import and export of files and links
	60	+ def import_files(self, external_pathspec, *, files=None,
	61	+ report_written=True, update_utimes=False,
	62	+ can_link=False):
	63	+ """Imports some or all files from external_path into this directory.
	64	+
	65	+ Args:
	66	+ external_pathspec: Either a string containing a pathname, or a
	67	+ Directory object, to use as the source.
	68	+ files (list of str): A list of all the files relative to
	69	+ the external_pathspec to copy. If 'None' is supplied, all
	70	+ files are copied.
	71	+ report_written (bool): Return the full list of files
	72	+ written. Defaults to true. If false, only a list of
	73	+ overwritten files is returned.
	74	+ update_utimes (bool): Update the access and modification time
	75	+ of each file copied to the current time.
	76	+ can_link (bool): Whether it's OK to create a hard link to the
	77	+ original content, meaning the stored copy will change when the
	78	+ original files change. Setting this doesn't guarantee hard
	79	+ links will be made. can_link will never be used if
	80	+ update_utimes is set.
	81	+
	82	+ Yields:
	83	+ (FileListResult) - A report of files imported and overwritten.
	84	+
	85	+ """
	86	+
	87	+ raise NotImplementedError()
	88	+
	89	+ def export_files(self, to_directory, *, can_link=False, can_destroy=False):
	90	+ """Copies everything from this into to_directory.
	91	+
	92	+ Args:
	93	+ to_directory (string): a path outside this directory object
	94	+ where the contents will be copied to.
	95	+ can_link (bool): Whether we can create hard links in to_directory
	96	+ instead of copying. Setting this does not guarantee hard links will be used.
	97	+ can_destroy (bool): Can we destroy the data already in this
	98	+ directory when exporting? If set, this may allow data to be
	99	+ moved rather than copied which will be quicker.
	100	+ """
	101	+
	102	+ raise NotImplementedError()
	103	+
	104	+ def export_to_tar(self, tarfile, destination_dir, mtime=0):
	105	+ """ Exports this directory into the given tar file.
	106	+
	107	+ Args:
	108	+ tarfile (TarFile): A Python TarFile object to export into.
	109	+ destination_dir (str): The prefix for all filenames inside the archive.
	110	+ mtime (int): mtimes of all files in the archive are set to this.
	111	+ """
	112	+ raise NotImplementedError()
	113	+
	114	+ # Convenience functions
	115	+ def is_empty(self):
	116	+ """ Return true if this directory has no files, subdirectories or links in it.
	117	+ """
	118	+ raise NotImplementedError()
	119	+
	120	+ def set_deterministic_mtime(self):
	121	+ """ Sets a static modification time for all regular files in this directory.
	122	+ The magic number for timestamps is 2011-11-11 11:11:11.
	123	+ """
	124	+ raise NotImplementedError()
	125	+
	126	+ def set_deterministic_user(self):
	127	+ """ Sets all files in this directory to the current user's euid/egid.
	128	+ """
	129	+ raise NotImplementedError()
	130	+
	131	+ def mark_unmodified(self):
	132	+ """ Marks all files in this directory (recursively) as unmodified.
	133	+ """
	134	+ raise NotImplementedError()
	135	+
	136	+ def list_modified_paths(self):
	137	+ """Provide a list of relative paths which have been modified since the
	138	+ last call to mark_unmodified. Includes directories only if
	139	+ they are empty.
	140	+
	141	+ Yields:
	142	+ (List(str)) - list of all modified files with relative paths.
	143	+
	144	+ """
	145	+ raise NotImplementedError()
	146	+
	147	+ def list_relative_paths(self):
	148	+ """Provide a list of all relative paths in this directory. Includes
	149	+ directories only if they are empty.
	150	+
	151	+ Yields:
	152	+ (List(str)) - list of all files with relative paths.
	153	+
	154	+ """
	155	+ raise NotImplementedError()

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.