[Notes] [Git][BuildStream/buildstream][valentindavid/ruamel_0_15_41] 26 commits: tests/integration/manual.py: Add test for command logging

[Valentin David <gitlab mg gitlab com>]

Title: GitLab

Valentin David pushed to branch valentindavid/ruamel_0_15_41 at BuildStream / buildstream

Commits:

• 223c0073
  by Jürg Billeter at 2018-11-27T13:41:09Z

  tests/integration/manual.py: Add test for command logging
  

• 38d16c62
  by Jürg Billeter at 2018-11-27T13:41:09Z

  sandbox: Deduplicate code to process cwd, env, and command arguments
  

• 85f5b574
  by Jürg Billeter at 2018-11-27T13:41:09Z

  sandbox/sandbox.py: Add SandboxFlags.NONE
  
  This improves readability.
  

• bdd6925f
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Use SandboxFlags.NONE instead of 0
  

• 69005c76
  by Jürg Billeter at 2018-11-27T13:41:09Z

  scriptelement.py: Use SandboxFlags.NONE instead of 0
  

• f1767de2
  by Jürg Billeter at 2018-11-27T13:41:09Z

  sandbox/sandbox.py: Add command batching API
  
  This adds the batch() context manager.
  

• 87f7c99a
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Pass element object to sandbox
  
  This allows the sandbox to output messages using the plugin id.
  

• 024d584d
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Cache artifact also on SandboxCommandError
  

• bb7c3213
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Defer workspace.prepared setting in case of pending commands
  

• 04f05b1a
  by Jürg Billeter at 2018-11-27T13:41:09Z

  element.py: Support command batching for integration commands
  

• 7827ac50
  by Jürg Billeter at 2018-11-27T13:44:49Z

  element.py: Add batch_prepare_assemble() method
  
  This allows batching of all sandbox commands aross prepare() and
  assemble().
  

• 63db90b2
  by Jürg Billeter at 2018-11-27T13:44:49Z

  buildelement.py: Support batching for integration and build commands
  

• 73dec86e
  by Jürg Billeter at 2018-11-27T13:44:49Z

  scriptelement.py: Support batching for integration and build commands
  

• 79442e0a
  by Jürg Billeter at 2018-11-27T13:44:49Z

  compose.py: Support batching for integration commands
  

• 6018b3bd
  by Jürg Billeter at 2018-11-27T13:44:49Z

  sandbox/_sandboxremote.py: Implement command batching
  
  Execute batched commands in a single shell script.
  

• ec3d4486
  by Jürg Billeter at 2018-11-27T13:44:49Z

  NEWS: Add entry for command batching
  

• 127d332f
  by Jürg Billeter at 2018-11-27T14:16:50Z

  Merge branch 'juerg/command-batching' into 'master'
  
  Command batching
  
  Closes #675
  
  See merge request BuildStream/buildstream!915

• d5b5538d
  by Jim MacArthur at 2018-11-27T15:32:19Z

  artifactcache: Move ArtifactCacheSpec code into CASRemoteSpec.
  
  There is nothing in ArtifactCacheSpec that's actually specific to
  artifacts, so I've made it a CAS class so we can use the same
  spec for remote execution.
  

• f224d4cb
  by Jim MacArthur at 2018-11-27T15:32:19Z

  artifactcache.py, _context.py: Move CASCache object into context
  
  Since the artifact cache and remote execution share the same
  local CAS store, they should share the same CASCache object. Moving
  this into context allows us to do this.
  

• 926c1446
  by Jim MacArthur at 2018-11-27T15:33:34Z

  _sandboxremote: Remove references to artifact cache.
  
  Uses the new config options introduced earlier to specify remote
  exec storage and execution service.
  

• d7cdb03e
  by Jim MacArthur at 2018-11-27T15:33:36Z

  Documentation: Update to show new config options
  

• 95d8244f
  by Jim MacArthur at 2018-11-27T15:33:36Z

  tests/sandboxes/remote-exec-config.py: New test.
  
  Tests that we get a sensible error message when we specify incomplete
  or wrong information in the configuration for remote execution.
  

• 7f2f62eb
  by Jim MacArthur at 2018-11-27T15:33:36Z

  artifactcache.py: Remove unused verify_digest_pushed
  

• c55ca3a5
  by Jim MacArthur at 2018-11-27T15:33:36Z

  Docs: Correct URLs in artifact cache example
  
  The port number must come after the hostname and before the path.
  The path is ignored by CASCache anyway, so including it is misleading.
  

• 2a8a3b19
  by Jim MacArthur at 2018-11-27T18:17:03Z

  Merge branch 'jmac/remote_execution_split' into 'master'
  
  Split remote execution from artifact cache
  
  Closes #750
  
  See merge request BuildStream/buildstream!946

• 499c70fd
  by Valentin David at 2018-11-28T09:11:21Z

  setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52
  

25 changed files:

• NEWS
• buildstream/__init__.py
• buildstream/_artifactcache/artifactcache.py
• buildstream/_artifactcache/cascache.py
• buildstream/_context.py
• buildstream/_project.py
• buildstream/buildelement.py
• buildstream/data/projectconfig.yaml
• buildstream/element.py
• buildstream/plugins/elements/compose.py
• buildstream/sandbox/__init__.py
• buildstream/sandbox/_sandboxbwrap.py
• buildstream/sandbox/_sandboxchroot.py
• buildstream/sandbox/_sandboxdummy.py
• buildstream/sandbox/_sandboxremote.py
• buildstream/sandbox/sandbox.py
• buildstream/scriptelement.py
• doc/source/format_project.rst
• setup.py
• tests/integration/manual.py
• tests/integration/sandbox-bwrap.py
• + tests/sandboxes/remote-exec-config.py
• + tests/sandboxes/remote-exec-config/missing-certs/certificates/client.crt
• + tests/sandboxes/remote-exec-config/missing-certs/certificates/client.key
• + tests/sandboxes/remote-exec-config/missing-certs/element.bst

Changes:

NEWS

---

...	...	@@ -67,6 +67,9 @@ buildstream 1.3.1
67	67	allows the user to set a default location for their creation. This has meant
68	68	that the new CLI is no longer backwards compatible with buildstream 1.2.
69	69
	70	+ o Add sandbox API for command batching and use it for build, script, and
	71	+ compose elements.
	72	+
70	73
71	74	=================
72	75	buildstream 1.1.5

buildstream/__init__.py

---

...	...	@@ -27,7 +27,7 @@ if "_BST_COMPLETION" not in os.environ:
27	27	del get_versions
28	28
29	29	from .utils import UtilError, ProgramNotFoundError
30		- from .sandbox import Sandbox, SandboxFlags
	30	+ from .sandbox import Sandbox, SandboxFlags, SandboxCommandError
31	31	from .types import Scope, Consistency
32	32	from .plugin import Plugin
33	33	from .source import Source, SourceError, SourceFetcher

buildstream/_artifactcache/artifactcache.py

---

...	...	@@ -21,7 +21,6 @@ import multiprocessing
21	21	import os
22	22	import signal
23	23	import string
24		-from collections import namedtuple
25	24	from collections.abc import Mapping
26	25
27	26	from ..types import _KeyStrength
...	...	@@ -31,7 +30,7 @@ from .. import _signals
31	30	from .. import utils
32	31	from .. import _yaml
33	32
34		-from .cascache import CASCache, CASRemote
	33	+from .cascache import CASRemote, CASRemoteSpec
35	34
36	35
37	36	CACHE_SIZE_FILE = "cache_size"
...	...	@@ -45,48 +44,8 @@ CACHE_SIZE_FILE = "cache_size"
45	44	# push (bool): Whether we should attempt to push artifacts to this cache,
46	45	# in addition to pulling from it.
47	46	#
48		-class ArtifactCacheSpec(namedtuple('ArtifactCacheSpec', 'url push server_cert client_key client_cert')):
49		-
50		- # _new_from_config_node
51		- #
52		- # Creates an ArtifactCacheSpec() from a YAML loaded node
53		- #
54		- @staticmethod
55		- def _new_from_config_node(spec_node, basedir=None):
56		- _yaml.node_validate(spec_node, ['url', 'push', 'server-cert', 'client-key', 'client-cert'])
57		- url = _yaml.node_get(spec_node, str, 'url')
58		- push = _yaml.node_get(spec_node, bool, 'push', default_value=False)
59		- if not url:
60		- provenance = _yaml.node_get_provenance(spec_node, 'url')
61		- raise LoadError(LoadErrorReason.INVALID_DATA,
62		- "{}: empty artifact cache URL".format(provenance))
63		-
64		- server_cert = _yaml.node_get(spec_node, str, 'server-cert', default_value=None)
65		- if server_cert and basedir:
66		- server_cert = os.path.join(basedir, server_cert)
67		-
68		- client_key = _yaml.node_get(spec_node, str, 'client-key', default_value=None)
69		- if client_key and basedir:
70		- client_key = os.path.join(basedir, client_key)
71		-
72		- client_cert = _yaml.node_get(spec_node, str, 'client-cert', default_value=None)
73		- if client_cert and basedir:
74		- client_cert = os.path.join(basedir, client_cert)
75		-
76		- if client_key and not client_cert:
77		- provenance = _yaml.node_get_provenance(spec_node, 'client-key')
78		- raise LoadError(LoadErrorReason.INVALID_DATA,
79		- "{}: 'client-key' was specified without 'client-cert'".format(provenance))
80		-
81		- if client_cert and not client_key:
82		- provenance = _yaml.node_get_provenance(spec_node, 'client-cert')
83		- raise LoadError(LoadErrorReason.INVALID_DATA,
84		- "{}: 'client-cert' was specified without 'client-key'".format(provenance))
85		-
86		- return ArtifactCacheSpec(url, push, server_cert, client_key, client_cert)
87		-
88		-
89		-ArtifactCacheSpec.__new__.__defaults__ = (None, None, None)
	47	+class ArtifactCacheSpec(CASRemoteSpec):
	48	+ pass
90	49
91	50
92	51	# An ArtifactCache manages artifacts.
...	...	@@ -99,7 +58,7 @@ class ArtifactCache():
99	58	self.context = context
100	59	self.extractdir = os.path.join(context.artifactdir, 'extract')
101	60
102		- self.cas = CASCache(context.artifactdir)
	61	+ self.cas = context.get_cascache()
103	62
104	63	self.global_remote_specs = []
105	64	self.project_remote_specs = {}
...	...	@@ -792,34 +751,6 @@ class ArtifactCache():
792	751
793	752	return message_digest
794	753
795		- # verify_digest_pushed():
796		- #
797		- # Check whether the object is already on the server in which case
798		- # there is no need to upload it.
799		- #
800		- # Args:
801		- # project (Project): The current project
802		- # digest (Digest): The object digest.
803		- #
804		- def verify_digest_pushed(self, project, digest):
805		-
806		- if self._has_push_remotes:
807		- push_remotes = [r for r in self._remotes[project] if r.spec.push]
808		- else:
809		- push_remotes = []
810		-
811		- if not push_remotes:
812		- raise ArtifactError("verify_digest_pushed was called, but no remote artifact " +
813		- "servers are configured as push remotes.")
814		-
815		- pushed = False
816		-
817		- for remote in push_remotes:
818		- if self.cas.verify_digest_on_remote(remote, digest):
819		- pushed = True
820		-
821		- return pushed
822		-
823	754	# link_key():
824	755	#
825	756	# Add a key for an existing artifact.

buildstream/_artifactcache/cascache.py

---

...	...	@@ -17,6 +17,7 @@
17	17	# Authors:
18	18	# Jürg Billeter <juerg billeter codethink co uk>
19	19
	20	+from collections import namedtuple
20	21	import hashlib
21	22	import itertools
22	23	import io
...	...	@@ -34,7 +35,8 @@ from .._protos.build.bazel.remote.execution.v2 import remote_execution_pb2, remo
34	35	from .._protos.buildstream.v2 import buildstream_pb2, buildstream_pb2_grpc
35	36
36	37	from .. import utils
37		-from .._exceptions import CASError
	38	+from .._exceptions import CASError, LoadError, LoadErrorReason
	39	+from .. import _yaml
38	40
39	41
40	42	# The default limit for gRPC messages is 4 MiB.
...	...	@@ -42,6 +44,50 @@ from .._exceptions import CASError
42	44	_MAX_PAYLOAD_BYTES = 1024 * 1024
43	45
44	46
	47	+class CASRemoteSpec(namedtuple('CASRemoteSpec', 'url push server_cert client_key client_cert')):
	48	+
	49	+ # _new_from_config_node
	50	+ #
	51	+ # Creates an CASRemoteSpec() from a YAML loaded node
	52	+ #
	53	+ @staticmethod
	54	+ def _new_from_config_node(spec_node, basedir=None):
	55	+ _yaml.node_validate(spec_node, ['url', 'push', 'server-cert', 'client-key', 'client-cert'])
	56	+ url = _yaml.node_get(spec_node, str, 'url')
	57	+ push = _yaml.node_get(spec_node, bool, 'push', default_value=False)
	58	+ if not url:
	59	+ provenance = _yaml.node_get_provenance(spec_node, 'url')
	60	+ raise LoadError(LoadErrorReason.INVALID_DATA,
	61	+ "{}: empty artifact cache URL".format(provenance))
	62	+
	63	+ server_cert = _yaml.node_get(spec_node, str, 'server-cert', default_value=None)
	64	+ if server_cert and basedir:
	65	+ server_cert = os.path.join(basedir, server_cert)
	66	+
	67	+ client_key = _yaml.node_get(spec_node, str, 'client-key', default_value=None)
	68	+ if client_key and basedir:
	69	+ client_key = os.path.join(basedir, client_key)
	70	+
	71	+ client_cert = _yaml.node_get(spec_node, str, 'client-cert', default_value=None)
	72	+ if client_cert and basedir:
	73	+ client_cert = os.path.join(basedir, client_cert)
	74	+
	75	+ if client_key and not client_cert:
	76	+ provenance = _yaml.node_get_provenance(spec_node, 'client-key')
	77	+ raise LoadError(LoadErrorReason.INVALID_DATA,
	78	+ "{}: 'client-key' was specified without 'client-cert'".format(provenance))
	79	+
	80	+ if client_cert and not client_key:
	81	+ provenance = _yaml.node_get_provenance(spec_node, 'client-cert')
	82	+ raise LoadError(LoadErrorReason.INVALID_DATA,
	83	+ "{}: 'client-cert' was specified without 'client-key'".format(provenance))
	84	+
	85	+ return CASRemoteSpec(url, push, server_cert, client_key, client_cert)
	86	+
	87	+
	88	+CASRemoteSpec.__new__.__defaults__ = (None, None, None)
	89	+
	90	+
45	91	# A CASCache manages a CAS repository as specified in the Remote Execution API.
46	92	#
47	93	# Args:

buildstream/_context.py

---

...	...	@@ -31,6 +31,7 @@ from ._exceptions import LoadError, LoadErrorReason, BstError
31	31	from ._message import Message, MessageType
32	32	from ._profile import Topics, profile_start, profile_end
33	33	from ._artifactcache import ArtifactCache
	34	+from ._artifactcache.cascache import CASCache
34	35	from ._workspaces import Workspaces
35	36	from .plugin import _plugin_lookup
36	37
...	...	@@ -141,6 +142,7 @@ class Context():
141	142	self._workspaces = None
142	143	self._log_handle = None
143	144	self._log_filename = None
	145	+ self._cascache = None
144	146
145	147	# load()
146	148	#
...	...	@@ -620,6 +622,11 @@ class Context():
620	622	if not os.environ.get('XDG_DATA_HOME'):
621	623	os.environ['XDG_DATA_HOME'] = os.path.expanduser('~/.local/share')
622	624
	625	+ def get_cascache(self):
	626	+ if self._cascache is None:
	627	+ self._cascache = CASCache(self.artifactdir)
	628	+ return self._cascache
	629	+
623	630
624	631	# _node_get_option_str()
625	632	#

buildstream/_project.py

---

...	...	@@ -30,6 +30,7 @@ from ._profile import Topics, profile_start, profile_end
30	30	from ._exceptions import LoadError, LoadErrorReason
31	31	from ._options import OptionPool
32	32	from ._artifactcache import ArtifactCache
	33	+from .sandbox import SandboxRemote
33	34	from ._elementfactory import ElementFactory
34	35	from ._sourcefactory import SourceFactory
35	36	from .plugin import CoreWarnings
...	...	@@ -130,7 +131,7 @@ class Project():
130	131	self._shell_host_files = [] # A list of HostMount objects
131	132
132	133	self.artifact_cache_specs = None
133		- self.remote_execution_url = None
	134	+ self.remote_execution_specs = None
134	135	self._sandbox = None
135	136	self._splits = None
136	137
...	...	@@ -493,9 +494,7 @@ class Project():
493	494	self.artifact_cache_specs = ArtifactCache.specs_from_config_node(config, self.directory)
494	495
495	496	# Load remote-execution configuration for this project
496		- remote_execution = _yaml.node_get(config, Mapping, 'remote-execution')
497		- _yaml.node_validate(remote_execution, ['url'])
498		- self.remote_execution_url = _yaml.node_get(remote_execution, str, 'url')
	497	+ self.remote_execution_specs = SandboxRemote.specs_from_config_node(config, self.directory)
499	498
500	499	# Load sandbox environment variables
501	500	self.base_environment = _yaml.node_get(config, Mapping, 'environment')

buildstream/buildelement.py

---

...	...	@@ -127,7 +127,7 @@ artifact collection purposes.
127	127	"""
128	128
129	129	import os
130		-from . import Element, Scope, ElementError
	130	+from . import Element, Scope
131	131	from . import SandboxFlags
132	132
133	133
...	...	@@ -207,6 +207,10 @@ class BuildElement(Element):
207	207	# Setup environment
208	208	sandbox.set_environment(self.get_environment())
209	209
	210	+ # Enable command batching across prepare() and assemble()
	211	+ self.batch_prepare_assemble(SandboxFlags.ROOT_READ_ONLY,
	212	+ collect=self.get_variable('install-root'))
	213	+
210	214	def stage(self, sandbox):
211	215
212	216	# Stage deps in the sandbox root
...	...	@@ -215,7 +219,7 @@ class BuildElement(Element):
215	219
216	220	# Run any integration commands provided by the dependencies
217	221	# once they are all staged and ready
218		- with self.timed_activity("Integrating sandbox"):
	222	+ with sandbox.batch(SandboxFlags.NONE, label="Integrating sandbox"):
219	223	for dep in self.dependencies(Scope.BUILD):
220	224	dep.integrate(sandbox)
221	225
...	...	@@ -223,14 +227,13 @@ class BuildElement(Element):
223	227	self.stage_sources(sandbox, self.get_variable('build-root'))
224	228
225	229	def assemble(self, sandbox):
226		-
227	230	# Run commands
228	231	for command_name in _command_steps:
229	232	commands = self.__commands[command_name]
230	233	if not commands or command_name == 'configure-commands':
231	234	continue
232	235
233		- with self.timed_activity("Running {}".format(command_name)):
	236	+ with sandbox.batch(SandboxFlags.ROOT_READ_ONLY, label="Running {}".format(command_name)):
234	237	for cmd in commands:
235	238	self.__run_command(sandbox, cmd, command_name)
236	239
...	...	@@ -254,7 +257,7 @@ class BuildElement(Element):
254	257	def prepare(self, sandbox):
255	258	commands = self.__commands['configure-commands']
256	259	if commands:
257		- with self.timed_activity("Running configure-commands"):
	260	+ with sandbox.batch(SandboxFlags.ROOT_READ_ONLY, label="Running configure-commands"):
258	261	for cmd in commands:
259	262	self.__run_command(sandbox, cmd, 'configure-commands')
260	263
...	...	@@ -282,13 +285,9 @@ class BuildElement(Element):
282	285	return commands
283	286
284	287	def __run_command(self, sandbox, cmd, cmd_name):
285		- self.status("Running {}".format(cmd_name), detail=cmd)
286		-
287	288	# Note the -e switch to 'sh' means to exit with an error
288	289	# if any untested command fails.
289	290	#
290		- exitcode = sandbox.run(['sh', '-c', '-e', cmd + '\n'],
291		- SandboxFlags.ROOT_READ_ONLY)
292		- if exitcode != 0:
293		- raise ElementError("Command '{}' failed with exitcode {}".format(cmd, exitcode),
294		- collect=self.get_variable('install-root'))
	291	+ sandbox.run(['sh', '-c', '-e', cmd + '\n'],
	292	+ SandboxFlags.ROOT_READ_ONLY,
	293	+ label=cmd)

buildstream/data/projectconfig.yaml

---

...	...	@@ -196,7 +196,4 @@ shell:
196	196
197	197	# Command to run when `bst shell` does not provide a command
198	198	#
199		- command: [ 'sh', '-i' ]
200		-
201		-remote-execution:
202		- url: ""
		\ No newline at end of file
	199	+ command: [ 'sh', '-i' ]
		\ No newline at end of file

buildstream/element.py

---

...	...	@@ -78,6 +78,7 @@ import stat
78	78	import copy
79	79	from collections import OrderedDict
80	80	from collections.abc import Mapping
	81	+import contextlib
81	82	from contextlib import contextmanager
82	83	import tempfile
83	84	import shutil
...	...	@@ -89,7 +90,7 @@ from ._exceptions import BstError, LoadError, LoadErrorReason, ImplError, \
89	90	ErrorDomain
90	91	from .utils import UtilError
91	92	from . import Plugin, Consistency, Scope
92		-from . import SandboxFlags
	93	+from . import SandboxFlags, SandboxCommandError
93	94	from . import utils
94	95	from . import _cachekey
95	96	from . import _signals
...	...	@@ -217,6 +218,10 @@ class Element(Plugin):
217	218	self.__build_result = None # The result of assembling this Element (success, description, detail)
218	219	self._build_log_path = None # The path of the build log for this Element
219	220
	221	+ self.__batch_prepare_assemble = False # Whether batching across prepare()/assemble() is configured
	222	+ self.__batch_prepare_assemble_flags = 0 # Sandbox flags for batching across prepare()/assemble()
	223	+ self.__batch_prepare_assemble_collect = None # Collect dir for batching across prepare()/assemble()
	224	+
220	225	# hash tables of loaded artifact metadata, hashed by key
221	226	self.__metadata_keys = {} # Strong and weak keys for this key
222	227	self.__metadata_dependencies = {} # Dictionary of dependency strong keys
...	...	@@ -250,9 +255,9 @@ class Element(Plugin):
250	255
251	256	# Extract remote execution URL
252	257	if not self.__is_junction:
253		- self.__remote_execution_url = project.remote_execution_url
	258	+ self.__remote_execution_specs = project.remote_execution_specs
254	259	else:
255		- self.__remote_execution_url = None
	260	+ self.__remote_execution_specs = None
256	261
257	262	# Extract Sandbox config
258	263	self.__sandbox_config = self.__extract_sandbox_config(meta)
...	...	@@ -770,13 +775,13 @@ class Element(Plugin):
770	775	environment = self.get_environment()
771	776
772	777	if bstdata is not None:
773		- commands = self.node_get_member(bstdata, list, 'integration-commands', [])
774		- for i in range(len(commands)):
775		- cmd = self.node_subst_list_element(bstdata, 'integration-commands', [i])
776		- self.status("Running integration command", detail=cmd)
777		- exitcode = sandbox.run(['sh', '-e', '-c', cmd], 0, env=environment, cwd='/')
778		- if exitcode != 0:
779		- raise ElementError("Command '{}' failed with exitcode {}".format(cmd, exitcode))
	778	+ with sandbox.batch(SandboxFlags.NONE):
	779	+ commands = self.node_get_member(bstdata, list, 'integration-commands', [])
	780	+ for i in range(len(commands)):
	781	+ cmd = self.node_subst_list_element(bstdata, 'integration-commands', [i])
	782	+
	783	+ sandbox.run(['sh', '-e', '-c', cmd], 0, env=environment, cwd='/',
	784	+ label=cmd)
780	785
781	786	def stage_sources(self, sandbox, directory):
782	787	"""Stage this element's sources to a directory in the sandbox
...	...	@@ -863,6 +868,24 @@ class Element(Plugin):
863	868
864	869	return None
865	870
	871	+ def batch_prepare_assemble(self, flags, *, collect=None):
	872	+ """ Configure command batching across prepare() and assemble()
	873	+
	874	+ Args:
	875	+ flags (:class:`.SandboxFlags`): The sandbox flags for the command batch
	876	+ collect (str): An optional directory containing partial install contents
	877	+ on command failure.
	878	+
	879	+ This may be called in :func:`Element.configure_sandbox() <buildstream.element.Element.configure_sandbox>`
	880	+ to enable batching of all sandbox commands issued in prepare() and assemble().
	881	+ """
	882	+ if self.__batch_prepare_assemble:
	883	+ raise ElementError("{}: Command batching for prepare/assemble is already configured".format(self))
	884	+
	885	+ self.__batch_prepare_assemble = True
	886	+ self.__batch_prepare_assemble_flags = flags
	887	+ self.__batch_prepare_assemble_collect = collect
	888	+
866	889	#############################################################
867	890	# Private Methods used in BuildStream #
868	891	#############################################################
...	...	@@ -1323,7 +1346,7 @@ class Element(Plugin):
1323	1346	bare_directory=bare_directory) as sandbox:
1324	1347
1325	1348	# Configure always comes first, and we need it.
1326		- self.configure_sandbox(sandbox)
	1349	+ self.__configure_sandbox(sandbox)
1327	1350
1328	1351	# Stage something if we need it
1329	1352	if not directory:
...	...	@@ -1556,15 +1579,24 @@ class Element(Plugin):
1556	1579	# Call the abstract plugin methods
1557	1580	try:
1558	1581	# Step 1 - Configure
1559		- self.configure_sandbox(sandbox)
	1582	+ self.__configure_sandbox(sandbox)
1560	1583	# Step 2 - Stage
1561	1584	self.stage(sandbox)
1562		- # Step 3 - Prepare
1563		- self.__prepare(sandbox)
1564		- # Step 4 - Assemble
1565		- collect = self.assemble(sandbox) # pylint: disable=assignment-from-no-return
	1585	+
	1586	+ if self.__batch_prepare_assemble:
	1587	+ cm = sandbox.batch(self.__batch_prepare_assemble_flags,
	1588	+ collect=self.__batch_prepare_assemble_collect)
	1589	+ else:
	1590	+ cm = contextlib.suppress()
	1591	+
	1592	+ with cm:
	1593	+ # Step 3 - Prepare
	1594	+ self.__prepare(sandbox)
	1595	+ # Step 4 - Assemble
	1596	+ collect = self.assemble(sandbox) # pylint: disable=assignment-from-no-return
	1597	+
1566	1598	self.__set_build_result(success=True, description="succeeded")
1567		- except ElementError as e:
	1599	+ except (ElementError, SandboxCommandError) as e:
1568	1600	# Shelling into a sandbox is useful to debug this error
1569	1601	e.sandbox = True
1570	1602
...	...	@@ -2059,6 +2091,15 @@ class Element(Plugin):
2059	2091	def __can_build_incrementally(self):
2060	2092	return bool(self._get_workspace())
2061	2093
	2094	+ # __configure_sandbox():
	2095	+ #
	2096	+ # Internal method for calling public abstract configure_sandbox() method.
	2097	+ #
	2098	+ def __configure_sandbox(self, sandbox):
	2099	+ self.__batch_prepare_assemble = False
	2100	+
	2101	+ self.configure_sandbox(sandbox)
	2102	+
2062	2103	# __prepare():
2063	2104	#
2064	2105	# Internal method for calling public abstract prepare() method.
...	...	@@ -2074,7 +2115,12 @@ class Element(Plugin):
2074	2115	self.prepare(sandbox)
2075	2116
2076	2117	if workspace:
2077		- workspace.prepared = True
	2118	+ def mark_workspace_prepared():
	2119	+ workspace.prepared = True
	2120	+
	2121	+ # Defer workspace.prepared setting until pending batch commands
	2122	+ # have been executed.
	2123	+ sandbox._callback(mark_workspace_prepared)
2078	2124
2079	2125	def __is_cached(self, keystrength):
2080	2126	if keystrength is None:
...	...	@@ -2125,7 +2171,7 @@ class Element(Plugin):
2125	2171	# supports it.
2126	2172	#
2127	2173	def __use_remote_execution(self):
2128		- return self.__remote_execution_url and self.BST_VIRTUAL_DIRECTORY
	2174	+ return self.__remote_execution_specs and self.BST_VIRTUAL_DIRECTORY
2129	2175
2130	2176	# __sandbox():
2131	2177	#
...	...	@@ -2157,16 +2203,17 @@ class Element(Plugin):
2157	2203
2158	2204	sandbox = SandboxRemote(context, project,
2159	2205	directory,
	2206	+ plugin=self,
2160	2207	stdout=stdout,
2161	2208	stderr=stderr,
2162	2209	config=config,
2163		- server_url=self.__remote_execution_url,
	2210	+ specs=self.__remote_execution_specs,
2164	2211	bare_directory=bare_directory,
2165	2212	allow_real_directory=False)
2166	2213	yield sandbox
2167	2214
2168	2215	elif directory is not None and os.path.exists(directory):
2169		- if allow_remote and self.__remote_execution_url:
	2216	+ if allow_remote and self.__remote_execution_specs:
2170	2217	self.warn("Artifact {} is configured to use remote execution but element plugin does not support it."
2171	2218	.format(self.name), detail="Element plugin '{kind}' does not support virtual directories."
2172	2219	.format(kind=self.get_kind()), warning_token="remote-failure")
...	...	@@ -2175,6 +2222,7 @@ class Element(Plugin):
2175	2222
2176	2223	sandbox = platform.create_sandbox(context, project,
2177	2224	directory,
	2225	+ plugin=self,
2178	2226	stdout=stdout,
2179	2227	stderr=stderr,
2180	2228	config=config,

buildstream/plugins/elements/compose.py

---

...	...	@@ -122,8 +122,9 @@ class ComposeElement(Element):
122	122	snapshot = set(vbasedir.list_relative_paths())
123	123	vbasedir.mark_unmodified()
124	124
125		- for dep in self.dependencies(Scope.BUILD):
126		- dep.integrate(sandbox)
	125	+ with sandbox.batch(0):
	126	+ for dep in self.dependencies(Scope.BUILD):
	127	+ dep.integrate(sandbox)
127	128
128	129	if require_split:
129	130	# Calculate added, modified and removed files

buildstream/sandbox/__init__.py

---

...	...	@@ -17,6 +17,6 @@
17	17	# Authors:
18	18	# Tristan Maat <tristan maat codethink co uk>
19	19
20		-from .sandbox import Sandbox, SandboxFlags
	20	+from .sandbox import Sandbox, SandboxFlags, SandboxCommandError
21	21	from ._sandboxremote import SandboxRemote
22	22	from ._sandboxdummy import SandboxDummy

buildstream/sandbox/_sandboxbwrap.py

---

...	...	@@ -58,22 +58,12 @@ class SandboxBwrap(Sandbox):
58	58	self.die_with_parent_available = kwargs['die_with_parent_available']
59	59	self.json_status_available = kwargs['json_status_available']
60	60
61		- def run(self, command, flags, *, cwd=None, env=None):
	61	+ def _run(self, command, flags, *, cwd, env):
62	62	stdout, stderr = self._get_output()
63	63
64	64	# Allowable access to underlying storage as we're part of the sandbox
65	65	root_directory = self.get_virtual_directory()._get_underlying_directory()
66	66
67		- # Fallback to the sandbox default settings for
68		- # the cwd and env.
69		- #
70		- cwd = self._get_work_directory(cwd=cwd)
71		- env = self._get_environment(cwd=cwd, env=env)
72		-
73		- # Convert single-string argument to a list
74		- if isinstance(command, str):
75		- command = [command]
76		-
77	67	if not self._has_command(command[0], env):
78	68	raise SandboxError("Staged artifacts do not provide command "
79	69	"'{}'".format(command[0]),

buildstream/sandbox/_sandboxchroot.py

---

...	...	@@ -49,17 +49,7 @@ class SandboxChroot(Sandbox):
49	49
50	50	self.mount_map = None
51	51
52		- def run(self, command, flags, *, cwd=None, env=None):
53		-
54		- # Fallback to the sandbox default settings for
55		- # the cwd and env.
56		- #
57		- cwd = self._get_work_directory(cwd=cwd)
58		- env = self._get_environment(cwd=cwd, env=env)
59		-
60		- # Convert single-string argument to a list
61		- if isinstance(command, str):
62		- command = [command]
	52	+ def _run(self, command, flags, *, cwd, env):
63	53
64	54	if not self._has_command(command[0], env):
65	55	raise SandboxError("Staged artifacts do not provide command "

buildstream/sandbox/_sandboxdummy.py

---

...	...	@@ -25,17 +25,7 @@ class SandboxDummy(Sandbox):
25	25	super().__init__(*args, **kwargs)
26	26	self._reason = kwargs.get("dummy_reason", "no reason given")
27	27
28		- def run(self, command, flags, *, cwd=None, env=None):
29		-
30		- # Fallback to the sandbox default settings for
31		- # the cwd and env.
32		- #
33		- cwd = self._get_work_directory(cwd=cwd)
34		- env = self._get_environment(cwd=cwd, env=env)
35		-
36		- # Convert single-string argument to a list
37		- if isinstance(command, str):
38		- command = [command]
	28	+ def _run(self, command, flags, *, cwd, env):
39	29
40	30	if not self._has_command(command[0], env):
41	31	raise SandboxError("Staged artifacts do not provide command "

buildstream/sandbox/_sandboxremote.py

---

...	...	@@ -19,19 +19,28 @@
19	19	# Jim MacArthur <jim macarthur codethink co uk>
20	20
21	21	import os
	22	+import shlex
	23	+from collections import namedtuple
22	24	from urllib.parse import urlparse
23	25	from functools import partial
24	26
25	27	import grpc
26	28
27		-from . import Sandbox
	29	+from . import Sandbox, SandboxCommandError
	30	+from .sandbox import _SandboxBatch
28	31	from ..storage._filebaseddirectory import FileBasedDirectory
29	32	from ..storage._casbaseddirectory import CasBasedDirectory
30	33	from .. import _signals
31	34	from .._protos.build.bazel.remote.execution.v2 import remote_execution_pb2, remote_execution_pb2_grpc
32	35	from .._protos.google.rpc import code_pb2
33	36	from .._exceptions import SandboxError
	37	+from .. import _yaml
34	38	from .._protos.google.longrunning import operations_pb2, operations_pb2_grpc
	39	+from .._artifactcache.cascache import CASRemote, CASRemoteSpec
	40	+
	41	+
	42	+class RemoteExecutionSpec(namedtuple('RemoteExecutionSpec', 'exec_service storage_service')):
	43	+ pass
35	44
36	45
37	46	# SandboxRemote()
...	...	@@ -44,18 +53,70 @@ class SandboxRemote(Sandbox):
44	53	def __init__(self, *args, **kwargs):
45	54	super().__init__(*args, **kwargs)
46	55
47		- url = urlparse(kwargs['server_url'])
48		- if not url.scheme or not url.hostname or not url.port:
49		- raise SandboxError("Configured remote URL '{}' does not match the expected layout. "
50		- .format(kwargs['server_url']) +
51		- "It should be of the form <protocol>://<domain name>:<port>.")
52		- elif url.scheme != 'http':
53		- raise SandboxError("Configured remote '{}' uses an unsupported protocol. "
54		- "Only plain HTTP is currenlty supported (no HTTPS).")
	56	+ config = kwargs['specs'] # This should be a RemoteExecutionSpec
	57	+ if config is None:
	58	+ return
	59	+
	60	+ self.storage_url = config.storage_service['url']
	61	+ self.exec_url = config.exec_service['url']
55	62
56		- self.server_url = '{}:{}'.format(url.hostname, url.port)
	63	+ self.storage_remote_spec = CASRemoteSpec(self.storage_url, push=True,
	64	+ server_cert=config.storage_service['server-cert'],
	65	+ client_key=config.storage_service['client-key'],
	66	+ client_cert=config.storage_service['client-cert'])
57	67	self.operation_name = None
58	68
	69	+ @staticmethod
	70	+ def specs_from_config_node(config_node, basedir):
	71	+
	72	+ def require_node(config, keyname):
	73	+ val = config.get(keyname)
	74	+ if val is None:
	75	+ provenance = _yaml.node_get_provenance(remote_config, key=keyname)
	76	+ raise _yaml.LoadError(_yaml.LoadErrorReason.INVALID_DATA,
	77	+ "{}: '{}' was not present in the remote "
	78	+ "execution configuration (remote-execution). "
	79	+ .format(str(provenance), keyname))
	80	+ return val
	81	+
	82	+ remote_config = config_node.get("remote-execution", None)
	83	+ if remote_config is None:
	84	+ return None
	85	+
	86	+ # Maintain some backwards compatibility with older configs, in which 'url' was the only valid key for
	87	+ # remote-execution.
	88	+
	89	+ tls_keys = ['client-key', 'client-cert', 'server-cert']
	90	+
	91	+ _yaml.node_validate(remote_config, ['execution-service', 'storage-service', 'url'])
	92	+ remote_exec_service_config = require_node(remote_config, 'execution-service')
	93	+ remote_exec_storage_config = require_node(remote_config, 'storage-service')
	94	+
	95	+ _yaml.node_validate(remote_exec_service_config, ['url'])
	96	+ _yaml.node_validate(remote_exec_storage_config, ['url'] + tls_keys)
	97	+
	98	+ if 'url' in remote_config:
	99	+ if 'execution-service' not in remote_config:
	100	+ remote_config['execution-service'] = {'url': remote_config['url']}
	101	+ else:
	102	+ provenance = _yaml.node_get_provenance(remote_config, key='url')
	103	+ raise _yaml.LoadError(_yaml.LoadErrorReason.INVALID_DATA,
	104	+ "{}: 'url' and 'execution-service' keys were found in the remote "
	105	+ "execution configuration (remote-execution). "
	106	+ "You can only specify one of these."
	107	+ .format(str(provenance)))
	108	+
	109	+ for key in tls_keys:
	110	+ if key not in remote_exec_storage_config:
	111	+ provenance = _yaml.node_get_provenance(remote_config, key='storage-service')
	112	+ raise _yaml.LoadError(_yaml.LoadErrorReason.INVALID_DATA,
	113	+ "{}: The keys {} are necessary for the storage-service section of "
	114	+ "remote-execution configuration. Your config is missing '{}'."
	115	+ .format(str(provenance), tls_keys, key))
	116	+
	117	+ spec = RemoteExecutionSpec(remote_config['execution-service'], remote_config['storage-service'])
	118	+ return spec
	119	+
59	120	def run_remote_command(self, command, input_root_digest, working_directory, environment):
60	121	# Sends an execution request to the remote execution server.
61	122	#
...	...	@@ -73,12 +134,13 @@ class SandboxRemote(Sandbox):
73	134	output_directories=[self._output_directory],
74	135	platform=None)
75	136	context = self._get_context()
76		- cascache = context.artifactcache
	137	+ cascache = context.get_cascache()
	138	+ casremote = CASRemote(self.storage_remote_spec)
	139	+
77	140	# Upload the Command message to the remote CAS server
78		- command_digest = cascache.push_message(self._get_project(), remote_command)
79		- if not command_digest or not cascache.verify_digest_pushed(self._get_project(), command_digest):
	141	+ command_digest = cascache.push_message(casremote, remote_command)
	142	+ if not command_digest or not cascache.verify_digest_on_remote(casremote, command_digest):
80	143	raise SandboxError("Failed pushing build command to remote CAS.")
81		-
82	144	# Create and send the action.
83	145	action = remote_execution_pb2.Action(command_digest=command_digest,
84	146	input_root_digest=input_root_digest,
...	...	@@ -86,12 +148,21 @@ class SandboxRemote(Sandbox):
86	148	do_not_cache=False)
87	149
88	150	# Upload the Action message to the remote CAS server
89		- action_digest = cascache.push_message(self._get_project(), action)
90		- if not action_digest or not cascache.verify_digest_pushed(self._get_project(), action_digest):
	151	+ action_digest = cascache.push_message(casremote, action)
	152	+ if not action_digest or not cascache.verify_digest_on_remote(casremote, action_digest):
91	153	raise SandboxError("Failed pushing build action to remote CAS.")
92	154
93	155	# Next, try to create a communication channel to the BuildGrid server.
94		- channel = grpc.insecure_channel(self.server_url)
	156	+ url = urlparse(self.exec_url)
	157	+ if not url.port:
	158	+ raise SandboxError("You must supply a protocol and port number in the execution-service url, "
	159	+ "for example: http://buildservice:50051.")
	160	+ if url.scheme == 'http':
	161	+ channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
	162	+ else:
	163	+ raise SandboxError("Remote execution currently only supports the 'http' protocol "
	164	+ "and '{}' was supplied.".format(url.scheme))
	165	+
95	166	stub = remote_execution_pb2_grpc.ExecutionStub(channel)
96	167	request = remote_execution_pb2.ExecuteRequest(action_digest=action_digest,
97	168	skip_cache_lookup=False)
...	...	@@ -117,7 +188,7 @@ class SandboxRemote(Sandbox):
117	188	status_code = e.code()
118	189	if status_code == grpc.StatusCode.UNAVAILABLE:
119	190	raise SandboxError("Failed contacting remote execution server at {}."
120		- .format(self.server_url))
	191	+ .format(self.exec_url))
121	192
122	193	elif status_code in (grpc.StatusCode.INVALID_ARGUMENT,
123	194	grpc.StatusCode.FAILED_PRECONDITION,
...	...	@@ -188,9 +259,11 @@ class SandboxRemote(Sandbox):
188	259	raise SandboxError("Output directory structure had no digest attached.")
189	260
190	261	context = self._get_context()
191		- cascache = context.artifactcache
	262	+ cascache = context.get_cascache()
	263	+ casremote = CASRemote(self.storage_remote_spec)
	264	+
192	265	# Now do a pull to ensure we have the necessary parts.
193		- dir_digest = cascache.pull_tree(self._get_project(), tree_digest)
	266	+ dir_digest = cascache.pull_tree(casremote, tree_digest)
194	267	if dir_digest is None or not dir_digest.hash or not dir_digest.size_bytes:
195	268	raise SandboxError("Output directory structure pulling from remote failed.")
196	269
...	...	@@ -212,33 +285,28 @@ class SandboxRemote(Sandbox):
212	285	new_dir = CasBasedDirectory(self._get_context().artifactcache.cas, ref=dir_digest)
213	286	self._set_virtual_directory(new_dir)
214	287
215		- def run(self, command, flags, *, cwd=None, env=None):
	288	+ def _run(self, command, flags, *, cwd, env):
216	289	# Upload sources
217	290	upload_vdir = self.get_virtual_directory()
218	291
	292	+ cascache = self._get_context().get_cascache()
219	293	if isinstance(upload_vdir, FileBasedDirectory):
220	294	# Make a new temporary directory to put source in
221		- upload_vdir = CasBasedDirectory(self._get_context().artifactcache.cas, ref=None)
	295	+ upload_vdir = CasBasedDirectory(cascache, ref=None)
222	296	upload_vdir.import_files(self.get_virtual_directory()._get_underlying_directory())
223	297
224	298	upload_vdir.recalculate_hash()
225	299
226		- context = self._get_context()
227		- cascache = context.artifactcache
	300	+ casremote = CASRemote(self.storage_remote_spec)
228	301	# Now, push that key (without necessarily needing a ref) to the remote.
229		- cascache.push_directory(self._get_project(), upload_vdir)
230		- if not cascache.verify_digest_pushed(self._get_project(), upload_vdir.ref):
231		- raise SandboxError("Failed to verify that source has been pushed to the remote artifact cache.")
232	302
233		- # Fallback to the sandbox default settings for
234		- # the cwd and env.
235		- #
236		- cwd = self._get_work_directory(cwd=cwd)
237		- env = self._get_environment(cwd=cwd, env=env)
	303	+ try:
	304	+ cascache.push_directory(casremote, upload_vdir)
	305	+ except grpc.RpcError as e:
	306	+ raise SandboxError("Failed to push source directory to remote: {}".format(e)) from e
238	307
239		- # We want command args as a list of strings
240		- if isinstance(command, str):
241		- command = [command]
	308	+ if not cascache.verify_digest_on_remote(casremote, upload_vdir.ref):
	309	+ raise SandboxError("Failed to verify that source has been pushed to the remote artifact cache.")
242	310
243	311	# Now transmit the command to execute
244	312	operation = self.run_remote_command(command, upload_vdir.ref, cwd, env)
...	...	@@ -275,3 +343,69 @@ class SandboxRemote(Sandbox):
275	343	self.process_job_output(action_result.output_directories, action_result.output_files)
276	344
277	345	return 0
	346	+
	347	+ def _create_batch(self, main_group, flags, *, collect=None):
	348	+ return _SandboxRemoteBatch(self, main_group, flags, collect=collect)
	349	+
	350	+
	351	+# _SandboxRemoteBatch()
	352	+#
	353	+# Command batching by shell script generation.
	354	+#
	355	+class _SandboxRemoteBatch(_SandboxBatch):
	356	+
	357	+ def __init__(self, sandbox, main_group, flags, *, collect=None):
	358	+ super().__init__(sandbox, main_group, flags, collect=collect)
	359	+
	360	+ self.script = None
	361	+ self.first_command = None
	362	+ self.cwd = None
	363	+ self.env = None
	364	+
	365	+ def execute(self):
	366	+ self.script = ""
	367	+
	368	+ self.main_group.execute(self)
	369	+
	370	+ first = self.first_command
	371	+ if first and self.sandbox.run(['sh', '-c', '-e', self.script], self.flags, cwd=first.cwd, env=first.env) != 0:
	372	+ raise SandboxCommandError("Command execution failed", collect=self.collect)
	373	+
	374	+ def execute_group(self, group):
	375	+ group.execute_children(self)
	376	+
	377	+ def execute_command(self, command):
	378	+ if self.first_command is None:
	379	+ # First command in batch
	380	+ # Initial working directory and environment of script already matches
	381	+ # the command configuration.
	382	+ self.first_command = command
	383	+ else:
	384	+ # Change working directory for this command
	385	+ if command.cwd != self.cwd:
	386	+ self.script += "mkdir -p {}\n".format(command.cwd)
	387	+ self.script += "cd {}\n".format(command.cwd)
	388	+
	389	+ # Update environment for this command
	390	+ for key in self.env.keys():
	391	+ if key not in command.env:
	392	+ self.script += "unset {}\n".format(key)
	393	+ for key, value in command.env.items():
	394	+ if key not in self.env or self.env[key] != value:
	395	+ self.script += "export {}={}\n".format(key, shlex.quote(value))
	396	+
	397	+ # Keep track of current working directory and environment
	398	+ self.cwd = command.cwd
	399	+ self.env = command.env
	400	+
	401	+ # Actual command execution
	402	+ cmdline = ' '.join(shlex.quote(cmd) for cmd in command.command)
	403	+ self.script += "(set -ex; {})".format(cmdline)
	404	+
	405	+ # Error handling
	406	+ label = command.label or cmdline
	407	+ quoted_label = shlex.quote("'{}'".format(label))
	408	+ self.script += " || (echo Command {} failed with exitcode $? >&2 ; exit 1)\n".format(quoted_label)
	409	+
	410	+ def execute_call(self, call):
	411	+ raise SandboxError("SandboxRemote does not support callbacks in command batches")

buildstream/sandbox/sandbox.py

---

1	1	#
2	2	# Copyright (C) 2017 Codethink Limited
	3	+# Copyright (C) 2018 Bloomberg Finance LP
3	4	#
4	5	# This program is free software; you can redistribute it and/or
5	6	# modify it under the terms of the GNU Lesser General Public
...	...	@@ -29,7 +30,12 @@ See also: :ref:`sandboxing`.
29	30	"""
30	31
31	32	import os
32		-from .._exceptions import ImplError, BstError
	33	+import shlex
	34	+import contextlib
	35	+from contextlib import contextmanager
	36	+
	37	+from .._exceptions import ImplError, BstError, SandboxError
	38	+from .._message import Message, MessageType
33	39	from ..storage._filebaseddirectory import FileBasedDirectory
34	40	from ..storage._casbaseddirectory import CasBasedDirectory
35	41
...	...	@@ -38,6 +44,10 @@ class SandboxFlags():
38	44	"""Flags indicating how the sandbox should be run.
39	45	"""
40	46
	47	+ NONE = 0
	48	+ """Use default sandbox configuration.
	49	+ """
	50	+
41	51	ROOT_READ_ONLY = 0x01
42	52	"""The root filesystem is read only.
43	53
...	...	@@ -71,6 +81,19 @@ class SandboxFlags():
71	81	"""
72	82
73	83
	84	+class SandboxCommandError(SandboxError):
	85	+ """Raised by :class:`.Sandbox` implementations when a command fails.
	86	+
	87	+ Args:
	88	+ message (str): The error message to report to the user
	89	+ collect (str): An optional directory containing partial install contents
	90	+ """
	91	+ def __init__(self, message, *, collect=None):
	92	+ super().__init__(message, reason='command-failed')
	93	+
	94	+ self.collect = collect
	95	+
	96	+
74	97	class Sandbox():
75	98	"""Sandbox()
76	99
...	...	@@ -94,6 +117,13 @@ class Sandbox():
94	117	self.__mount_sources = {}
95	118	self.__allow_real_directory = kwargs['allow_real_directory']
96	119
	120	+ # Plugin ID for logging
	121	+ plugin = kwargs.get('plugin', None)
	122	+ if plugin:
	123	+ self.__plugin_id = plugin._get_unique_id()
	124	+ else:
	125	+ self.__plugin_id = None
	126	+
97	127	# Configuration from kwargs common to all subclasses
98	128	self.__config = kwargs['config']
99	129	self.__stdout = kwargs['stdout']
...	...	@@ -121,6 +151,9 @@ class Sandbox():
121	151	# directory via get_directory.
122	152	self._never_cache_vdirs = False
123	153
	154	+ # Pending command batch
	155	+ self.__batch = None
	156	+
124	157	def get_directory(self):
125	158	"""Fetches the sandbox root directory
126	159
...	...	@@ -209,9 +242,16 @@ class Sandbox():
209	242	'artifact': artifact
210	243	})
211	244
212		- def run(self, command, flags, *, cwd=None, env=None):
	245	+ def run(self, command, flags, *, cwd=None, env=None, label=None):
213	246	"""Run a command in the sandbox.
214	247
	248	+ If this is called outside a batch context, the command is immediately
	249	+ executed.
	250	+
	251	+ If this is called in a batch context, the command is added to the batch
	252	+ for later execution. If the command fails, later commands will not be
	253	+ executed. Command flags must match batch flags.
	254	+
215	255	Args:
216	256	command (list): The command to run in the sandboxed environment, as a list
217	257	of strings starting with the binary to run.
...	...	@@ -219,9 +259,10 @@ class Sandbox():
219	259	cwd (str): The sandbox relative working directory in which to run the command.
220	260	env (dict): A dictionary of string key, value pairs to set as environment
221	261	variables inside the sandbox environment.
	262	+ label (str): An optional label for the command, used for logging. (*Since: 1.4*)
222	263
223	264	Returns:
224		- (int): The program exit code.
	265	+ (int|None): The program exit code, or None if running in batch context.
225	266
226	267	Raises:
227	268	(:class:`.ProgramNotFoundError`): If a host tool which the given sandbox
...	...	@@ -234,9 +275,115 @@ class Sandbox():
234	275	function must make sure the directory will be created if it does
235	276	not exist yet, even if a workspace is being used.
236	277	"""
237		- raise ImplError("Sandbox of type '{}' does not implement run()"
	278	+
	279	+ # Fallback to the sandbox default settings for
	280	+ # the cwd and env.
	281	+ #
	282	+ cwd = self._get_work_directory(cwd=cwd)
	283	+ env = self._get_environment(cwd=cwd, env=env)
	284	+
	285	+ # Convert single-string argument to a list
	286	+ if isinstance(command, str):
	287	+ command = [command]
	288	+
	289	+ if self.__batch:
	290	+ if flags != self.__batch.flags:
	291	+ raise SandboxError("Inconsistent sandbox flags in single command batch")
	292	+
	293	+ batch_command = _SandboxBatchCommand(command, cwd=cwd, env=env, label=label)
	294	+
	295	+ current_group = self.__batch.current_group
	296	+ current_group.append(batch_command)
	297	+ return None
	298	+ else:
	299	+ return self._run(command, flags, cwd=cwd, env=env)
	300	+
	301	+ @contextmanager
	302	+ def batch(self, flags, *, label=None, collect=None):
	303	+ """Context manager for command batching
	304	+
	305	+ This provides a batch context that defers execution of commands until
	306	+ the end of the context. If a command fails, the batch will be aborted
	307	+ and subsequent commands will not be executed.
	308	+
	309	+ Command batches may be nested. Execution will start only when the top
	310	+ level batch context ends.
	311	+
	312	+ Args:
	313	+ flags (:class:`.SandboxFlags`): The flags for this command batch.
	314	+ label (str): An optional label for the batch group, used for logging.
	315	+ collect (str): An optional directory containing partial install contents
	316	+ on command failure.
	317	+
	318	+ Raises:
	319	+ (:class:`.SandboxCommandError`): If a command fails.
	320	+
	321	+ *Since: 1.4*
	322	+ """
	323	+
	324	+ group = _SandboxBatchGroup(label=label)
	325	+
	326	+ if self.__batch:
	327	+ # Nested batch
	328	+ if flags != self.__batch.flags:
	329	+ raise SandboxError("Inconsistent sandbox flags in single command batch")
	330	+
	331	+ parent_group = self.__batch.current_group
	332	+ parent_group.append(group)
	333	+ self.__batch.current_group = group
	334	+ try:
	335	+ yield
	336	+ finally:
	337	+ self.__batch.current_group = parent_group
	338	+ else:
	339	+ # Top-level batch
	340	+ batch = self._create_batch(group, flags, collect=collect)
	341	+
	342	+ self.__batch = batch
	343	+ try:
	344	+ yield
	345	+ finally:
	346	+ self.__batch = None
	347	+
	348	+ batch.execute()
	349	+
	350	+ #####################################################
	351	+ # Abstract Methods for Sandbox implementations #
	352	+ #####################################################
	353	+
	354	+ # _run()
	355	+ #
	356	+ # Abstract method for running a single command
	357	+ #
	358	+ # Args:
	359	+ # command (list): The command to run in the sandboxed environment, as a list
	360	+ # of strings starting with the binary to run.
	361	+ # flags (:class:`.SandboxFlags`): The flags for running this command.
	362	+ # cwd (str): The sandbox relative working directory in which to run the command.
	363	+ # env (dict): A dictionary of string key, value pairs to set as environment
	364	+ # variables inside the sandbox environment.
	365	+ #
	366	+ # Returns:
	367	+ # (int): The program exit code.
	368	+ #
	369	+ def _run(self, command, flags, *, cwd, env):
	370	+ raise ImplError("Sandbox of type '{}' does not implement _run()"
238	371	.format(type(self).__name__))
239	372
	373	+ # _create_batch()
	374	+ #
	375	+ # Abstract method for creating a batch object. Subclasses can override
	376	+ # this method to instantiate a subclass of _SandboxBatch.
	377	+ #
	378	+ # Args:
	379	+ # main_group (:class:`_SandboxBatchGroup`): The top level batch group.
	380	+ # flags (:class:`.SandboxFlags`): The flags for commands in this batch.
	381	+ # collect (str): An optional directory containing partial install contents
	382	+ # on command failure.
	383	+ #
	384	+ def _create_batch(self, main_group, flags, *, collect=None):
	385	+ return _SandboxBatch(self, main_group, flags, collect=collect)
	386	+
240	387	################################################
241	388	# Private methods #
242	389	################################################
...	...	@@ -385,3 +532,138 @@ class Sandbox():
385	532	return True
386	533
387	534	return False
	535	+
	536	+ # _get_plugin_id()
	537	+ #
	538	+ # Get the plugin's unique identifier
	539	+ #
	540	+ def _get_plugin_id(self):
	541	+ return self.__plugin_id
	542	+
	543	+ # _callback()
	544	+ #
	545	+ # If this is called outside a batch context, the specified function is
	546	+ # invoked immediately.
	547	+ #
	548	+ # If this is called in a batch context, the function is added to the batch
	549	+ # for later invocation.
	550	+ #
	551	+ # Args:
	552	+ # callback (callable): The function to invoke
	553	+ #
	554	+ def _callback(self, callback):
	555	+ if self.__batch:
	556	+ batch_call = _SandboxBatchCall(callback)
	557	+
	558	+ current_group = self.__batch.current_group
	559	+ current_group.append(batch_call)
	560	+ else:
	561	+ callback()
	562	+
	563	+
	564	+# _SandboxBatch()
	565	+#
	566	+# A batch of sandbox commands.
	567	+#
	568	+class _SandboxBatch():
	569	+
	570	+ def __init__(self, sandbox, main_group, flags, *, collect=None):
	571	+ self.sandbox = sandbox
	572	+ self.main_group = main_group
	573	+ self.current_group = main_group
	574	+ self.flags = flags
	575	+ self.collect = collect
	576	+
	577	+ def execute(self):
	578	+ self.main_group.execute(self)
	579	+
	580	+ def execute_group(self, group):
	581	+ if group.label:
	582	+ context = self.sandbox._get_context()
	583	+ cm = context.timed_activity(group.label, unique_id=self.sandbox._get_plugin_id())
	584	+ else:
	585	+ cm = contextlib.suppress()
	586	+
	587	+ with cm:
	588	+ group.execute_children(self)
	589	+
	590	+ def execute_command(self, command):
	591	+ if command.label:
	592	+ context = self.sandbox._get_context()
	593	+ message = Message(self.sandbox._get_plugin_id(), MessageType.STATUS,
	594	+ 'Running {}'.format(command.label))
	595	+ context.message(message)
	596	+
	597	+ exitcode = self.sandbox._run(command.command, self.flags, cwd=command.cwd, env=command.env)
	598	+ if exitcode != 0:
	599	+ cmdline = ' '.join(shlex.quote(cmd) for cmd in command.command)
	600	+ label = command.label or cmdline
	601	+ raise SandboxCommandError("Command '{}' failed with exitcode {}".format(label, exitcode),
	602	+ collect=self.collect)
	603	+
	604	+ def execute_call(self, call):
	605	+ call.callback()
	606	+
	607	+
	608	+# _SandboxBatchItem()
	609	+#
	610	+# An item in a command batch.
	611	+#
	612	+class _SandboxBatchItem():
	613	+
	614	+ def __init__(self, *, label=None):
	615	+ self.label = label
	616	+
	617	+
	618	+# _SandboxBatchCommand()
	619	+#
	620	+# A command item in a command batch.
	621	+#
	622	+class _SandboxBatchCommand(_SandboxBatchItem):
	623	+
	624	+ def __init__(self, command, *, cwd, env, label=None):
	625	+ super().__init__(label=label)
	626	+
	627	+ self.command = command
	628	+ self.cwd = cwd
	629	+ self.env = env
	630	+
	631	+ def execute(self, batch):
	632	+ batch.execute_command(self)
	633	+
	634	+
	635	+# _SandboxBatchGroup()
	636	+#
	637	+# A group in a command batch.
	638	+#
	639	+class _SandboxBatchGroup(_SandboxBatchItem):
	640	+
	641	+ def __init__(self, *, label=None):
	642	+ super().__init__(label=label)
	643	+
	644	+ self.children = []
	645	+
	646	+ def append(self, item):
	647	+ self.children.append(item)
	648	+
	649	+ def execute(self, batch):
	650	+ batch.execute_group(self)
	651	+
	652	+ def execute_children(self, batch):
	653	+ for item in self.children:
	654	+ item.execute(batch)
	655	+
	656	+
	657	+# _SandboxBatchCall()
	658	+#
	659	+# A call item in a command batch.
	660	+#
	661	+class _SandboxBatchCall(_SandboxBatchItem):
	662	+
	663	+ def __init__(self, callback):
	664	+ super().__init__()
	665	+
	666	+ self.callback = callback
	667	+
	668	+ def execute(self, batch):
	669	+ batch.execute_call(self)

buildstream/scriptelement.py

---

...	...	@@ -226,10 +226,11 @@ class ScriptElement(Element):
226	226	.format(build_dep.name), silent_nested=True):
227	227	build_dep.stage_dependency_artifacts(sandbox, Scope.RUN, path="/")
228	228
229		- for build_dep in self.dependencies(Scope.BUILD, recurse=False):
230		- with self.timed_activity("Integrating {}".format(build_dep.name), silent_nested=True):
231		- for dep in build_dep.dependencies(Scope.RUN):
232		- dep.integrate(sandbox)
	229	+ with sandbox.batch(SandboxFlags.NONE):
	230	+ for build_dep in self.dependencies(Scope.BUILD, recurse=False):
	231	+ with self.timed_activity("Integrating {}".format(build_dep.name), silent_nested=True):
	232	+ for dep in build_dep.dependencies(Scope.RUN):
	233	+ dep.integrate(sandbox)
233	234	else:
234	235	# If layout, follow its rules.
235	236	for item in self.__layout:
...	...	@@ -251,37 +252,40 @@ class ScriptElement(Element):
251	252	virtual_dstdir.descend(item['destination'].lstrip(os.sep).split(os.sep), create=True)
252	253	element.stage_dependency_artifacts(sandbox, Scope.RUN, path=item['destination'])
253	254
254		- for item in self.__layout:
	255	+ with sandbox.batch(SandboxFlags.NONE):
	256	+ for item in self.__layout:
255	257
256		- # Skip layout members which dont stage an element
257		- if not item['element']:
258		- continue
	258	+ # Skip layout members which dont stage an element
	259	+ if not item['element']:
	260	+ continue
259	261
260		- element = self.search(Scope.BUILD, item['element'])
	262	+ element = self.search(Scope.BUILD, item['element'])
261	263
262		- # Integration commands can only be run for elements staged to /
263		- if item['destination'] == '/':
264		- with self.timed_activity("Integrating {}".format(element.name),
265		- silent_nested=True):
266		- for dep in element.dependencies(Scope.RUN):
267		- dep.integrate(sandbox)
	264	+ # Integration commands can only be run for elements staged to /
	265	+ if item['destination'] == '/':
	266	+ with self.timed_activity("Integrating {}".format(element.name),
	267	+ silent_nested=True):
	268	+ for dep in element.dependencies(Scope.RUN):
	269	+ dep.integrate(sandbox)
268	270
269	271	install_root_path_components = self.__install_root.lstrip(os.sep).split(os.sep)
270	272	sandbox.get_virtual_directory().descend(install_root_path_components, create=True)
271	273
272	274	def assemble(self, sandbox):
273	275
274		- for groupname, commands in self.__commands.items():
275		- with self.timed_activity("Running '{}'".format(groupname)):
276		- for cmd in commands:
277		- self.status("Running command", detail=cmd)
278		- # Note the -e switch to 'sh' means to exit with an error
279		- # if any untested command fails.
280		- exitcode = sandbox.run(['sh', '-c', '-e', cmd + '\n'],
281		- SandboxFlags.ROOT_READ_ONLY if self.__root_read_only else 0)
282		- if exitcode != 0:
283		- raise ElementError("Command '{}' failed with exitcode {}".format(cmd, exitcode),
284		- collect=self.__install_root)
	276	+ flags = SandboxFlags.NONE
	277	+ if self.__root_read_only:
	278	+ flags |= SandboxFlags.ROOT_READ_ONLY
	279	+
	280	+ with sandbox.batch(flags, collect=self.__install_root):
	281	+ for groupname, commands in self.__commands.items():
	282	+ with sandbox.batch(flags, label="Running '{}'".format(groupname)):
	283	+ for cmd in commands:
	284	+ # Note the -e switch to 'sh' means to exit with an error
	285	+ # if any untested command fails.
	286	+ sandbox.run(['sh', '-c', '-e', cmd + '\n'],
	287	+ flags,
	288	+ label=cmd)
285	289
286	290	# Return where the result can be collected from
287	291	return self.__install_root

doc/source/format_project.rst

---

...	...	@@ -201,10 +201,10 @@ with an artifact share.
201	201	#
202	202	artifacts:
203	203	# A remote cache from which to download prebuilt artifacts
204		- - url: https://foo.com/artifacts:11001
	204	+ - url: https://foo.com:11001
205	205	server.cert: server.crt
206	206	# A remote cache from which to upload/download built/prebuilt artifacts
207		- - url: https://foo.com/artifacts:11002
	207	+ - url: https://foo.com:11002
208	208	server-cert: server.crt
209	209	client-cert: client.crt
210	210	client-key: client.key
...	...	@@ -231,10 +231,24 @@ using the `remote-execution` option:
231	231	remote-execution:
232	232
233	233	# A url defining a remote execution server
234		- url: http://buildserver.example.com:50051
	234	+ execution-service:
	235	+ url: http://buildserver.example.com:50051
	236	+ storage-service:
	237	+ - url: https://foo.com:11002/
	238	+ server-cert: server.crt
	239	+ client-cert: client.crt
	240	+ client-key: client.key
	241	+
	242	+The execution-service part of remote execution does not support encrypted
	243	+connections yet, so the protocol must always be http.
	244	+
	245	+storage-service specifies a remote CAS store and the parameters are the
	246	+same as those used to specify an :ref:`artifact server <artifacts>`.
235	247
236		-The url should contain a hostname and port separated by ':'. Only plain HTTP is
237		-currently suported (no HTTPS).
	248	+The storage service may be the same endpoint used for artifact
	249	+caching. Remote execution cannot work without push access to the
	250	+storage endpoint, so you must specify a client certificate and key,
	251	+and a server certificate.
238	252
239	253	The Remote Execution API can be found via https://github.com/bazelbuild/remote-apis.
240	254

setup.py

---

...	...	@@ -337,7 +337,14 @@ setup(name='BuildStream',
337	337	install_requires=[
338	338	'setuptools',
339	339	'psutil',
340		- 'ruamel.yaml < 0.15.52',
	340	+ # According to ruamel.yaml's PyPI page, we are suppose to use
	341	+ # "<=0.15" in production until 0.15 becomes API stable.
	342	+ # However we need ruamel.yaml 0.15.41 or greater for Python 3.7.
	343	+ # We know that ruamel.yaml 0.15.52 breaks API in a way that
	344	+ # is incompatible with BuildStream.
	345	+ #
	346	+ # See issues #571 and #790.
	347	+ 'ruamel.yaml >= 0.15.41, < 0.15.52',
341	348	'pluginbase',
342	349	'Click',
343	350	'jinja2 >= 2.10',

tests/integration/manual.py

---

...	...	@@ -128,3 +128,28 @@ def test_manual_element_noparallel(cli, tmpdir, datafiles):
128	128	assert text == """-j1 -Wall
129	129	2
130	130	"""
	131	+
	132	+
	133	+@pytest.mark.datafiles(DATA_DIR)
	134	+@pytest.mark.skipif(IS_LINUX and not HAVE_BWRAP, reason='Only available with bubblewrap on Linux')
	135	+def test_manual_element_logging(cli, tmpdir, datafiles):
	136	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	137	+ checkout = os.path.join(cli.directory, 'checkout')
	138	+ element_path = os.path.join(project, 'elements')
	139	+ element_name = 'import/import.bst'
	140	+
	141	+ create_manual_element(element_name, element_path, {
	142	+ 'configure-commands': ["echo configure"],
	143	+ 'build-commands': ["echo build"],
	144	+ 'install-commands': ["echo install"],
	145	+ 'strip-commands': ["echo strip"]
	146	+ }, {}, {})
	147	+
	148	+ res = cli.run(project=project, args=['build', element_name])
	149	+ assert res.exit_code == 0
	150	+
	151	+ # Verify that individual commands are logged
	152	+ assert "echo configure" in res.stderr
	153	+ assert "echo build" in res.stderr
	154	+ assert "echo install" in res.stderr
	155	+ assert "echo strip" in res.stderr

tests/integration/sandbox-bwrap.py

---

...	...	@@ -58,5 +58,5 @@ def test_sandbox_bwrap_return_subprocess(cli, tmpdir, datafiles):
58	58	})
59	59
60	60	result = cli.run(project=project, args=['build', element_name])
61		- result.assert_task_error(error_domain=ErrorDomain.ELEMENT, error_reason=None)
	61	+ result.assert_task_error(error_domain=ErrorDomain.SANDBOX, error_reason="command-failed")
62	62	assert "sandbox-bwrap/command-exit-42.bst|Command 'exit 42' failed with exitcode 42" in result.stderr

tests/sandboxes/remote-exec-config.py

---

	1	+import pytest
	2	+
	3	+import itertools
	4	+import os
	5	+
	6	+from buildstream import _yaml
	7	+from buildstream._exceptions import ErrorDomain, LoadErrorReason
	8	+
	9	+from tests.testutils.runcli import cli
	10	+
	11	+DATA_DIR = os.path.join(
	12	+ os.path.dirname(os.path.realpath(__file__)),
	13	+ "remote-exec-config"
	14	+)
	15	+
	16	+# Tests that we get a useful error message when supplying invalid
	17	+# remote execution configurations.
	18	+
	19	+
	20	+# Assert that if both 'url' (the old style) and 'execution-service' (the new style)
	21	+# are used at once, a LoadError results.
	22	+@pytest.mark.datafiles(DATA_DIR)
	23	+def test_old_and_new_configs(cli, datafiles):
	24	+ project = os.path.join(datafiles.dirname, datafiles.basename, 'missing-certs')
	25	+
	26	+ project_conf = {
	27	+ 'name': 'test',
	28	+
	29	+ 'remote-execution': {
	30	+ 'url': 'https://cache.example.com:12345',
	31	+ 'execution-service': {
	32	+ 'url': 'http://localhost:8088'
	33	+ },
	34	+ 'storage-service': {
	35	+ 'url': 'http://charactron:11001',
	36	+ }
	37	+ }
	38	+ }
	39	+ project_conf_file = os.path.join(project, 'project.conf')
	40	+ _yaml.dump(project_conf, project_conf_file)
	41	+
	42	+ # Use `pull` here to ensure we try to initialize the remotes, triggering the error
	43	+ #
	44	+ # This does not happen for a simple `bst show`.
	45	+ result = cli.run(project=project, args=['pull', 'element.bst'])
	46	+ result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA, "specify one")
	47	+
	48	+
	49	+# Assert that if either the client key or client cert is specified
	50	+# without specifying its counterpart, we get a comprehensive LoadError
	51	+# instead of an unhandled exception.
	52	+@pytest.mark.datafiles(DATA_DIR)
	53	+@pytest.mark.parametrize('config_key, config_value', [
	54	+ ('client-cert', 'client.crt'),
	55	+ ('client-key', 'client.key')
	56	+])
	57	+def test_missing_certs(cli, datafiles, config_key, config_value):
	58	+ project = os.path.join(datafiles.dirname, datafiles.basename, 'missing-certs')
	59	+
	60	+ project_conf = {
	61	+ 'name': 'test',
	62	+
	63	+ 'remote-execution': {
	64	+ 'execution-service': {
	65	+ 'url': 'http://localhost:8088'
	66	+ },
	67	+ 'storage-service': {
	68	+ 'url': 'http://charactron:11001',
	69	+ config_key: config_value,
	70	+ }
	71	+ }
	72	+ }
	73	+ project_conf_file = os.path.join(project, 'project.conf')
	74	+ _yaml.dump(project_conf, project_conf_file)
	75	+
	76	+ # Use `pull` here to ensure we try to initialize the remotes, triggering the error
	77	+ #
	78	+ # This does not happen for a simple `bst show`.
	79	+ result = cli.run(project=project, args=['show', 'element.bst'])
	80	+ result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA, "Your config is missing")
	81	+
	82	+
	83	+# Assert that if incomplete information is supplied we get a sensible error message.
	84	+@pytest.mark.datafiles(DATA_DIR)
	85	+def test_empty_config(cli, datafiles):
	86	+ project = os.path.join(datafiles.dirname, datafiles.basename, 'missing-certs')
	87	+
	88	+ project_conf = {
	89	+ 'name': 'test',
	90	+
	91	+ 'remote-execution': {
	92	+ }
	93	+ }
	94	+ project_conf_file = os.path.join(project, 'project.conf')
	95	+ _yaml.dump(project_conf, project_conf_file)
	96	+
	97	+ # Use `pull` here to ensure we try to initialize the remotes, triggering the error
	98	+ #
	99	+ # This does not happen for a simple `bst show`.
	100	+ result = cli.run(project=project, args=['pull', 'element.bst'])
	101	+ result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA, "specify one")

tests/sandboxes/remote-exec-config/missing-certs/certificates/client.crt

---

tests/sandboxes/remote-exec-config/missing-certs/certificates/client.key

---

tests/sandboxes/remote-exec-config/missing-certs/element.bst

---

1

+kind: autotools

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.