[Notes] [Git][BuildStream/buildstream][lachlanmackenzie/debug_benchmark_failure] Deleted 15 commits: format_project.rst: Add docs for OptionOS

[Lachlan <gitlab mg gitlab com>]

Title: GitLab

Lachlan pushed to branch lachlanmackenzie/debug_benchmark_failure at BuildStream / buildstream

WARNING: The push did not contain any new commits, but force pushed to delete the commits and changes below.

Deleted commits:

• 23c27580
  by Raoul Hidalgo Charman at 2018-12-05T14:58:53Z

  format_project.rst: Add docs for OptionOS
  

• eb1ed410
  by Jürg Billeter at 2018-12-05T15:28:52Z

  Merge branch 'raoul/775-execution-environment-reqs' into 'master'
  
  Execution environment reqs
  
  Closes #775
  
  See merge request BuildStream/buildstream!969

• 9ef1a8a2
  by Raoul Hidalgo Charman at 2018-12-05T15:32:22Z

  utils.py: Add message digest function
  
  Get cascache.py to use this
  

• 4a56bf2a
  by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

  _sandboxremote.py: Add checks to action cache before attempting to push.
  
  Stops unneccesary pushing of builds that have already been built, just checks
  the action cache to begin with.
  
  Fixes #628
  

• 3641b6f6
  by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

  _sandboxremote.py: Add action-cache-service endpoint
  
  The endpoint may not necessarily be the same as execution-service endpoint.
  
  Part of #628
  

• 9d77351f
  by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

  format_project.rst: Add docs for action-cache-service
  
  Part of #628
  

• a53d6d1f
  by Raoul Hidalgo Charman at 2018-12-05T16:36:00Z

  Merge branch 'raoul/628-RE-flow-optimisation' into 'master'
  
  Remote-execution client flow optimisation
  
  Closes #628
  
  See merge request BuildStream/buildstream!982

• 4688b2c9
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  tests/sources/git.py: Refactor ref-not-in-track test to use parameterization
  
  Instead of duplicating the whole test body, run it twice while
  checking for a warning or an error depending on the parameter.
  

• 03028c7f
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  types.py: Moving CoreWarnings to the types.py module
  
  This also ensures it is exposed via the main buildstream __init__.py
  file, fixes the imports of CoreWarnings, and adjusts the documentation
  links to point to the right place.
  

• 3b7f8df8
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  Source: Fix typo in documentation, and make it a link to `bst track`
  
  Now the commands have documentation linkability so we can link to
  them when referring to them anywhere in the docs. This reference
  was previously referring to a bogus non-existent `build-stream track`,
  maybe this dates way back to the days before we named the frontend `bst`.
  

• e5c0f067
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  git source plugin: Fixing documentation linking and typo
  
  Fixes the configurable warning part of this to properly link
  to the project.conf documentation describing what configurable
  warnings actually are.
  
  Also fix the plugin raise the actual configurable warning which
  it advertizes that it raises; which is 'inconsistent-submodule',
  not 'inconsistent-submodules'.
  

• 205c27d8
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  git source plugin: Rename symbol for consistency and readability
  
  Change 'INCONSISTENT_SUBMODULE' for 'WARN_INCONSISTENT_SUBMODULE' to
  make things a bit more readable and comprehensive.
  

• bea4d4f5
  by Tristan Van Berkom at 2018-12-06T09:37:50Z

  Merge branch 'tristan/misc-cleanup' into 'master'
  
  Misc cleanups
  
  See merge request BuildStream/buildstream!993

• b1d2f001
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  source.py: Add new delegate method validate_cache()
  
  This is guaranteed to be called only once for a given session once the
  sources are known to be Consistency.CACHED, if source tracking is enabled
  in the session for this source, then this will only be called if the
  sources become cached after tracking completes.
  

• ee7fc47f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  git source plugin: Implementing submodule warnings
  
   o Unlisted submodule warning
  
     Now the git plugin will issue a configurable warning if a submodule
     exists and is used (checking out the submodule is not disabled),
     but is not specified in the source configuration.
  
   o Invalid submodule warning
  
     Now the git source plugin will issue a warning if the configuration
     specified a submodule which does not exist in the underlying git repository.
  
  As a side effect, this patch also changes the flow control of the git
  plugin such that submodules which are explicitly set to not be
  checked out, are also not fetched but instead ignored completely.
  

14 changed files:

• buildstream/__init__.py
• buildstream/_artifactcache/cascache.py
• buildstream/_loader/loader.py
• buildstream/_project.py
• buildstream/_versions.py
• buildstream/element.py
• buildstream/plugin.py
• buildstream/plugins/sources/git.py
• buildstream/sandbox/_sandboxremote.py
• buildstream/source.py
• buildstream/types.py
• buildstream/utils.py
• doc/source/format_project.rst
• tests/sources/git.py

Changes:

buildstream/__init__.py

---

...	...	@@ -28,7 +28,7 @@ if "_BST_COMPLETION" not in os.environ:
28	28
29	29	from .utils import UtilError, ProgramNotFoundError
30	30	from .sandbox import Sandbox, SandboxFlags, SandboxCommandError
31		- from .types import Scope, Consistency
	31	+ from .types import Scope, Consistency, CoreWarnings
32	32	from .plugin import Plugin
33	33	from .source import Source, SourceError, SourceFetcher
34	34	from .element import Element, ElementError

buildstream/_artifactcache/cascache.py

---

...	...	@@ -427,10 +427,7 @@ class CASCache():
427	427	def push_message(self, remote, message):
428	428
429	429	message_buffer = message.SerializeToString()
430		- message_sha = hashlib.sha256(message_buffer)
431		- message_digest = remote_execution_pb2.Digest()
432		- message_digest.hash = message_sha.hexdigest()
433		- message_digest.size_bytes = len(message_buffer)
	430	+ message_digest = utils._message_digest(message_buffer)
434	431
435	432	remote.init()
436	433

buildstream/_loader/loader.py

---

...	...	@@ -36,7 +36,7 @@ from .types import Symbol, Dependency
36	36	from .loadelement import LoadElement
37	37	from . import MetaElement
38	38	from . import MetaSource
39		-from ..plugin import CoreWarnings
	39	+from ..types import CoreWarnings
40	40	from .._message import Message, MessageType
41	41
42	42

buildstream/_project.py

---

...	...	@@ -33,7 +33,7 @@ from ._artifactcache import ArtifactCache
33	33	from .sandbox import SandboxRemote
34	34	from ._elementfactory import ElementFactory
35	35	from ._sourcefactory import SourceFactory
36		-from .plugin import CoreWarnings
	36	+from .types import CoreWarnings
37	37	from ._projectrefs import ProjectRefs, ProjectRefStorage
38	38	from ._versions import BST_FORMAT_VERSION
39	39	from ._loader import Loader

buildstream/_versions.py

---

...	...	@@ -23,7 +23,7 @@
23	23	# This version is bumped whenever enhancements are made
24	24	# to the `project.conf` format or the core element format.
25	25	#
26		-BST_FORMAT_VERSION = 19
	26	+BST_FORMAT_VERSION = 20
27	27
28	28
29	29	# The base BuildStream artifact version

buildstream/element.py

---

...	...	@@ -96,10 +96,9 @@ from . import _cachekey
96	96	from . import _signals
97	97	from . import _site
98	98	from ._platform import Platform
99		-from .plugin import CoreWarnings
100	99	from .sandbox._config import SandboxConfig
101	100	from .sandbox._sandboxremote import SandboxRemote
102		-from .types import _KeyStrength
	101	+from .types import _KeyStrength, CoreWarnings
103	102
104	103	from .storage.directory import Directory
105	104	from .storage._filebaseddirectory import FileBasedDirectory

buildstream/plugin.py

---

...	...	@@ -119,6 +119,7 @@ from . import _yaml
119	119	from . import utils
120	120	from ._exceptions import PluginError, ImplError
121	121	from ._message import Message, MessageType
	122	+from .types import CoreWarnings
122	123
123	124
124	125	class Plugin():
...	...	@@ -766,38 +767,6 @@ class Plugin():
766	767	return self.name
767	768
768	769
769		-class CoreWarnings():
770		- """CoreWarnings()
771		-
772		- Some common warnings which are raised by core functionalities within BuildStream are found in this class.
773		- """
774		-
775		- OVERLAPS = "overlaps"
776		- """
777		- This warning will be produced when buildstream detects an overlap on an element
778		- which is not whitelisted. See :ref:`Overlap Whitelist <public_overlap_whitelist>`
779		- """
780		-
781		- REF_NOT_IN_TRACK = "ref-not-in-track"
782		- """
783		- This warning will be produced when a source is configured with a reference
784		- which is found to be invalid based on the configured track
785		- """
786		-
787		- BAD_ELEMENT_SUFFIX = "bad-element-suffix"
788		- """
789		- This warning will be produced when an element whose name does not end in .bst
790		- is referenced either on the command line or by another element
791		- """
792		-
793		-
794		-__CORE_WARNINGS = [
795		- value
796		- for name, value in CoreWarnings.__dict__.items()
797		- if not name.startswith("__")
798		-]
799		-
800		-
801	770	# Hold on to a lookup table by counter of all instantiated plugins.
802	771	# We use this to send the id back from child processes so we can lookup
803	772	# corresponding element/source in the master process.
...	...	@@ -828,6 +797,24 @@ def _plugin_lookup(unique_id):
828	797	return __PLUGINS_TABLE[unique_id]
829	798
830	799
	800	+# No need for unregister, WeakValueDictionary() will remove entries
	801	+# in itself when the referenced plugins are garbage collected.
	802	+def _plugin_register(plugin):
	803	+ global __PLUGINS_UNIQUE_ID # pylint: disable=global-statement
	804	+ __PLUGINS_UNIQUE_ID += 1
	805	+ __PLUGINS_TABLE[__PLUGINS_UNIQUE_ID] = plugin
	806	+ return __PLUGINS_UNIQUE_ID
	807	+
	808	+
	809	+# A local table for _prefix_warning()
	810	+#
	811	+__CORE_WARNINGS = [
	812	+ value
	813	+ for name, value in CoreWarnings.__dict__.items()
	814	+ if not name.startswith("__")
	815	+]
	816	+
	817	+
831	818	# _prefix_warning():
832	819	#
833	820	# Prefix a warning with the plugin kind. CoreWarnings are not prefixed.
...	...	@@ -843,12 +830,3 @@ def _prefix_warning(plugin, warning):
843	830	if any((warning is core_warning for core_warning in __CORE_WARNINGS)):
844	831	return warning
845	832	return "{}:{}".format(plugin.get_kind(), warning)
846		-
847		-
848		-# No need for unregister, WeakValueDictionary() will remove entries
849		-# in itself when the referenced plugins are garbage collected.
850		-def _plugin_register(plugin):
851		- global __PLUGINS_UNIQUE_ID # pylint: disable=global-statement
852		- __PLUGINS_UNIQUE_ID += 1
853		- __PLUGINS_TABLE[__PLUGINS_UNIQUE_ID] = plugin
854		- return __PLUGINS_UNIQUE_ID

buildstream/plugins/sources/git.py

---

...	...	@@ -131,13 +131,29 @@ details on common configuration options for sources.
131	131
132	132	**Configurable Warnings:**
133	133
134		-This plugin provides the following configurable warnings:
	134	+This plugin provides the following :ref:`configurable warnings <configurable_warnings>`:
135	135
136		-- 'git:inconsistent-submodule' - A submodule was found to be missing from the underlying git repository.
	136	+- ``git:inconsistent-submodule`` - A submodule present in the git repository's .gitmodules was never
	137	+ added with `git submodule add`.
137	138
138		-This plugin also utilises the following configurable core plugin warnings:
	139	+- ``git:unlisted-submodule`` - A submodule is present in the git repository but was not specified in
	140	+ the source configuration and was not disabled for checkout.
139	141
140		-- 'ref-not-in-track' - The provided ref was not found in the provided track in the element's git repository.
	142	+ .. note::
	143	+
	144	+ The ``git:unlisted-submodule`` warning is available since :ref:`format version 20 <project_format_version>`
	145	+
	146	+- ``git:invalid-submodule`` - A submodule is specified in the source configuration but does not exist
	147	+ in the repository.
	148	+
	149	+ .. note::
	150	+
	151	+ The ``git:invalid-submodule`` warning is available since :ref:`format version 20 <project_format_version>`
	152	+
	153	+This plugin also utilises the following configurable :class:`core warnings <buildstream.types.CoreWarnings>`:
	154	+
	155	+- :attr:`ref-not-in-track <buildstream.types.CoreWarnings.REF_NOT_IN_TRACK>` - The provided ref was not
	156	+ found in the provided track in the element's git repository.
141	157	"""
142	158
143	159	import os
...	...	@@ -149,15 +165,16 @@ from tempfile import TemporaryFile
149	165
150	166	from configparser import RawConfigParser
151	167
152		-from buildstream import Source, SourceError, Consistency, SourceFetcher
	168	+from buildstream import Source, SourceError, Consistency, SourceFetcher, CoreWarnings
153	169	from buildstream import utils
154		-from buildstream.plugin import CoreWarnings
155	170	from buildstream.utils import move_atomic, DirectoryExistsError
156	171
157	172	GIT_MODULES = '.gitmodules'
158	173
159	174	# Warnings
160		-INCONSISTENT_SUBMODULE = "inconsistent-submodules"
	175	+WARN_INCONSISTENT_SUBMODULE = "inconsistent-submodule"
	176	+WARN_UNLISTED_SUBMODULE = "unlisted-submodule"
	177	+WARN_INVALID_SUBMODULE = "invalid-submodule"
161	178
162	179
163	180	# Because of handling of submodules, we maintain a GitMirror
...	...	@@ -408,7 +425,8 @@ class GitMirror(SourceFetcher):
408	425	"underlying git repository with `git submodule add`."
409	426
410	427	self.source.warn("{}: Ignoring inconsistent submodule '{}'"
411		- .format(self.source, submodule), detail=detail, warning_token=INCONSISTENT_SUBMODULE)
	428	+ .format(self.source, submodule), detail=detail,
	429	+ warning_token=WARN_INCONSISTENT_SUBMODULE)
412	430
413	431	return None
414	432
...	...	@@ -679,13 +697,7 @@ class GitSource(Source):
679	697	with self.timed_activity("Staging {}".format(self.mirror.url), silent_nested=True):
680	698	self.mirror.stage(directory, track=(self.tracking if not self.tracked else None))
681	699	for mirror in self.submodules:
682		- if mirror.path in self.submodule_checkout_overrides:
683		- checkout = self.submodule_checkout_overrides[mirror.path]
684		- else:
685		- checkout = self.checkout_submodules
686		-
687		- if checkout:
688		- mirror.stage(directory)
	700	+ mirror.stage(directory)
689	701
690	702	def get_source_fetchers(self):
691	703	yield self.mirror
...	...	@@ -693,6 +705,48 @@ class GitSource(Source):
693	705	for submodule in self.submodules:
694	706	yield submodule
695	707
	708	+ def validate_cache(self):
	709	+ discovered_submodules = {}
	710	+ unlisted_submodules = []
	711	+ invalid_submodules = []
	712	+
	713	+ for path, url in self.mirror.submodule_list():
	714	+ discovered_submodules[path] = url
	715	+ if self.ignore_submodule(path):
	716	+ continue
	717	+
	718	+ override_url = self.submodule_overrides.get(path)
	719	+ if not override_url:
	720	+ unlisted_submodules.append((path, url))
	721	+
	722	+ # Warn about submodules which are explicitly configured but do not exist
	723	+ for path, url in self.submodule_overrides.items():
	724	+ if path not in discovered_submodules:
	725	+ invalid_submodules.append((path, url))
	726	+
	727	+ if invalid_submodules:
	728	+ detail = []
	729	+ for path, url in invalid_submodules:
	730	+ detail.append(" Submodule URL '{}' at path '{}'".format(url, path))
	731	+
	732	+ self.warn("{}: Invalid submodules specified".format(self),
	733	+ warning_token=WARN_INVALID_SUBMODULE,
	734	+ detail="The following submodules are specified in the source "
	735	+ "description but do not exist according to the repository\n\n" +
	736	+ "\n".join(detail))
	737	+
	738	+ # Warn about submodules which exist but have not been explicitly configured
	739	+ if unlisted_submodules:
	740	+ detail = []
	741	+ for path, url in unlisted_submodules:
	742	+ detail.append(" Submodule URL '{}' at path '{}'".format(url, path))
	743	+
	744	+ self.warn("{}: Unlisted submodules exist".format(self),
	745	+ warning_token=WARN_UNLISTED_SUBMODULE,
	746	+ detail="The following submodules exist but are not specified " +
	747	+ "in the source description\n\n" +
	748	+ "\n".join(detail))
	749	+
696	750	###########################################################
697	751	# Local Functions #
698	752	###########################################################
...	...	@@ -717,12 +771,12 @@ class GitSource(Source):
717	771	self.mirror.ensure()
718	772	submodules = []
719	773
720		- # XXX Here we should issue a warning if either:
721		- # A.) A submodule exists but is not defined in the element configuration
722		- # B.) The element configuration configures submodules which dont exist at the current ref
723		- #
724	774	for path, url in self.mirror.submodule_list():
725	775
	776	+ # Completely ignore submodules which are disabled for checkout
	777	+ if self.ignore_submodule(path):
	778	+ continue
	779	+
726	780	# Allow configuration to override the upstream
727	781	# location of the submodules.
728	782	override_url = self.submodule_overrides.get(path)
...	...	@@ -746,6 +800,16 @@ class GitSource(Source):
746	800	tags.append((tag, commit_ref, annotated))
747	801	return tags
748	802
	803	+ # Checks whether the plugin configuration has explicitly
	804	+ # configured this submodule to be ignored
	805	+ def ignore_submodule(self, path):
	806	+ try:
	807	+ checkout = self.submodule_checkout_overrides[path]
	808	+ except KeyError:
	809	+ checkout = self.checkout_submodules
	810	+
	811	+ return not checkout
	812	+
749	813
750	814	# Plugin entry point
751	815	def setup():

buildstream/sandbox/_sandboxremote.py

---

...	...	@@ -26,6 +26,8 @@ from functools import partial
26	26
27	27	import grpc
28	28
	29	+from .. import utils
	30	+from .._message import Message, MessageType
29	31	from . import Sandbox, SandboxCommandError
30	32	from .sandbox import _SandboxBatch
31	33	from ..storage._filebaseddirectory import FileBasedDirectory
...	...	@@ -39,7 +41,7 @@ from .._protos.google.longrunning import operations_pb2, operations_pb2_grpc
39	41	from .._artifactcache.cascache import CASRemote, CASRemoteSpec
40	42
41	43
42		-class RemoteExecutionSpec(namedtuple('RemoteExecutionSpec', 'exec_service storage_service')):
	44	+class RemoteExecutionSpec(namedtuple('RemoteExecutionSpec', 'exec_service storage_service action_service')):
43	45	pass
44	46
45	47
...	...	@@ -59,6 +61,10 @@ class SandboxRemote(Sandbox):
59	61
60	62	self.storage_url = config.storage_service['url']
61	63	self.exec_url = config.exec_service['url']
	64	+ if config.action_service:
	65	+ self.action_url = config.action_service['url']
	66	+ else:
	67	+ self.action_url = None
62	68
63	69	self.storage_remote_spec = CASRemoteSpec(self.storage_url, push=True,
64	70	server_cert=config.storage_service['server-cert'],
...	...	@@ -66,6 +72,9 @@ class SandboxRemote(Sandbox):
66	72	client_cert=config.storage_service['client-cert'])
67	73	self.operation_name = None
68	74
	75	+ def info(self, msg):
	76	+ self._get_context().message(Message(None, MessageType.INFO, msg))
	77	+
69	78	@staticmethod
70	79	def specs_from_config_node(config_node, basedir):
71	80
...	...	@@ -88,12 +97,19 @@ class SandboxRemote(Sandbox):
88	97
89	98	tls_keys = ['client-key', 'client-cert', 'server-cert']
90	99
91		- _yaml.node_validate(remote_config, ['execution-service', 'storage-service', 'url'])
	100	+ _yaml.node_validate(
	101	+ remote_config,
	102	+ ['execution-service', 'storage-service', 'url', 'action-cache-service'])
92	103	remote_exec_service_config = require_node(remote_config, 'execution-service')
93	104	remote_exec_storage_config = require_node(remote_config, 'storage-service')
	105	+ remote_exec_action_config = remote_config.get('action-cache-service')
94	106
95	107	_yaml.node_validate(remote_exec_service_config, ['url'])
96	108	_yaml.node_validate(remote_exec_storage_config, ['url'] + tls_keys)
	109	+ if remote_exec_action_config:
	110	+ _yaml.node_validate(remote_exec_action_config, ['url'])
	111	+ else:
	112	+ remote_config['action-service'] = None
97	113
98	114	if 'url' in remote_config:
99	115	if 'execution-service' not in remote_config:
...	...	@@ -114,59 +130,17 @@ class SandboxRemote(Sandbox):
114	130	"remote-execution configuration. Your config is missing '{}'."
115	131	.format(str(provenance), tls_keys, key))
116	132
117		- spec = RemoteExecutionSpec(remote_config['execution-service'], remote_config['storage-service'])
	133	+ spec = RemoteExecutionSpec(remote_config['execution-service'],
	134	+ remote_config['storage-service'],
	135	+ remote_config['action-cache-service'])
118	136	return spec
119	137
120		- def run_remote_command(self, command, input_root_digest, working_directory, environment):
	138	+ def run_remote_command(self, channel, action_digest):
121	139	# Sends an execution request to the remote execution server.
122	140	#
123	141	# This function blocks until it gets a response from the server.
124		- #
125		- environment_variables = [remote_execution_pb2.Command.
126		- EnvironmentVariable(name=k, value=v)
127		- for (k, v) in environment.items()]
128		-
129		- config = self._get_config()
130		- platform = remote_execution_pb2.Platform()
131		- platform.properties.extend([remote_execution_pb2.Platform.
132		- Property(name="OSFamily", value=config.build_os),
133		- remote_execution_pb2.Platform.
134		- Property(name="ISA", value=config.build_arch)])
135		-
136		- # Create and send the Command object.
137		- remote_command = remote_execution_pb2.Command(arguments=command,
138		- working_directory=working_directory,
139		- environment_variables=environment_variables,
140		- output_files=[],
141		- output_directories=[self._output_directory],
142		- platform=platform)
143		- context = self._get_context()
144		- cascache = context.get_cascache()
145		- casremote = CASRemote(self.storage_remote_spec)
146		-
147		- # Upload the Command message to the remote CAS server
148		- command_digest = cascache.push_message(casremote, remote_command)
149		-
150		- # Create and send the action.
151		- action = remote_execution_pb2.Action(command_digest=command_digest,
152		- input_root_digest=input_root_digest,
153		- timeout=None,
154		- do_not_cache=False)
155		-
156		- # Upload the Action message to the remote CAS server
157		- action_digest = cascache.push_message(casremote, action)
158		-
159		- # Next, try to create a communication channel to the BuildGrid server.
160		- url = urlparse(self.exec_url)
161		- if not url.port:
162		- raise SandboxError("You must supply a protocol and port number in the execution-service url, "
163		- "for example: http://buildservice:50051.")
164		- if url.scheme == 'http':
165		- channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
166		- else:
167		- raise SandboxError("Remote execution currently only supports the 'http' protocol "
168		- "and '{}' was supplied.".format(url.scheme))
169	142
	143	+ # Try to create a communication channel to the BuildGrid server.
170	144	stub = remote_execution_pb2_grpc.ExecutionStub(channel)
171	145	request = remote_execution_pb2.ExecuteRequest(action_digest=action_digest,
172	146	skip_cache_lookup=False)
...	...	@@ -286,13 +260,12 @@ class SandboxRemote(Sandbox):
286	260	# to replace the sandbox's virtual directory with that. Creating a new virtual directory object
287	261	# from another hash will be interesting, though...
288	262
289		- new_dir = CasBasedDirectory(self._get_context().artifactcache.cas, ref=dir_digest)
	263	+ new_dir = CasBasedDirectory(context.artifactcache.cas, ref=dir_digest)
290	264	self._set_virtual_directory(new_dir)
291	265
292	266	def _run(self, command, flags, *, cwd, env):
293		- # Upload sources
	267	+ # set up virtual dircetory
294	268	upload_vdir = self.get_virtual_directory()
295		-
296	269	cascache = self._get_context().get_cascache()
297	270	if isinstance(upload_vdir, FileBasedDirectory):
298	271	# Make a new temporary directory to put source in
...	...	@@ -301,16 +274,111 @@ class SandboxRemote(Sandbox):
301	274
302	275	upload_vdir.recalculate_hash()
303	276
304		- casremote = CASRemote(self.storage_remote_spec)
305		- # Now, push that key (without necessarily needing a ref) to the remote.
	277	+ # Generate action_digest first
	278	+ input_root_digest = upload_vdir.ref
	279	+ command_proto = self._create_command(command, cwd, env)
	280	+ command_digest = utils._message_digest(command_proto.SerializeToString())
	281	+ action = remote_execution_pb2.Action(command_digest=command_digest,
	282	+ input_root_digest=input_root_digest)
	283	+ action_digest = utils._message_digest(action.SerializeToString())
	284	+
	285	+ # Next, try to create a communication channel to the BuildGrid server.
	286	+ url = urlparse(self.exec_url)
	287	+ if not url.port:
	288	+ raise SandboxError("You must supply a protocol and port number in the execution-service url, "
	289	+ "for example: http://buildservice:50051.")
	290	+ if url.scheme == 'http':
	291	+ channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
	292	+ else:
	293	+ raise SandboxError("Remote execution currently only supports the 'http' protocol "
	294	+ "and '{}' was supplied.".format(url.scheme))
	295	+
	296	+ # check action cache download and download if there
	297	+ action_result = self._check_action_cache(action_digest)
	298	+
	299	+ if not action_result:
	300	+ casremote = CASRemote(self.storage_remote_spec)
	301	+
	302	+ # Now, push that key (without necessarily needing a ref) to the remote.
	303	+ try:
	304	+ cascache.push_directory(casremote, upload_vdir)
	305	+ except grpc.RpcError as e:
	306	+ raise SandboxError("Failed to push source directory to remote: {}".format(e)) from e
	307	+
	308	+ if not cascache.verify_digest_on_remote(casremote, upload_vdir.ref):
	309	+ raise SandboxError("Failed to verify that source has been pushed to the remote artifact cache.")
	310	+
	311	+ # Push command and action
	312	+ try:
	313	+ cascache.push_message(casremote, command_proto)
	314	+ except grpc.RpcError as e:
	315	+ raise SandboxError("Failed to push command to remote: {}".format(e))
	316	+
	317	+ try:
	318	+ cascache.push_message(casremote, action)
	319	+ except grpc.RpcError as e:
	320	+ raise SandboxError("Failed to push action to remote: {}".format(e))
	321	+
	322	+ # Now request to execute the action
	323	+ operation = self.run_remote_command(channel, action_digest)
	324	+ action_result = self._extract_action_result(operation)
	325	+
	326	+ if action_result.exit_code != 0:
	327	+ # A normal error during the build: the remote execution system
	328	+ # has worked correctly but the command failed.
	329	+ # action_result.stdout and action_result.stderr also contains
	330	+ # build command outputs which we ignore at the moment.
	331	+ return action_result.exit_code
	332	+
	333	+ # Get output of build
	334	+ self.process_job_output(action_result.output_directories, action_result.output_files)
	335	+
	336	+ return 0
	337	+
	338	+ def _check_action_cache(self, action_digest):
	339	+ # Checks the action cache to see if this artifact has already been built
	340	+ #
	341	+ # Should return either the action response or None if not found, raise
	342	+ # Sandboxerror if other grpc error was raised
	343	+ if not self.action_url:
	344	+ return None
	345	+ url = urlparse(self.action_url)
	346	+ if not url.port:
	347	+ raise SandboxError("You must supply a protocol and port number in the action-cache-service url, "
	348	+ "for example: http://buildservice:50051.")
	349	+ if not url.scheme == "http":
	350	+ raise SandboxError("Currently only support http for the action cache"
	351	+ "and {} was supplied".format(url.scheme))
	352	+
	353	+ channel = grpc.insecure_channel('{}:{}'.format(url.hostname, url.port))
	354	+ request = remote_execution_pb2.GetActionResultRequest(action_digest=action_digest)
	355	+ stub = remote_execution_pb2_grpc.ActionCacheStub(channel)
306	356	try:
307		- cascache.push_directory(casremote, upload_vdir)
	357	+ result = stub.GetActionResult(request)
308	358	except grpc.RpcError as e:
309		- raise SandboxError("Failed to push source directory to remote: {}".format(e)) from e
	359	+ if e.code() != grpc.StatusCode.NOT_FOUND:
	360	+ raise SandboxError("Failed to query action cache: {} ({})"
	361	+ .format(e.code(), e.details()))
	362	+ else:
	363	+ return None
	364	+ else:
	365	+ self.info("Action result found in action cache")
	366	+ return result
310	367
311		- # Now transmit the command to execute
312		- operation = self.run_remote_command(command, upload_vdir.ref, cwd, env)
	368	+ def _create_command(self, command, working_directory, environment):
	369	+ # Creates a command proto
	370	+ environment_variables = [remote_execution_pb2.Command.
	371	+ EnvironmentVariable(name=k, value=v)
	372	+ for (k, v) in environment.items()]
	373	+ return remote_execution_pb2.Command(arguments=command,
	374	+ working_directory=working_directory,
	375	+ environment_variables=environment_variables,
	376	+ output_files=[],
	377	+ output_directories=[self._output_directory],
	378	+ platform=None)
313	379
	380	+ @staticmethod
	381	+ def _extract_action_result(operation):
314	382	if operation is None:
315	383	# Failure of remote execution, usually due to an error in BuildStream
316	384	raise SandboxError("No response returned from server")
...	...	@@ -331,18 +399,7 @@ class SandboxRemote(Sandbox):
331	399	else:
332	400	raise SandboxError("Remote server failed at executing the build request.")
333	401
334		- action_result = execution_response.result
335		-
336		- if action_result.exit_code != 0:
337		- # A normal error during the build: the remote execution system
338		- # has worked correctly but the command failed.
339		- # action_result.stdout and action_result.stderr also contains
340		- # build command outputs which we ignore at the moment.
341		- return action_result.exit_code
342		-
343		- self.process_job_output(action_result.output_directories, action_result.output_files)
344		-
345		- return 0
	402	+ return execution_response.result
346	403
347	404	def _create_batch(self, main_group, flags, *, collect=None):
348	405	return _SandboxRemoteBatch(self, main_group, flags, collect=collect)

buildstream/source.py

---

...	...	@@ -102,6 +102,11 @@ these methods are mandatory to implement.
102	102	submodules). For details on how to define a SourceFetcher, see
103	103	:ref:`SourceFetcher <core_source_fetcher>`.
104	104
	105	+* :func:`Source.validate_cache() <buildstream.source.Source.validate_cache>`
	106	+
	107	+ Perform any validations which require the sources to be cached.
	108	+
	109	+ **Optional**: This is completely optional and will do nothing if left unimplemented.
105	110
106	111	Accessing previous sources
107	112	--------------------------
...	...	@@ -391,7 +396,8 @@ class Source(Plugin):
391	396
392	397	If the backend in question supports resolving references from
393	398	a symbolic tracking branch or tag, then this should be implemented
394		- to perform this task on behalf of ``build-stream track`` commands.
	399	+ to perform this task on behalf of :ref:`bst track <invoking_track>`
	400	+ commands.
395	401
396	402	This usually requires fetching new content from a remote origin
397	403	to see if a new ref has appeared for your branch or tag. If the
...	...	@@ -479,9 +485,22 @@ class Source(Plugin):
479	485
480	486	*Since: 1.2*
481	487	"""
482		-
483	488	return []
484	489
	490	+ def validate_cache(self):
	491	+ """Implement any validations once we know the sources are cached
	492	+
	493	+ This is guaranteed to be called only once for a given session
	494	+ once the sources are known to be
	495	+ :attr:`Consistency.CACHED <buildstream.types.Consistency.CACHED>`,
	496	+ if source tracking is enabled in the session for this source,
	497	+ then this will only be called if the sources become cached after
	498	+ tracking completes.
	499	+
	500	+ *Since: 1.4*
	501	+ """
	502	+ pass
	503	+
485	504	#############################################################
486	505	# Public Methods #
487	506	#############################################################
...	...	@@ -658,6 +677,11 @@ class Source(Plugin):
658	677	with context.silence():
659	678	self.__consistency = self.get_consistency() # pylint: disable=assignment-from-no-return
660	679
	680	+ # Give the Source an opportunity to validate the cached
	681	+ # sources as soon as the Source becomes Consistency.CACHED.
	682	+ if self.__consistency == Consistency.CACHED:
	683	+ self.validate_cache()
	684	+
661	685	# Return cached consistency
662	686	#
663	687	def _get_consistency(self):

buildstream/types.py

---

...	...	@@ -81,6 +81,31 @@ class Consistency():
81	81	"""
82	82
83	83
	84	+class CoreWarnings():
	85	+ """CoreWarnings()
	86	+
	87	+ Some common warnings which are raised by core functionalities within BuildStream are found in this class.
	88	+ """
	89	+
	90	+ OVERLAPS = "overlaps"
	91	+ """
	92	+ This warning will be produced when buildstream detects an overlap on an element
	93	+ which is not whitelisted. See :ref:`Overlap Whitelist <public_overlap_whitelist>`
	94	+ """
	95	+
	96	+ REF_NOT_IN_TRACK = "ref-not-in-track"
	97	+ """
	98	+ This warning will be produced when a source is configured with a reference
	99	+ which is found to be invalid based on the configured track
	100	+ """
	101	+
	102	+ BAD_ELEMENT_SUFFIX = "bad-element-suffix"
	103	+ """
	104	+ This warning will be produced when an element whose name does not end in .bst
	105	+ is referenced either on the command line or by another element
	106	+ """
	107	+
	108	+
84	109	# _KeyStrength():
85	110	#
86	111	# Strength of cache key

buildstream/utils.py

---

...	...	@@ -41,6 +41,7 @@ import psutil
41	41
42	42	from . import _signals
43	43	from ._exceptions import BstError, ErrorDomain
	44	+from ._protos.build.bazel.remote.execution.v2 import remote_execution_pb2
44	45
45	46	# The magic number for timestamps: 2011-11-11 11:11:11
46	47	_magic_timestamp = calendar.timegm([2011, 11, 11, 11, 11, 11])
...	...	@@ -1242,3 +1243,19 @@ def _deduplicate(iterable, key=None):
1242	1243	def _get_link_mtime(path):
1243	1244	path_stat = os.lstat(path)
1244	1245	return path_stat.st_mtime
	1246	+
	1247	+
	1248	+# _message_digest()
	1249	+#
	1250	+# Args:
	1251	+# message_buffer (str): String to create digest of
	1252	+#
	1253	+# Returns:
	1254	+# (remote_execution_pb2.Digest): Content digest
	1255	+#
	1256	+def _message_digest(message_buffer):
	1257	+ sha = hashlib.sha256(message_buffer)
	1258	+ digest = remote_execution_pb2.Digest()
	1259	+ digest.hash = sha.hexdigest()
	1260	+ digest.size_bytes = len(message_buffer)
	1261	+ return digest

doc/source/format_project.rst

---

...	...	@@ -143,7 +143,7 @@ Individual warnings can be configured as fatal by setting ``fatal-warnings`` to
143	143	- ref-not-in-track
144	144	- <plugin>:<warning>
145	145
146		-BuildStream provides a collection of :class:`Core Warnings <buildstream.plugin.CoreWarnings>` which may be raised
	146	+BuildStream provides a collection of :class:`Core Warnings <buildstream.types.CoreWarnings>` which may be raised
147	147	by a variety of plugins. Other configurable warnings are plugin specific and should be noted within their individual documentation.
148	148
149	149	.. note::
...	...	@@ -238,6 +238,8 @@ using the `remote-execution` option:
238	238	server-cert: server.crt
239	239	client-cert: client.crt
240	240	client-key: client.key
	241	+ action-cache-service:
	242	+ url: http://bar.action.com:50052
241	243
242	244	The execution-service part of remote execution does not support encrypted
243	245	connections yet, so the protocol must always be http.
...	...	@@ -245,6 +247,11 @@ connections yet, so the protocol must always be http.
245	247	storage-service specifies a remote CAS store and the parameters are the
246	248	same as those used to specify an :ref:`artifact server <artifacts>`.
247	249
	250	+The action-cache-service specifies where built actions are cached, allowing
	251	+buildstream to check whether an action has already been executed and download it
	252	+if so. This is similar to the artifact cache but REAPI specified, and is
	253	+optional for remote execution to work.
	254	+
248	255	The storage service may be the same endpoint used for artifact
249	256	caching. Remote execution cannot work without push access to the
250	257	storage endpoint, so you must specify a client certificate and key,
...	...	@@ -586,6 +593,30 @@ Architecture options can be tested with the same expressions
586	593	as other Enumeration options.
587	594
588	595
	596	+.. _project_options_os:
	597	+
	598	+OS
	599	+~~
	600	+
	601	+The ``os`` option type is a special enumeration option, which defaults to the
	602	+results of `uname -s`. It does not support assigning any default in the project
	603	+configuration.
	604	+
	605	+.. code:: yaml
	606	+
	607	+ options:
	608	+ machine_os:
	609	+ type: os
	610	+ description: The machine OS
	611	+ values:
	612	+ - Linux
	613	+ - SunOS
	614	+ - Darwin
	615	+ - FreeBSD
	616	+
	617	+Os options can be tested with the same expressions as other Enumeration options.
	618	+
	619	+
589	620	.. _project_options_element_mask:
590	621
591	622	Element mask

tests/sources/git.py

---

...	...	@@ -414,45 +414,17 @@ def test_submodule_track_no_ref_or_track(cli, tmpdir, datafiles):
414	414
415	415	@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
416	416	@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
417		-def test_ref_not_in_track_warn(cli, tmpdir, datafiles):
	417	+@pytest.mark.parametrize("fail", ['warn', 'error'])
	418	+def test_ref_not_in_track(cli, tmpdir, datafiles, fail):
418	419	project = os.path.join(datafiles.dirname, datafiles.basename)
419	420
420		- # Create the repo from 'repofiles', create a branch without latest commit
421		- repo = create_repo('git', str(tmpdir))
422		- ref = repo.create(os.path.join(project, 'repofiles'))
423		-
424		- gitsource = repo.source_config(ref=ref)
425		-
426		- # Overwrite the track value to the added branch
427		- gitsource['track'] = 'foo'
428		-
429		- # Write out our test target
430		- element = {
431		- 'kind': 'import',
432		- 'sources': [
433		- gitsource
434		- ]
435		- }
436		- _yaml.dump(element, os.path.join(project, 'target.bst'))
437		-
438		- # Assert the warning is raised as ref is not in branch foo.
439		- # Assert warning not error to the user, when not set as fatal.
440		- result = cli.run(project=project, args=['build', 'target.bst'])
441		- assert "The ref provided for the element does not exist locally" in result.stderr
442		-
443		-
444		-@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
445		-@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
446		-def test_ref_not_in_track_warn_error(cli, tmpdir, datafiles):
447		- project = os.path.join(datafiles.dirname, datafiles.basename)
448		-
449		- # Add fatal-warnings ref-not-in-track to project.conf
450		- project_template = {
451		- "name": "foo",
452		- "fatal-warnings": [CoreWarnings.REF_NOT_IN_TRACK]
453		- }
454		-
455		- _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	421	+ # Make the warning an error if we're testing errors
	422	+ if fail == 'error':
	423	+ project_template = {
	424	+ "name": "foo",
	425	+ "fatal-warnings": [CoreWarnings.REF_NOT_IN_TRACK]
	426	+ }
	427	+ _yaml.dump(project_template, os.path.join(project, 'project.conf'))
456	428
457	429	# Create the repo from 'repofiles', create a branch without latest commit
458	430	repo = create_repo('git', str(tmpdir))
...	...	@@ -472,11 +444,15 @@ def test_ref_not_in_track_warn_error(cli, tmpdir, datafiles):
472	444	}
473	445	_yaml.dump(element, os.path.join(project, 'target.bst'))
474	446
475		- # Assert that build raises a warning here that is captured
476		- # as plugin error, due to the fatal warning being set
477	447	result = cli.run(project=project, args=['build', 'target.bst'])
478		- result.assert_main_error(ErrorDomain.STREAM, None)
479		- result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.REF_NOT_IN_TRACK)
	448	+
	449	+ # Assert a warning or an error depending on what we're checking
	450	+ if fail == 'error':
	451	+ result.assert_main_error(ErrorDomain.STREAM, None)
	452	+ result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.REF_NOT_IN_TRACK)
	453	+ else:
	454	+ result.assert_success()
	455	+ assert "ref-not-in-track" in result.stderr
480	456
481	457
482	458	@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.