[Notes] [Git][BuildStream/buildstream][mablanch/448-autocompletion-broken-defaults] 12 commits: element.py: Cache the result of checking whether an artifact is cached weakly

[Phil Dawson <gitlab mg gitlab com>]

Title: GitLab

Phil Dawson pushed to branch mablanch/448-autocompletion-broken-defaults at BuildStream / buildstream

Commits:

• 3d308894
  by Richard Maw at 2018-07-31T16:31:36Z

  element.py: Cache the result of checking whether an artifact is cached weakly
  
  Normally we'd only need it in the case of scheduling a weakly cached build,
  but to allow caching of failed builds we need to be able to distinguish
  between cached successes and cached failures
  for both strong and weak cache keys.
  
  To allow other cache lookup codepaths to look up via the weak key
  requires changes through the call stack to consult which key to use,
  and cache invalidation of the saved state when it changes.
  

• b3a68e28
  by Richard Maw at 2018-07-31T16:31:36Z

  element.py: Add metadata to distinguish between successful and failed builds
  
  This just puts the metadata in place,
  we're adding code paths to add failed builds later.
  

• f4573df3
  by Richard Maw at 2018-07-31T16:31:36Z

  Convert call-sites of Element._cached() that assume success
  
  When we later add cached failures it needs to not treat them as successes.
  

• 553df108
  by Richard Maw at 2018-07-31T16:31:36Z

  _frontend/widget.py: Render cached failures differently to successes
  

• d14d8ee2
  by Richard Maw at 2018-07-31T16:32:35Z

  Cache failed builds
  
  This creates an artifact when element assembly fails too,
  and if it's the right kind of exception uses the now-included install directory
  similarly to if it had returned successfully.
  
  If there's a failure during install the artifact contains any installed files,
  but may contain nothing at all.
  

• d83122bb
  by Richard Maw at 2018-07-31T16:32:35Z

  _scheduler/queues/buildqueue.py: Skip rebuilding cached failures
  
  This flags up a failure and if run in an interactive prompt
  permits the user to attempt a rebuild.
  

• 78944e9a
  by Richard Maw at 2018-07-31T16:32:35Z

  tests: Add tests for cached behaviours
  

• 3fa79d8d
  by Richard Maw at 2018-07-31T16:32:35Z

  _scheduler/queues: Add failed builds to "done" queue
  
  This allows the scheduler to move jobs from the current queue to the next.
  
  As a result of this change later queues than the build queue
  mustn't skip a cached failure, so the logic is specialised to build queues only.
  

• e8cd43dc
  by Richard Maw at 2018-07-31T16:32:58Z

  NEWS: Describe caching of failures
  
  Closes #76.
  

• 88cd61ea
  by knownexus at 2018-07-31T16:32:58Z

  Added the missing `_get_build_log` function
  

• 4fc1f5d1
  by Phillip Smyth at 2018-07-31T17:17:34Z

  Merge branch 'richardmaw/cache-fail' into 'master'
  
  Store failed builds in the cache
  
  Closes #76
  
  See merge request BuildStream/buildstream!475

• c9762187
  by Martin Blanchard at 2018-08-01T08:38:46Z

  complete.py: Ensure paths get completed from 'element-path'
  
  Element paths should always be completed from the root element folder
  defined by the element-path key in project.conf. Fix complete_path() to
  always search into its given base_directory argument.
  
  See issue BuildStream/buildstream#448
  

16 changed files:

• NEWS
• buildstream/_frontend/complete.py
• buildstream/_frontend/widget.py
• buildstream/_pipeline.py
• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/queues/queue.py
• buildstream/buildelement.py
• buildstream/element.py
• buildstream/scriptelement.py
• tests/completions/completions.py
• + tests/completions/sub-folders/base/unwanted.bst
• + tests/completions/sub-folders/elements/base.bst
• + tests/completions/sub-folders/elements/base/wanted.bst
• + tests/completions/sub-folders/elements/hello.bst
• + tests/completions/sub-folders/project.conf
• + tests/integration/cachedfail.py

Changes:

NEWS

---

...	...	@@ -11,6 +11,13 @@ buildstream 1.1.5
11	11
12	12	o Added new `remote` source plugin for downloading file blobs
13	13
	14	+ o Failed builds are included in the cache as well.
	15	+ `bst checkout` will provide anything in `%{install-root}`.
	16	+ A build including cached fails will cause any dependant elements
	17	+ to not be scheduled and fail during artifact assembly,
	18	+ and display the retry prompt during an interactive session.
	19	+
	20	+
14	21	=================
15	22	buildstream 1.1.4
16	23	=================

buildstream/_frontend/complete.py

---

...	...	@@ -68,9 +68,10 @@ def complete_path(path_type, incomplete, base_directory='.'):
68	68	# If there was nothing on the left of the last separator,
69	69	# we are completing files in the filesystem root
70	70	base_path = os.path.join(base_directory, base_path)
71		-
72		- elif os.path.isdir(incomplete):
73		- base_path = incomplete
	71	+ else:
	72	+ incomplete_base_path = os.path.join(base_directory, incomplete)
	73	+ if os.path.isdir(incomplete_base_path):
	74	+ base_path = incomplete_base_path
74	75
75	76	try:
76	77	if base_path:

buildstream/_frontend/widget.py

---

...	...	@@ -368,7 +368,9 @@ class LogLine(Widget):
368	368	if consistency == Consistency.INCONSISTENT:
369	369	line = p.fmt_subst(line, 'state', "no reference", fg='red')
370	370	else:
371		- if element._cached():
	371	+ if element._cached_failure():
	372	+ line = p.fmt_subst(line, 'state', "failed", fg='red')
	373	+ elif element._cached_success():
372	374	line = p.fmt_subst(line, 'state', "cached", fg='magenta')
373	375	elif consistency == Consistency.RESOLVED:
374	376	line = p.fmt_subst(line, 'state', "fetch needed", fg='red')

buildstream/_pipeline.py

---

...	...	@@ -489,7 +489,7 @@ class _Planner():
489	489	self.plan_element(dep, depth)
490	490
491	491	# Dont try to plan builds of elements that are cached already
492		- if not element._cached():
	492	+ if not element._cached_success():
493	493	for dep in element.dependencies(Scope.BUILD, recurse=False):
494	494	self.plan_element(dep, depth + 1)
495	495
...	...	@@ -501,4 +501,4 @@ class _Planner():
501	501	self.plan_element(root, 0)
502	502
503	503	depth_sorted = sorted(self.depth_map.items(), key=itemgetter(1), reverse=True)
504		- return [item[0] for item in depth_sorted if plan_cached or not item[0]._cached()]
	504	+ return [item[0] for item in depth_sorted if plan_cached or not item[0]._cached_success()]

buildstream/_scheduler/queues/buildqueue.py

---

...	...	@@ -18,8 +18,12 @@
18	18	# Tristan Van Berkom <tristan vanberkom codethink co uk>
19	19	# Jürg Billeter <juerg billeter codethink co uk>
20	20
	21	+from datetime import timedelta
	22	+
21	23	from . import Queue, QueueStatus
	24	+from ..jobs import ElementJob
22	25	from ..resources import ResourceType
	26	+from ..._message import MessageType
23	27
24	28
25	29	# A queue which assembles elements
...	...	@@ -30,6 +34,38 @@ class BuildQueue(Queue):
30	34	complete_name = "Built"
31	35	resources = [ResourceType.PROCESS]
32	36
	37	+ def __init__(self, *args, **kwargs):
	38	+ super().__init__(*args, **kwargs)
	39	+ self._tried = set()
	40	+
	41	+ def enqueue(self, elts):
	42	+ to_queue = []
	43	+
	44	+ for element in elts:
	45	+ if not element._cached_failure() or element in self._tried:
	46	+ to_queue.append(element)
	47	+ continue
	48	+
	49	+ # Bypass queue processing entirely the first time it's tried.
	50	+ self._tried.add(element)
	51	+ _, description, detail = element._get_build_result()
	52	+ logfile = element._get_build_log()
	53	+ self._message(element, MessageType.FAIL, description,
	54	+ detail=detail, action_name=self.action_name,
	55	+ elapsed=timedelta(seconds=0),
	56	+ logfile=logfile)
	57	+ job = ElementJob(self._scheduler, self.action_name,
	58	+ logfile, element=element, queue=self,
	59	+ resources=self.resources,
	60	+ action_cb=self.process,
	61	+ complete_cb=self._job_done,
	62	+ max_retries=self._max_retries)
	63	+ self._done_queue.append(job)
	64	+ self.failed_elements.append(element)
	65	+ self._scheduler._job_complete_callback(job, False)
	66	+
	67	+ return super().enqueue(to_queue)
	68	+
33	69	def process(self, element):
34	70	element._assemble()
35	71	return element._get_unique_id()
...	...	@@ -43,7 +79,7 @@ class BuildQueue(Queue):
43	79	# Keep it in the queue.
44	80	return QueueStatus.WAIT
45	81
46		- if element._cached():
	82	+ if element._cached_success():
47	83	return QueueStatus.SKIP
48	84
49	85	if not element._buildable():

buildstream/_scheduler/queues/queue.py

---

...	...	@@ -296,6 +296,7 @@ class Queue():
296	296	# See the Job object for an explanation of the call signature
297	297	#
298	298	def _job_done(self, job, element, success, result):
	299	+ element._update_state()
299	300
300	301	# Update values that need to be synchronized in the main task
301	302	# before calling any queue implementation
...	...	@@ -335,8 +336,9 @@ class Queue():
335	336
336	337	# No exception occured, handle the success/failure state in the normal way
337	338	#
	339	+ self._done_queue.append(job)
	340	+
338	341	if success:
339		- self._done_queue.append(job)
340	342	if processed:
341	343	self.processed_elements.append(element)
342	344	else:

buildstream/buildelement.py

---

...	...	@@ -233,12 +233,14 @@ class BuildElement(Element):
233	233	return commands
234	234
235	235	def __run_command(self, sandbox, cmd, cmd_name):
236		- self.status("Running {}".format(cmd_name), detail=cmd)
237		-
238		- # Note the -e switch to 'sh' means to exit with an error
239		- # if any untested command fails.
240		- #
241		- exitcode = sandbox.run(['sh', '-c', '-e', cmd + '\n'],
242		- SandboxFlags.ROOT_READ_ONLY)
243		- if exitcode != 0:
244		- raise ElementError("Command '{}' failed with exitcode {}".format(cmd, exitcode))
	236	+ with self.timed_activity("Running {}".format(cmd_name)):
	237	+ self.status("Running {}".format(cmd_name), detail=cmd)
	238	+
	239	+ # Note the -e switch to 'sh' means to exit with an error
	240	+ # if any untested command fails.
	241	+ #
	242	+ exitcode = sandbox.run(['sh', '-c', '-e', cmd + '\n'],
	243	+ SandboxFlags.ROOT_READ_ONLY)
	244	+ if exitcode != 0:
	245	+ raise ElementError("Command '{}' failed with exitcode {}".format(cmd, exitcode),
	246	+ collect=self.get_variable('install-root'))

buildstream/element.py

---

...	...	@@ -140,11 +140,14 @@ class ElementError(BstError):
140	140	message (str): The error message to report to the user
141	141	detail (str): A possibly multiline, more detailed error message
142	142	reason (str): An optional machine readable reason string, used for test cases
	143	+ collect (str): An optional directory containing partial install contents
143	144	temporary (bool): An indicator to whether the error may occur if the operation was run again. (*Since: 1.2*)
144	145	"""
145		- def __init__(self, message, *, detail=None, reason=None, temporary=False):
	146	+ def __init__(self, message, *, detail=None, reason=None, collect=None, temporary=False):
146	147	super().__init__(message, detail=detail, domain=ErrorDomain.ELEMENT, reason=reason, temporary=temporary)
147	148
	149	+ self.collect = collect
	150	+
148	151
149	152	class Element(Plugin):
150	153	"""Element()
...	...	@@ -216,6 +219,7 @@ class Element(Plugin):
216	219	self.__consistency = Consistency.INCONSISTENT # Cached overall consistency state
217	220	self.__cached = None # Whether we have a cached artifact
218	221	self.__strong_cached = None # Whether we have a cached artifact
	222	+ self.__weak_cached = None # Whether we have a cached artifact
219	223	self.__assemble_scheduled = False # Element is scheduled to be assembled
220	224	self.__assemble_done = False # Element is assembled
221	225	self.__tracking_scheduled = False # Sources are scheduled to be tracked
...	...	@@ -227,6 +231,8 @@ class Element(Plugin):
227	231	self.__tainted = None # Whether the artifact is tainted and should not be shared
228	232	self.__required = False # Whether the artifact is required in the current session
229	233	self.__artifact_size = None # The size of data committed to the artifact cache
	234	+ self.__build_result = None # The result of assembling this Element
	235	+ self._build_log_path = None # The path of the build log for this Element
230	236
231	237	# hash tables of loaded artifact metadata, hashed by key
232	238	self.__metadata_keys = {} # Strong and weak keys for this key
...	...	@@ -951,7 +957,51 @@ class Element(Plugin):
951	957	# the artifact cache
952	958	#
953	959	def _cached(self):
954		- return self.__cached
	960	+ return self.__is_cached(keystrength=None)
	961	+
	962	+ # _get_build_result():
	963	+ #
	964	+ # Returns:
	965	+ # (bool): Whether the artifact of this element present in the artifact cache is of a success
	966	+ # (str): Short description of the result
	967	+ # (str): Detailed description of the result
	968	+ #
	969	+ def _get_build_result(self):
	970	+ return self.__get_build_result(keystrength=None)
	971	+
	972	+ # __set_build_result():
	973	+ #
	974	+ # Sets the assembly result
	975	+ #
	976	+ # Args:
	977	+ # success (bool): Whether the result is a success
	978	+ # description (str): Short description of the result
	979	+ # detail (str): Detailed description of the result
	980	+ #
	981	+ def __set_build_result(self, success, description, detail=None):
	982	+ self.__build_result = (success, description, detail)
	983	+
	984	+ # _cached_success():
	985	+ #
	986	+ # Returns:
	987	+ # (bool): Whether this element is already present in
	988	+ # the artifact cache and the element assembled successfully
	989	+ #
	990	+ def _cached_success(self):
	991	+ return self.__cached_success(keystrength=None)
	992	+
	993	+ # _cached_failure():
	994	+ #
	995	+ # Returns:
	996	+ # (bool): Whether this element is already present in
	997	+ # the artifact cache and the element did not assemble successfully
	998	+ #
	999	+ def _cached_failure(self):
	1000	+ if not self._cached():
	1001	+ return False
	1002	+
	1003	+ success, _, _ = self._get_build_result()
	1004	+ return not success
955	1005
956	1006	# _buildable():
957	1007	#
...	...	@@ -968,7 +1018,7 @@ class Element(Plugin):
968	1018	# if the pull job is still pending as the remote cache may have an artifact
969	1019	# that matches the strict cache key, which is preferred over a locally
970	1020	# cached artifact with a weak cache key match.
971		- if not dependency._cached() or not dependency._get_cache_key(strength=_KeyStrength.STRONG):
	1021	+ if not dependency._cached_success() or not dependency._get_cache_key(strength=_KeyStrength.STRONG):
972	1022	return False
973	1023
974	1024	if not self.__assemble_scheduled:
...	...	@@ -1039,6 +1089,8 @@ class Element(Plugin):
1039	1089	self.__weak_cache_key = None
1040	1090	self.__strict_cache_key = None
1041	1091	self.__strong_cached = None
	1092	+ self.__weak_cached = None
	1093	+ self.__build_result = None
1042	1094	return
1043	1095
1044	1096	if self.__weak_cache_key is None:
...	...	@@ -1061,6 +1113,9 @@ class Element(Plugin):
1061	1113	# Weak cache key could not be calculated yet
1062	1114	return
1063	1115
	1116	+ if not self.__weak_cached:
	1117	+ self.__weak_cached = self.__artifacts.contains(self, self.__weak_cache_key)
	1118	+
1064	1119	if not context.get_strict():
1065	1120	# Full cache query in non-strict mode requires both the weak and
1066	1121	# strict cache keys. However, we need to determine as early as
...	...	@@ -1068,9 +1123,9 @@ class Element(Plugin):
1068	1123	# for workspaced elements. For this cache check the weak cache keys
1069	1124	# are sufficient. However, don't update the `cached` attributes
1070	1125	# until the full cache query below.
1071		- cached = self.__artifacts.contains(self, self.__weak_cache_key)
1072	1126	if (not self.__assemble_scheduled and not self.__assemble_done and
1073		- not cached and not self._pull_pending() and self._is_required()):
	1127	+ not self.__cached_success(keystrength=_KeyStrength.WEAK) and
	1128	+ not self._pull_pending() and self._is_required()):
1074	1129	self._schedule_assemble()
1075	1130	return
1076	1131
...	...	@@ -1090,9 +1145,12 @@ class Element(Plugin):
1090	1145	self.__cached = self.__artifacts.contains(self, key_for_cache_lookup)
1091	1146	if not self.__strong_cached:
1092	1147	self.__strong_cached = self.__artifacts.contains(self, self.__strict_cache_key)
	1148	+ if key_for_cache_lookup == self.__weak_cache_key:
	1149	+ if not self.__weak_cached:
	1150	+ self.__weak_cached = self.__artifacts.contains(self, self.__weak_cache_key)
1093	1151
1094	1152	if (not self.__assemble_scheduled and not self.__assemble_done and
1095		- not self.__cached and not self._pull_pending() and self._is_required()):
	1153	+ not self._cached_success() and not self._pull_pending() and self._is_required()):
1096	1154	# Workspaced sources are considered unstable if a build is pending
1097	1155	# as the build will modify the contents of the workspace.
1098	1156	# Determine as early as possible if a build is pending to discard
...	...	@@ -1434,7 +1492,7 @@ class Element(Plugin):
1434	1492	def _assemble(self):
1435	1493
1436	1494	# Assert call ordering
1437		- assert not self._cached()
	1495	+ assert not self._cached_success()
1438	1496
1439	1497	context = self._get_context()
1440	1498	with self._output_file() as output_file:
...	...	@@ -1457,6 +1515,7 @@ class Element(Plugin):
1457	1515	self.__dynamic_public = _yaml.node_copy(self.__public)
1458	1516
1459	1517	# Call the abstract plugin methods
	1518	+ collect = None
1460	1519	try:
1461	1520	# Step 1 - Configure
1462	1521	self.configure_sandbox(sandbox)
...	...	@@ -1466,6 +1525,7 @@ class Element(Plugin):
1466	1525	self.__prepare(sandbox)
1467	1526	# Step 4 - Assemble
1468	1527	collect = self.assemble(sandbox)
	1528	+ self.__set_build_result(success=True, description="succeeded")
1469	1529	except BstError as e:
1470	1530	# If an error occurred assembling an element in a sandbox,
1471	1531	# then tack on the sandbox directory to the error
...	...	@@ -1489,80 +1549,95 @@ class Element(Plugin):
1489	1549	self.warn("Failed to preserve workspace state for failed build sysroot: {}"
1490	1550	.format(e))
1491	1551
1492		- raise
	1552	+ if isinstance(e, ElementError):
	1553	+ collect = e.collect # pylint: disable=no-member
1493	1554
1494		- collectdir = os.path.join(sandbox_root, collect.lstrip(os.sep))
1495		- if not os.path.exists(collectdir):
1496		- raise ElementError(
1497		- "Directory '{}' was not found inside the sandbox, "
1498		- "unable to collect artifact contents"
1499		- .format(collect))
1500		-
1501		- # At this point, we expect an exception was raised leading to
1502		- # an error message, or we have good output to collect.
1503		-
1504		- # Create artifact directory structure
1505		- assembledir = os.path.join(rootdir, 'artifact')
1506		- filesdir = os.path.join(assembledir, 'files')
1507		- logsdir = os.path.join(assembledir, 'logs')
1508		- metadir = os.path.join(assembledir, 'meta')
1509		- buildtreedir = os.path.join(assembledir, 'buildtree')
1510		- os.mkdir(assembledir)
1511		- os.mkdir(filesdir)
1512		- os.mkdir(logsdir)
1513		- os.mkdir(metadir)
1514		- os.mkdir(buildtreedir)
1515		-
1516		- # Hard link files from collect dir to files directory
1517		- utils.link_files(collectdir, filesdir)
1518		-
1519		- sandbox_build_dir = os.path.join(sandbox_root, self.get_variable('build-root').lstrip(os.sep))
1520		- # Hard link files from build-root dir to buildtreedir directory
1521		- if os.path.isdir(sandbox_build_dir):
1522		- utils.link_files(sandbox_build_dir, buildtreedir)
1523		-
1524		- # Copy build log
1525		- log_filename = context.get_log_filename()
1526		- if log_filename:
1527		- shutil.copyfile(log_filename, os.path.join(logsdir, 'build.log'))
1528		-
1529		- # Store public data
1530		- _yaml.dump(_yaml.node_sanitize(self.__dynamic_public), os.path.join(metadir, 'public.yaml'))
1531		-
1532		- # ensure we have cache keys
1533		- self._assemble_done()
1534		-
1535		- # Store keys.yaml
1536		- _yaml.dump(_yaml.node_sanitize({
1537		- 'strong': self._get_cache_key(),
1538		- 'weak': self._get_cache_key(_KeyStrength.WEAK),
1539		- }), os.path.join(metadir, 'keys.yaml'))
1540		-
1541		- # Store dependencies.yaml
1542		- _yaml.dump(_yaml.node_sanitize({
1543		- e.name: e._get_cache_key() for e in self.dependencies(Scope.BUILD)
1544		- }), os.path.join(metadir, 'dependencies.yaml'))
1545		-
1546		- # Store workspaced.yaml
1547		- _yaml.dump(_yaml.node_sanitize({
1548		- 'workspaced': True if self._get_workspace() else False
1549		- }), os.path.join(metadir, 'workspaced.yaml'))
1550		-
1551		- # Store workspaced-dependencies.yaml
1552		- _yaml.dump(_yaml.node_sanitize({
1553		- 'workspaced-dependencies': [
1554		- e.name for e in self.dependencies(Scope.BUILD)
1555		- if e._get_workspace()
1556		- ]
1557		- }), os.path.join(metadir, 'workspaced-dependencies.yaml'))
1558		-
1559		- with self.timed_activity("Caching artifact"):
1560		- self.__artifact_size = utils._get_dir_size(assembledir)
1561		- self.__artifacts.commit(self, assembledir, self.__get_cache_keys_for_commit())
	1555	+ self.__set_build_result(success=False, description=str(e), detail=e.detail)
	1556	+ raise
	1557	+ finally:
	1558	+ if collect is not None:
	1559	+ collectdir = os.path.join(sandbox_root, collect.lstrip(os.sep))
	1560	+
	1561	+ # Create artifact directory structure
	1562	+ assembledir = os.path.join(rootdir, 'artifact')
	1563	+ filesdir = os.path.join(assembledir, 'files')
	1564	+ logsdir = os.path.join(assembledir, 'logs')
	1565	+ metadir = os.path.join(assembledir, 'meta')
	1566	+ buildtreedir = os.path.join(assembledir, 'buildtree')
	1567	+ os.mkdir(assembledir)
	1568	+ if collect is not None and os.path.exists(collectdir):
	1569	+ os.mkdir(filesdir)
	1570	+ os.mkdir(logsdir)
	1571	+ os.mkdir(metadir)
	1572	+ os.mkdir(buildtreedir)
	1573	+
	1574	+ # Hard link files from collect dir to files directory
	1575	+ if collect is not None and os.path.exists(collectdir):
	1576	+ utils.link_files(collectdir, filesdir)
	1577	+
	1578	+ sandbox_build_dir = os.path.join(sandbox_root, self.get_variable('build-root').lstrip(os.sep))
	1579	+ # Hard link files from build-root dir to buildtreedir directory
	1580	+ if os.path.isdir(sandbox_build_dir):
	1581	+ utils.link_files(sandbox_build_dir, buildtreedir)
	1582	+
	1583	+ # Copy build log
	1584	+ log_filename = context.get_log_filename()
	1585	+ self._build_log_path = os.path.join(logsdir, 'build.log')
	1586	+ if log_filename:
	1587	+ shutil.copyfile(log_filename, self._build_log_path)
	1588	+
	1589	+ # Store public data
	1590	+ _yaml.dump(_yaml.node_sanitize(self.__dynamic_public), os.path.join(metadir, 'public.yaml'))
	1591	+
	1592	+ # Store result
	1593	+ build_result_dict = {"success": self.__build_result[0], "description": self.__build_result[1]}
	1594	+ if self.__build_result[2] is not None:
	1595	+ build_result_dict["detail"] = self.__build_result[2]
	1596	+ _yaml.dump(build_result_dict, os.path.join(metadir, 'build-result.yaml'))
	1597	+
	1598	+ # ensure we have cache keys
	1599	+ self._assemble_done()
	1600	+
	1601	+ # Store keys.yaml
	1602	+ _yaml.dump(_yaml.node_sanitize({
	1603	+ 'strong': self._get_cache_key(),
	1604	+ 'weak': self._get_cache_key(_KeyStrength.WEAK),
	1605	+ }), os.path.join(metadir, 'keys.yaml'))
	1606	+
	1607	+ # Store dependencies.yaml
	1608	+ _yaml.dump(_yaml.node_sanitize({
	1609	+ e.name: e._get_cache_key() for e in self.dependencies(Scope.BUILD)
	1610	+ }), os.path.join(metadir, 'dependencies.yaml'))
	1611	+
	1612	+ # Store workspaced.yaml
	1613	+ _yaml.dump(_yaml.node_sanitize({
	1614	+ 'workspaced': True if self._get_workspace() else False
	1615	+ }), os.path.join(metadir, 'workspaced.yaml'))
	1616	+
	1617	+ # Store workspaced-dependencies.yaml
	1618	+ _yaml.dump(_yaml.node_sanitize({
	1619	+ 'workspaced-dependencies': [
	1620	+ e.name for e in self.dependencies(Scope.BUILD)
	1621	+ if e._get_workspace()
	1622	+ ]
	1623	+ }), os.path.join(metadir, 'workspaced-dependencies.yaml'))
	1624	+
	1625	+ with self.timed_activity("Caching artifact"):
	1626	+ self.__artifact_size = utils._get_dir_size(assembledir)
	1627	+ self.__artifacts.commit(self, assembledir, self.__get_cache_keys_for_commit())
	1628	+
	1629	+ if collect is not None and not os.path.exists(collectdir):
	1630	+ raise ElementError(
	1631	+ "Directory '{}' was not found inside the sandbox, "
	1632	+ "unable to collect artifact contents"
	1633	+ .format(collect))
1562	1634
1563	1635	# Finally cleanup the build dir
1564	1636	cleanup_rootdir()
1565	1637
	1638	+ def _get_build_log(self):
	1639	+ return self._build_log_path
	1640	+
1566	1641	# _pull_pending()
1567	1642	#
1568	1643	# Check whether the artifact will be pulled.
...	...	@@ -1983,12 +2058,19 @@ class Element(Plugin):
1983	2058	if workspace:
1984	2059	workspace.prepared = True
1985	2060
	2061	+ def __is_cached(self, keystrength):
	2062	+ if keystrength is None:
	2063	+ return self.__cached
	2064	+
	2065	+ return self.__strong_cached if keystrength == _KeyStrength.STRONG else self.__weak_cached
	2066	+
1986	2067	# __assert_cached()
1987	2068	#
1988	2069	# Raises an error if the artifact is not cached.
1989	2070	#
1990		- def __assert_cached(self):
1991		- assert self._cached(), "{}: Missing artifact {}".format(self, self._get_brief_display_key())
	2071	+ def __assert_cached(self, keystrength=_KeyStrength.STRONG):
	2072	+ assert self.__is_cached(keystrength=keystrength), "{}: Missing artifact {}".format(
	2073	+ self, self._get_brief_display_key())
1992	2074
1993	2075	# __get_tainted():
1994	2076	#
...	...	@@ -2448,6 +2530,38 @@ class Element(Plugin):
2448	2530	metadir = os.path.join(artifact_base, 'meta')
2449	2531	self.__dynamic_public = _yaml.load(os.path.join(metadir, 'public.yaml'))
2450	2532
	2533	+ def __load_build_result(self, keystrength):
	2534	+ self.__assert_cached(keystrength=keystrength)
	2535	+ assert self.__build_result is None
	2536	+
	2537	+ artifact_base, _ = self.__extract(key=self.__weak_cache_key if keystrength is _KeyStrength.WEAK
	2538	+ else self.__strict_cache_key)
	2539	+
	2540	+ metadir = os.path.join(artifact_base, 'meta')
	2541	+ result_path = os.path.join(metadir, 'build-result.yaml')
	2542	+ if not os.path.exists(result_path):
	2543	+ self.__build_result = (True, "succeeded", None)
	2544	+ return
	2545	+
	2546	+ data = _yaml.load(result_path)
	2547	+ self.__build_result = (data["success"], data.get("description"), data.get("detail"))
	2548	+
	2549	+ def __get_build_result(self, keystrength):
	2550	+ if keystrength is None:
	2551	+ keystrength = _KeyStrength.STRONG if self._get_context().get_strict() else _KeyStrength.WEAK
	2552	+
	2553	+ if self.__build_result is None:
	2554	+ self.__load_build_result(keystrength)
	2555	+
	2556	+ return self.__build_result
	2557	+
	2558	+ def __cached_success(self, keystrength):
	2559	+ if not self.__is_cached(keystrength=keystrength):
	2560	+ return False
	2561	+
	2562	+ success, _, _ = self.__get_build_result(keystrength=keystrength)
	2563	+ return success
	2564	+
2451	2565	def __get_cache_keys_for_commit(self):
2452	2566	keys = []
2453	2567

buildstream/scriptelement.py

---

...	...	@@ -277,7 +277,8 @@ class ScriptElement(Element):
277	277	exitcode = sandbox.run(['sh', '-c', '-e', cmd + '\n'],
278	278	SandboxFlags.ROOT_READ_ONLY if self.__root_read_only else 0)
279	279	if exitcode != 0:
280		- raise ElementError("Command '{}' failed with exitcode {}".format(cmd, exitcode))
	280	+ raise ElementError("Command '{}' failed with exitcode {}".format(cmd, exitcode),
	281	+ collect=self.__install_root)
281	282
282	283	# Return where the result can be collected from
283	284	return self.__install_root

tests/completions/completions.py

---

...	...	@@ -212,6 +212,10 @@ def test_option_directory(datafiles, cli, cmd, word_idx, expected, subdir):
212	212	# Also try multi arguments together
213	213	('no-element-path', 'bst --directory ../ checkout t ', 4, ['target.bst '], 'files'),
214	214	('no-element-path', 'bst --directory ../ checkout target.bst ', 5, ['bin-files/', 'dev-files/'], 'files'),
	215	+
	216	+ # When element-path have sub-folders
	217	+ ('sub-folders', 'bst show base', 2, ['base/wanted.bst '], None),
	218	+ ('sub-folders', 'bst show base/', 2, ['base/wanted.bst '], None),
215	219	])
216	220	def test_argument_element(datafiles, cli, project, cmd, word_idx, expected, subdir):
217	221	cwd = os.path.join(str(datafiles), project)

tests/completions/sub-folders/base/unwanted.bst

---

	1	+kind: autotools
	2	+description: |
	3	+
	4	+ Not auto-completed element

tests/completions/sub-folders/elements/base.bst

---

	1	+kind: stack
	2	+description: Base stack
	3	+
	4	+depends:
	5	+- base/wanted.bst

tests/completions/sub-folders/elements/base/wanted.bst

---

	1	+kind: autotools
	2	+description: |
	3	+
	4	+ Auto-completed element

tests/completions/sub-folders/elements/hello.bst

---

	1	+kind: autotools
	2	+description: |
	3	+
	4	+ Hello world

tests/completions/sub-folders/project.conf

---

	1	+# Project config for frontend build test
	2	+name: test
	3	+
	4	+element-path: elements

tests/integration/cachedfail.py

---

	1	+import os
	2	+import pytest
	3	+
	4	+from buildstream import _yaml
	5	+from buildstream._exceptions import ErrorDomain
	6	+
	7	+from tests.testutils import cli_integration as cli, create_artifact_share
	8	+from tests.testutils.site import IS_LINUX
	9	+
	10	+
	11	+pytestmark = pytest.mark.integration
	12	+
	13	+
	14	+DATA_DIR = os.path.join(
	15	+ os.path.dirname(os.path.realpath(__file__)),
	16	+ "project"
	17	+)
	18	+
	19	+
	20	+@pytest.mark.datafiles(DATA_DIR)
	21	+def test_build_checkout_cached_fail(cli, tmpdir, datafiles):
	22	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	23	+ element_path = os.path.join(project, 'elements', 'element.bst')
	24	+ workspace = os.path.join(cli.directory, 'workspace')
	25	+ checkout = os.path.join(cli.directory, 'checkout')
	26	+
	27	+ # Write out our test target
	28	+ element = {
	29	+ 'kind': 'script',
	30	+ 'depends': [
	31	+ {
	32	+ 'filename': 'base.bst',
	33	+ 'type': 'build',
	34	+ },
	35	+ ],
	36	+ 'config': {
	37	+ 'commands': [
	38	+ 'touch %{install-root}/foo',
	39	+ 'false',
	40	+ ],
	41	+ },
	42	+ }
	43	+ _yaml.dump(element, element_path)
	44	+
	45	+ # Try to build it, this should result in a failure that contains the content
	46	+ result = cli.run(project=project, args=['build', 'element.bst'])
	47	+ result.assert_main_error(ErrorDomain.STREAM, None)
	48	+
	49	+ # Assert that it's cached in a failed artifact
	50	+ assert cli.get_element_state(project, 'element.bst') == 'failed'
	51	+
	52	+ # Now check it out
	53	+ result = cli.run(project=project, args=[
	54	+ 'checkout', 'element.bst', checkout
	55	+ ])
	56	+ result.assert_success()
	57	+
	58	+ # Check that the checkout contains the file created before failure
	59	+ filename = os.path.join(checkout, 'foo')
	60	+ assert os.path.exists(filename)
	61	+
	62	+
	63	+@pytest.mark.datafiles(DATA_DIR)
	64	+def test_build_depend_on_cached_fail(cli, tmpdir, datafiles):
	65	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	66	+ dep_path = os.path.join(project, 'elements', 'dep.bst')
	67	+ target_path = os.path.join(project, 'elements', 'target.bst')
	68	+ workspace = os.path.join(cli.directory, 'workspace')
	69	+ checkout = os.path.join(cli.directory, 'checkout')
	70	+
	71	+ dep = {
	72	+ 'kind': 'script',
	73	+ 'depends': [
	74	+ {
	75	+ 'filename': 'base.bst',
	76	+ 'type': 'build',
	77	+ },
	78	+ ],
	79	+ 'config': {
	80	+ 'commands': [
	81	+ 'touch %{install-root}/foo',
	82	+ 'false',
	83	+ ],
	84	+ },
	85	+ }
	86	+ _yaml.dump(dep, dep_path)
	87	+ target = {
	88	+ 'kind': 'script',
	89	+ 'depends': [
	90	+ {
	91	+ 'filename': 'base.bst',
	92	+ 'type': 'build',
	93	+ },
	94	+ {
	95	+ 'filename': 'dep.bst',
	96	+ 'type': 'build',
	97	+ },
	98	+ ],
	99	+ 'config': {
	100	+ 'commands': [
	101	+ 'test -e /foo',
	102	+ ],
	103	+ },
	104	+ }
	105	+ _yaml.dump(target, target_path)
	106	+
	107	+ # Try to build it, this should result in caching a failure to build dep
	108	+ result = cli.run(project=project, args=['build', 'dep.bst'])
	109	+ result.assert_main_error(ErrorDomain.STREAM, None)
	110	+
	111	+ # Assert that it's cached in a failed artifact
	112	+ assert cli.get_element_state(project, 'dep.bst') == 'failed'
	113	+
	114	+ # Now we should fail because we've a cached fail of dep
	115	+ result = cli.run(project=project, args=['build', 'target.bst'])
	116	+ result.assert_main_error(ErrorDomain.STREAM, None)
	117	+
	118	+ # Assert that it's not yet built, since one of its dependencies isn't ready.
	119	+ assert cli.get_element_state(project, 'target.bst') == 'waiting'
	120	+
	121	+
	122	+@pytest.mark.skipif(not IS_LINUX, reason='Only available on linux')
	123	+@pytest.mark.datafiles(DATA_DIR)
	124	+@pytest.mark.parametrize("on_error", ("continue",))
	125	+def test_push_cached_fail(cli, tmpdir, datafiles, on_error):
	126	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	127	+ element_path = os.path.join(project, 'elements', 'element.bst')
	128	+ workspace = os.path.join(cli.directory, 'workspace')
	129	+ checkout = os.path.join(cli.directory, 'checkout')
	130	+
	131	+ # Write out our test target
	132	+ element = {
	133	+ 'kind': 'script',
	134	+ 'depends': [
	135	+ {
	136	+ 'filename': 'base.bst',
	137	+ 'type': 'build',
	138	+ },
	139	+ ],
	140	+ 'config': {
	141	+ 'commands': [
	142	+ 'false',
	143	+ ],
	144	+ },
	145	+ }
	146	+ _yaml.dump(element, element_path)
	147	+
	148	+ with create_artifact_share(os.path.join(str(tmpdir), 'remote')) as share:
	149	+ cli.configure({
	150	+ 'artifacts': {'url': share.repo, 'push': True},
	151	+ })
	152	+
	153	+ # Build the element, continuing to finish active jobs on error.
	154	+ result = cli.run(project=project, args=['--on-error={}'.format(on_error), 'build', 'element.bst'])
	155	+ result.assert_main_error(ErrorDomain.STREAM, None)
	156	+
	157	+ # This element should have failed
	158	+ assert cli.get_element_state(project, 'element.bst') == 'failed'
	159	+ # This element should have been pushed to the remote
	160	+ assert share.has_artifact('test', 'element.bst', cli.get_element_key(project, 'element.bst'))

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.