[Notes] [Git][BuildStream/buildstream][master] 13 commits: _frontend/app.py: Initialize logging before preflighting the artifact cache

[Tristan Van Berkom <gitlab mg gitlab com>]

Title: GitLab

Tristan Van Berkom pushed to branch master at BuildStream / buildstream

Commits:

• 41f03296
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _frontend/app.py: Initialize logging before preflighting the artifact cache
  
  The artifact cache emits messages, and we want to allow that in preflight.
  

• 24ca2f46
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  utils.py: Add _get_volume_size()
  
  We can streamline this call to os.statvfs() in a few places.
  

• 9fd9fbb9
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Refactored to use utils._get_volume_size()
  
  This will benefit from a better UtilError being raised, and
  and turns the artifact cache's local function into a one liner.
  
  The loop which finds the first existing directory in the
  given path has been removed, being meaningless due to the
  call to os.makedirs() in ArtifactCache.__init__().
  
  The local function was renamed to _get_cache_volume_size() and
  no longer takes any arguments, which is more suitable for the
  function as it serves as a testing override surface for
  unittest.mock().
  
  The following test cases which use the function to override
  the ArtifactCache behavior have been updated to use the new
  overridable function name:
  
     tests/artifactcache/cache_size.py
     tests/artifactcache/expiry.py
  

• 7ee0c579
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Added ArtifactCacheUsage()
  
  A simple object which creates a snapshot of current
  usage statistics for easy reporting in the frontend.
  

• 353293b6
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _context.py: Added get_artifact_cache_usage()
  
  A frontend facing API for obtaining usage statistics.
  
  I would have put this on Stream instead, but the Context
  seems to be the de facto place for looking up the artifact cache
  in general so let's put it here.
  

• 51ed36de
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _frontend/widget.py: Added cache usage entry in the startup heading
  

• 5797238b
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _frontend/status.py: Added Cache size usage indicator to status bar
  
  This also adds some comments around the main status bar heading
  rendering function.
  

• 8074ebf4
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Add status messages in cache management operations
  
  Added some useful status messages when:
  
    * Calculating a new artifact cache usage size
  
    * Starting a cleanup
  
    * Finishing a cleanup
  
  Also enhanced messaging about what was cleaned up so far when
  aborting a cleanup.
  

• 49c11bc8
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Added client progress callback to ArtifactCache.clean()
  

• 3616e939
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _scheduler/jobs/job.py: Allow subclasses to message the frontend
  

• bcd19266
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _scheduler/jobs/cleanupjob.py: Update cache size while processing
  
  Updates the known cache size in the main process while the cleanup
  process is ongoing, so that the status indicators update live
  while the cleanup happens.
  

• 9c33107f
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Correcting API documenting comment for remove()
  
  This seems to have been copy/pasted from cascache, and
  documents the function to possibly return None if defer_prune
  was specified, but this function does not expose defer_prune.
  

• a2140d74
  by Tristan Van Berkom at 2019-01-24T19:44:52Z

  Merge branch 'tristan/cache-management-logging' into 'master'
  
  Cache management logging enhancements
  
  See merge request BuildStream/buildstream!1105

10 changed files:

• buildstream/_artifactcache.py
• buildstream/_context.py
• buildstream/_frontend/app.py
• buildstream/_frontend/status.py
• buildstream/_frontend/widget.py
• buildstream/_scheduler/jobs/cleanupjob.py
• buildstream/_scheduler/jobs/job.py
• buildstream/utils.py
• tests/artifactcache/cache_size.py
• tests/artifactcache/expiry.py

Changes:

buildstream/_artifactcache.py

---

...	...	@@ -46,6 +46,39 @@ class ArtifactCacheSpec(CASRemoteSpec):
46	46	pass
47	47
48	48
	49	+# ArtifactCacheUsage
	50	+#
	51	+# A simple object to report the current artifact cache
	52	+# usage details.
	53	+#
	54	+# Note that this uses the user configured cache quota
	55	+# rather than the internal quota with protective headroom
	56	+# removed, to provide a more sensible value to display to
	57	+# the user.
	58	+#
	59	+# Args:
	60	+# artifacts (ArtifactCache): The artifact cache to get the status of
	61	+#
	62	+class ArtifactCacheUsage():
	63	+
	64	+ def __init__(self, artifacts):
	65	+ context = artifacts.context
	66	+ self.quota_config = context.config_cache_quota # Configured quota
	67	+ self.quota_size = artifacts._cache_quota_original # Resolved cache quota in bytes
	68	+ self.used_size = artifacts.get_cache_size() # Size used by artifacts in bytes
	69	+ self.used_percent = 0 # Percentage of the quota used
	70	+ if self.quota_size is not None:
	71	+ self.used_percent = int(self.used_size * 100 / self.quota_size)
	72	+
	73	+ # Formattable into a human readable string
	74	+ #
	75	+ def __str__(self):
	76	+ return "{} / {} ({}%)" \
	77	+ .format(utils._pretty_size(self.used_size, dec_places=1),
	78	+ self.quota_config,
	79	+ self.used_percent)
	80	+
	81	+
49	82	# An ArtifactCache manages artifacts.
50	83	#
51	84	# Args:
...	...	@@ -64,6 +97,7 @@ class ArtifactCache():
64	97	self._required_elements = set() # The elements required for this session
65	98	self._cache_size = None # The current cache size, sometimes it's an estimate
66	99	self._cache_quota = None # The cache quota
	100	+ self._cache_quota_original = None # The cache quota as specified by the user, in bytes
67	101	self._cache_lower_threshold = None # The target cache size for a cleanup
68	102	self._remotes_setup = False # Check to prevent double-setup of remotes
69	103
...	...	@@ -216,11 +250,33 @@ class ArtifactCache():
216	250	#
217	251	# Clean the artifact cache as much as possible.
218	252	#
	253	+ # Args:
	254	+ # progress (callable): A callback to call when a ref is removed
	255	+ #
219	256	# Returns:
220	257	# (int): The size of the cache after having cleaned up
221	258	#
222		- def clean(self):
	259	+ def clean(self, progress=None):
223	260	artifacts = self.list_artifacts()
	261	+ context = self.context
	262	+
	263	+ # Some accumulative statistics
	264	+ removed_ref_count = 0
	265	+ space_saved = 0
	266	+
	267	+ # Start off with an announcement with as much info as possible
	268	+ volume_size, volume_avail = self._get_cache_volume_size()
	269	+ self._message(MessageType.STATUS, "Starting cache cleanup",
	270	+ detail=("Elements required by the current build plan: {}\n" +
	271	+ "User specified quota: {} ({})\n" +
	272	+ "Cache usage: {}\n" +
	273	+ "Cache volume: {} total, {} available")
	274	+ .format(len(self._required_elements),
	275	+ context.config_cache_quota,
	276	+ utils._pretty_size(self._cache_quota_original, dec_places=2),
	277	+ utils._pretty_size(self.get_cache_size(), dec_places=2),
	278	+ utils._pretty_size(volume_size, dec_places=2),
	279	+ utils._pretty_size(volume_avail, dec_places=2)))
224	280
225	281	# Build a set of the cache keys which are required
226	282	# based on the required elements at cleanup time
...	...	@@ -245,11 +301,18 @@ class ArtifactCache():
245	301	# can't remove them, we have to abort the build.
246	302	#
247	303	# FIXME: Asking the user what to do may be neater
	304	+ #
248	305	default_conf = os.path.join(os.environ['XDG_CONFIG_HOME'],
249	306	'buildstream.conf')
250		- detail = ("There is not enough space to complete the build.\n"
251		- "Please increase the cache-quota in {}."
252		- .format(self.context.config_origin or default_conf))
	307	+ detail = ("Aborted after removing {} refs and saving {} disk space.\n"
	308	+ "The remaining {} in the cache is required by the {} elements in your build plan\n\n"
	309	+ "There is not enough space to complete the build.\n"
	310	+ "Please increase the cache-quota in {} and/or make more disk space."
	311	+ .format(removed_ref_count,
	312	+ utils._pretty_size(space_saved, dec_places=2),
	313	+ utils._pretty_size(self.get_cache_size(), dec_places=2),
	314	+ len(self._required_elements),
	315	+ (context.config_origin or default_conf)))
253	316
254	317	if self.has_quota_exceeded():
255	318	raise ArtifactError("Cache too full. Aborting.",
...	...	@@ -264,10 +327,33 @@ class ArtifactCache():
264	327	# Remove the actual artifact, if it's not required.
265	328	size = self.remove(to_remove)
266	329
	330	+ removed_ref_count += 1
	331	+ space_saved += size
	332	+
	333	+ self._message(MessageType.STATUS,
	334	+ "Freed {: <7} {}".format(
	335	+ utils._pretty_size(size, dec_places=2),
	336	+ to_remove))
	337	+
267	338	# Remove the size from the removed size
268	339	self.set_cache_size(self._cache_size - size)
269	340
270		- # This should be O(1) if implemented correctly
	341	+ # User callback
	342	+ #
	343	+ # Currently this process is fairly slow, but we should
	344	+ # think about throttling this progress() callback if this
	345	+ # becomes too intense.
	346	+ if progress:
	347	+ progress()
	348	+
	349	+ # Informational message about the side effects of the cleanup
	350	+ self._message(MessageType.INFO, "Cleanup completed",
	351	+ detail=("Removed {} refs and saving {} disk space.\n" +
	352	+ "Cache usage is now: {}")
	353	+ .format(removed_ref_count,
	354	+ utils._pretty_size(space_saved, dec_places=2),
	355	+ utils._pretty_size(self.get_cache_size(), dec_places=2)))
	356	+
271	357	return self.get_cache_size()
272	358
273	359	# compute_cache_size()
...	...	@@ -279,7 +365,14 @@ class ArtifactCache():
279	365	# (int): The size of the artifact cache.
280	366	#
281	367	def compute_cache_size(self):
282		- self._cache_size = self.cas.calculate_cache_size()
	368	+ old_cache_size = self._cache_size
	369	+ new_cache_size = self.cas.calculate_cache_size()
	370	+
	371	+ if old_cache_size != new_cache_size:
	372	+ self._cache_size = new_cache_size
	373	+
	374	+ usage = ArtifactCacheUsage(self)
	375	+ self._message(MessageType.STATUS, "Cache usage recomputed: {}".format(usage))
283	376
284	377	return self._cache_size
285	378
...	...	@@ -307,7 +400,7 @@ class ArtifactCache():
307	400	# it is greater than the actual cache size.
308	401	#
309	402	# Returns:
310		- # (int) An approximation of the artifact cache size.
	403	+ # (int) An approximation of the artifact cache size, in bytes.
311	404	#
312	405	def get_cache_size(self):
313	406
...	...	@@ -459,8 +552,7 @@ class ArtifactCache():
459	552	# `ArtifactCache.get_artifact_fullname`)
460	553	#
461	554	# Returns:
462		- # (int|None) The amount of space pruned from the repository in
463		- # Bytes, or None if defer_prune is True
	555	+ # (int): The amount of space recovered in the cache, in bytes
464	556	#
465	557	def remove(self, ref):
466	558
...	...	@@ -848,19 +940,16 @@ class ArtifactCache():
848	940	else:
849	941	headroom = 2e9
850	942
851		- artifactdir_volume = self.context.artifactdir
852		- while not os.path.exists(artifactdir_volume):
853		- artifactdir_volume = os.path.dirname(artifactdir_volume)
854		-
855	943	try:
856		- cache_quota = utils._parse_size(self.context.config_cache_quota, artifactdir_volume)
	944	+ cache_quota = utils._parse_size(self.context.config_cache_quota,
	945	+ self.context.artifactdir)
857	946	except utils.UtilError as e:
858	947	raise LoadError(LoadErrorReason.INVALID_DATA,
859	948	"{}\nPlease specify the value in bytes or as a % of full disk space.\n"
860	949	"\nValid values are, for example: 800M 10G 1T 50%\n"
861	950	.format(str(e))) from e
862	951
863		- available_space, total_size = self._get_volume_space_info_for(artifactdir_volume)
	952	+ total_size, available_space = self._get_cache_volume_size()
864	953	cache_size = self.get_cache_size()
865	954
866	955	# Ensure system has enough storage for the cache_quota
...	...	@@ -900,22 +989,25 @@ class ArtifactCache():
900	989	# if we end up writing more than 2G, but hey, this stuff is
901	990	# already really fuzzy.
902	991	#
	992	+ self._cache_quota_original = cache_quota
903	993	self._cache_quota = cache_quota - headroom
904	994	self._cache_lower_threshold = self._cache_quota / 2
905	995
906		- # _get_volume_space_info_for
907		- #
908		- # Get the available space and total space for the given volume
	996	+ # _get_cache_volume_size()
909	997	#
910		- # Args:
911		- # volume: volume for which to get the size
	998	+ # Get the available space and total space for the volume on
	999	+ # which the artifact cache is located.
912	1000	#
913	1001	# Returns:
914		- # A tuple containing first the availabe number of bytes on the requested
915		- # volume, then the total number of bytes of the volume.
916		- def _get_volume_space_info_for(self, volume):
917		- stat = os.statvfs(volume)
918		- return stat.f_bsize * stat.f_bavail, stat.f_bsize * stat.f_blocks
	1002	+ # (int): The total number of bytes on the volume
	1003	+ # (int): The number of available bytes on the volume
	1004	+ #
	1005	+ # NOTE: We use this stub to allow the test cases
	1006	+ # to override what an artifact cache thinks
	1007	+ # about it's disk size and available bytes.
	1008	+ #
	1009	+ def _get_cache_volume_size(self):
	1010	+ return utils._get_volume_size(self.context.artifactdir)
919	1011
920	1012
921	1013	# _configured_remote_artifact_cache_specs():

buildstream/_context.py

---

...	...	@@ -30,7 +30,7 @@ from . import _yaml
30	30	from ._exceptions import LoadError, LoadErrorReason, BstError
31	31	from ._message import Message, MessageType
32	32	from ._profile import Topics, profile_start, profile_end
33		-from ._artifactcache import ArtifactCache
	33	+from ._artifactcache import ArtifactCache, ArtifactCacheUsage
34	34	from ._cas import CASCache
35	35	from ._workspaces import Workspaces, WorkspaceProjectCache
36	36	from .plugin import _plugin_lookup
...	...	@@ -289,6 +289,16 @@ class Context():
289	289
290	290	return self._artifactcache
291	291
	292	+ # get_artifact_cache_usage()
	293	+ #
	294	+ # Fetches the current usage of the artifact cache
	295	+ #
	296	+ # Returns:
	297	+ # (ArtifactCacheUsage): The current status
	298	+ #
	299	+ def get_artifact_cache_usage(self):
	300	+ return ArtifactCacheUsage(self.artifactcache)
	301	+
292	302	# add_project():
293	303	#
294	304	# Add a project to the context.

buildstream/_frontend/app.py

---

...	...	@@ -194,11 +194,6 @@ class App():
194	194	except BstError as e:
195	195	self._error_exit(e, "Error instantiating platform")
196	196
197		- try:
198		- self.context.artifactcache.preflight()
199		- except BstError as e:
200		- self._error_exit(e, "Error instantiating artifact cache")
201		-
202	197	# Create the logger right before setting the message handler
203	198	self.logger = LogLine(self.context,
204	199	self._content_profile,
...	...	@@ -211,6 +206,13 @@ class App():
211	206	# Propagate pipeline feedback to the user
212	207	self.context.set_message_handler(self._message_handler)
213	208
	209	+ # Preflight the artifact cache after initializing logging,
	210	+ # this can cause messages to be emitted.
	211	+ try:
	212	+ self.context.artifactcache.preflight()
	213	+ except BstError as e:
	214	+ self._error_exit(e, "Error instantiating artifact cache")
	215	+
214	216	#
215	217	# Load the Project
216	218	#

buildstream/_frontend/status.py

---

...	...	@@ -353,13 +353,17 @@ class _StatusHeader():
353	353	def render(self, line_length, elapsed):
354	354	project = self._context.get_toplevel_project()
355	355	line_length = max(line_length, 80)
356		- size = 0
357		- text = ''
358	356
	357	+ #
	358	+ # Line 1: Session time, project name, session / total elements
	359	+ #
	360	+ # ========= 00:00:00 project-name (143/387) =========
	361	+ #
359	362	session = str(len(self._stream.session_elements))
360	363	total = str(len(self._stream.total_elements))
361	364
362		- # Format and calculate size for target and overall time code
	365	+ size = 0
	366	+ text = ''
363	367	size += len(total) + len(session) + 4 # Size for (N/N) with a leading space
364	368	size += 8 # Size of time code
365	369	size += len(project.name) + 1
...	...	@@ -372,6 +376,12 @@ class _StatusHeader():
372	376	self._format_profile.fmt(')')
373	377
374	378	line1 = self._centered(text, size, line_length, '=')
	379	+
	380	+ #
	381	+ # Line 2: Dynamic list of queue status reports
	382	+ #
	383	+ # (Fetched:0 117 0)→ (Built:4 0 0)
	384	+ #
375	385	size = 0
376	386	text = ''
377	387
...	...	@@ -389,10 +399,28 @@ class _StatusHeader():
389	399
390	400	line2 = self._centered(text, size, line_length, ' ')
391	401
392		- size = 24
393		- text = self._format_profile.fmt("~~~~~ ") + \
394		- self._content_profile.fmt('Active Tasks') + \
395		- self._format_profile.fmt(" ~~~~~")
	402	+ #
	403	+ # Line 3: Cache usage percentage report
	404	+ #
	405	+ # ~~~~~~ cache: 69% ~~~~~~
	406	+ #
	407	+ usage = self._context.get_artifact_cache_usage()
	408	+ usage_percent = '{}%'.format(usage.used_percent)
	409	+
	410	+ size = 21
	411	+ size += len(usage_percent)
	412	+ if usage.used_percent >= 95:
	413	+ formatted_usage_percent = self._error_profile.fmt(usage_percent)
	414	+ elif usage.used_percent >= 80:
	415	+ formatted_usage_percent = self._content_profile.fmt(usage_percent)
	416	+ else:
	417	+ formatted_usage_percent = self._success_profile.fmt(usage_percent)
	418	+
	419	+ text = self._format_profile.fmt("~~~~~~ ") + \
	420	+ self._content_profile.fmt('cache') + \
	421	+ self._format_profile.fmt(': ') + \
	422	+ formatted_usage_percent + \
	423	+ self._format_profile.fmt(' ~~~~~~')
396	424	line3 = self._centered(text, size, line_length, ' ')
397	425
398	426	return line1 + '\n' + line2 + '\n' + line3

buildstream/_frontend/widget.py

---

...	...	@@ -452,6 +452,7 @@ class LogLine(Widget):
452	452	values["Session Start"] = starttime.strftime('%A, %d-%m-%Y at %H:%M:%S')
453	453	values["Project"] = "{} ({})".format(project.name, project.directory)
454	454	values["Targets"] = ", ".join([t.name for t in stream.targets])
	455	+ values["Cache Usage"] = "{}".format(context.get_artifact_cache_usage())
455	456	text += self._format_values(values)
456	457
457	458	# User configurations

buildstream/_scheduler/jobs/cleanupjob.py

---

...	...	@@ -28,7 +28,20 @@ class CleanupJob(Job):
28	28	self._artifacts = context.artifactcache
29	29
30	30	def child_process(self):
31		- return self._artifacts.clean()
	31	+ def progress():
	32	+ self.send_message('update-cache-size',
	33	+ self._artifacts.get_cache_size())
	34	+ return self._artifacts.clean(progress)
	35	+
	36	+ def handle_message(self, message_type, message):
	37	+
	38	+ # Update the cache size in the main process as we go,
	39	+ # this provides better feedback in the UI.
	40	+ if message_type == 'update-cache-size':
	41	+ self._artifacts.set_cache_size(message)
	42	+ return True
	43	+
	44	+ return False
32	45
33	46	def parent_complete(self, status, result):
34	47	if status == JobStatus.OK:

buildstream/_scheduler/jobs/job.py

---

...	...	@@ -58,10 +58,10 @@ class JobStatus():
58	58
59	59
60	60	# Used to distinguish between status messages and return values
61		-class Envelope():
	61	+class _Envelope():
62	62	def __init__(self, message_type, message):
63		- self._message_type = message_type
64		- self._message = message
	63	+ self.message_type = message_type
	64	+ self.message = message
65	65
66	66
67	67	# Process class that doesn't call waitpid on its own.
...	...	@@ -275,10 +275,37 @@ class Job():
275	275	def set_task_id(self, task_id):
276	276	self._task_id = task_id
277	277
	278	+ # send_message()
	279	+ #
	280	+ # To be called from inside Job.child_process() implementations
	281	+ # to send messages to the main process during processing.
	282	+ #
	283	+ # These messages will be processed by the class's Job.handle_message()
	284	+ # implementation.
	285	+ #
	286	+ def send_message(self, message_type, message):
	287	+ self._queue.put(_Envelope(message_type, message))
	288	+
278	289	#######################################################
279	290	# Abstract Methods #
280	291	#######################################################
281	292
	293	+ # handle_message()
	294	+ #
	295	+ # Handle a custom message. This will be called in the main process in
	296	+ # response to any messages sent to the main proces using the
	297	+ # Job.send_message() API from inside a Job.child_process() implementation
	298	+ #
	299	+ # Args:
	300	+ # message_type (str): A string to identify the message type
	301	+ # message (any): A simple serializable object
	302	+ #
	303	+ # Returns:
	304	+ # (bool): Should return a truthy value if message_type is handled.
	305	+ #
	306	+ def handle_message(self, message_type, message):
	307	+ return False
	308	+
282	309	# parent_complete()
283	310	#
284	311	# This will be executed after the job finishes, and is expected to
...	...	@@ -416,7 +443,7 @@ class Job():
416	443	elapsed=elapsed, detail=e.detail,
417	444	logfile=filename, sandbox=e.sandbox)
418	445
419		- self._queue.put(Envelope('child_data', self.child_process_data()))
	446	+ self._queue.put(_Envelope('child_data', self.child_process_data()))
420	447
421	448	# Report the exception to the parent (for internal testing purposes)
422	449	self._child_send_error(e)
...	...	@@ -442,7 +469,7 @@ class Job():
442	469
443	470	else:
444	471	# No exception occurred in the action
445		- self._queue.put(Envelope('child_data', self.child_process_data()))
	472	+ self._queue.put(_Envelope('child_data', self.child_process_data()))
446	473	self._child_send_result(result)
447	474
448	475	elapsed = datetime.datetime.now() - starttime
...	...	@@ -469,7 +496,7 @@ class Job():
469	496	domain = e.domain
470	497	reason = e.reason
471	498
472		- envelope = Envelope('error', {
	499	+ envelope = _Envelope('error', {
473	500	'domain': domain,
474	501	'reason': reason
475	502	})
...	...	@@ -487,7 +514,7 @@ class Job():
487	514	#
488	515	def _child_send_result(self, result):
489	516	if result is not None:
490		- envelope = Envelope('result', result)
	517	+ envelope = _Envelope('result', result)
491	518	self._queue.put(envelope)
492	519
493	520	# _child_shutdown()
...	...	@@ -524,7 +551,7 @@ class Job():
524	551	if message.message_type == MessageType.LOG:
525	552	return
526	553
527		- self._queue.put(Envelope('message', message))
	554	+ self._queue.put(_Envelope('message', message))
528	555
529	556	# _parent_shutdown()
530	557	#
...	...	@@ -588,24 +615,28 @@ class Job():
588	615	if not self._listening:
589	616	return
590	617
591		- if envelope._message_type == 'message':
	618	+ if envelope.message_type == 'message':
592	619	# Propagate received messages from children
593	620	# back through the context.
594		- self._scheduler.context.message(envelope._message)
595		- elif envelope._message_type == 'error':
	621	+ self._scheduler.context.message(envelope.message)
	622	+ elif envelope.message_type == 'error':
596	623	# For regression tests only, save the last error domain / reason
597	624	# reported from a child task in the main process, this global state
598	625	# is currently managed in _exceptions.py
599		- set_last_task_error(envelope._message['domain'],
600		- envelope._message['reason'])
601		- elif envelope._message_type == 'result':
	626	+ set_last_task_error(envelope.message['domain'],
	627	+ envelope.message['reason'])
	628	+ elif envelope.message_type == 'result':
602	629	assert self._result is None
603		- self._result = envelope._message
604		- elif envelope._message_type == 'child_data':
	630	+ self._result = envelope.message
	631	+ elif envelope.message_type == 'child_data':
605	632	# If we retry a job, we assign a new value to this
606		- self.child_data = envelope._message
607		- else:
608		- raise Exception()
	633	+ self.child_data = envelope.message
	634	+
	635	+ # Try Job subclass specific messages now
	636	+ elif not self.handle_message(envelope.message_type,
	637	+ envelope.message):
	638	+ assert 0, "Unhandled message type '{}': {}" \
	639	+ .format(envelope.message_type, envelope.message)
609	640
610	641	# _parent_process_queue()
611	642	#

buildstream/utils.py

---

...	...	@@ -633,6 +633,27 @@ def _get_dir_size(path):
633	633	return get_size(path)
634	634
635	635
	636	+# _get_volume_size():
	637	+#
	638	+# Gets the overall usage and total size of a mounted filesystem in bytes.
	639	+#
	640	+# Args:
	641	+# path (str): The path to check
	642	+#
	643	+# Returns:
	644	+# (int): The total number of bytes on the volume
	645	+# (int): The number of available bytes on the volume
	646	+#
	647	+def _get_volume_size(path):
	648	+ try:
	649	+ stat_ = os.statvfs(path)
	650	+ except OSError as e:
	651	+ raise UtilError("Failed to retrieve stats on volume for path '{}': {}"
	652	+ .format(path, e)) from e
	653	+
	654	+ return stat_.f_bsize * stat_.f_blocks, stat_.f_bsize * stat_.f_bavail
	655	+
	656	+
636	657	# _parse_size():
637	658	#
638	659	# Convert a string representing data size to a number of
...	...	@@ -667,8 +688,7 @@ def _parse_size(size, volume):
667	688	if num > 100:
668	689	raise UtilError("{}% is not a valid percentage value.".format(num))
669	690
670		- stat_ = os.statvfs(volume)
671		- disk_size = stat_.f_blocks * stat_.f_bsize
	691	+ disk_size, _ = _get_volume_size(volume)
672	692
673	693	return disk_size * (num / 100)
674	694

tests/artifactcache/cache_size.py

---

...	...	@@ -80,7 +80,7 @@ def test_quota_over_1024T(cli, tmpdir):
80	80	_yaml.dump({'name': 'main'}, str(project.join("project.conf")))
81	81
82	82	volume_space_patch = mock.patch(
83		- "buildstream._artifactcache.ArtifactCache._get_volume_space_info_for",
	83	+ "buildstream._artifactcache.ArtifactCache._get_cache_volume_size",
84	84	autospec=True,
85	85	return_value=(1025 * TiB, 1025 * TiB)
86	86	)

tests/artifactcache/expiry.py

---

...	...	@@ -358,9 +358,9 @@ def test_invalid_cache_quota(cli, datafiles, tmpdir, quota, err_domain, err_reas
358	358	total_space = 10000
359	359
360	360	volume_space_patch = mock.patch(
361		- "buildstream._artifactcache.ArtifactCache._get_volume_space_info_for",
	361	+ "buildstream._artifactcache.ArtifactCache._get_cache_volume_size",
362	362	autospec=True,
363		- return_value=(free_space, total_space),
	363	+ return_value=(total_space, free_space),
364	364	)
365	365
366	366	cache_size_patch = mock.patch(

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.