[Notes] [Git][BuildStream/buildstream][tristan/fix-cache-size-race] 5 commits: _scheduler/scheduler.py: Ignore interrupt events while terminating.

[Tristan Van Berkom <gitlab mg gitlab com>]

Title: GitLab

Tristan Van Berkom pushed to branch tristan/fix-cache-size-race at BuildStream / buildstream

Commits:

• a0814399
  by Tristan Van Berkom at 2018-10-03T13:05:52Z

  _scheduler/scheduler.py: Ignore interrupt events while terminating.
  
  For some reason, we now receive a SIGINT from the main loop even
  when the SIGINT occurred with the handler disconnected in an interactive
  prompt. This patch simply ignores any received SIGINT events from the
  main loop in the case that we are already in the process of terminating.
  
  This fixes issue #693
  

• 0a1f8e3c
  by Tristan Van Berkom at 2018-10-03T13:42:20Z

  Merge branch 'tristan/fix-double-terminate-prompt' into 'master'
  
  _scheduler/scheduler.py: Ignore interrupt events while terminating.
  
  Closes #693
  
  See merge request BuildStream/buildstream!852

• 11161f99
  by Tristan Van Berkom at 2018-10-03T13:44:02Z

  utils.py: Give save_file_atomic() a tempdir argument
  
  Allow callers to decide where the temporary file will be created.
  

• 3e797bb9
  by Tristan Van Berkom at 2018-10-03T13:44:02Z

  _artifactcache/cascache.py: Don't create temporary files in the CAS storage
  
  Use the designated tempdir when creating refs, we expect that temporary
  files are not created in the storage directory ever, they should be
  only ever created in the designated temporary directory.
  
  This fixes race conditions where utils._get_dir_size() throws an
  unhandled exception when attempting to stat the file which inadvertantly
  disappears.
  

• d9020e43
  by Tristan Van Berkom at 2018-10-03T13:44:02Z

  utils.py: Document _get_dir_size() expectations.
  
  This function assumes that files do not disappear while
  walking the given directory.
  

3 changed files:

• buildstream/_artifactcache/cascache.py
• buildstream/_scheduler/scheduler.py
• buildstream/utils.py

Changes:

buildstream/_artifactcache/cascache.py

---

...	...	@@ -506,7 +506,7 @@ class CASCache(ArtifactCache):
506	506	def set_ref(self, ref, tree):
507	507	refpath = self._refpath(ref)
508	508	os.makedirs(os.path.dirname(refpath), exist_ok=True)
509		- with utils.save_file_atomic(refpath, 'wb') as f:
	509	+ with utils.save_file_atomic(refpath, 'wb', tempdir=self.tmpdir) as f:
510	510	f.write(tree.SerializeToString())
511	511
512	512	# resolve_ref():

buildstream/_scheduler/scheduler.py

---

...	...	@@ -387,6 +387,15 @@ class Scheduler():
387	387	# A loop registered event callback for keyboard interrupts
388	388	#
389	389	def _interrupt_event(self):
	390	+
	391	+ # FIXME: This should not be needed, but for some reason we receive an
	392	+ # additional SIGINT event when the user hits ^C a second time
	393	+ # to inform us that they really intend to terminate; even though
	394	+ # we have disconnected our handlers at this time.
	395	+ #
	396	+ if self.terminated:
	397	+ return
	398	+
390	399	# Leave this to the frontend to decide, if no
391	400	# interrrupt callback was specified, then just terminate.
392	401	if self._interrupt_callback:

buildstream/utils.py

---

...	...	@@ -502,7 +502,7 @@ def get_bst_version():
502	502
503	503	@contextmanager
504	504	def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
505		- errors=None, newline=None, closefd=True, opener=None):
	505	+ errors=None, newline=None, closefd=True, opener=None, tempdir=None):
506	506	"""Save a file with a temporary name and rename it into place when ready.
507	507
508	508	This is a context manager which is meant for saving data to files.
...	...	@@ -529,8 +529,9 @@ def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
529	529	# https://bugs.python.org/issue8604
530	530
531	531	assert os.path.isabs(filename), "The utils.save_file_atomic() parameter ``filename`` must be an absolute path"
532		- dirname = os.path.dirname(filename)
533		- fd, tempname = tempfile.mkstemp(dir=dirname)
	532	+ if tempdir is None:
	533	+ tempdir = os.path.dirname(filename)
	534	+ fd, tempname = tempfile.mkstemp(dir=tempdir)
534	535	os.close(fd)
535	536
536	537	f = open(tempname, mode=mode, buffering=buffering, encoding=encoding,
...	...	@@ -562,6 +563,9 @@ def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
562	563	#
563	564	# Get the disk usage of a given directory in bytes.
564	565	#
	566	+# This function assumes that files do not inadvertantly
	567	+# disappear while this function is running.
	568	+#
565	569	# Arguments:
566	570	# (str) The path whose size to check.
567	571	#

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.