[Notes] [Git][BuildStream/buildstream][master] 4 commits: utils.py: Give save_file_atomic() a tempdir argument

[Tristan Van Berkom <gitlab mg gitlab com>]

Title: GitLab

Tristan Van Berkom pushed to branch master at BuildStream / buildstream

Commits:

• 11161f99
  by Tristan Van Berkom at 2018-10-03T13:44:02Z

  utils.py: Give save_file_atomic() a tempdir argument
  
  Allow callers to decide where the temporary file will be created.
  

• 3e797bb9
  by Tristan Van Berkom at 2018-10-03T13:44:02Z

  _artifactcache/cascache.py: Don't create temporary files in the CAS storage
  
  Use the designated tempdir when creating refs, we expect that temporary
  files are not created in the storage directory ever, they should be
  only ever created in the designated temporary directory.
  
  This fixes race conditions where utils._get_dir_size() throws an
  unhandled exception when attempting to stat the file which inadvertantly
  disappears.
  

• d9020e43
  by Tristan Van Berkom at 2018-10-03T13:44:02Z

  utils.py: Document _get_dir_size() expectations.
  
  This function assumes that files do not disappear while
  walking the given directory.
  

• 3e5ff5a9
  by Tristan Van Berkom at 2018-10-03T14:09:51Z

  Merge branch 'tristan/fix-cache-size-race' into 'master'
  
  fix cache size race
  
  See merge request BuildStream/buildstream!854

2 changed files:

• buildstream/_artifactcache/cascache.py
• buildstream/utils.py

Changes:

buildstream/_artifactcache/cascache.py

---

...	...	@@ -506,7 +506,7 @@ class CASCache(ArtifactCache):
506	506	def set_ref(self, ref, tree):
507	507	refpath = self._refpath(ref)
508	508	os.makedirs(os.path.dirname(refpath), exist_ok=True)
509		- with utils.save_file_atomic(refpath, 'wb') as f:
	509	+ with utils.save_file_atomic(refpath, 'wb', tempdir=self.tmpdir) as f:
510	510	f.write(tree.SerializeToString())
511	511
512	512	# resolve_ref():

buildstream/utils.py

---

...	...	@@ -502,7 +502,7 @@ def get_bst_version():
502	502
503	503	@contextmanager
504	504	def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
505		- errors=None, newline=None, closefd=True, opener=None):
	505	+ errors=None, newline=None, closefd=True, opener=None, tempdir=None):
506	506	"""Save a file with a temporary name and rename it into place when ready.
507	507
508	508	This is a context manager which is meant for saving data to files.
...	...	@@ -529,8 +529,9 @@ def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
529	529	# https://bugs.python.org/issue8604
530	530
531	531	assert os.path.isabs(filename), "The utils.save_file_atomic() parameter ``filename`` must be an absolute path"
532		- dirname = os.path.dirname(filename)
533		- fd, tempname = tempfile.mkstemp(dir=dirname)
	532	+ if tempdir is None:
	533	+ tempdir = os.path.dirname(filename)
	534	+ fd, tempname = tempfile.mkstemp(dir=tempdir)
534	535	os.close(fd)
535	536
536	537	f = open(tempname, mode=mode, buffering=buffering, encoding=encoding,
...	...	@@ -562,6 +563,9 @@ def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
562	563	#
563	564	# Get the disk usage of a given directory in bytes.
564	565	#
	566	+# This function assumes that files do not inadvertantly
	567	+# disappear while this function is running.
	568	+#
565	569	# Arguments:
566	570	# (str) The path whose size to check.
567	571	#

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.