[Notes] [Git][BuildStream/buildstream][bst-1.2] 4 commits: utils.py: Give save_file_atomic() a tempdir argument

[Tristan Van Berkom <gitlab mg gitlab com>]

Title: GitLab

Tristan Van Berkom pushed to branch bst-1.2 at BuildStream / buildstream

Commits:

• 262e789f
  by Tristan Van Berkom at 2018-10-03T13:03:52Z

  utils.py: Give save_file_atomic() a tempdir argument
  
  Allow callers to decide where the temporary file will be created.
  

• 10d69988
  by Tristan Van Berkom at 2018-10-03T13:03:56Z

  _artifactcache/cascache.py: Don't create temporary files in the CAS storage
  
  Use the designated tempdir when creating refs, we expect that temporary
  files are not created in the storage directory ever, they should be
  only ever created in the designated temporary directory.
  
  This fixes race conditions where utils._get_dir_size() throws an
  unhandled exception when attempting to stat the file which inadvertantly
  disappears.
  

• c02a1ae8
  by Tristan Van Berkom at 2018-10-03T13:04:03Z

  utils.py: Document _get_dir_size() expectations.
  
  This function assumes that files do not disappear while
  walking the given directory.
  

• eb92e8e9
  by Tristan Van Berkom at 2018-10-03T13:44:31Z

  Merge branch 'tristan/fix-cache-size-race-1.2' into 'bst-1.2'
  
  Tristan/fix cache size race 1.2
  
  See merge request BuildStream/buildstream!855

2 changed files:

• buildstream/_artifactcache/cascache.py
• buildstream/utils.py

Changes:

buildstream/_artifactcache/cascache.py

---

...	...	@@ -418,7 +418,7 @@ class CASCache(ArtifactCache):
418	418	def set_ref(self, ref, tree):
419	419	refpath = self._refpath(ref)
420	420	os.makedirs(os.path.dirname(refpath), exist_ok=True)
421		- with utils.save_file_atomic(refpath, 'wb') as f:
	421	+ with utils.save_file_atomic(refpath, 'wb', tempdir=self.tmpdir) as f:
422	422	f.write(tree.SerializeToString())
423	423
424	424	# resolve_ref():

buildstream/utils.py

---

...	...	@@ -496,7 +496,7 @@ def get_bst_version():
496	496
497	497	@contextmanager
498	498	def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
499		- errors=None, newline=None, closefd=True, opener=None):
	499	+ errors=None, newline=None, closefd=True, opener=None, tempdir=None):
500	500	"""Save a file with a temporary name and rename it into place when ready.
501	501
502	502	This is a context manager which is meant for saving data to files.
...	...	@@ -523,8 +523,9 @@ def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
523	523	# https://bugs.python.org/issue8604
524	524
525	525	assert os.path.isabs(filename), "The utils.save_file_atomic() parameter ``filename`` must be an absolute path"
526		- dirname = os.path.dirname(filename)
527		- fd, tempname = tempfile.mkstemp(dir=dirname)
	526	+ if tempdir is None:
	527	+ tempdir = os.path.dirname(filename)
	528	+ fd, tempname = tempfile.mkstemp(dir=tempdir)
528	529	os.close(fd)
529	530
530	531	f = open(tempname, mode=mode, buffering=buffering, encoding=encoding,
...	...	@@ -556,6 +557,9 @@ def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
556	557	#
557	558	# Get the disk usage of a given directory in bytes.
558	559	#
	560	+# This function assumes that files do not inadvertantly
	561	+# disappear while this function is running.
	562	+#
559	563	# Arguments:
560	564	# (str) The path whose size to check.
561	565	#

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.