[Notes] [Git][BuildStream/buildstream][master] 3 commits: utils.py: Increase buffer size in sha256sum()

[Jürg Billeter <gitlab mg gitlab com>]

Title: GitLab

Jürg Billeter pushed to branch master at BuildStream / buildstream

Commits:

• a7aed65a
  by Jürg Billeter at 2019-02-12T05:52:07Z

  utils.py: Increase buffer size in sha256sum()
  
  Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
  factor 4, according to a very simple benchmark.
  

• 8b9e1d24
  by Jürg Billeter at 2019-02-12T05:52:07Z

  _cas/cascache.py: Increase buffer size in add_object()
  
  Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
  factor 4, according to a very simple benchmark.
  

• 86a9048a
  by Jürg Billeter at 2019-02-12T07:08:18Z

  Merge branch 'juerg/buffer-size' into 'master'
  
  Increase read buffer size to improve performance
  
  See merge request BuildStream/buildstream!1142

2 changed files:

• buildstream/_cas/cascache.py
• buildstream/utils.py

Changes:

buildstream/_cas/cascache.py

---

...	...	@@ -35,6 +35,8 @@ from .._exceptions import CASCacheError
35	35
36	36	from .casremote import BlobNotFound, _CASBatchRead, _CASBatchUpdate
37	37
	38	+_BUFFER_SIZE = 65536
	39	+
38	40
39	41	# A CASCache manages a CAS repository as specified in the Remote Execution API.
40	42	#
...	...	@@ -371,7 +373,7 @@ class CASCache():
371	373	with contextlib.ExitStack() as stack:
372	374	if path is not None and link_directly:
373	375	tmp = stack.enter_context(open(path, 'rb'))
374		- for chunk in iter(lambda: tmp.read(4096), b""):
	376	+ for chunk in iter(lambda: tmp.read(_BUFFER_SIZE), b""):
375	377	h.update(chunk)
376	378	else:
377	379	tmp = stack.enter_context(utils._tempnamedfile(dir=self.tmpdir))
...	...	@@ -380,7 +382,7 @@ class CASCache():
380	382
381	383	if path:
382	384	with open(path, 'rb') as f:
383		- for chunk in iter(lambda: f.read(4096), b""):
	385	+ for chunk in iter(lambda: f.read(_BUFFER_SIZE), b""):
384	386	h.update(chunk)
385	387	tmp.write(chunk)
386	388	else:

buildstream/utils.py

---

...	...	@@ -235,7 +235,7 @@ def sha256sum(filename):
235	235	try:
236	236	h = hashlib.sha256()
237	237	with open(filename, "rb") as f:
238		- for chunk in iter(lambda: f.read(4096), b""):
	238	+ for chunk in iter(lambda: f.read(65536), b""):
239	239	h.update(chunk)
240	240
241	241	except OSError as e:

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.