[Notes] [Git][BuildStream/buildstream][jjardon/fedora_29] 6 commits: utils.py: Increase buffer size in sha256sum()

[Javier Jardón <gitlab mg gitlab com>]

Title: GitLab

Javier Jardón pushed to branch jjardon/fedora_29 at BuildStream / buildstream

Commits:

• a7aed65a
  by Jürg Billeter at 2019-02-12T05:52:07Z

  utils.py: Increase buffer size in sha256sum()
  
  Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
  factor 4, according to a very simple benchmark.
  

• 8b9e1d24
  by Jürg Billeter at 2019-02-12T05:52:07Z

  _cas/cascache.py: Increase buffer size in add_object()
  
  Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
  factor 4, according to a very simple benchmark.
  

• 86a9048a
  by Jürg Billeter at 2019-02-12T07:08:18Z

  Merge branch 'juerg/buffer-size' into 'master'
  
  Increase read buffer size to improve performance
  
  See merge request BuildStream/buildstream!1142

• 406efd57
  by Javier Jardón at 2019-02-12T12:08:59Z

  .gitlab-ci.yml: Add job to run test in current Fedora:29
  

• a6427838
  by Javier Jardón at 2019-02-12T12:08:59Z

  .gitlab-ci.yml: run all generic fedora test in current stable version:29
  

• 995cc132
  by Javier Jardón at 2019-02-12T12:09:40Z

  .gitlab-ci.yml: Remove "tests-python-3.7-stretch" job
  
  Fedora 29 includes python 3.7 so this is not needed anymore
  

3 changed files:

• .gitlab-ci.yml
• buildstream/_cas/cascache.py
• buildstream/utils.py

Changes:

.gitlab-ci.yml

---

...	...	@@ -53,26 +53,20 @@ tests-fedora-28:
53	53	image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:28-master-46405991
54	54	<<: *tests
55	55
56		-tests-ubuntu-18.04:
57		- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-ubuntu:18.04-master-46405991
	56	+tests-fedora-29:
	57	+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:29-master-47052095
58	58	<<: *tests
59	59
60		-tests-python-3.7-stretch:
61		- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-python:3.7-stretch-master-46405991
	60	+tests-ubuntu-18.04:
	61	+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-ubuntu:18.04-master-46405991
62	62	<<: *tests
63	63
64		- variables:
65		- # Note that we explicitly specify TOXENV in this case because this
66		- # image has both 3.6 and 3.7 versions. python3.6 cannot be removed because
67		- # some of our base dependencies declare it as their runtime dependency.
68		- TOXENV: py37
69		-
70	64	tests-centos-7.6:
71	65	<<: *tests
72	66	image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-centos:7.6.1810-master-46405991
73	67
74		-overnight-fedora-28-aarch64:
75		- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:aarch64-28-master-46405991
	68	+overnight-fedora-29-aarch64:
	69	+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:aarch64-29-master-47052095
76	70	tags:
77	71	- aarch64
78	72	<<: *tests
...	...	@@ -91,7 +85,7 @@ overnight-fedora-28-aarch64:
91	85	tests-unix:
92	86	# Use fedora here, to a) run a test on fedora and b) ensure that we
93	87	# can get rid of ostree - this is not possible with debian-8
94		- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:28-master-46405991
	88	+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:29-master-47052095
95	89	<<: *tests
96	90	variables:
97	91	BST_FORCE_BACKEND: "unix"
...	...	@@ -109,7 +103,7 @@ tests-unix:
109	103
110	104	tests-fedora-missing-deps:
111	105	# Ensure that tests behave nicely while missing bwrap and ostree
112		- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:28-master-46405991
	106	+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:29-master-47052095
113	107	<<: *tests
114	108
115	109	script:
...	...	@@ -128,7 +122,7 @@ tests-fedora-update-deps:
128	122	# Check if the tests pass after updating requirements to their latest
129	123	# allowed version.
130	124	allow_failure: true
131		- image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:28-master-46405991
	125	+ image: registry.gitlab.com/buildstream/buildstream-docker-images/testsuite-fedora:29-master-47052095
132	126	<<: *tests
133	127
134	128	script:
...	...	@@ -289,6 +283,7 @@ coverage:
289	283	dependencies:
290	284	- tests-debian-9
291	285	- tests-fedora-28
	286	+ - tests-fedora-29
292	287	- tests-fedora-missing-deps
293	288	- tests-ubuntu-18.04
294	289	- tests-unix

buildstream/_cas/cascache.py

---

...	...	@@ -35,6 +35,8 @@ from .._exceptions import CASCacheError
35	35
36	36	from .casremote import BlobNotFound, _CASBatchRead, _CASBatchUpdate
37	37
	38	+_BUFFER_SIZE = 65536
	39	+
38	40
39	41	# A CASCache manages a CAS repository as specified in the Remote Execution API.
40	42	#
...	...	@@ -371,7 +373,7 @@ class CASCache():
371	373	with contextlib.ExitStack() as stack:
372	374	if path is not None and link_directly:
373	375	tmp = stack.enter_context(open(path, 'rb'))
374		- for chunk in iter(lambda: tmp.read(4096), b""):
	376	+ for chunk in iter(lambda: tmp.read(_BUFFER_SIZE), b""):
375	377	h.update(chunk)
376	378	else:
377	379	tmp = stack.enter_context(utils._tempnamedfile(dir=self.tmpdir))
...	...	@@ -380,7 +382,7 @@ class CASCache():
380	382
381	383	if path:
382	384	with open(path, 'rb') as f:
383		- for chunk in iter(lambda: f.read(4096), b""):
	385	+ for chunk in iter(lambda: f.read(_BUFFER_SIZE), b""):
384	386	h.update(chunk)
385	387	tmp.write(chunk)
386	388	else:

buildstream/utils.py

---

...	...	@@ -235,7 +235,7 @@ def sha256sum(filename):
235	235	try:
236	236	h = hashlib.sha256()
237	237	with open(filename, "rb") as f:
238		- for chunk in iter(lambda: f.read(4096), b""):
	238	+ for chunk in iter(lambda: f.read(65536), b""):
239	239	h.update(chunk)
240	240
241	241	except OSError as e:

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.