[Notes] [Git][BuildStream/buildstream][jjardon/max-jobs] 4 commits: Keep original flags for create in SafeHardlinks.

[Javier Jardón <gitlab mg gitlab com>]

Title: GitLab

Javier Jardón pushed to branch jjardon/max-jobs at BuildStream / buildstream

Commits:

• 8aa33e23
  by Valentin David at 2018-08-09T12:49:17Z

  Keep original flags for create in SafeHardlinks.
  
  When open(2) is used with flags O_CREAT|O_RDWR, the file descriptor
  must be readable. Unfortunately O_RDWR was not passed which made
  read fail with EBADF and mmap to signal SIGBUS.
  
  This issue happened with man-db for example.
  
  Fixes #143.
  

• ef7810f3
  by Valentin David at 2018-08-09T14:06:50Z

  Merge branch 'valentindavid/fuse-create-flags' into 'master'
  
  Keep original flags for create in SafeHardlinks.
  
  Closes #143
  
  See merge request BuildStream/buildstream!624

• 2d061173
  by Javier Jardón at 2018-08-09T16:31:15Z

  buildstream/_project.py:
  
  Multiprocessing with n_jobs given by multiprocessing.cpu_count() is not optimal for
  systems where not all CPU can be used (in particular, Docker, CI etc).
  
  As suggested in the multiprocessing docs
  (http://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count)
  using len(os.sched_getaffinity(0)) might be better.
  

• 4e1488ee
  by Javier Jardón at 2018-08-09T16:31:15Z

  buildstream/_project.py: Restrict max-jobs
  
  even if the machine have mores cores available
  
  Patch taken from YBD: https://gitlab.com/baserock/ybd/blob/master/ybd/app.py#L227
  

3 changed files:

• buildstream/_fuse/fuse.py
• buildstream/_fuse/hardlinks.py
• buildstream/_project.py

Changes:

buildstream/_fuse/fuse.py

---

...	...	@@ -757,7 +757,11 @@ class FUSE(object):
757	757	if self.raw_fi:
758	758	return self.operations('create', path, mode, fi)
759	759	else:
760		- fi.fh = self.operations('create', path, mode)
	760	+ # This line is different from upstream to fix issues
	761	+ # reading file opened with O_CREAT|O_RDWR.
	762	+ # See issue #143.
	763	+ fi.fh = self.operations('create', path, mode, fi.flags)
	764	+ # END OF MODIFICATION
761	765	return 0
762	766
763	767	def ftruncate(self, path, length, fip):

buildstream/_fuse/hardlinks.py

---

...	...	@@ -185,12 +185,12 @@ class SafeHardlinkOps(Operations):
185	185
186	186	return os.open(full_path, flags)
187	187
188		- def create(self, path, mode, fi=None):
	188	+ def create(self, path, mode, flags):
189	189	full_path = self._full_path(path)
190	190
191	191	# If it already exists, ensure it's a copy first
192	192	self._ensure_copy(full_path)
193		- return os.open(full_path, os.O_WRONLY | os.O_CREAT, mode)
	193	+ return os.open(full_path, flags, mode)
194	194
195	195	def read(self, path, length, offset, fh):
196	196	os.lseek(fh, offset, os.SEEK_SET)

buildstream/_project.py

---

...	...	@@ -19,7 +19,6 @@
19	19	# Tiago Gomes <tiago gomes codethink co uk>
20	20
21	21	import os
22		-import multiprocessing # for cpu_count()
23	22	from collections import Mapping, OrderedDict
24	23	from pluginbase import PluginBase
25	24	from . import utils
...	...	@@ -572,7 +571,10 @@ class Project():
572	571
573	572	# Extend variables with automatic variables and option exports
574	573	# Initialize it as a string as all variables are processed as strings.
575		- output.base_variables['max-jobs'] = str(multiprocessing.cpu_count())
	574	+ # Based on some testing (mainly on AWS), maximum effective
	575	+ # max-jobs value seems to be around 8-10 if we have enough cores
	576	+ # users should set values based on workload and build infrastructure
	577	+ output.base_variables['max-jobs'] = str(min(len(os.sched_getaffinity(0)), 8))
576	578
577	579	# Export options into variables, if that was requested
578	580	output.options.export_variables(output.base_variables)

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.