[Notes] [Git][BuildStream/buildstream][jjardon/max-jobs] 2 commits: buildstream/_project.py:

[Javier Jardón <gitlab mg gitlab com>]

Title: GitLab

Javier Jardón pushed to branch jjardon/max-jobs at BuildStream / buildstream

Commits:

• 27c0aacc
  by Javier Jardón at 2018-08-09T13:23:13Z

  buildstream/_project.py:
  
  Multiprocessing with n_jobs given by multiprocessing.cpu_count() is not optimal for
  systems where not all CPU can be used (in particular, Docker, CI etc).
  
  As suggested in the multiprocessing docs
  (http://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count)
  using len(os.sched_getaffinity(0)) might be better.
  

• c728d0f2
  by Javier Jardón at 2018-08-09T13:26:04Z

  buildstream/_project.py: Restrict max-jobs
  
  even if the machine have mores cores available
  
  Patch taken from YBD: https://gitlab.com/baserock/ybd/blob/master/ybd/app.py#L227
  

1 changed file:

• buildstream/_project.py

Changes:

buildstream/_project.py

---

...	...	@@ -19,7 +19,6 @@
19	19	# Tiago Gomes <tiago gomes codethink co uk>
20	20
21	21	import os
22		-import multiprocessing # for cpu_count()
23	22	from collections import Mapping, OrderedDict
24	23	from pluginbase import PluginBase
25	24	from . import utils
...	...	@@ -572,7 +571,10 @@ class Project():
572	571
573	572	# Extend variables with automatic variables and option exports
574	573	# Initialize it as a string as all variables are processed as strings.
575		- output.base_variables['max-jobs'] = str(multiprocessing.cpu_count())
	574	+ # Based on some testing (mainly on AWS), maximum effective
	575	+ # max-jobs value seems to be around 8-10 if we have enough cores
	576	+ # users should set values based on workload and build infrastructure
	577	+ output.base_variables['max-jobs'] = str(min(len(os.sched_getaffinity(0), 8))
576	578
577	579	# Export options into variables, if that was requested
578	580	output.options.export_variables(output.base_variables)

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.