[Notes] [Git][BuildStream/buildstream][tristan/fix-cache-exclusivity] 3 commits: _scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE

[Tristan Van Berkom <gitlab mg gitlab com>]

Title: GitLab

Tristan Van Berkom pushed to branch tristan/fix-cache-exclusivity at BuildStream / buildstream

Commits:

• a569b561
  by Tristan Van Berkom at 2018-09-05T12:20:35Z

  _scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE
  
  This is what the whole resource.py thing was created for, the
  cleanup job must have exclusive access to the cache, while the pull
  and build jobs which result in adding new artifacts, must only require
  shared access.
  
  This is a part of #623
  

• fe8e397b
  by Tristan Van Berkom at 2018-09-05T12:20:43Z

  _scheduler/resources.py: Set CACHE max resources to 0
  
  This means there is no cap for shared resource requests.
  
  Together with the previous commit, this causes the cleanup
  job and the pull/build jobs to all require unlimited shared
  access to the CACHE resource; while only the cleanup job
  requires exclusive access to the resource.
  
  This is a part of #623
  

• 3dba9c76
  by Tristan Van Berkom at 2018-09-05T12:20:45Z

  _scheduler/scheduler.py: Make CacheSizeJob() not require the CACHE exclusively
  
  This runs after every pull, and does not need the cache exclusively,
  only the cleanup job requires the cache exclusively.
  
  Without this, every time a cache_size job is queued, all pull and
  build jobs need to complete before cache_size job can run exclusively,
  which is not good.
  
  This is a part of #623
  

4 changed files:

• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/queues/pullqueue.py
• buildstream/_scheduler/resources.py
• buildstream/_scheduler/scheduler.py

Changes:

buildstream/_scheduler/queues/buildqueue.py

---

...	...	@@ -32,7 +32,7 @@ class BuildQueue(Queue):
32	32
33	33	action_name = "Build"
34	34	complete_name = "Built"
35		- resources = [ResourceType.PROCESS]
	35	+ resources = [ResourceType.PROCESS, ResourceType.CACHE]
36	36
37	37	def __init__(self, *args, **kwargs):
38	38	super().__init__(*args, **kwargs)

buildstream/_scheduler/queues/pullqueue.py

---

...	...	@@ -29,7 +29,7 @@ class PullQueue(Queue):
29	29
30	30	action_name = "Pull"
31	31	complete_name = "Pulled"
32		- resources = [ResourceType.DOWNLOAD]
	32	+ resources = [ResourceType.DOWNLOAD, ResourceType.CACHE]
33	33
34	34	def process(self, element):
35	35	# returns whether an artifact was downloaded or not

buildstream/_scheduler/resources.py

---

...	...	@@ -8,7 +8,7 @@ class ResourceType():
8	8	class Resources():
9	9	def __init__(self, num_builders, num_fetchers, num_pushers):
10	10	self._max_resources = {
11		- ResourceType.CACHE: 1,
	11	+ ResourceType.CACHE: 0,
12	12	ResourceType.DOWNLOAD: num_fetchers,
13	13	ResourceType.PROCESS: num_builders,
14	14	ResourceType.UPLOAD: num_pushers

buildstream/_scheduler/scheduler.py

---

...	...	@@ -332,7 +332,6 @@ class Scheduler():
332	332	job = CacheSizeJob(self, 'cache_size', 'cache_size/cache_size',
333	333	resources=[ResourceType.CACHE,
334	334	ResourceType.PROCESS],
335		- exclusive_resources=[ResourceType.CACHE],
336	335	complete_cb=self._run_cleanup)
337	336	self.schedule_jobs([job])
338	337

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.