[Notes] [Git][BuildStream/buildstream][bschubert/pipeline] 2 commits: buildqueue: only update state of direct reverse dependencies

[Benjamin Schubert <gitlab mg gitlab com>]

Title: GitLab

Benjamin Schubert pushed to branch bschubert/pipeline at BuildStream / buildstream

Commits:

• 07ce3f9f
  by Benjamin Schubert at 2019-01-05T10:55:43Z

  buildqueue: only update state of direct reverse dependencies
  
  Previously, we would update the state of all elements in the pipeline,
  everytime an element in the queue would change state.
  
  This is inefficient. We know that the only impacted elements will
  be the direct build reverse dependencies. We can therefore, by recording
  the direct build reverse dependencies, only update those and reduce
  the time we spend updating states.
  

• c980f8d1
  by Benjamin Schubert at 2019-01-05T10:55:43Z

  WIP: notes
  

4 changed files:

• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/queues/fetchqueue.py
• buildstream/_scheduler/queues/pullqueue.py
• buildstream/element.py

Changes:

buildstream/_scheduler/queues/buildqueue.py

---

...	...	@@ -71,9 +71,6 @@ class BuildQueue(Queue):
71	71	return element._assemble()
72	72
73	73	def status(self, element):
74		- # state of dependencies may have changed, recalculate element state
75		- element._update_state()
76		-
77	74	if not element._is_required():
78	75	# Artifact is not currently required but it may be requested later.
79	76	# Keep it in the queue.
...	...	@@ -109,6 +106,10 @@ class BuildQueue(Queue):
109	106	# Inform element in main process that assembly is done
110	107	element._assemble_done()
111	108
	109	+ # Update the state of all reverse dependencies
	110	+ for reverse_dependency in element.reverse_build_dependencies:
	111	+ reverse_dependency._update_state()
	112	+
112	113	# This has to be done after _assemble_done, such that the
113	114	# element may register its cache key as required
114	115	#

buildstream/_scheduler/queues/fetchqueue.py

---

...	...	@@ -44,6 +44,7 @@ class FetchQueue(Queue):
44	44
45	45	def status(self, element):
46	46	# state of dependencies may have changed, recalculate element state
	47	+ # FIXME: we should never be doing this
47	48	element._update_state()
48	49
49	50	if not element._is_required():

buildstream/_scheduler/queues/pullqueue.py

---

...	...	@@ -39,6 +39,7 @@ class PullQueue(Queue):
39	39
40	40	def status(self, element):
41	41	# state of dependencies may have changed, recalculate element state
	42	+ # FIXME: this should never be needed
42	43	element._update_state()
43	44
44	45	if not element._is_required():

buildstream/element.py

---

...	...	@@ -197,6 +197,7 @@ class Element(Plugin):
197	197
198	198	self.__runtime_dependencies = [] # Direct runtime dependency Elements
199	199	self.__build_dependencies = [] # Direct build dependency Elements
	200	+ self.reverse_build_dependencies = [] # Direct reverse build dependency Elements
200	201	self.__sources = [] # List of Sources
201	202	self.__weak_cache_key = None # Our cached weak cache key
202	203	self.__strict_cache_key = None # Our cached cache key for strict builds
...	...	@@ -930,6 +931,7 @@ class Element(Plugin):
930	931	for meta_dep in meta.build_dependencies:
931	932	dependency = Element._new_from_meta(meta_dep)
932	933	element.__build_dependencies.append(dependency)
	934	+ dependency.reverse_build_dependencies.append(element)
933	935
934	936	return element
935	937
...	...	@@ -1446,6 +1448,7 @@ class Element(Plugin):
1446	1448	# This unblocks pull/fetch/build.
1447	1449	#
1448	1450	def _set_required(self):
	1451	+ # FIXME: this should enqueue stuff in the queue, it should not be here by default
1449	1452	if self.__required:
1450	1453	# Already done
1451	1454	return
...	...	@@ -1456,6 +1459,7 @@ class Element(Plugin):
1456	1459	for dep in self.dependencies(Scope.RUN, recurse=False):
1457	1460	dep._set_required()
1458	1461
	1462	+ # FIXME: this should not be done at all here
1459	1463	self._update_state()
1460	1464
1461	1465	# _is_required():

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.