[Notes] [Git][BuildStream/buildstream][master] 3 commits: requirements: Add roaringbitmap to the requirements

[Benjamin Schubert <gitlab mg gitlab com>]

Title: GitLab

Benjamin Schubert pushed to branch master at BuildStream / buildstream

Commits:

• 9c94e8e5
  by Daniel Silverstone at 2019-02-05T13:15:24Z

  requirements: Add roaringbitmap to the requirements
  
  In order to support use of roaring bitmaps in the loader, we need to
  depend on it here.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 7682ef49
  by Daniel Silverstone at 2019-02-05T13:15:24Z

  loadelement.py: Use roaring bitmaps in dep cache
  
  The dependency cache in LoadElement causes the peak RAM consumption of
  the loader to be exceedingly large, upwards of 25GB for a test of a
  Debian stack.  By switching from the old dict cache to a roaring bitmap
  based cache, we reduce the cost of the loader cache in RAM terms to
  around 5.5GB for the same stack, which makes it plausible for the short
  term.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 0e471144
  by Benjamin Schubert at 2019-02-05T15:45:37Z

  Merge branch 'danielsilverstone-ct/roaring-bitmaps' into 'master'
  
  Switch to roaring bitmaps for the loader dependency caches
  
  See merge request BuildStream/buildstream!1128

3 changed files:

• buildstream/_loader/loadelement.py
• requirements/requirements.in
• requirements/requirements.txt

Changes:

buildstream/_loader/loadelement.py

---

...	...	@@ -19,6 +19,9 @@
19	19
20	20	# System imports
21	21	from collections.abc import Mapping
	22	+from itertools import count
	23	+
	24	+from roaringbitmap import RoaringBitmap, ImmutableRoaringBitmap # pylint: disable=no-name-in-module
22	25
23	26	# BuildStream toplevel imports
24	27	from .._exceptions import LoadError, LoadErrorReason
...	...	@@ -54,6 +57,8 @@ class LoadElement():
54	57	self.element = element
55	58	self.dep_type = dep_type
56	59
	60	+ _counter = count()
	61	+
57	62	def __init__(self, node, filename, loader):
58	63
59	64	#
...	...	@@ -63,6 +68,7 @@ class LoadElement():
63	68	self.name = filename # The element name
64	69	self.full_name = None # The element full name (with associated junction)
65	70	self.deps = None # The list of Dependency objects
	71	+ self.node_id = next(self._counter)
66	72
67	73	#
68	74	# Private members
...	...	@@ -107,7 +113,7 @@ class LoadElement():
107	113	#
108	114	def depends(self, other):
109	115	self._ensure_depends_cache()
110		- return self._dep_cache.get(other.full_name) is not None
	116	+ return other.node_id in self._dep_cache
111	117
112	118	###########################################
113	119	# Private Methods #
...	...	@@ -117,7 +123,8 @@ class LoadElement():
117	123	if self._dep_cache:
118	124	return
119	125
120		- self._dep_cache = {}
	126	+ self._dep_cache = RoaringBitmap()
	127	+
121	128	for dep in self.dependencies:
122	129	elt = dep.element
123	130
...	...	@@ -125,11 +132,13 @@ class LoadElement():
125	132	elt._ensure_depends_cache()
126	133
127	134	# We depend on this element
128		- self._dep_cache[elt.full_name] = True
	135	+ self._dep_cache.add(elt.node_id)
129	136
130	137	# And we depend on everything this element depends on
131	138	self._dep_cache.update(elt._dep_cache)
132	139
	140	+ self._dep_cache = ImmutableRoaringBitmap(self._dep_cache)
	141	+
133	142
134	143	# _extract_depends_from_node():
135	144	#

requirements/requirements.in

---

...	...	@@ -13,3 +13,6 @@ psutil
13	13	# See issues #571 and #790.
14	14	ruamel.yaml >= 0.15.41, < 0.15.52
15	15	setuptools
	16	+# (Potentially) short-term need for roaring bitmaps for the
	17	+# loader dependency sorting
	18	+roaringbitmap

requirements/requirements.txt

---

...	...	@@ -13,6 +13,9 @@ psutil==5.4.8
13	13	# See issues #571 and #790.
14	14	ruamel.yaml==0.15.51
15	15	setuptools==39.0.1
	16	+# (Potentially) short-term need for roaring bitmaps for the
	17	+# loader dependency sorting
	18	+roaringbitmap==0.6
16	19	## The following requirements were added by pip freeze:
17	20	MarkupSafe==1.1.0
18	21	six==1.12.0

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.