[Notes] [Git][BuildStream/buildstream][danielsilverstone-ct/roaring-bitmaps] loadelement.py: Use roaring bitmaps in dep cache

[Daniel Silverstone <gitlab mg gitlab com>]

Title: GitLab

Daniel Silverstone pushed to branch danielsilverstone-ct/roaring-bitmaps at BuildStream / buildstream

Commits:

• a0529039
  by Daniel Silverstone at 2019-02-05T11:44:21Z

  loadelement.py: Use roaring bitmaps in dep cache
  
  The dependency cache in LoadElement causes the peak RAM consumption of
  the loader to be exceedingly large, upwards of 25GB for a test of a
  Debian stack.  By switching from the old dict cache to a roaring bitmap
  based cache, we reduce the cost of the loader cache in RAM terms to
  around 5.5GB for the same stack, which makes it plausible for the short
  term.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

1 changed file:

• buildstream/_loader/loadelement.py

Changes:

buildstream/_loader/loadelement.py

---

...	...	@@ -19,6 +19,9 @@
19	19
20	20	# System imports
21	21	from collections.abc import Mapping
	22	+from itertools import count
	23	+
	24	+from roaringbitmap import RoaringBitmap, ImmutableRoaringBitmap # pylint: disable=no-name-in-module
22	25
23	26	# BuildStream toplevel imports
24	27	from .._exceptions import LoadError, LoadErrorReason
...	...	@@ -54,6 +57,8 @@ class LoadElement():
54	57	self.element = element
55	58	self.dep_type = dep_type
56	59
	60	+ _counter = count()
	61	+
57	62	def __init__(self, node, filename, loader):
58	63
59	64	#
...	...	@@ -63,6 +68,7 @@ class LoadElement():
63	68	self.name = filename # The element name
64	69	self.full_name = None # The element full name (with associated junction)
65	70	self.deps = None # The list of Dependency objects
	71	+ self.node_id = next(self._counter)
66	72
67	73	#
68	74	# Private members
...	...	@@ -107,7 +113,7 @@ class LoadElement():
107	113	#
108	114	def depends(self, other):
109	115	self._ensure_depends_cache()
110		- return self._dep_cache.get(other.full_name) is not None
	116	+ return other.node_id in self._dep_cache
111	117
112	118	###########################################
113	119	# Private Methods #
...	...	@@ -117,7 +123,8 @@ class LoadElement():
117	123	if self._dep_cache:
118	124	return
119	125
120		- self._dep_cache = {}
	126	+ self._dep_cache = RoaringBitmap()
	127	+
121	128	for dep in self.dependencies:
122	129	elt = dep.element
123	130
...	...	@@ -125,11 +132,13 @@ class LoadElement():
125	132	elt._ensure_depends_cache()
126	133
127	134	# We depend on this element
128		- self._dep_cache[elt.full_name] = True
	135	+ self._dep_cache.add(elt.node_id)
129	136
130	137	# And we depend on everything this element depends on
131	138	self._dep_cache.update(elt._dep_cache)
132	139
	140	+ self._dep_cache = ImmutableRoaringBitmap(self._dep_cache)
	141	+
133	142
134	143	# _extract_depends_from_node():
135	144	#

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.