[Notes] [Git][BuildStream/buildstream][master] 3 commits: Cleanup internal Loader cache after loading elements

[Benjamin Schubert <gitlab mg gitlab com>]

Title: GitLab

Benjamin Schubert pushed to branch master at BuildStream / buildstream

Commits:

• ec4bbf35
  by Benjamin Schubert at 2019-02-13T14:59:47Z

  Cleanup internal Loader cache after loading elements
  

• b41a82d3
  by Benjamin Schubert at 2019-02-13T14:59:47Z

  Don't register exceptions when not running the testsuite
  
  This fix a problem with the garbage collector not being able to
  clean the MetaElements that are loaded.
  
  On small projects this is not a problem, but in bigger projects, this
  can save a few hundred of MBs at runtime
  
  The reason behind this is, whenever we have a "stack" element, which
  has no stack.yaml configuration, since it doesn't need it, we would
  get an exception thrown when initiating the first one, as loading the
  yaml file would fail.
  
  This would capture the frame in which this command was executed, which
  references meta_elements. Therefore, as long as another exception is not
  thrown, the garbage collector would not be able to clean all the
  MetaElements.
  

• 9db7f489
  by Benjamin Schubert at 2019-02-13T16:13:16Z

  Merge branch 'bschubert/cleanup-local-state' into 'master'
  
  Cleanup MetaElement local state
  
  See merge request BuildStream/buildstream!1147

4 changed files:

• buildstream/_exceptions.py
• buildstream/_loader/loader.py
• buildstream/_project.py
• buildstream/element.py

Changes:

buildstream/_exceptions.py

---

...	...	@@ -19,6 +19,7 @@
19	19	# Tiago Gomes <tiago gomes codethink co uk>
20	20
21	21	from enum import Enum
	22	+import os
22	23
23	24	# Disable pylint warnings for whole file here:
24	25	# pylint: disable=global-statement
...	...	@@ -50,6 +51,9 @@ def get_last_exception():
50	51	# Used by regression tests
51	52	#
52	53	def get_last_task_error():
	54	+ if 'BST_TEST_SUITE' not in os.environ:
	55	+ raise BstError("Getting the last task error is only supported when running tests")
	56	+
53	57	global _last_task_error_domain
54	58	global _last_task_error_reason
55	59
...	...	@@ -67,11 +71,12 @@ def get_last_task_error():
67	71	# tests about how things failed in a machine readable way
68	72	#
69	73	def set_last_task_error(domain, reason):
70		- global _last_task_error_domain
71		- global _last_task_error_reason
	74	+ if 'BST_TEST_SUITE' in os.environ:
	75	+ global _last_task_error_domain
	76	+ global _last_task_error_reason
72	77
73		- _last_task_error_domain = domain
74		- _last_task_error_reason = reason
	78	+ _last_task_error_domain = domain
	79	+ _last_task_error_reason = reason
75	80
76	81
77	82	class ErrorDomain(Enum):
...	...	@@ -126,7 +131,8 @@ class BstError(Exception):
126	131	self.reason = reason
127	132
128	133	# Hold on to the last raised exception for testing purposes
129		- _last_exception = self
	134	+ if 'BST_TEST_SUITE' in os.environ:
	135	+ _last_exception = self
130	136
131	137
132	138	# PluginError

buildstream/_loader/loader.py

---

...	...	@@ -152,8 +152,27 @@ class Loader():
152	152	#
153	153	ret.append(loader._collect_element(element))
154	154
	155	+ self._clean_caches()
	156	+
155	157	return ret
156	158
	159	+ # clean_caches()
	160	+ #
	161	+ # Clean internal loader caches, recursively
	162	+ #
	163	+ # When loading the elements, the loaders use caches in order to not load the
	164	+ # same element twice. These are kept after loading and prevent garbage
	165	+ # collection. Cleaning them explicitely is required.
	166	+ #
	167	+ def _clean_caches(self):
	168	+ for loader in self._loaders.values():
	169	+ # value may be None with nested junctions without overrides
	170	+ if loader is not None:
	171	+ loader._clean_caches()
	172	+
	173	+ self._meta_elements = {}
	174	+ self._elements = {}
	175	+
157	176	###########################################
158	177	# Private Methods #
159	178	###########################################

buildstream/_project.py

---

...	...	@@ -358,6 +358,8 @@ class Project():
358	358	for meta in meta_elements
359	359	]
360	360
	361	+ Element._clear_meta_elements_cache()
	362	+
361	363	# Now warn about any redundant source references which may have
362	364	# been discovered in the resolve() phase.
363	365	redundant_refs = Element._get_redundant_source_refs()

buildstream/element.py

---

...	...	@@ -966,6 +966,21 @@ class Element(Plugin):
966	966
967	967	return element
968	968
	969	+ # _clear_meta_elements_cache()
	970	+ #
	971	+ # Clear the internal meta elements cache.
	972	+ #
	973	+ # When loading elements from meta, we cache already instantiated elements
	974	+ # in order to not have to load the same elements twice.
	975	+ # This clears the cache.
	976	+ #
	977	+ # It should be called whenever we are done loading all elements in order
	978	+ # to save memory.
	979	+ #
	980	+ @classmethod
	981	+ def _clear_meta_elements_cache(cls):
	982	+ cls.__instantiated_elements = {}
	983	+
969	984	# _get_redundant_source_refs()
970	985	#
971	986	# Fetches a list of (Source, ref) tuples of all the Sources

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.