[Notes] [Git][BuildStream/buildstream][chandan/dot-graph] 4 commits: Cleanup internal Loader cache after loading elements

[Chandan Singh <gitlab mg gitlab com>]

Title: GitLab

Chandan Singh pushed to branch chandan/dot-graph at BuildStream / buildstream

Commits:

• ec4bbf35
  by Benjamin Schubert at 2019-02-13T14:59:47Z

  Cleanup internal Loader cache after loading elements
  

• b41a82d3
  by Benjamin Schubert at 2019-02-13T14:59:47Z

  Don't register exceptions when not running the testsuite
  
  This fix a problem with the garbage collector not being able to
  clean the MetaElements that are loaded.
  
  On small projects this is not a problem, but in bigger projects, this
  can save a few hundred of MBs at runtime
  
  The reason behind this is, whenever we have a "stack" element, which
  has no stack.yaml configuration, since it doesn't need it, we would
  get an exception thrown when initiating the first one, as loading the
  yaml file would fail.
  
  This would capture the frame in which this command was executed, which
  references meta_elements. Therefore, as long as another exception is not
  thrown, the garbage collector would not be able to clean all the
  MetaElements.
  

• 9db7f489
  by Benjamin Schubert at 2019-02-13T16:13:16Z

  Merge branch 'bschubert/cleanup-local-state' into 'master'
  
  Cleanup MetaElement local state
  
  See merge request BuildStream/buildstream!1147

• f2d72ba9
  by Chandan Singh at 2019-02-13T16:15:03Z

  contrib/bst-graph: Add script to print graph in DOT format
  
  This script leverages the recently added format strings
  (`%{build-deps}`, `%{runtime-deps}`) to `bst show` to print a graph in
  DOT format. This requires users to have the `graphviz` python package
  installed.
  
  Additionally, users can also render the graph using the `--format`
  option if they have the `graphviz` command line tool installed.
  

5 changed files:

• buildstream/_exceptions.py
• buildstream/_loader/loader.py
• buildstream/_project.py
• buildstream/element.py
• + contrib/bst-graph

Changes:

buildstream/_exceptions.py

---

...	...	@@ -19,6 +19,7 @@
19	19	# Tiago Gomes <tiago gomes codethink co uk>
20	20
21	21	from enum import Enum
	22	+import os
22	23
23	24	# Disable pylint warnings for whole file here:
24	25	# pylint: disable=global-statement
...	...	@@ -50,6 +51,9 @@ def get_last_exception():
50	51	# Used by regression tests
51	52	#
52	53	def get_last_task_error():
	54	+ if 'BST_TEST_SUITE' not in os.environ:
	55	+ raise BstError("Getting the last task error is only supported when running tests")
	56	+
53	57	global _last_task_error_domain
54	58	global _last_task_error_reason
55	59
...	...	@@ -67,11 +71,12 @@ def get_last_task_error():
67	71	# tests about how things failed in a machine readable way
68	72	#
69	73	def set_last_task_error(domain, reason):
70		- global _last_task_error_domain
71		- global _last_task_error_reason
	74	+ if 'BST_TEST_SUITE' in os.environ:
	75	+ global _last_task_error_domain
	76	+ global _last_task_error_reason
72	77
73		- _last_task_error_domain = domain
74		- _last_task_error_reason = reason
	78	+ _last_task_error_domain = domain
	79	+ _last_task_error_reason = reason
75	80
76	81
77	82	class ErrorDomain(Enum):
...	...	@@ -126,7 +131,8 @@ class BstError(Exception):
126	131	self.reason = reason
127	132
128	133	# Hold on to the last raised exception for testing purposes
129		- _last_exception = self
	134	+ if 'BST_TEST_SUITE' in os.environ:
	135	+ _last_exception = self
130	136
131	137
132	138	# PluginError

buildstream/_loader/loader.py

---

...	...	@@ -152,8 +152,27 @@ class Loader():
152	152	#
153	153	ret.append(loader._collect_element(element))
154	154
	155	+ self._clean_caches()
	156	+
155	157	return ret
156	158
	159	+ # clean_caches()
	160	+ #
	161	+ # Clean internal loader caches, recursively
	162	+ #
	163	+ # When loading the elements, the loaders use caches in order to not load the
	164	+ # same element twice. These are kept after loading and prevent garbage
	165	+ # collection. Cleaning them explicitely is required.
	166	+ #
	167	+ def _clean_caches(self):
	168	+ for loader in self._loaders.values():
	169	+ # value may be None with nested junctions without overrides
	170	+ if loader is not None:
	171	+ loader._clean_caches()
	172	+
	173	+ self._meta_elements = {}
	174	+ self._elements = {}
	175	+
157	176	###########################################
158	177	# Private Methods #
159	178	###########################################

buildstream/_project.py

---

...	...	@@ -358,6 +358,8 @@ class Project():
358	358	for meta in meta_elements
359	359	]
360	360
	361	+ Element._clear_meta_elements_cache()
	362	+
361	363	# Now warn about any redundant source references which may have
362	364	# been discovered in the resolve() phase.
363	365	redundant_refs = Element._get_redundant_source_refs()

buildstream/element.py

---

...	...	@@ -966,6 +966,21 @@ class Element(Plugin):
966	966
967	967	return element
968	968
	969	+ # _clear_meta_elements_cache()
	970	+ #
	971	+ # Clear the internal meta elements cache.
	972	+ #
	973	+ # When loading elements from meta, we cache already instantiated elements
	974	+ # in order to not have to load the same elements twice.
	975	+ # This clears the cache.
	976	+ #
	977	+ # It should be called whenever we are done loading all elements in order
	978	+ # to save memory.
	979	+ #
	980	+ @classmethod
	981	+ def _clear_meta_elements_cache(cls):
	982	+ cls.__instantiated_elements = {}
	983	+
969	984	# _get_redundant_source_refs()
970	985	#
971	986	# Fetches a list of (Source, ref) tuples of all the Sources

contrib/bst-graph

---

	1	+#!/usr/bin/env python3
	2	+'''Print dependency graph of given element(s) in DOT format.
	3	+
	4	+This script must be run from the same directory where you would normally
	5	+run `bst` commands.
	6	+
	7	+When `--format` option is specified, the output will also be rendered in the
	8	+given format. A file with name `bst-graph.{format}` will be created in the same
	9	+directory. To use this option, you must have the `graphviz` command line tool
	10	+installed.
	11	+'''
	12	+
	13	+import argparse
	14	+import subprocess
	15	+
	16	+from graphviz import Digraph
	17	+
	18	+
	19	+def parse_args():
	20	+ '''Handle parsing of command line arguments.
	21	+
	22	+ Returns:
	23	+ A argparse.Namespace object
	24	+ '''
	25	+ parser = argparse.ArgumentParser(description=__doc__)
	26	+ parser.add_argument(
	27	+ 'ELEMENT', nargs='*',
	28	+ help='Name of the element'
	29	+ )
	30	+ parser.add_argument(
	31	+ '--format',
	32	+ help='Redner the graph in given format (`pdf`, `png`, `svg` etc)'
	33	+ )
	34	+ parser.add_argument(
	35	+ '--view', action='store_true',
	36	+ help='Open the rendered graph with the default application'
	37	+ )
	38	+ return parser.parse_args()
	39	+
	40	+
	41	+def parse_graph(lines):
	42	+ '''Return nodes and edges of the parsed grpah.
	43	+
	44	+ Args:
	45	+ lines: List of lines in format 'NAME|BUILD-DEPS|RUNTIME-DEPS'
	46	+
	47	+ Returns:
	48	+ Tuple of format (nodes,build_deps,runtime_deps)
	49	+ Each member of build_deps and runtime_deps is also a tuple.
	50	+ '''
	51	+ nodes = set()
	52	+ build_deps = set()
	53	+ runtime_deps = set()
	54	+ for line in lines:
	55	+ # It is safe to split on '|' as it is not a valid character for
	56	+ # element names.
	57	+ name, build_dep, runtime_dep = line.split('|')
	58	+ build_dep = build_dep.lstrip('[').rstrip(']').split(',')
	59	+ runtime_dep = runtime_dep.lstrip('[').rstrip(']').split(',')
	60	+ nodes.add(name)
	61	+ [build_deps.add((name, dep)) for dep in build_dep if dep]
	62	+ [runtime_deps.add((name, dep)) for dep in runtime_dep if dep]
	63	+
	64	+ return nodes, build_deps, runtime_deps
	65	+
	66	+
	67	+def generate_graph(nodes, build_deps, runtime_deps):
	68	+ '''Generate graph from given nodes and edges.
	69	+
	70	+ Args:
	71	+ nodes: set of nodes
	72	+ build_deps: set of tuples of build depdencies
	73	+ runtime_deps: set of tuples of runtime depdencies
	74	+
	75	+ Returns:
	76	+ A graphviz.Digraph object
	77	+ '''
	78	+ graph = Digraph()
	79	+ for node in nodes:
	80	+ graph.node(node)
	81	+ for source, target in build_deps:
	82	+ graph.edge(source, target, label='build-dep')
	83	+ for source, target in runtime_deps:
	84	+ graph.edge(source, target, label='runtime-dep')
	85	+ return graph
	86	+
	87	+
	88	+def main():
	89	+ args = parse_args()
	90	+ cmd = ['bst', 'show', '--format', '%{name}|%{build-deps}|%{runtime-deps}']
	91	+ if 'element' in args:
	92	+ cmd += args.element
	93	+ graph_lines = subprocess.check_output(cmd, universal_newlines=True)
	94	+ # NOTE: We generate nodes and edges before giving them to graphviz as
	95	+ # the library does not de-deuplicate them.
	96	+ nodes, build_deps, runtime_deps = parse_graph(graph_lines.splitlines())
	97	+ graph = generate_graph(nodes, build_deps, runtime_deps)
	98	+ print(graph.source)
	99	+ if args.format:
	100	+ graph.render(cleanup=True,
	101	+ filename='bst-graph',
	102	+ format=args.format,
	103	+ view=args.view)
	104	+
	105	+
	106	+if __name__ == '__main__':
	107	+ main()

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.