[Notes] [Git][BuildStream/buildstream][danielsilverstone-ct/gc-play] 7 commits: element.py: Expect absolute paths in overlap whitelists

[Daniel Silverstone <gitlab mg gitlab com>]

Title: GitLab

Daniel Silverstone pushed to branch danielsilverstone-ct/gc-play at BuildStream / buildstream

Commits:

• 1ae17968
  by Valentin David at 2019-02-19T09:01:48Z

  element.py: Expect absolute paths in overlap whitelists
  
  This allows use of variables such as %{prefix} and matches the
  documentation. This is a breaking change.
  
  Fixes #721
  

• afe823e8
  by Jürg Billeter at 2019-02-19T10:33:59Z

  Merge branch 'valentindavid/absolute-whitelist' into 'master'
  
  Allow absolute paths in whitelist
  
  Closes #721
  
  See merge request BuildStream/buildstream!968

• d70bfc38
  by Adam Jones at 2019-02-19T10:34:37Z

  Add a link to the label descriptions in the issue templates
  

• 857e7414
  by Tom Pollard at 2019-02-19T11:37:15Z

  Merge branch 'adamjones/labels' into 'master'
  
  Add a link to the label descriptions in the issue templates
  
  See merge request BuildStream/buildstream!913

• 9bc389a8
  by Jürg Billeter at 2019-02-19T11:39:44Z

  tests: Don't use integration test for artifact log
  
  A non-integration test works just as well for this and is a lot quicker,
  especially on WSL.
  

• 4a002bee
  by Jürg Billeter at 2019-02-19T13:17:00Z

  Merge branch 'juerg/artifact-test' into 'master'
  
  tests: Don't use integration test for artifact log
  
  See merge request BuildStream/buildstream!1162

• afd15d7c
  by Daniel Silverstone at 2019-02-19T13:59:51Z

  _stream.py, _project.py: Manage GC during pipeline load
  
  Since during pipeline load we are generating a lot of objects
  which need to live for the duration of the BuildStream instance
  there is benefit to disabling GC for the duration of the load since
  that drastically reduces wasted CPU cycles iterating data which
  will remain around always.  In order to limit the increase in
  the peak memory consumption though, we do an explicit gc.collect()
  after loading the YAML in, since without that, we use 60% more
  memory at peak, and with it, only 20%.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co UK.
  

11 changed files:

• .gitlab/issue_templates/bst_bug.md
• .gitlab/issue_templates/bst_task.md
• NEWS
• buildstream/_project.py
• buildstream/_stream.py
• buildstream/element.py
• + tests/frontend/artifact.py
• tests/frontend/overlaps/a-whitelisted.bst
• tests/frontend/overlaps/b-whitelisted.bst
• tests/frontend/overlaps/c-whitelisted.bst
• tests/integration/artifact.py

Changes:

.gitlab/issue_templates/bst_bug.md

---

...	...	@@ -33,4 +33,6 @@
33	33	* BuildStream version affected: /milestone %BuildStream_v1.x
34	34
35	35	----
	36	+[//]: # (To review information about possible relevant labels for this issue please view the list of labels: https://gitlab.com/BuildStream/buildstream/labels)
	37	+
36	38	/label ~bug

.gitlab/issue_templates/bst_task.md

---

...	...	@@ -15,3 +15,5 @@
15	15	[//]: # (Acceptance criteria should follow the S.M.A.R.T. principle https://en.wikipedia.org/wiki/SMART_criteria )
16	16
17	17	----
	18	+[//]: # (To review information about possible relevant labels for this issue please view the list of labels: https://gitlab.com/BuildStream/buildstream/labels)
	19	+

NEWS

---

...	...	@@ -138,6 +138,9 @@ buildstream 1.3.1
138	138	o BREAKING CHANGE: Symlinks are no longer resolved during staging and absolute
139	139	symlinks are now preserved instead of being converted to relative symlinks.
140	140
	141	+ o BREAKING CHANGE: Overlap whitelists now require absolute paths. This allows
	142	+ use of variables such as %{prefix} and matches the documentation.
	143	+
141	144
142	145	=================
143	146	buildstream 1.1.5

buildstream/_project.py

---

...	...	@@ -18,6 +18,7 @@
18	18	# Tristan Van Berkom <tristan vanberkom codethink co uk>
19	19	# Tiago Gomes <tiago gomes codethink co uk>
20	20
	21	+import gc
21	22	import os
22	23	from collections import OrderedDict
23	24	from collections.abc import Mapping
...	...	@@ -352,6 +353,9 @@ class Project():
352	353	ticker=None,
353	354	fetch_subprojects=fetch_subprojects)
354	355
	356	+ # Loading elements generates a lot of garbage, clear it now
	357	+ gc.collect()
	358	+
355	359	with self._context.timed_activity("Resolving elements"):
356	360	elements = [
357	361	Element._new_from_meta(meta)

buildstream/_stream.py

---

...	...	@@ -19,6 +19,7 @@
19	19	# Jürg Billeter <juerg billeter codethink co uk>
20	20	# Tristan Maat <tristan maat codethink co uk>
21	21
	22	+import gc
22	23	import os
23	24	import sys
24	25	import stat
...	...	@@ -115,6 +116,11 @@ class Stream():
115	116
116	117	profile_start(Topics.LOAD_SELECTION, "_".join(t.replace(os.sep, '-') for t in targets))
117	118
	119	+ # Disable the GC for the duration of the load. We make many objects
	120	+ # during load and this gives us a significant speedup at the cost of
	121	+ # an additional ca. 20% of memory being used at peak.
	122	+ gc.disable()
	123	+
118	124	target_objects, _ = self._load(targets, (),
119	125	selection=selection,
120	126	except_targets=except_targets,
...	...	@@ -122,6 +128,12 @@ class Stream():
122	128	use_artifact_config=use_artifact_config,
123	129	load_refs=load_refs)
124	130
	131	+ # Re-enable the GC now that we've finished making lots of objects
	132	+ gc.enable()
	133	+ # And clean up to ensure we don't grow any more, freeing up room to be
	134	+ # used by other objects during the course of running BuildStream.
	135	+ gc.collect()
	136	+
125	137	profile_end(Topics.LOAD_SELECTION, "_".join(t.replace(os.sep, '-') for t in targets))
126	138
127	139	return target_objects

buildstream/element.py

---

...	...	@@ -2609,7 +2609,7 @@ class Element(Plugin):
2609	2609	if include_file and not exclude_file:
2610	2610	yield filename.lstrip(os.sep)
2611	2611
2612		- def __file_is_whitelisted(self, pattern):
	2612	+ def __file_is_whitelisted(self, path):
2613	2613	# Considered storing the whitelist regex for re-use, but public data
2614	2614	# can be altered mid-build.
2615	2615	# Public data is not guaranteed to stay the same for the duration of
...	...	@@ -2621,7 +2621,7 @@ class Element(Plugin):
2621	2621	whitelist_expressions = [utils._glob2re(self.__variables.subst(exp.strip())) for exp in whitelist]
2622	2622	_expression_ = ('^(?:' + '|'.join(whitelist_expressions) + ')$')
2623	2623	self.__whitelist_regex = re.compile(_expression_)
2624		- return self.__whitelist_regex.match(pattern)
	2624	+ return self.__whitelist_regex.match(os.path.join(os.sep, path))
2625	2625
2626	2626	# __extract():
2627	2627	#

tests/frontend/artifact.py

---

	1	+#
	2	+# Copyright (C) 2018 Codethink Limited
	3	+# Copyright (C) 2018 Bloomberg Finance LP
	4	+#
	5	+# This program is free software; you can redistribute it and/or
	6	+# modify it under the terms of the GNU Lesser General Public
	7	+# License as published by the Free Software Foundation; either
	8	+# version 2 of the License, or (at your option) any later version.
	9	+#
	10	+# This library is distributed in the hope that it will be useful,
	11	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	12	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	13	+# Lesser General Public License for more details.
	14	+#
	15	+# You should have received a copy of the GNU Lesser General Public
	16	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	17	+#
	18	+# Authors: Richard Maw <richard maw codethink co uk>
	19	+#
	20	+
	21	+import os
	22	+import pytest
	23	+
	24	+from buildstream.plugintestutils import cli
	25	+
	26	+
	27	+# Project directory
	28	+DATA_DIR = os.path.join(
	29	+ os.path.dirname(os.path.realpath(__file__)),
	30	+ "project",
	31	+)
	32	+
	33	+
	34	+@pytest.mark.datafiles(DATA_DIR)
	35	+def test_artifact_log(cli, tmpdir, datafiles):
	36	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	37	+
	38	+ # Get the cache key of our test element
	39	+ result = cli.run(project=project, silent=True, args=[
	40	+ '--no-colors',
	41	+ 'show', '--deps', 'none', '--format', '%{full-key}',
	42	+ 'target.bst'
	43	+ ])
	44	+ key = result.output.strip()
	45	+
	46	+ # Ensure we have an artifact to read
	47	+ result = cli.run(project=project, args=['build', 'target.bst'])
	48	+ assert result.exit_code == 0
	49	+
	50	+ # Read the log via the element name
	51	+ result = cli.run(project=project, args=['artifact', 'log', 'target.bst'])
	52	+ assert result.exit_code == 0
	53	+ log = result.output
	54	+
	55	+ # Read the log via the key
	56	+ result = cli.run(project=project, args=['artifact', 'log', 'test/target/' + key])
	57	+ assert result.exit_code == 0
	58	+ assert log == result.output
	59	+
	60	+ # Read the log via glob
	61	+ result = cli.run(project=project, args=['artifact', 'log', 'test/target/*'])
	62	+ assert result.exit_code == 0
	63	+ # The artifact is cached under both a strong key and a weak key
	64	+ assert (log + log) == result.output

tests/frontend/overlaps/a-whitelisted.bst

---

...	...	@@ -10,4 +10,4 @@ sources:
10	10	public:
11	11	bst:
12	12	overlap-whitelist:
13		- - "file*"
	13	+ - "/file*"

tests/frontend/overlaps/b-whitelisted.bst

---

...	...	@@ -8,9 +8,9 @@ sources:
8	8	- kind: local
9	9	path: "b"
10	10	variables:
11		- FILE: file
	11	+ FILE: /file
12	12	public:
13	13	bst:
14	14	overlap-whitelist:
15		- - file2
	15	+ - /file2
16	16	- "%{FILE}3"

tests/frontend/overlaps/c-whitelisted.bst

---

...	...	@@ -8,4 +8,4 @@ sources:
8	8	public:
9	9	bst:
10	10	overlap-whitelist:
11		- - "file*"
	11	+ - "/file*"

tests/integration/artifact.py

---

...	...	@@ -37,40 +37,6 @@ DATA_DIR = os.path.join(
37	37	)
38	38
39	39
40		-@pytest.mark.integration
41		-@pytest.mark.datafiles(DATA_DIR)
42		-def test_artifact_log(cli, tmpdir, datafiles):
43		- project = os.path.join(datafiles.dirname, datafiles.basename)
44		-
45		- # Get the cache key of our test element
46		- result = cli.run(project=project, silent=True, args=[
47		- '--no-colors',
48		- 'show', '--deps', 'none', '--format', '%{full-key}',
49		- 'base.bst'
50		- ])
51		- key = result.output.strip()
52		-
53		- # Ensure we have an artifact to read
54		- result = cli.run(project=project, args=['build', 'base.bst'])
55		- assert result.exit_code == 0
56		-
57		- # Read the log via the element name
58		- result = cli.run(project=project, args=['artifact', 'log', 'base.bst'])
59		- assert result.exit_code == 0
60		- log = result.output
61		-
62		- # Read the log via the key
63		- result = cli.run(project=project, args=['artifact', 'log', 'test/base/' + key])
64		- assert result.exit_code == 0
65		- assert log == result.output
66		-
67		- # Read the log via glob
68		- result = cli.run(project=project, args=['artifact', 'log', 'test/base/*'])
69		- assert result.exit_code == 0
70		- # The artifact is cached under both a strong key and a weak key
71		- assert (log + log) == result.output
72		-
73		-
74	40	# A test to capture the integration of the cachebuildtrees
75	41	# behaviour, which by default is to include the buildtree
76	42	# content of an element on caching.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.