[Notes] [Git][BuildStream/buildstream][valentindavid/fix-broken-indentation-after-track] 11 commits: _frontend/app.py: Notify session completions

[Valentin David <gitlab mg gitlab com>]

Title: GitLab

Valentin David pushed to branch valentindavid/fix-broken-indentation-after-track at BuildStream / buildstream

Commits:

• 10f27f3f
  by Tristan Van Berkom at 2018-08-17T06:20:40Z

  _frontend/app.py: Notify session completions
  
  Use the optionally implemented desktop notification method
  to notify when a session completes.
  
  Previously, this used to only notify when an interactive prompt
  appears.
  
  This is an attempt to fix #385
  

• c102b92f
  by Tristan Van Berkom at 2018-08-17T08:52:07Z

  Merge branch 'tristan/notifications' into 'master'
  
  _frontend/app.py: Notify session completions
  
  Closes #385
  
  See merge request BuildStream/buildstream!672

• 9b6fac5a
  by William Salmon at 2018-08-17T09:48:01Z

  Add Error to git and ostree sources configure
  
  Raise a error at configure time if the track and ref properties are
  not present in the sources.
  
  This is to address https://gitlab.com/BuildStream/buildstream/issues/471
  that documented unhelpful behaviour when tracking git sources. However
  the issue was also identified in ostree.
  

• d5ed4f45
  by William Salmon at 2018-08-17T09:49:23Z

  Improve error message for build if there are refs missing
  

• c43af4eb
  by Will Salmon at 2018-08-17T10:51:38Z

  Merge branch 'willsalmon/APIFix' into 'master'
  
  Updating the no track or ref fix to match the buildstream style
  
  See merge request BuildStream/buildstream!628

• 7263ea8c
  by Josh Smith at 2018-08-17T13:19:23Z

  job.py: Prevent terminated jobs retrying
  
  Fixes #531: Jobs were retrying when terminated, this lead to the process
  being spawned again and starting up fresh.
  

• 30f696f2
  by Qinusty at 2018-08-17T14:29:45Z

  Merge branch 'Qinusty/531-fetch-retries-on-terminate' into 'master'
  
  Prevent jobs retrying on terminate
  
  Closes #531
  
  See merge request BuildStream/buildstream!662

• c15cb951
  by Tristan Van Berkom at 2018-08-20T10:19:03Z

  setup.cfg: Add tests/integration/project to norecursedirs
  
  This is causing pytest to try to import the recently added app1.py
  from the pip related tests. This fails because it's not meant to be
  imported outside of the sandbox environment.
  

• 372abed5
  by Tristan Van Berkom at 2018-08-20T11:47:53Z

  Merge branch 'tristan/exclude-project-py-from-tests' into 'master'
  
  setup.cfg: Add tests/integration/project to norecursedirs
  
  See merge request BuildStream/buildstream!682

• b9986cfc
  by Valentin David at 2018-08-20T15:38:10Z

  Fix broken indentation after tracking.
  
  Issue was introduced by 171e803f (include directive) and the fix was
  found courtesy of @Qinusty. This fixes also the include
  feature. Because elements are to be serialized, the included fragments
  need to use copy_tree when loaded.
  
  Related to #470.
  

• 165083ca
  by Valentin David at 2018-08-20T15:38:10Z

  buildstream/_includes.py: Fix caching of included fragments.
  

14 changed files:

• buildstream/_frontend/app.py
• buildstream/_includes.py
• buildstream/_loader/loader.py
• buildstream/_pipeline.py
• buildstream/_project.py
• buildstream/_scheduler/jobs/job.py
• buildstream/plugins/sources/git.py
• buildstream/plugins/sources/ostree.py
• buildstream/source.py
• setup.cfg
• tests/sources/git.py
• + tests/sources/ostree.py
• + tests/sources/ostree/template/project.conf
• + tests/sources/ostree/template/repofiles/file

Changes:

buildstream/_frontend/app.py

---

...	...	@@ -269,6 +269,9 @@ class App():
269	269	else:
270	270	self._message(MessageType.FAIL, session_name, elapsed=elapsed)
271	271
	272	+ # Notify session failure
	273	+ self._notify("{} failed".format(session_name), "{}".format(e))
	274	+
272	275	if self._started:
273	276	self._print_summary()
274	277
...	...	@@ -286,6 +289,9 @@ class App():
286	289	if self._started:
287	290	self._print_summary()
288	291
	292	+ # Notify session success
	293	+ self._notify("{} succeeded".format(session_name), "")
	294	+
289	295	# init_project()
290	296	#
291	297	# Initialize a new BuildStream project, either with the explicitly passed options,
...	...	@@ -419,6 +425,12 @@ class App():
419	425	# Local Functions #
420	426	############################################################
421	427
	428	+ # Local function for calling the notify() virtual method
	429	+ #
	430	+ def _notify(self, title, text):
	431	+ if self.interactive:
	432	+ self.notify(title, text)
	433	+
422	434	# Local message propagator
423	435	#
424	436	def _message(self, message_type, message, **kwargs):
...	...	@@ -571,8 +583,8 @@ class App():
571	583	while choice not in ['continue', 'quit', 'terminate', 'retry']:
572	584	click.echo(summary, err=True)
573	585
574		- self.notify("BuildStream failure", "{} on element {}"
575		- .format(failure.action_name, element.name))
	586	+ self._notify("BuildStream failure", "{} on element {}"
	587	+ .format(failure.action_name, element.name))
576	588
577	589	try:
578	590	choice = click.prompt("Choice:", default='continue', err=True,

buildstream/_includes.py

---

...	...	@@ -10,11 +10,15 @@ from ._exceptions import LoadError, LoadErrorReason
10	10	#
11	11	# Args:
12	12	# loader (Loader): The Loader object
	13	+# copy_tree (bool): Whether to make a copy, of tree in
	14	+# provenance. Should be true if intended to be
	15	+# serialized.
13	16	class Includes:
14	17
15		- def __init__(self, loader):
	18	+ def __init__(self, loader, *, copy_tree=False):
16	19	self._loader = loader
17	20	self._loaded = {}
	21	+ self._copy_tree = copy_tree
18	22
19	23	# process()
20	24	#
...	...	@@ -96,10 +100,11 @@ class Includes:
96	100	directory = project.directory
97	101	file_path = os.path.join(directory, include)
98	102	key = (current_loader, file_path)
99		- if file_path not in self._loaded:
	103	+ if key not in self._loaded:
100	104	self._loaded[key] = _yaml.load(os.path.join(directory, include),
101	105	shortname=shortname,
102		- project=project)
	106	+ project=project,
	107	+ copy_tree=self._copy_tree)
103	108	return self._loaded[key], file_path, current_loader
104	109
105	110	# _process_value()

buildstream/_loader/loader.py

---

...	...	@@ -78,7 +78,7 @@ class Loader():
78	78	self._elements = {} # Dict of elements
79	79	self._loaders = {} # Dict of junction loaders
80	80
81		- self._includes = Includes(self)
	81	+ self._includes = Includes(self, copy_tree=True)
82	82
83	83	# load():
84	84	#

buildstream/_pipeline.py

---

...	...	@@ -359,23 +359,14 @@ class Pipeline():
359	359
360	360	if inconsistent:
361	361	detail = "Exact versions are missing for the following elements:\n\n"
362		-
363		- missingTrack = 0
364	362	for element in inconsistent:
365		- detail += " " + element._get_full_name()
	363	+ detail += " Element: {} is inconsistent\n".format(element._get_full_name())
366	364	for source in element.sources():
367		- if not source._get_consistency() and not source.get_ref():
368		- if hasattr(source, 'tracking') and source.tracking is None:
369		- detail += ": Source {} is missing ref and track. ".format(source._get_full_name()) + \
370		- "Please specify a ref or branch/tag to track."
371		- missingTrack = 1
372		-
373		- detail += "\n"
	365	+ if source._get_consistency() == Consistency.INCONSISTENT:
	366	+ detail += " Source {} is missing ref\n".format(source)
	367	+ detail += '\n'
	368	+ detail += "Try tracking these elements first with `bst track`\n"
374	369
375		- if missingTrack:
376		- detail += "\nThen track these elements with `bst track`\n"
377		- else:
378		- detail += "\nTry tracking these elements first with `bst track`\n"
379	370	raise PipelineError("Inconsistent pipeline", detail=detail, reason="inconsistent-pipeline")
380	371
381	372	#############################################################

buildstream/_project.py

---

...	...	@@ -419,7 +419,7 @@ class Project():
419	419	parent=parent_loader,
420	420	tempdir=tempdir)
421	421
422		- self._project_includes = Includes(self.loader)
	422	+ self._project_includes = Includes(self.loader, copy_tree=False)
423	423
424	424	project_conf_first_pass = _yaml.node_copy(self._project_conf)
425	425	self._project_includes.process(project_conf_first_pass, only_local=True)

buildstream/_scheduler/jobs/job.py

---

...	...	@@ -250,7 +250,7 @@ class Job():
250	250	#
251	251	def resume(self, silent=False):
252	252	if self._suspended:
253		- if not silent:
	253	+ if not silent and not self._scheduler.terminated:
254	254	self.message(MessageType.STATUS,
255	255	"{} resuming".format(self.action_name))
256	256
...	...	@@ -549,7 +549,7 @@ class Job():
549	549	#
550	550	self._retry_flag = returncode not in (RC_OK, RC_PERM_FAIL)
551	551
552		- if self._retry_flag and (self._tries <= self._max_retries):
	552	+ if self._retry_flag and (self._tries <= self._max_retries) and not self._scheduler.terminated:
553	553	self.spawn()
554	554	return
555	555

buildstream/plugins/sources/git.py

---

...	...	@@ -309,6 +309,13 @@ class GitSource(Source):
309	309	self.original_url = self.node_get_member(node, str, 'url')
310	310	self.mirror = GitMirror(self, '', self.original_url, ref)
311	311	self.tracking = self.node_get_member(node, str, 'track', None)
	312	+
	313	+ # At this point we now know if the source has a ref and/or a track.
	314	+ # If it is missing both then we will be unable to track or build.
	315	+ if self.mirror.ref is None and self.tracking is None:
	316	+ raise SourceError("{}: Git sources require a ref and/or track".format(self),
	317	+ reason="missing-track-and-ref")
	318	+
312	319	self.checkout_submodules = self.node_get_member(node, bool, 'checkout-submodules', True)
313	320	self.submodules = []
314	321

buildstream/plugins/sources/ostree.py

---

...	...	@@ -73,6 +73,12 @@ class OSTreeSource(Source):
73	73	self.mirror = os.path.join(self.get_mirror_directory(),
74	74	utils.url_directory_name(self.original_url))
75	75
	76	+ # At this point we now know if the source has a ref and/or a track.
	77	+ # If it is missing both then we will be unable to track or build.
	78	+ if self.ref is None and self.tracking is None:
	79	+ raise SourceError("{}: OSTree sources require a ref and/or track".format(self),
	80	+ reason="missing-track-and-ref")
	81	+
76	82	# (optional) Not all repos are signed. But if they are, get the gpg key
77	83	self.gpg_key_path = None
78	84	if self.node_get_member(node, str, 'gpg-key', None):

buildstream/source.py

---

...	...	@@ -794,7 +794,7 @@ class Source(Plugin):
794	794	# Save the ref in the originating file
795	795	#
796	796	try:
797		- _yaml.dump(_yaml.node_sanitize(provenance.toplevel), provenance.filename.name)
	797	+ _yaml.dump(provenance.toplevel, provenance.filename.name)
798	798	except OSError as e:
799	799	raise SourceError("{}: Error saving source reference to '{}': {}"
800	800	.format(self, provenance.filename.name, e),

setup.cfg

---

...	...	@@ -11,7 +11,7 @@ test=pytest
11	11
12	12	[tool:pytest]
13	13	addopts = --verbose --basetemp ./tmp --pep8 --pylint --pylint-rcfile=.pylintrc --cov=buildstream --cov-config .coveragerc
14		-norecursedirs = integration-cache tmp __pycache__ .eggs
	14	+norecursedirs = tests/integration/project integration-cache tmp __pycache__ .eggs
15	15	python_files = tests/*/*.py
16	16	pep8maxlinelength = 119
17	17	pep8ignore =

tests/sources/git.py

---

	1	+#
	2	+# Copyright (C) 2018 Codethink Limited
	3	+# Copyright (C) 2018 Bloomberg Finance LP
	4	+#
	5	+# This program is free software; you can redistribute it and/or
	6	+# modify it under the terms of the GNU Lesser General Public
	7	+# License as published by the Free Software Foundation; either
	8	+# version 2 of the License, or (at your option) any later version.
	9	+#
	10	+# This library is distributed in the hope that it will be useful,
	11	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	12	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	13	+# Lesser General Public License for more details.
	14	+#
	15	+# You should have received a copy of the GNU Lesser General Public
	16	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	17	+#
	18	+# Authors: Tristan Van Berkom <tristan vanberkom codethink co uk>
	19	+# Jonathan Maw <jonathan maw codethink co uk>
	20	+# William Salmon <will salmon codethink co uk>
	21	+#
	22	+
1	23	import os
2	24	import pytest
3	25
...	...	@@ -383,21 +405,6 @@ def test_submodule_track_no_ref_or_track(cli, tmpdir, datafiles):
383	405	_yaml.dump(element, os.path.join(project, 'target.bst'))
384	406
385	407	# Track will encounter an inconsistent submodule without any ref
386		- result = cli.run(project=project, args=['track', 'target.bst'])
387		- result.assert_main_error(ErrorDomain.STREAM, None)
388		- result.assert_task_error(ErrorDomain.SOURCE, 'track-attempt-no-track')
389		-
390		- # Assert that we are just fine without it, and emit a warning to the user.
391		- assert "FAILURE git source at" in result.stderr
392		- assert "Without a tracking branch ref can not be updated. Please " + \
393		- "provide a ref or a track." in result.stderr
394		-
395		- # Track will encounter an inconsistent submodule without any ref
396		- result = cli.run(project=project, args=['build', 'target.bst'])
397		- result.assert_main_error(ErrorDomain.PIPELINE, 'inconsistent-pipeline')
	408	+ result = cli.run(project=project, args=['show', 'target.bst'])
	409	+ result.assert_main_error(ErrorDomain.SOURCE, "missing-track-and-ref")
398	410	result.assert_task_error(None, None)
399		-
400		- # Assert that we are just fine without it, and emit a warning to the user.
401		- assert "Exact versions are missing for the following elements" in result.stderr
402		- assert "is missing ref and track." in result.stderr
403		- assert "Then track these elements with `bst track`" in result.stderr

tests/sources/ostree.py

---

	1	+#
	2	+# Copyright (C) 2018 Bloomberg Finance LP
	3	+#
	4	+# This program is free software; you can redistribute it and/or
	5	+# modify it under the terms of the GNU Lesser General Public
	6	+# License as published by the Free Software Foundation; either
	7	+# version 2 of the License, or (at your option) any later version.
	8	+#
	9	+# This library is distributed in the hope that it will be useful,
	10	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	11	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	12	+# Lesser General Public License for more details.
	13	+#
	14	+# You should have received a copy of the GNU Lesser General Public
	15	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	16	+#
	17	+# Authors: William Salmon <will salmon codethink co uk>
	18	+#
	19	+
	20	+import os
	21	+import pytest
	22	+
	23	+from buildstream._exceptions import ErrorDomain
	24	+from buildstream import _yaml
	25	+
	26	+from tests.testutils import cli, create_repo
	27	+
	28	+DATA_DIR = os.path.join(
	29	+ os.path.dirname(os.path.realpath(__file__)),
	30	+ 'ostree',
	31	+)
	32	+
	33	+
	34	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	35	+def test_submodule_track_no_ref_or_track(cli, tmpdir, datafiles):
	36	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	37	+
	38	+ # Create the repo from 'repofiles' subdir
	39	+ repo = create_repo('ostree', str(tmpdir))
	40	+ ref = repo.create(os.path.join(project, 'repofiles'))
	41	+
	42	+ # Write out our test target
	43	+ ostreesource = repo.source_config(ref=None)
	44	+ ostreesource.pop('track')
	45	+ element = {
	46	+ 'kind': 'import',
	47	+ 'sources': [
	48	+ ostreesource
	49	+ ]
	50	+ }
	51	+
	52	+ _yaml.dump(element, os.path.join(project, 'target.bst'))
	53	+
	54	+ # Track will encounter an inconsistent submodule without any ref
	55	+ result = cli.run(project=project, args=['show', 'target.bst'])
	56	+ result.assert_main_error(ErrorDomain.SOURCE, "missing-track-and-ref")
	57	+ result.assert_task_error(None, None)

tests/sources/ostree/template/project.conf

---

	1	+# Basic project
	2	+name: foo

tests/sources/ostree/template/repofiles/file

---

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.