[Notes] [Git][BuildStream/buildstream][jonathan/workspace-fragment-guess-element] 32 commits: tests/sources/git.py: Refactor ref-not-in-track test to use parameterization

[Jonathan Maw <gitlab mg gitlab com>]

Title: GitLab

Jonathan Maw pushed to branch jonathan/workspace-fragment-guess-element at BuildStream / buildstream

Commits:

• 4688b2c9
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  tests/sources/git.py: Refactor ref-not-in-track test to use parameterization
  
  Instead of duplicating the whole test body, run it twice while
  checking for a warning or an error depending on the parameter.
  

• 03028c7f
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  types.py: Moving CoreWarnings to the types.py module
  
  This also ensures it is exposed via the main buildstream __init__.py
  file, fixes the imports of CoreWarnings, and adjusts the documentation
  links to point to the right place.
  

• 3b7f8df8
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  Source: Fix typo in documentation, and make it a link to `bst track`
  
  Now the commands have documentation linkability so we can link to
  them when referring to them anywhere in the docs. This reference
  was previously referring to a bogus non-existent `build-stream track`,
  maybe this dates way back to the days before we named the frontend `bst`.
  

• e5c0f067
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  git source plugin: Fixing documentation linking and typo
  
  Fixes the configurable warning part of this to properly link
  to the project.conf documentation describing what configurable
  warnings actually are.
  
  Also fix the plugin raise the actual configurable warning which
  it advertizes that it raises; which is 'inconsistent-submodule',
  not 'inconsistent-submodules'.
  

• 205c27d8
  by Tristan Van Berkom at 2018-12-06T07:35:59Z

  git source plugin: Rename symbol for consistency and readability
  
  Change 'INCONSISTENT_SUBMODULE' for 'WARN_INCONSISTENT_SUBMODULE' to
  make things a bit more readable and comprehensive.
  

• bea4d4f5
  by Tristan Van Berkom at 2018-12-06T09:37:50Z

  Merge branch 'tristan/misc-cleanup' into 'master'
  
  Misc cleanups
  
  See merge request BuildStream/buildstream!993

• b1d2f001
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  source.py: Add new delegate method validate_cache()
  
  This is guaranteed to be called only once for a given session once the
  sources are known to be Consistency.CACHED, if source tracking is enabled
  in the session for this source, then this will only be called if the
  sources become cached after tracking completes.
  

• ee7fc47f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  git source plugin: Implementing submodule warnings
  
   o Unlisted submodule warning
  
     Now the git plugin will issue a configurable warning if a submodule
     exists and is used (checking out the submodule is not disabled),
     but is not specified in the source configuration.
  
   o Invalid submodule warning
  
     Now the git source plugin will issue a warning if the configuration
     specified a submodule which does not exist in the underlying git repository.
  
  As a side effect, this patch also changes the flow control of the git
  plugin such that submodules which are explicitly set to not be
  checked out, are also not fetched but instead ignored completely.
  

• 5fc9a1da
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  git source plugin: Emmit the ref-not-in-track warning from validate_cache()
  
  Now that we have Source.validate_cache(), this is a better place to emmit
  the ref-not-in-track warning, since it will be emmitted at the earliest
  opportunity and not only at Source.stage() or Source.init_workspace().
  
  This also allows us to remove the `self.tracked` local state, and cleanup
  some convoluted calling paths, removing some unnecessary parameters from
  the usual codepaths and making the plugin overall more readable.
  

• 50cb2706
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/testutils/repo/git.py: Adding remove_path() helper
  
  A function for removing paths from a git repo, can also
  be used to remove submodules.
  

• f139ca8f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Testing the git:unlisted-submodule warning
  
   o Test that it is not triggered in show before fetch, because we
     don't know about the unlisted submodules yet
  
   o Test that it is triggered by a fetch command
  
   o Test that it is triggered by `show` after having completed a
     fetch command, since now we have the repository and know about
     the unlisted submodule
  
   o Test all of this under warning or error conditions (parameterized
     for fatal-warnings)
  

• afe1aa9f
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Testing the git:invalid-submodule warning
  
     o Test that it is not triggered in show before fetch, because we
       don't know the submodules yet so we cannot know if they are
       valid or not.
  
     o Test that it is triggered by a fetch command
  
     o Test that it is triggered by `show` after having completed a
       fetch command, since now we have the repository and know which
       specified submodules are invalid
  
     o Test all of this under warning or error conditions (parameterized
       for fatal-warnings)
  

• 57300201
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Test unlisted submodules warning appearing after track
  

• a0297625
  by Tristan Van Berkom at 2018-12-06T14:18:31Z

  tests/sources/git.py: Test invalid submodules warning appearing after track
  

• 60e96781
  by Tristan Van Berkom at 2018-12-06T15:14:37Z

  Merge branch 'tristan/submodule-warnings' into 'master'
  
  Implement submodule warnings
  
  See merge request BuildStream/buildstream!996

• 61b2635c
  by Jonathan Maw at 2018-12-06T17:04:45Z

  utils.py: Add a helper for searching upwards for files
  
  i.e. with a given directory and filename, check parent directories until
  either a directory with the filename is found, or you reach the root of
  the filesystem.
  
  This is a part of #222
  

• a047d12a
  by Jonathan Maw at 2018-12-06T17:04:45Z

  _workspaces.py: Add WorkspaceProject and WorkspaceProjectCache objects
  
  WorkspaceProject contains all the information needed to refer back to a
  project from its workspace (currently this is the project path and the
  element used to create this workspace)
  
  This is stored within a WorkspaceProjectCache object, which keeps
  WorkspaceProjects around so they don't need to be loaded from disk
  repeatedly.
  
  Workspaces has been extended to contain the WorkspaceProjectCache, and
  will use it when opening and closing workspaces.
  
  Context has been extended to contain a WorkspaceProjectCache, as there
  are times when we want to use it before a Workspaces can be initialised
  (looking up a WorkspaceProject to find the directory that the project is
  in)
  
  This is a part of #222
  

• 17f5c25d
  by Jonathan Maw at 2018-12-06T17:04:45Z

  _project.py: Rename _ensure_project_dir to _find_project_dir
  
  This is a part of #222
  

• 2eac2939
  by Jonathan Maw at 2018-12-06T17:04:45Z

  cli.py: Use utils' search upwards helper when searching for project.conf
  
  This is a part of #222
  

• c1202abf
  by Jonathan Maw at 2018-12-06T17:04:45Z

  _project.py: Find project from workspace if outside of a project
  
  This is a part of #222
  

• e868f400
  by Jonathan Maw at 2018-12-06T17:04:46Z

  _stream.py: Opened workspaces store the project that opened them
  
  This is a part of #222
  

• a64cc945
  by Jonathan Maw at 2018-12-06T17:04:46Z

  _stream.py: Re-create the .bstproject when resetting the workspace
  
  This is a part of #222
  

• b722785e
  by Jonathan Maw at 2018-12-06T17:04:46Z

  _workspaces.py: Do not include .bstproject.yaml in the cache key
  
  This is a part of #222
  

• 4446a7b2
  by Jonathan Maw at 2018-12-06T17:04:46Z

  cli: Interactively warn if the user is trying to close the workspace they're using to load the project
  
  This involves changes in:
  * _stream.py:
    * Add the helper Stream.workspace_is_required()
  * userconfig.yaml:
    * Add a default value for prompt.really-workspace-close-project-inaccessible
  * _context.py:
    * Load the prompt 'really-workspace-close-project-inaccessible' from
      user config.
  * cli.py:
    * If buildstream is invoked interactively, prompt the user to confirm
      that they want to close the workspace they're using to load this
      project.
  
  This is a part of #222
  

• 1d1602ad
  by Jonathan Maw at 2018-12-06T17:04:46Z

  tests: Test bst commands from an external workspace
  
  This is a part of #222
  

• ff8f1909
  by Jonathan Maw at 2018-12-06T17:04:46Z

  NEWS: Add an entry for being able to run commands from a workspace
  
  This is a part of #222
  

• f8fe4c3a
  by Jonathan Maw at 2018-12-06T17:34:56Z

  _stream.py: Fix ugly error when opening a workspace using a relative path
  

• 52069032
  by Jonathan Maw at 2018-12-06T17:42:59Z

  _context.py: Store the directory buildstream was invoked on
  

• b6c6ffc2
  by Jonathan Maw at 2018-12-06T17:42:59Z

  Make specifying elements optional in bst commands
  
  Known issues:
  * `bst shell` works, but `bst shell COMMANDS...` doesn't, because click
    has no way of separating optional args from variable-length args.
  * `bst checkout` and `bst source-checkout`'s usage strings mark LOCATION
    as an optional argument. Because click gets confused if there's an
    optional argument before a mandatory argument, I had to mark LOCATION
    as optional internally.
  * `bst workspace open` makes no sense with element being optional, so
    I skipped it.
  * `bst workspace close` will probably need to be revisited when multiple
    projects can own one workspace.
  * `bst workspace reset` will happily delete the directory you're
    currently in, requiring you to `cd $PWD` to see the contents of your
    directory.
    I could exclude the top-level directory of the workspace being
    deleted, but it is entirely valid to run workspace commands from deeper
    in the workspace.
  
  This is a part of #222
  

• f5fc538d
  by Jonathan Maw at 2018-12-06T17:44:08Z

  tests: Add tests for guessing element names
  
  This is a part of #222
  

• ffca42c9
  by Jonathan Maw at 2018-12-06T17:45:09Z

  Add NEWS for guessing the element in commands
  
  This is a part of #222
  

• 675791e7
  by Jonathan Maw at 2018-12-06T17:45:09Z

  doc: Point out that element is optional when commands are run from the workspace
  
  This is a part of #222
  

26 changed files:

• NEWS
• buildstream/__init__.py
• buildstream/_context.py
• buildstream/_frontend/app.py
• buildstream/_frontend/cli.py
• buildstream/_loader/loader.py
• buildstream/_project.py
• buildstream/_stream.py
• buildstream/_versions.py
• buildstream/_workspaces.py
• buildstream/data/userconfig.yaml
• buildstream/element.py
• buildstream/plugin.py
• buildstream/plugins/sources/git.py
• buildstream/source.py
• buildstream/types.py
• buildstream/utils.py
• doc/sessions/developing.run
• doc/source/developing/workspaces.rst
• doc/source/format_project.rst
• + doc/source/sessions-stored/developing-build-after-changes-workspace.html
• tests/frontend/source_checkout.py
• tests/frontend/workspace.py
• tests/integration/shell.py
• tests/sources/git.py
• tests/testutils/repo/git.py

Changes:

NEWS

---

...	...	@@ -79,6 +79,15 @@ buildstream 1.3.1
79	79	plugin has now a tag tracking feature instead. This can be enabled
80	80	by setting 'track-tags'.
81	81
	82	+ o Opening a workspace now creates a .bstproject.yaml file that allows buildstream
	83	+ commands to be run from a workspace that is not inside a project.
	84	+
	85	+ o Specifying an element is now optional for some commands when buildstream is run
	86	+ from inside a workspace - the 'build', 'checkout', 'fetch', 'pull', 'push',
	87	+ 'shell', 'show', 'source-checkout', 'track', 'workspace close' and 'workspace reset'
	88	+ commands are affected.
	89	+
	90	+
82	91	=================
83	92	buildstream 1.1.5
84	93	=================

buildstream/__init__.py

---

...	...	@@ -28,7 +28,7 @@ if "_BST_COMPLETION" not in os.environ:
28	28
29	29	from .utils import UtilError, ProgramNotFoundError
30	30	from .sandbox import Sandbox, SandboxFlags, SandboxCommandError
31		- from .types import Scope, Consistency
	31	+ from .types import Scope, Consistency, CoreWarnings
32	32	from .plugin import Plugin
33	33	from .source import Source, SourceError, SourceFetcher
34	34	from .element import Element, ElementError

buildstream/_context.py

---

...	...	@@ -32,7 +32,7 @@ from ._message import Message, MessageType
32	32	from ._profile import Topics, profile_start, profile_end
33	33	from ._artifactcache import ArtifactCache
34	34	from ._artifactcache.cascache import CASCache
35		-from ._workspaces import Workspaces
	35	+from ._workspaces import Workspaces, WorkspaceProjectCache
36	36	from .plugin import _plugin_lookup
37	37
38	38
...	...	@@ -47,9 +47,12 @@ from .plugin import _plugin_lookup
47	47	# verbosity levels and basically anything pertaining to the context
48	48	# in which BuildStream was invoked.
49	49	#
	50	+# Args:
	51	+# directory (str): The directory that buildstream was invoked in
	52	+#
50	53	class Context():
51	54
52		- def __init__(self):
	55	+ def __init__(self, directory=None):
53	56
54	57	# Filename indicating which configuration file was used, or None for the defaults
55	58	self.config_origin = None
...	...	@@ -122,6 +125,10 @@ class Context():
122	125	# remove a workspace directory.
123	126	self.prompt_workspace_close_remove_dir = None
124	127
	128	+ # Boolean, whether we double-check with the user that they meant to
	129	+ # close the workspace when they're using it to access the project.
	130	+ self.prompt_workspace_close_project_inaccessible = None
	131	+
125	132	# Boolean, whether we double-check with the user that they meant to do
126	133	# a hard reset of a workspace, potentially losing changes.
127	134	self.prompt_workspace_reset_hard = None
...	...	@@ -140,9 +147,11 @@ class Context():
140	147	self._projects = []
141	148	self._project_overrides = {}
142	149	self._workspaces = None
	150	+ self._workspace_project_cache = WorkspaceProjectCache()
143	151	self._log_handle = None
144	152	self._log_filename = None
145	153	self._cascache = None
	154	+ self._directory = directory
146	155
147	156	# load()
148	157	#
...	...	@@ -250,12 +259,15 @@ class Context():
250	259	defaults, Mapping, 'prompt')
251	260	_yaml.node_validate(prompt, [
252	261	'auto-init', 'really-workspace-close-remove-dir',
	262	+ 'really-workspace-close-project-inaccessible',
253	263	'really-workspace-reset-hard',
254	264	])
255	265	self.prompt_auto_init = _node_get_option_str(
256	266	prompt, 'auto-init', ['ask', 'no']) == 'ask'
257	267	self.prompt_workspace_close_remove_dir = _node_get_option_str(
258	268	prompt, 'really-workspace-close-remove-dir', ['ask', 'yes']) == 'ask'
	269	+ self.prompt_workspace_close_project_inaccessible = _node_get_option_str(
	270	+ prompt, 'really-workspace-close-project-inaccessible', ['ask', 'yes']) == 'ask'
259	271	self.prompt_workspace_reset_hard = _node_get_option_str(
260	272	prompt, 'really-workspace-reset-hard', ['ask', 'yes']) == 'ask'
261	273
...	...	@@ -285,7 +297,7 @@ class Context():
285	297	#
286	298	def add_project(self, project):
287	299	if not self._projects:
288		- self._workspaces = Workspaces(project)
	300	+ self._workspaces = Workspaces(project, self._workspace_project_cache)
289	301	self._projects.append(project)
290	302
291	303	# get_projects():
...	...	@@ -312,6 +324,16 @@ class Context():
312	324	def get_workspaces(self):
313	325	return self._workspaces
314	326
	327	+ # get_workspace_project_cache():
	328	+ #
	329	+ # Return the WorkspaceProjectCache object used for this BuildStream invocation
	330	+ #
	331	+ # Returns:
	332	+ # (WorkspaceProjectCache): The WorkspaceProjectCache object
	333	+ #
	334	+ def get_workspace_project_cache(self):
	335	+ return self._workspace_project_cache
	336	+
315	337	# get_overrides():
316	338	#
317	339	# Fetch the override dictionary for the active project. This returns
...	...	@@ -627,6 +649,19 @@ class Context():
627	649	self._cascache = CASCache(self.artifactdir)
628	650	return self._cascache
629	651
	652	+ # guess_element()
	653	+ #
	654	+ # Attempts to interpret which element the user intended to run commands on
	655	+ #
	656	+ # Returns:
	657	+ # (str) The name of the element, or None if no element can be guessed
	658	+ def guess_element(self):
	659	+ workspace_project = self._workspace_project_cache.get(self._directory)
	660	+ if workspace_project:
	661	+ return workspace_project.get_default_element()
	662	+ else:
	663	+ return None
	664	+
630	665
631	666	# _node_get_option_str()
632	667	#

buildstream/_frontend/app.py

---

...	...	@@ -164,7 +164,7 @@ class App():
164	164	# Load the Context
165	165	#
166	166	try:
167		- self.context = Context()
	167	+ self.context = Context(directory)
168	168	self.context.load(config)
169	169	except BstError as e:
170	170	self._error_exit(e, "Error loading user configuration")

buildstream/_frontend/cli.py

---

...	...	@@ -59,18 +59,9 @@ def complete_target(args, incomplete):
59	59	:return: all the possible user-specified completions for the param
60	60	"""
61	61
	62	+ from .. import utils
62	63	project_conf = 'project.conf'
63	64
64		- def ensure_project_dir(directory):
65		- directory = os.path.abspath(directory)
66		- while not os.path.isfile(os.path.join(directory, project_conf)):
67		- parent_dir = os.path.dirname(directory)
68		- if directory == parent_dir:
69		- break
70		- directory = parent_dir
71		-
72		- return directory
73		-
74	65	# First resolve the directory, in case there is an
75	66	# active --directory/-C option
76	67	#
...	...	@@ -89,7 +80,7 @@ def complete_target(args, incomplete):
89	80	else:
90	81	# Check if this directory or any of its parent directories
91	82	# contain a project config file
92		- base_directory = ensure_project_dir(base_directory)
	83	+ base_directory = utils._search_upward_for_file(base_directory, project_conf)
93	84
94	85	# Now parse the project.conf just to find the element path,
95	86	# this is unfortunately a bit heavy.
...	...	@@ -325,10 +316,15 @@ def build(app, elements, all_, track_, track_save, track_all, track_except, trac
325	316	if track_save:
326	317	click.echo("WARNING: --track-save is deprecated, saving is now unconditional", err=True)
327	318
328		- if track_all:
329		- track_ = elements
330		-
331	319	with app.initialized(session_name="Build"):
	320	+ if not all_ and not elements:
	321	+ guessed_target = app.context.guess_element()
	322	+ if guessed_target:
	323	+ elements = (guessed_target,)
	324	+
	325	+ if track_all:
	326	+ track_ = elements
	327	+
332	328	app.stream.build(elements,
333	329	track_targets=track_,
334	330	track_except=track_except,
...	...	@@ -380,6 +376,11 @@ def fetch(app, elements, deps, track_, except_, track_cross_junctions):
380	376	deps = PipelineSelection.ALL
381	377
382	378	with app.initialized(session_name="Fetch"):
	379	+ if not elements:
	380	+ guessed_target = app.context.guess_element()
	381	+ if guessed_target:
	382	+ elements = (guessed_target,)
	383	+
383	384	app.stream.fetch(elements,
384	385	selection=deps,
385	386	except_targets=except_,
...	...	@@ -416,6 +417,11 @@ def track(app, elements, deps, except_, cross_junctions):
416	417	all: All dependencies of all specified elements
417	418	"""
418	419	with app.initialized(session_name="Track"):
	420	+ if not elements:
	421	+ guessed_target = app.context.guess_element()
	422	+ if guessed_target:
	423	+ elements = (guessed_target,)
	424	+
419	425	# Substitute 'none' for 'redirect' so that element redirections
420	426	# will be done
421	427	if deps == 'none':
...	...	@@ -451,7 +457,13 @@ def pull(app, elements, deps, remote):
451	457	none: No dependencies, just the element itself
452	458	all: All dependencies
453	459	"""
	460	+
454	461	with app.initialized(session_name="Pull"):
	462	+ if not elements:
	463	+ guessed_target = app.context.guess_element()
	464	+ if guessed_target:
	465	+ elements = (guessed_target,)
	466	+
455	467	app.stream.pull(elements, selection=deps, remote=remote)
456	468
457	469
...	...	@@ -484,6 +496,11 @@ def push(app, elements, deps, remote):
484	496	all: All dependencies
485	497	"""
486	498	with app.initialized(session_name="Push"):
	499	+ if not elements:
	500	+ guessed_target = app.context.guess_element()
	501	+ if guessed_target:
	502	+ elements = (guessed_target,)
	503	+
487	504	app.stream.push(elements, selection=deps, remote=remote)
488	505
489	506
...	...	@@ -554,6 +571,11 @@ def show(app, elements, deps, except_, order, format_):
554	571	$'---------- %{name} ----------\\n%{vars}'
555	572	"""
556	573	with app.initialized():
	574	+ if not elements:
	575	+ guessed_target = app.context.guess_element()
	576	+ if guessed_target:
	577	+ elements = (guessed_target,)
	578	+
557	579	dependencies = app.stream.load_selection(elements,
558	580	selection=deps,
559	581	except_targets=except_)
...	...	@@ -582,7 +604,7 @@ def show(app, elements, deps, except_, order, format_):
582	604	help="Mount a file or directory into the sandbox")
583	605	@click.option('--isolate', is_flag=True, default=False,
584	606	help='Create an isolated build sandbox')
585		-@click.argument('element',
	607	+@click.argument('element', required=False,
586	608	type=click.Path(readable=False))
587	609	@click.argument('command', type=click.STRING, nargs=-1)
588	610	@click.pass_obj
...	...	@@ -613,6 +635,11 @@ def shell(app, element, sysroot, mount, isolate, build_, command):
613	635	scope = Scope.RUN
614	636
615	637	with app.initialized():
	638	+ if not element:
	639	+ element = app.context.guess_element()
	640	+ if not element:
	641	+ raise AppError('Missing argument "ELEMENT".')
	642	+
616	643	dependencies = app.stream.load_selection((element,), selection=PipelineSelection.NONE)
617	644	element = dependencies[0]
618	645	prompt = app.shell_prompt(element)
...	...	@@ -650,15 +677,24 @@ def shell(app, element, sysroot, mount, isolate, build_, command):
650	677	help="Create a tarball from the artifact contents instead "
651	678	"of a file tree. If LOCATION is '-', the tarball "
652	679	"will be dumped to the standard output.")
653		-@click.argument('element',
	680	+@click.argument('element', required=False,
654	681	type=click.Path(readable=False))
655		-@click.argument('location', type=click.Path())
	682	+@click.argument('location', type=click.Path(), required=False)
656	683	@click.pass_obj
657	684	def checkout(app, element, location, force, deps, integrate, hardlinks, tar):
658	685	"""Checkout a built artifact to the specified location
659	686	"""
660	687	from ..element import Scope
661	688
	689	+ if not element and not location:
	690	+ click.echo("ERROR: LOCATION is not specified", err=True)
	691	+ sys.exit(-1)
	692	+
	693	+ if element and not location:
	694	+ # Nasty hack to get around click's optional args
	695	+ location = element
	696	+ element = None
	697	+
662	698	if hardlinks and tar:
663	699	click.echo("ERROR: options --hardlinks and --tar conflict", err=True)
664	700	sys.exit(-1)
...	...	@@ -671,6 +707,11 @@ def checkout(app, element, location, force, deps, integrate, hardlinks, tar):
671	707	scope = Scope.NONE
672	708
673	709	with app.initialized():
	710	+ if not element:
	711	+ element = app.context.guess_element()
	712	+ if not element:
	713	+ raise AppError('Missing argument "ELEMENT".')
	714	+
674	715	app.stream.checkout(element,
675	716	location=location,
676	717	force=force,
...	...	@@ -692,14 +733,28 @@ def checkout(app, element, location, force, deps, integrate, hardlinks, tar):
692	733	help='The dependencies whose sources to checkout (default: none)')
693	734	@click.option('--fetch', 'fetch_', default=False, is_flag=True,
694	735	help='Fetch elements if they are not fetched')
695		-@click.argument('element',
	736	+@click.argument('element', required=False,
696	737	type=click.Path(readable=False))
697		-@click.argument('location', type=click.Path())
	738	+@click.argument('location', type=click.Path(), required=False)
698	739	@click.pass_obj
699	740	def source_checkout(app, element, location, deps, fetch_, except_):
700	741	"""Checkout sources of an element to the specified location
701	742	"""
	743	+ if not element and not location:
	744	+ click.echo("ERROR: LOCATION is not specified", err=True)
	745	+ sys.exit(-1)
	746	+
	747	+ if element and not location:
	748	+ # Nasty hack to get around click's optional args
	749	+ location = element
	750	+ element = None
	751	+
702	752	with app.initialized():
	753	+ if not element:
	754	+ element = app.context.guess_element()
	755	+ if not element:
	756	+ raise AppError('Missing argument "ELEMENT".')
	757	+
703	758	app.stream.source_checkout(element,
704	759	location=location,
705	760	deps=deps,
...	...	@@ -756,11 +811,15 @@ def workspace_open(app, no_checkout, force, track_, directory, elements):
756	811	def workspace_close(app, remove_dir, all_, elements):
757	812	"""Close a workspace"""
758	813
759		- if not (all_ or elements):
760		- click.echo('ERROR: no elements specified', err=True)
761		- sys.exit(-1)
762		-
763	814	with app.initialized():
	815	+ if not (all_ or elements):
	816	+ # NOTE: I may need to revisit this when implementing multiple projects
	817	+ # opening one workspace.
	818	+ element = app.context.guess_element()
	819	+ if element:
	820	+ elements = (element,)
	821	+ else:
	822	+ raise AppError('No elements specified')
764	823
765	824	# Early exit if we specified `all` and there are no workspaces
766	825	if all_ and not app.stream.workspace_exists():
...	...	@@ -772,11 +831,18 @@ def workspace_close(app, remove_dir, all_, elements):
772	831
773	832	elements = app.stream.redirect_element_names(elements)
774	833
775		- # Check that the workspaces in question exist
	834	+ # Check that the workspaces in question exist, and that it's safe to
	835	+ # remove them.
776	836	nonexisting = []
777	837	for element_name in elements:
778	838	if not app.stream.workspace_exists(element_name):
779	839	nonexisting.append(element_name)
	840	+ if (app.stream.workspace_is_required(element_name) and app.interactive and
	841	+ app.context.prompt_workspace_close_project_inaccessible):
	842	+ click.echo("Removing '{}' will prevent you from running buildstream commands".format(element_name))
	843	+ if not click.confirm('Are you sure you want to close this workspace?'):
	844	+ click.echo('Aborting', err=True)
	845	+ sys.exit(-1)
780	846	if nonexisting:
781	847	raise AppError("Workspace does not exist", detail="\n".join(nonexisting))
782	848
...	...	@@ -809,7 +875,11 @@ def workspace_reset(app, soft, track_, all_, elements):
809	875	with app.initialized():
810	876
811	877	if not (all_ or elements):
812		- raise AppError('No elements specified to reset')
	878	+ element = app.context.guess_element()
	879	+ if element:
	880	+ elements = (element,)
	881	+ else:
	882	+ raise AppError('No elements specified to reset')
813	883
814	884	if all_ and not app.stream.workspace_exists():
815	885	raise AppError("No open workspaces to reset")

buildstream/_loader/loader.py

---

...	...	@@ -36,7 +36,7 @@ from .types import Symbol, Dependency
36	36	from .loadelement import LoadElement
37	37	from . import MetaElement
38	38	from . import MetaSource
39		-from ..plugin import CoreWarnings
	39	+from ..types import CoreWarnings
40	40	from .._message import Message, MessageType
41	41
42	42

buildstream/_project.py

---

...	...	@@ -33,7 +33,7 @@ from ._artifactcache import ArtifactCache
33	33	from .sandbox import SandboxRemote
34	34	from ._elementfactory import ElementFactory
35	35	from ._sourcefactory import SourceFactory
36		-from .plugin import CoreWarnings
	36	+from .types import CoreWarnings
37	37	from ._projectrefs import ProjectRefs, ProjectRefStorage
38	38	from ._versions import BST_FORMAT_VERSION
39	39	from ._loader import Loader
...	...	@@ -95,8 +95,10 @@ class Project():
95	95	# The project name
96	96	self.name = None
97	97
98		- # The project directory
99		- self.directory = self._ensure_project_dir(directory)
	98	+ self._context = context # The invocation Context, a private member
	99	+
	100	+ # The project directory, and whether the project was found from an external workspace
	101	+ self.directory, self._required_workspace_element = self._find_project_dir(directory)
100	102
101	103	# Absolute path to where elements are loaded from within the project
102	104	self.element_path = None
...	...	@@ -117,7 +119,6 @@ class Project():
117	119	#
118	120	# Private Members
119	121	#
120		- self._context = context # The invocation Context
121	122
122	123	self._default_mirror = default_mirror # The name of the preferred mirror.
123	124
...	...	@@ -371,6 +372,14 @@ class Project():
371	372
372	373	self._load_second_pass()
373	374
	375	+ # required_workspace_element()
	376	+ #
	377	+ # Returns the element whose workspace is required to load this project,
	378	+ # if any.
	379	+ #
	380	+ def required_workspace_element(self):
	381	+ return self._required_workspace_element
	382	+
374	383	# cleanup()
375	384	#
376	385	# Cleans up resources used loading elements
...	...	@@ -650,7 +659,7 @@ class Project():
650	659	# Source url aliases
651	660	output._aliases = _yaml.node_get(config, Mapping, 'aliases', default_value={})
652	661
653		- # _ensure_project_dir()
	662	+ # _find_project_dir()
654	663	#
655	664	# Returns path of the project directory, if a configuration file is found
656	665	# in given directory or any of its parent directories.
...	...	@@ -661,18 +670,26 @@ class Project():
661	670	# Raises:
662	671	# LoadError if project.conf is not found
663	672	#
664		- def _ensure_project_dir(self, directory):
665		- directory = os.path.abspath(directory)
666		- while not os.path.isfile(os.path.join(directory, _PROJECT_CONF_FILE)):
667		- parent_dir = os.path.dirname(directory)
668		- if directory == parent_dir:
	673	+ # Returns:
	674	+ # (str) - the directory that contains the project, and
	675	+ # (str) - the name of the element required to find the project, or an empty string
	676	+ #
	677	+ def _find_project_dir(self, directory):
	678	+ workspace_element = ""
	679	+ project_directory = utils._search_upward_for_file(directory, _PROJECT_CONF_FILE)
	680	+ if not project_directory:
	681	+ workspace_project_cache = self._context.get_workspace_project_cache()
	682	+ workspace_project = workspace_project_cache.get(directory)
	683	+ if workspace_project:
	684	+ project_directory = workspace_project.get_default_project_path()
	685	+ workspace_element = workspace_project.get_default_element()
	686	+ else:
669	687	raise LoadError(
670	688	LoadErrorReason.MISSING_PROJECT_CONF,
671	689	'{} not found in current directory or any of its parent directories'
672	690	.format(_PROJECT_CONF_FILE))
673		- directory = parent_dir
674	691
675		- return directory
	692	+ return project_directory, workspace_element
676	693
677	694	def _load_plugin_factories(self, config, output):
678	695	plugin_source_origins = [] # Origins of custom sources

buildstream/_stream.py

---

...	...	@@ -544,7 +544,8 @@ class Stream():
544	544	if len(elements) != 1:
545	545	raise StreamError("Exactly one element can be given if --directory is used",
546	546	reason='directory-with-multiple-elements')
547		- expanded_directories = [custom_dir, ]
	547	+ directory = os.path.abspath(custom_dir)
	548	+ expanded_directories = [directory, ]
548	549	else:
549	550	# If this fails it is a bug in what ever calls this, usually cli.py and so can not be tested for via the
550	551	# run bst test mechanism.
...	...	@@ -581,15 +582,7 @@ class Stream():
581	582	todo_elements = "\nDid not try to create workspaces for " + todo_elements
582	583	raise StreamError("Failed to create workspace directory: {}".format(e) + todo_elements) from e
583	584
584		- workspaces.create_workspace(target._get_full_name(), directory)
585		-
586		- if not no_checkout:
587		- with target.timed_activity("Staging sources to {}".format(directory)):
588		- target._open_workspace()
589		-
590		- # Saving the workspace once it is set up means that if the next workspace fails to be created before
591		- # the configuration gets saved. The successfully created workspace still gets saved.
592		- workspaces.save_config()
	585	+ workspaces.create_workspace(target, directory, not no_checkout)
593	586	self._message(MessageType.INFO, "Created a workspace for element: {}"
594	587	.format(target._get_full_name()))
595	588
...	...	@@ -672,10 +665,7 @@ class Stream():
672	665	.format(workspace_path, e)) from e
673	666
674	667	workspaces.delete_workspace(element._get_full_name())
675		- workspaces.create_workspace(element._get_full_name(), workspace_path)
676		-
677		- with element.timed_activity("Staging sources to {}".format(workspace_path)):
678		- element._open_workspace()
	668	+ workspaces.create_workspace(element, workspace_path, True)
679	669
680	670	self._message(MessageType.INFO,
681	671	"Reset workspace for {} at: {}".format(element.name,
...	...	@@ -707,6 +697,20 @@ class Stream():
707	697
708	698	return False
709	699
	700	+ # workspace_is_required()
	701	+ #
	702	+ # Checks whether the workspace belonging to element_name is required to
	703	+ # load the project
	704	+ #
	705	+ # Args:
	706	+ # element_name (str): The element whose workspace may be required
	707	+ #
	708	+ # Returns:
	709	+ # (bool): True if the workspace is required
	710	+ def workspace_is_required(self, element_name):
	711	+ required_elm = self._project.required_workspace_element()
	712	+ return required_elm == element_name
	713	+
710	714	# workspace_list
711	715	#
712	716	# Serializes the workspaces and dumps them in YAML to stdout.

buildstream/_versions.py

---

...	...	@@ -23,7 +23,7 @@
23	23	# This version is bumped whenever enhancements are made
24	24	# to the `project.conf` format or the core element format.
25	25	#
26		-BST_FORMAT_VERSION = 19
	26	+BST_FORMAT_VERSION = 20
27	27
28	28
29	29	# The base BuildStream artifact version

buildstream/_workspaces.py

---

...	...	@@ -25,6 +25,219 @@ from ._exceptions import LoadError, LoadErrorReason
25	25
26	26
27	27	BST_WORKSPACE_FORMAT_VERSION = 3
	28	+BST_WORKSPACE_PROJECT_FORMAT_VERSION = 1
	29	+WORKSPACE_PROJECT_FILE = ".bstproject.yaml"
	30	+
	31	+
	32	+# WorkspaceProject()
	33	+#
	34	+# An object to contain various helper functions and data required for
	35	+# referring from a workspace back to buildstream.
	36	+#
	37	+# Args:
	38	+# directory (str): The directory that the workspace exists in.
	39	+#
	40	+class WorkspaceProject():
	41	+ def __init__(self, directory):
	42	+ self._projects = []
	43	+ self._directory = directory
	44	+
	45	+ # get_default_project_path()
	46	+ #
	47	+ # Retrieves the default path to a project.
	48	+ #
	49	+ # Returns:
	50	+ # (str): The path to a project
	51	+ #
	52	+ def get_default_project_path(self):
	53	+ return self._projects[0]['project-path']
	54	+
	55	+ # get_default_element()
	56	+ #
	57	+ # Retrieves the name of the element that owns this workspace.
	58	+ #
	59	+ # Returns:
	60	+ # (str): The name of an element
	61	+ #
	62	+ def get_default_element(self):
	63	+ return self._projects[0]['element-name']
	64	+
	65	+ # to_dict()
	66	+ #
	67	+ # Turn the members data into a dict for serialization purposes
	68	+ #
	69	+ # Returns:
	70	+ # (dict): A dict representation of the WorkspaceProject
	71	+ #
	72	+ def to_dict(self):
	73	+ ret = {
	74	+ 'projects': self._projects,
	75	+ 'format-version': BST_WORKSPACE_PROJECT_FORMAT_VERSION,
	76	+ }
	77	+ return ret
	78	+
	79	+ # from_dict()
	80	+ #
	81	+ # Loads a new WorkspaceProject from a simple dictionary
	82	+ #
	83	+ # Args:
	84	+ # directory (str): The directory that the workspace exists in
	85	+ # dictionary (dict): The dict to generate a WorkspaceProject from
	86	+ #
	87	+ # Returns:
	88	+ # (WorkspaceProject): A newly instantiated WorkspaceProject
	89	+ #
	90	+ @classmethod
	91	+ def from_dict(cls, directory, dictionary):
	92	+ # Only know how to handle one format-version at the moment.
	93	+ format_version = int(dictionary['format-version'])
	94	+ assert format_version == BST_WORKSPACE_PROJECT_FORMAT_VERSION, \
	95	+ "Format version {} not found in {}".format(BST_WORKSPACE_PROJECT_FORMAT_VERSION, dictionary)
	96	+
	97	+ workspace_project = cls(directory)
	98	+ for item in dictionary['projects']:
	99	+ workspace_project.add_project(item['project-path'], item['element-name'])
	100	+
	101	+ return workspace_project
	102	+
	103	+ # load()
	104	+ #
	105	+ # Loads the WorkspaceProject for a given directory. This directory may be a
	106	+ # subdirectory of the workspace's directory.
	107	+ #
	108	+ # Args:
	109	+ # directory (str): The directory
	110	+ # Returns:
	111	+ # (WorkspaceProject): The created WorkspaceProject, if in a workspace, or
	112	+ # (NoneType): None, if the directory is not inside a workspace.
	113	+ #
	114	+ @classmethod
	115	+ def load(cls, directory):
	116	+ project_dir = cls.search_for_dir(directory)
	117	+ if project_dir:
	118	+ workspace_file = os.path.join(project_dir, WORKSPACE_PROJECT_FILE)
	119	+ data_dict = _yaml.load(workspace_file)
	120	+ return cls.from_dict(project_dir, data_dict)
	121	+ else:
	122	+ return None
	123	+
	124	+ # write()
	125	+ #
	126	+ # Writes the WorkspaceProject to disk
	127	+ #
	128	+ def write(self):
	129	+ os.makedirs(self._directory, exist_ok=True)
	130	+ _yaml.dump(self.to_dict(), self.get_filename())
	131	+
	132	+ # search_for_dir()
	133	+ #
	134	+ # Returns the directory that contains the workspace local project file,
	135	+ # searching upwards from search_dir.
	136	+ #
	137	+ @staticmethod
	138	+ def search_for_dir(search_dir):
	139	+ return utils._search_upward_for_file(search_dir, WORKSPACE_PROJECT_FILE)
	140	+
	141	+ # get_filename()
	142	+ #
	143	+ # Returns the full path to the workspace local project file
	144	+ #
	145	+ def get_filename(self):
	146	+ return os.path.join(self._directory, WORKSPACE_PROJECT_FILE)
	147	+
	148	+ # add_project()
	149	+ #
	150	+ # Adds an entry containing the project's path and element's name.
	151	+ #
	152	+ # Args:
	153	+ # project_path (str): The path to the project that opened the workspace.
	154	+ # element_name (str): The name of the element that the workspace belongs to.
	155	+ #
	156	+ def add_project(self, project_path, element_name):
	157	+ assert (project_path and element_name)
	158	+ self._projects.append({'project-path': project_path, 'element-name': element_name})
	159	+
	160	+
	161	+# WorkspaceProjectCache()
	162	+#
	163	+# A class to manage workspace project data for multiple workspaces.
	164	+#
	165	+class WorkspaceProjectCache():
	166	+ def __init__(self):
	167	+ self._projects = {} # Mapping of a workspace directory to its WorkspaceProject
	168	+
	169	+ # get()
	170	+ #
	171	+ # Returns a WorkspaceProject for a given directory, retrieving from the cache if
	172	+ # present, and searching the filesystem for the file and loading it if not.
	173	+ #
	174	+ # Args:
	175	+ # directory (str): The directory to search for a WorkspaceProject.
	176	+ #
	177	+ # Returns:
	178	+ # (WorkspaceProject): The WorkspaceProject that was found for that directory.
	179	+ # or (NoneType): None, if no WorkspaceProject can be found.
	180	+ #
	181	+ def get(self, directory):
	182	+ try:
	183	+ workspace_project = self._projects[directory]
	184	+ except KeyError:
	185	+ found_dir = WorkspaceProject.search_for_dir(directory)
	186	+ if found_dir:
	187	+ try:
	188	+ workspace_project = self._projects[found_dir]
	189	+ except KeyError:
	190	+ workspace_project = WorkspaceProject.load(found_dir)
	191	+ self._projects[found_dir] = workspace_project
	192	+ else:
	193	+ workspace_project = None
	194	+
	195	+ return workspace_project
	196	+
	197	+ # add()
	198	+ #
	199	+ # Adds the project path and element name to the WorkspaceProject that exists
	200	+ # for that directory
	201	+ #
	202	+ # Args:
	203	+ # directory (str): The directory to search for a WorkspaceProject.
	204	+ # project_path (str): The path to the project that refers to this workspace
	205	+ # element_name (str): The element in the project that was refers to this workspace
	206	+ #
	207	+ # Returns:
	208	+ # (WorkspaceProject): The WorkspaceProject that was found for that directory.
	209	+ #
	210	+ def add(self, directory, project_path, element_name):
	211	+ workspace_project = self.get(directory)
	212	+ if not workspace_project:
	213	+ workspace_project = WorkspaceProject(directory)
	214	+ self._projects[directory] = workspace_project
	215	+
	216	+ workspace_project.add_project(project_path, element_name)
	217	+ return workspace_project
	218	+
	219	+ # remove()
	220	+ #
	221	+ # Removes the project path and element name from the WorkspaceProject that exists
	222	+ # for that directory.
	223	+ #
	224	+ # NOTE: This currently just deletes the file, but with support for multiple
	225	+ # projects opening the same workspace, this will involve decreasing the count
	226	+ # and deleting the file if there are no more projects.
	227	+ #
	228	+ # Args:
	229	+ # directory (str): The directory to search for a WorkspaceProject.
	230	+ #
	231	+ def remove(self, directory):
	232	+ workspace_project = self.get(directory)
	233	+ if not workspace_project:
	234	+ raise LoadError(LoadErrorReason.MISSING_FILE,
	235	+ "Failed to find a {} file to remove".format(WORKSPACE_PROJECT_FILE))
	236	+ path = workspace_project.get_filename()
	237	+ try:
	238	+ os.unlink(path)
	239	+ except FileNotFoundError:
	240	+ pass
28	241
29	242
30	243	# Workspace()
...	...	@@ -174,10 +387,15 @@ class Workspace():
174	387	if recalculate or self._key is None:
175	388	fullpath = self.get_absolute_path()
176	389
	390	+ excluded_files = (WORKSPACE_PROJECT_FILE,)
	391	+
177	392	# Get a list of tuples of the the project relative paths and fullpaths
178	393	if os.path.isdir(fullpath):
179	394	filelist = utils.list_relative_paths(fullpath)
180		- filelist = [(relpath, os.path.join(fullpath, relpath)) for relpath in filelist]
	395	+ filelist = [
	396	+ (relpath, os.path.join(fullpath, relpath)) for relpath in filelist
	397	+ if relpath not in excluded_files
	398	+ ]
181	399	else:
182	400	filelist = [(self.get_absolute_path(), fullpath)]
183	401
...	...	@@ -199,12 +417,14 @@ class Workspace():
199	417	#
200	418	# Args:
201	419	# toplevel_project (Project): Top project used to resolve paths.
	420	+# workspace_project_cache (WorkspaceProjectCache): The cache of WorkspaceProjects
202	421	#
203	422	class Workspaces():
204		- def __init__(self, toplevel_project):
	423	+ def __init__(self, toplevel_project, workspace_project_cache):
205	424	self._toplevel_project = toplevel_project
206	425	self._bst_directory = os.path.join(toplevel_project.directory, ".bst")
207	426	self._workspaces = self._load_config()
	427	+ self._workspace_project_cache = workspace_project_cache
208	428
209	429	# list()
210	430	#
...	...	@@ -219,19 +439,36 @@ class Workspaces():
219	439
220	440	# create_workspace()
221	441	#
222		- # Create a workspace in the given path for the given element.
	442	+ # Create a workspace in the given path for the given element, and potentially
	443	+ # checks-out the target into it.
223	444	#
224	445	# Args:
225		- # element_name (str) - The element name to create a workspace for
	446	+ # target (Element) - The element to create a workspace for
226	447	# path (str) - The path in which the workspace should be kept
	448	+ # checkout (bool): Whether to check-out the element's sources into the directory
227	449	#
228		- def create_workspace(self, element_name, path):
229		- if path.startswith(self._toplevel_project.directory):
230		- path = os.path.relpath(path, self._toplevel_project.directory)
	450	+ def create_workspace(self, target, path, checkout):
	451	+ element_name = target._get_full_name()
	452	+ project_dir = self._toplevel_project.directory
	453	+ if path.startswith(project_dir):
	454	+ workspace_path = os.path.relpath(path, project_dir)
	455	+ else:
	456	+ workspace_path = path
231	457
232		- self._workspaces[element_name] = Workspace(self._toplevel_project, path=path)
	458	+ self._workspaces[element_name] = Workspace(self._toplevel_project, path=workspace_path)
233	459
234		- return self._workspaces[element_name]
	460	+ if checkout:
	461	+ with target.timed_activity("Staging sources to {}".format(path)):
	462	+ target._open_workspace()
	463	+
	464	+ workspace_project = self._workspace_project_cache.add(path, project_dir, element_name)
	465	+ project_file_path = workspace_project.get_filename()
	466	+
	467	+ if os.path.exists(project_file_path):
	468	+ target.warn("{} was staged from this element's sources".format(WORKSPACE_PROJECT_FILE))
	469	+ workspace_project.write()
	470	+
	471	+ self.save_config()
235	472
236	473	# get_workspace()
237	474	#
...	...	@@ -280,8 +517,19 @@ class Workspaces():
280	517	# element_name (str) - The element name whose workspace to delete
281	518	#
282	519	def delete_workspace(self, element_name):
	520	+ workspace = self.get_workspace(element_name)
283	521	del self._workspaces[element_name]
284	522
	523	+ # Remove from the cache if it exists
	524	+ try:
	525	+ self._workspace_project_cache.remove(workspace.get_absolute_path())
	526	+ except LoadError as e:
	527	+ # We might be closing a workspace with a deleted directory
	528	+ if e.reason == LoadErrorReason.MISSING_FILE:
	529	+ pass
	530	+ else:
	531	+ raise
	532	+
285	533	# save_config()
286	534	#
287	535	# Dump the current workspace element to the project configuration

buildstream/data/userconfig.yaml

---

...	...	@@ -128,6 +128,14 @@ prompt:
128	128	#
129	129	really-workspace-close-remove-dir: ask
130	130
	131	+ # Whether to really proceed with 'bst workspace close' when doing so would
	132	+ # stop them from running bst commands in this workspace.
	133	+ #
	134	+ # ask - Ask the user if they are sure.
	135	+ # yes - Always close, without asking.
	136	+ #
	137	+ really-workspace-close-project-inaccessible: ask
	138	+
131	139	# Whether to really proceed with 'bst workspace reset' doing a hard reset of
132	140	# a workspace, potentially losing changes.
133	141	#

buildstream/element.py

---

...	...	@@ -96,10 +96,9 @@ from . import _cachekey
96	96	from . import _signals
97	97	from . import _site
98	98	from ._platform import Platform
99		-from .plugin import CoreWarnings
100	99	from .sandbox._config import SandboxConfig
101	100	from .sandbox._sandboxremote import SandboxRemote
102		-from .types import _KeyStrength
	101	+from .types import _KeyStrength, CoreWarnings
103	102
104	103	from .storage.directory import Directory
105	104	from .storage._filebaseddirectory import FileBasedDirectory

buildstream/plugin.py

---

...	...	@@ -119,6 +119,7 @@ from . import _yaml
119	119	from . import utils
120	120	from ._exceptions import PluginError, ImplError
121	121	from ._message import Message, MessageType
	122	+from .types import CoreWarnings
122	123
123	124
124	125	class Plugin():
...	...	@@ -766,38 +767,6 @@ class Plugin():
766	767	return self.name
767	768
768	769
769		-class CoreWarnings():
770		- """CoreWarnings()
771		-
772		- Some common warnings which are raised by core functionalities within BuildStream are found in this class.
773		- """
774		-
775		- OVERLAPS = "overlaps"
776		- """
777		- This warning will be produced when buildstream detects an overlap on an element
778		- which is not whitelisted. See :ref:`Overlap Whitelist <public_overlap_whitelist>`
779		- """
780		-
781		- REF_NOT_IN_TRACK = "ref-not-in-track"
782		- """
783		- This warning will be produced when a source is configured with a reference
784		- which is found to be invalid based on the configured track
785		- """
786		-
787		- BAD_ELEMENT_SUFFIX = "bad-element-suffix"
788		- """
789		- This warning will be produced when an element whose name does not end in .bst
790		- is referenced either on the command line or by another element
791		- """
792		-
793		-
794		-__CORE_WARNINGS = [
795		- value
796		- for name, value in CoreWarnings.__dict__.items()
797		- if not name.startswith("__")
798		-]
799		-
800		-
801	770	# Hold on to a lookup table by counter of all instantiated plugins.
802	771	# We use this to send the id back from child processes so we can lookup
803	772	# corresponding element/source in the master process.
...	...	@@ -828,6 +797,24 @@ def _plugin_lookup(unique_id):
828	797	return __PLUGINS_TABLE[unique_id]
829	798
830	799
	800	+# No need for unregister, WeakValueDictionary() will remove entries
	801	+# in itself when the referenced plugins are garbage collected.
	802	+def _plugin_register(plugin):
	803	+ global __PLUGINS_UNIQUE_ID # pylint: disable=global-statement
	804	+ __PLUGINS_UNIQUE_ID += 1
	805	+ __PLUGINS_TABLE[__PLUGINS_UNIQUE_ID] = plugin
	806	+ return __PLUGINS_UNIQUE_ID
	807	+
	808	+
	809	+# A local table for _prefix_warning()
	810	+#
	811	+__CORE_WARNINGS = [
	812	+ value
	813	+ for name, value in CoreWarnings.__dict__.items()
	814	+ if not name.startswith("__")
	815	+]
	816	+
	817	+
831	818	# _prefix_warning():
832	819	#
833	820	# Prefix a warning with the plugin kind. CoreWarnings are not prefixed.
...	...	@@ -843,12 +830,3 @@ def _prefix_warning(plugin, warning):
843	830	if any((warning is core_warning for core_warning in __CORE_WARNINGS)):
844	831	return warning
845	832	return "{}:{}".format(plugin.get_kind(), warning)
846		-
847		-
848		-# No need for unregister, WeakValueDictionary() will remove entries
849		-# in itself when the referenced plugins are garbage collected.
850		-def _plugin_register(plugin):
851		- global __PLUGINS_UNIQUE_ID # pylint: disable=global-statement
852		- __PLUGINS_UNIQUE_ID += 1
853		- __PLUGINS_TABLE[__PLUGINS_UNIQUE_ID] = plugin
854		- return __PLUGINS_UNIQUE_ID

buildstream/plugins/sources/git.py

---

...	...	@@ -131,13 +131,29 @@ details on common configuration options for sources.
131	131
132	132	**Configurable Warnings:**
133	133
134		-This plugin provides the following configurable warnings:
	134	+This plugin provides the following :ref:`configurable warnings <configurable_warnings>`:
135	135
136		-- 'git:inconsistent-submodule' - A submodule was found to be missing from the underlying git repository.
	136	+- ``git:inconsistent-submodule`` - A submodule present in the git repository's .gitmodules was never
	137	+ added with `git submodule add`.
137	138
138		-This plugin also utilises the following configurable core plugin warnings:
	139	+- ``git:unlisted-submodule`` - A submodule is present in the git repository but was not specified in
	140	+ the source configuration and was not disabled for checkout.
139	141
140		-- 'ref-not-in-track' - The provided ref was not found in the provided track in the element's git repository.
	142	+ .. note::
	143	+
	144	+ The ``git:unlisted-submodule`` warning is available since :ref:`format version 20 <project_format_version>`
	145	+
	146	+- ``git:invalid-submodule`` - A submodule is specified in the source configuration but does not exist
	147	+ in the repository.
	148	+
	149	+ .. note::
	150	+
	151	+ The ``git:invalid-submodule`` warning is available since :ref:`format version 20 <project_format_version>`
	152	+
	153	+This plugin also utilises the following configurable :class:`core warnings <buildstream.types.CoreWarnings>`:
	154	+
	155	+- :attr:`ref-not-in-track <buildstream.types.CoreWarnings.REF_NOT_IN_TRACK>` - The provided ref was not
	156	+ found in the provided track in the element's git repository.
141	157	"""
142	158
143	159	import os
...	...	@@ -149,15 +165,16 @@ from tempfile import TemporaryFile
149	165
150	166	from configparser import RawConfigParser
151	167
152		-from buildstream import Source, SourceError, Consistency, SourceFetcher
	168	+from buildstream import Source, SourceError, Consistency, SourceFetcher, CoreWarnings
153	169	from buildstream import utils
154		-from buildstream.plugin import CoreWarnings
155	170	from buildstream.utils import move_atomic, DirectoryExistsError
156	171
157	172	GIT_MODULES = '.gitmodules'
158	173
159	174	# Warnings
160		-INCONSISTENT_SUBMODULE = "inconsistent-submodules"
	175	+WARN_INCONSISTENT_SUBMODULE = "inconsistent-submodule"
	176	+WARN_UNLISTED_SUBMODULE = "unlisted-submodule"
	177	+WARN_INVALID_SUBMODULE = "invalid-submodule"
161	178
162	179
163	180	# Because of handling of submodules, we maintain a GitMirror
...	...	@@ -305,7 +322,7 @@ class GitMirror(SourceFetcher):
305	322
306	323	return ref, list(tags)
307	324
308		- def stage(self, directory, track=None):
	325	+ def stage(self, directory):
309	326	fullpath = os.path.join(directory, self.path)
310	327
311	328	# Using --shared here avoids copying the objects into the checkout, in any
...	...	@@ -324,11 +341,7 @@ class GitMirror(SourceFetcher):
324	341
325	342	self._rebuild_git(fullpath)
326	343
327		- # Check that the user specified ref exists in the track if provided & not already tracked
328		- if track:
329		- self.assert_ref_in_track(fullpath, track)
330		-
331		- def init_workspace(self, directory, track=None):
	344	+ def init_workspace(self, directory):
332	345	fullpath = os.path.join(directory, self.path)
333	346	url = self.source.translate_url(self.url)
334	347
...	...	@@ -344,10 +357,6 @@ class GitMirror(SourceFetcher):
344	357	fail="Failed to checkout git ref {}".format(self.ref),
345	358	cwd=fullpath)
346	359
347		- # Check that the user specified ref exists in the track if provided & not already tracked
348		- if track:
349		- self.assert_ref_in_track(fullpath, track)
350		-
351	360	# List the submodules (path/url tuples) present at the given ref of this repo
352	361	def submodule_list(self):
353	362	modules = "{}:{}".format(self.ref, GIT_MODULES)
...	...	@@ -408,32 +417,11 @@ class GitMirror(SourceFetcher):
408	417	"underlying git repository with `git submodule add`."
409	418
410	419	self.source.warn("{}: Ignoring inconsistent submodule '{}'"
411		- .format(self.source, submodule), detail=detail, warning_token=INCONSISTENT_SUBMODULE)
	420	+ .format(self.source, submodule), detail=detail,
	421	+ warning_token=WARN_INCONSISTENT_SUBMODULE)
412	422
413	423	return None
414	424
415		- # Assert that ref exists in track, if track has been specified.
416		- def assert_ref_in_track(self, fullpath, track):
417		- _, branch = self.source.check_output([self.source.host_git, 'branch', '--list', track,
418		- '--contains', self.ref],
419		- cwd=fullpath,)
420		- if branch:
421		- return
422		- else:
423		- _, tag = self.source.check_output([self.source.host_git, 'tag', '--list', track,
424		- '--contains', self.ref],
425		- cwd=fullpath,)
426		- if tag:
427		- return
428		-
429		- detail = "The ref provided for the element does not exist locally in the provided track branch / tag " + \
430		- "'{}'.\nYou may wish to track the element to update the ref from '{}' ".format(track, track) + \
431		- "with `bst track`,\nor examine the upstream at '{}' for the specific ref.".format(self.url)
432		-
433		- self.source.warn("{}: expected ref '{}' was not found in given track '{}' for staged repository: '{}'\n"
434		- .format(self.source, self.ref, track, self.url),
435		- detail=detail, warning_token=CoreWarnings.REF_NOT_IN_TRACK)
436		-
437	425	def _rebuild_git(self, fullpath):
438	426	if not self.tags:
439	427	return
...	...	@@ -562,7 +550,6 @@ class GitSource(Source):
562	550	self.submodule_checkout_overrides[path] = checkout
563	551
564	552	self.mark_download_url(self.original_url)
565		- self.tracked = False
566	553
567	554	def preflight(self):
568	555	# Check if git is installed, get the binary at the same time
...	...	@@ -652,8 +639,6 @@ class GitSource(Source):
652	639	# Update self.mirror.ref and node.ref from the self.tracking branch
653	640	ret = self.mirror.latest_commit_with_tags(self.tracking, self.track_tags)
654	641
655		- # Set tracked attribute, parameter for if self.mirror.assert_ref_in_track is needed
656		- self.tracked = True
657	642	return ret
658	643
659	644	def init_workspace(self, directory):
...	...	@@ -661,7 +646,7 @@ class GitSource(Source):
661	646	self.refresh_submodules()
662	647
663	648	with self.timed_activity('Setting up workspace "{}"'.format(directory), silent_nested=True):
664		- self.mirror.init_workspace(directory, track=(self.tracking if not self.tracked else None))
	649	+ self.mirror.init_workspace(directory)
665	650	for mirror in self.submodules:
666	651	mirror.init_workspace(directory)
667	652
...	...	@@ -677,15 +662,9 @@ class GitSource(Source):
677	662	# Stage the main repo in the specified directory
678	663	#
679	664	with self.timed_activity("Staging {}".format(self.mirror.url), silent_nested=True):
680		- self.mirror.stage(directory, track=(self.tracking if not self.tracked else None))
	665	+ self.mirror.stage(directory)
681	666	for mirror in self.submodules:
682		- if mirror.path in self.submodule_checkout_overrides:
683		- checkout = self.submodule_checkout_overrides[mirror.path]
684		- else:
685		- checkout = self.checkout_submodules
686		-
687		- if checkout:
688		- mirror.stage(directory)
	667	+ mirror.stage(directory)
689	668
690	669	def get_source_fetchers(self):
691	670	yield self.mirror
...	...	@@ -693,6 +672,74 @@ class GitSource(Source):
693	672	for submodule in self.submodules:
694	673	yield submodule
695	674
	675	+ def validate_cache(self):
	676	+ discovered_submodules = {}
	677	+ unlisted_submodules = []
	678	+ invalid_submodules = []
	679	+
	680	+ for path, url in self.mirror.submodule_list():
	681	+ discovered_submodules[path] = url
	682	+ if self.ignore_submodule(path):
	683	+ continue
	684	+
	685	+ override_url = self.submodule_overrides.get(path)
	686	+ if not override_url:
	687	+ unlisted_submodules.append((path, url))
	688	+
	689	+ # Warn about submodules which are explicitly configured but do not exist
	690	+ for path, url in self.submodule_overrides.items():
	691	+ if path not in discovered_submodules:
	692	+ invalid_submodules.append((path, url))
	693	+
	694	+ if invalid_submodules:
	695	+ detail = []
	696	+ for path, url in invalid_submodules:
	697	+ detail.append(" Submodule URL '{}' at path '{}'".format(url, path))
	698	+
	699	+ self.warn("{}: Invalid submodules specified".format(self),
	700	+ warning_token=WARN_INVALID_SUBMODULE,
	701	+ detail="The following submodules are specified in the source "
	702	+ "description but do not exist according to the repository\n\n" +
	703	+ "\n".join(detail))
	704	+
	705	+ # Warn about submodules which exist but have not been explicitly configured
	706	+ if unlisted_submodules:
	707	+ detail = []
	708	+ for path, url in unlisted_submodules:
	709	+ detail.append(" Submodule URL '{}' at path '{}'".format(url, path))
	710	+
	711	+ self.warn("{}: Unlisted submodules exist".format(self),
	712	+ warning_token=WARN_UNLISTED_SUBMODULE,
	713	+ detail="The following submodules exist but are not specified " +
	714	+ "in the source description\n\n" +
	715	+ "\n".join(detail))
	716	+
	717	+ # Assert that the ref exists in the track tag/branch, if track has been specified.
	718	+ ref_in_track = False
	719	+ if self.tracking:
	720	+ _, branch = self.check_output([self.host_git, 'branch', '--list', self.tracking,
	721	+ '--contains', self.mirror.ref],
	722	+ cwd=self.mirror.mirror)
	723	+ if branch:
	724	+ ref_in_track = True
	725	+ else:
	726	+ _, tag = self.check_output([self.host_git, 'tag', '--list', self.tracking,
	727	+ '--contains', self.mirror.ref],
	728	+ cwd=self.mirror.mirror)
	729	+ if tag:
	730	+ ref_in_track = True
	731	+
	732	+ if not ref_in_track:
	733	+ detail = "The ref provided for the element does not exist locally " + \
	734	+ "in the provided track branch / tag '{}'.\n".format(self.tracking) + \
	735	+ "You may wish to track the element to update the ref from '{}' ".format(self.tracking) + \
	736	+ "with `bst track`,\n" + \
	737	+ "or examine the upstream at '{}' for the specific ref.".format(self.mirror.url)
	738	+
	739	+ self.warn("{}: expected ref '{}' was not found in given track '{}' for staged repository: '{}'\n"
	740	+ .format(self, self.mirror.ref, self.tracking, self.mirror.url),
	741	+ detail=detail, warning_token=CoreWarnings.REF_NOT_IN_TRACK)
	742	+
696	743	###########################################################
697	744	# Local Functions #
698	745	###########################################################
...	...	@@ -717,12 +764,12 @@ class GitSource(Source):
717	764	self.mirror.ensure()
718	765	submodules = []
719	766
720		- # XXX Here we should issue a warning if either:
721		- # A.) A submodule exists but is not defined in the element configuration
722		- # B.) The element configuration configures submodules which dont exist at the current ref
723		- #
724	767	for path, url in self.mirror.submodule_list():
725	768
	769	+ # Completely ignore submodules which are disabled for checkout
	770	+ if self.ignore_submodule(path):
	771	+ continue
	772	+
726	773	# Allow configuration to override the upstream
727	774	# location of the submodules.
728	775	override_url = self.submodule_overrides.get(path)
...	...	@@ -746,6 +793,16 @@ class GitSource(Source):
746	793	tags.append((tag, commit_ref, annotated))
747	794	return tags
748	795
	796	+ # Checks whether the plugin configuration has explicitly
	797	+ # configured this submodule to be ignored
	798	+ def ignore_submodule(self, path):
	799	+ try:
	800	+ checkout = self.submodule_checkout_overrides[path]
	801	+ except KeyError:
	802	+ checkout = self.checkout_submodules
	803	+
	804	+ return not checkout
	805	+
749	806
750	807	# Plugin entry point
751	808	def setup():

buildstream/source.py

---

...	...	@@ -102,6 +102,11 @@ these methods are mandatory to implement.
102	102	submodules). For details on how to define a SourceFetcher, see
103	103	:ref:`SourceFetcher <core_source_fetcher>`.
104	104
	105	+* :func:`Source.validate_cache() <buildstream.source.Source.validate_cache>`
	106	+
	107	+ Perform any validations which require the sources to be cached.
	108	+
	109	+ **Optional**: This is completely optional and will do nothing if left unimplemented.
105	110
106	111	Accessing previous sources
107	112	--------------------------
...	...	@@ -391,7 +396,8 @@ class Source(Plugin):
391	396
392	397	If the backend in question supports resolving references from
393	398	a symbolic tracking branch or tag, then this should be implemented
394		- to perform this task on behalf of ``build-stream track`` commands.
	399	+ to perform this task on behalf of :ref:`bst track <invoking_track>`
	400	+ commands.
395	401
396	402	This usually requires fetching new content from a remote origin
397	403	to see if a new ref has appeared for your branch or tag. If the
...	...	@@ -479,9 +485,22 @@ class Source(Plugin):
479	485
480	486	*Since: 1.2*
481	487	"""
482		-
483	488	return []
484	489
	490	+ def validate_cache(self):
	491	+ """Implement any validations once we know the sources are cached
	492	+
	493	+ This is guaranteed to be called only once for a given session
	494	+ once the sources are known to be
	495	+ :attr:`Consistency.CACHED <buildstream.types.Consistency.CACHED>`,
	496	+ if source tracking is enabled in the session for this source,
	497	+ then this will only be called if the sources become cached after
	498	+ tracking completes.
	499	+
	500	+ *Since: 1.4*
	501	+ """
	502	+ pass
	503	+
485	504	#############################################################
486	505	# Public Methods #
487	506	#############################################################
...	...	@@ -658,6 +677,11 @@ class Source(Plugin):
658	677	with context.silence():
659	678	self.__consistency = self.get_consistency() # pylint: disable=assignment-from-no-return
660	679
	680	+ # Give the Source an opportunity to validate the cached
	681	+ # sources as soon as the Source becomes Consistency.CACHED.
	682	+ if self.__consistency == Consistency.CACHED:
	683	+ self.validate_cache()
	684	+
661	685	# Return cached consistency
662	686	#
663	687	def _get_consistency(self):

buildstream/types.py

---

...	...	@@ -81,6 +81,31 @@ class Consistency():
81	81	"""
82	82
83	83
	84	+class CoreWarnings():
	85	+ """CoreWarnings()
	86	+
	87	+ Some common warnings which are raised by core functionalities within BuildStream are found in this class.
	88	+ """
	89	+
	90	+ OVERLAPS = "overlaps"
	91	+ """
	92	+ This warning will be produced when buildstream detects an overlap on an element
	93	+ which is not whitelisted. See :ref:`Overlap Whitelist <public_overlap_whitelist>`
	94	+ """
	95	+
	96	+ REF_NOT_IN_TRACK = "ref-not-in-track"
	97	+ """
	98	+ This warning will be produced when a source is configured with a reference
	99	+ which is found to be invalid based on the configured track
	100	+ """
	101	+
	102	+ BAD_ELEMENT_SUFFIX = "bad-element-suffix"
	103	+ """
	104	+ This warning will be produced when an element whose name does not end in .bst
	105	+ is referenced either on the command line or by another element
	106	+ """
	107	+
	108	+
84	109	# _KeyStrength():
85	110	#
86	111	# Strength of cache key

buildstream/utils.py

---

...	...	@@ -1259,3 +1259,17 @@ def _message_digest(message_buffer):
1259	1259	digest.hash = sha.hexdigest()
1260	1260	digest.size_bytes = len(message_buffer)
1261	1261	return digest
	1262	+
	1263	+
	1264	+# Returns the first directory to contain filename, or an empty string if
	1265	+# none found
	1266	+#
	1267	+def _search_upward_for_file(directory, filename):
	1268	+ directory = os.path.abspath(directory)
	1269	+ while not os.path.isfile(os.path.join(directory, filename)):
	1270	+ parent_dir = os.path.dirname(directory)
	1271	+ if directory == parent_dir:
	1272	+ return ""
	1273	+ directory = parent_dir
	1274	+
	1275	+ return directory

doc/sessions/developing.run

---

...	...	@@ -24,6 +24,11 @@ commands:
24	24	output: ../source/sessions/developing-build-after-changes.html
25	25	command: build hello.bst
26	26
	27	+# Rebuild, from the workspace
	28	+- directory: ../examples/developing/workspace_hello
	29	+ output: ../source/sessions/developing-build-after-changes-workspace.html
	30	+ command: build
	31	+
27	32	# Capture shell output with changes
28	33	- directory: ../examples/developing/
29	34	output: ../source/sessions/developing-shell-after-changes.html

doc/source/developing/workspaces.rst

---

...	...	@@ -50,11 +50,16 @@ We can open workspace_hello/hello.c and make the following change:
50	50	.. literalinclude:: ../../examples/developing/update.patch
51	51	:language: diff
52	52
53		-Now, rebuild the hello.bst element
	53	+Now, rebuild the hello.bst element.
54	54
55	55	.. raw:: html
56	56	:file: ../sessions/developing-build-after-changes.html
57	57
	58	+Note that if you run the command from inside the workspace, the element name is optional.
	59	+
	60	+.. raw:: html
	61	+ :file: ../sessions/developing-build-after-changes-workspace.html
	62	+
58	63	Now running the hello command using bst shell:
59	64
60	65	.. raw:: html

doc/source/format_project.rst

---

...	...	@@ -143,7 +143,7 @@ Individual warnings can be configured as fatal by setting ``fatal-warnings`` to
143	143	- ref-not-in-track
144	144	- <plugin>:<warning>
145	145
146		-BuildStream provides a collection of :class:`Core Warnings <buildstream.plugin.CoreWarnings>` which may be raised
	146	+BuildStream provides a collection of :class:`Core Warnings <buildstream.types.CoreWarnings>` which may be raised
147	147	by a variety of plugins. Other configurable warnings are plugin specific and should be noted within their individual documentation.
148	148
149	149	.. note::

doc/source/sessions-stored/developing-build-after-changes-workspace.html

---

	1	+<!--
	2	+ WARNING: This file was generated with bst2html.py
	3	+-->
	4	+<div class="highlight" style="font-size:x-small"><pre>
	5	+<span style="color:#C4A000;font-weight:bold">user@host</span>:<span style="color:#3456A4;font-weight:bold">~/workspace_hello</span>$ bst build
	6	+
	7	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#3465A4"><span style=""><span style="opacity:0.5">START </span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Build
	8	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#3465A4"><span style=""><span style="opacity:0.5">START </span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Loading elements
	9	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#4E9A06"><span style=""><span style="opacity:0.5">SUCCESS</span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Loading elements
	10	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#3465A4"><span style=""><span style="opacity:0.5">START </span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Resolving elements
	11	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#4E9A06"><span style=""><span style="opacity:0.5">SUCCESS</span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Resolving elements
	12	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#3465A4"><span style=""><span style="opacity:0.5">START </span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Resolving cached state
	13	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#4E9A06"><span style=""><span style="opacity:0.5">SUCCESS</span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Resolving cached state
	14	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#3465A4"><span style=""><span style="opacity:0.5">START </span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Checking sources
	15	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#4E9A06"><span style=""><span style="opacity:0.5">SUCCESS</span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Checking sources
	16	+
	17	+<span style="color:#C4A000"><span style="">BuildStream Version 1.3.0+1032.g41813d3a.dirty
	18	+</span></span><span style="color:#06989A"><span style="opacity:0.5"> Session Start: </span></span><span style="color:#C4A000">Wednesday, 05-12-2018 at 16:47:38</span>
	19	+<span style="color:#06989A"><span style="opacity:0.5"> Project: </span></span><span style="color:#C4A000">developing (/home/user/workspace/buildstream/buildstream/doc/examples/developing)</span>
	20	+<span style="color:#06989A"><span style="opacity:0.5"> Targets: </span></span><span style="color:#C4A000">hello.bst</span>
	21	+
	22	+<span style="color:#C4A000"><span style="">User Configuration
	23	+</span></span><span style="color:#06989A"><span style="opacity:0.5"> Configuration File: </span></span><span style="color:#C4A000">/home/user/workspace/buildstream/buildstream/doc/run-bst-7ocq4_a7/buildstream.conf</span>
	24	+<span style="color:#06989A"><span style="opacity:0.5"> Log Files: </span></span><span style="color:#C4A000">/home/user/workspace/buildstream/buildstream/doc/run-bst-7ocq4_a7/logs</span>
	25	+<span style="color:#06989A"><span style="opacity:0.5"> Source Mirrors: </span></span><span style="color:#C4A000">/home/user/workspace/buildstream/buildstream/doc/run-bst-7ocq4_a7/sources</span>
	26	+<span style="color:#06989A"><span style="opacity:0.5"> Build Area: </span></span><span style="color:#C4A000">/home/user/workspace/buildstream/buildstream/doc/run-bst-7ocq4_a7/build</span>
	27	+<span style="color:#06989A"><span style="opacity:0.5"> Artifact Cache: </span></span><span style="color:#C4A000">/home/user/workspace/buildstream/buildstream/doc/run-bst-7ocq4_a7/artifacts</span>
	28	+<span style="color:#06989A"><span style="opacity:0.5"> Strict Build Plan: </span></span><span style="color:#C4A000">Yes</span>
	29	+<span style="color:#06989A"><span style="opacity:0.5"> Maximum Fetch Tasks: </span></span><span style="color:#C4A000">10</span>
	30	+<span style="color:#06989A"><span style="opacity:0.5"> Maximum Build Tasks: </span></span><span style="color:#C4A000">4</span>
	31	+<span style="color:#06989A"><span style="opacity:0.5"> Maximum Push Tasks: </span></span><span style="color:#C4A000">4</span>
	32	+<span style="color:#06989A"><span style="opacity:0.5"> Maximum Network Retries: </span></span><span style="color:#C4A000">2</span>
	33	+
	34	+<span style="color:#C4A000"><span style="">Pipeline
	35	+</span></span><span style="color:#75507B"> cached</span> <span style="color:#C4A000">9afe69d645f0bee106749bc2101aae16ef437bb51e1b343ef1f16f04f0572efb</span> <span style="color:#3465A4"><span style="">base/alpine.bst</span></span>
	36	+<span style="color:#75507B"> cached</span> <span style="color:#C4A000">19f7c50c7a1db9ae4babe9d1f34f4cdbbf2428827d48673861fd1452d6c7e16b</span> <span style="color:#3465A4"><span style="">base.bst</span></span>
	37	+<span style="color:#75507B"> cached</span> <span style="color:#C4A000">faa419610e7309d36d15926a81a8d75bbc113443c23c8162e63843dd86b5f56a</span> <span style="color:#3465A4"><span style="">hello.bst</span></span> Workspace: /home/user/workspace/buildstream/buildstream/doc/examples/developing/workspace_hello
	38	+<span style="color:#06989A"><span style="opacity:0.5">===============================================================================
	39	+</span></span><span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#4E9A06"><span style=""><span style="opacity:0.5">SUCCESS</span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Build
	40	+
	41	+<span style="color:#C4A000"><span style="">Pipeline Summary
	42	+</span></span><span style="color:#06989A"><span style="opacity:0.5"> Total: </span></span><span style="color:#C4A000">3</span>
	43	+<span style="color:#06989A"><span style="opacity:0.5"> Session: </span></span><span style="color:#C4A000">0</span>
	44	+<span style="color:#06989A"><span style="opacity:0.5"> Fetch Queue: </span></span><span style="color:#C4A000">processed </span><span style="color:#4E9A06">0</span><span style="color:#06989A"><span style="opacity:0.5">, </span></span><span style="color:#C4A000">skipped </span><span style="color:#C4A000">0</span><span style="color:#06989A"><span style="opacity:0.5">, </span></span><span style="color:#C4A000">failed </span><span style="color:#CC0000"><span style="opacity:0.5">0</span></span>
	45	+<span style="color:#06989A"><span style="opacity:0.5"> Build Queue: </span></span><span style="color:#C4A000">processed </span><span style="color:#4E9A06">0</span><span style="color:#06989A"><span style="opacity:0.5">, </span></span><span style="color:#C4A000">skipped </span><span style="color:#C4A000">0</span><span style="color:#06989A"><span style="opacity:0.5">, </span></span><span style="color:#C4A000">failed </span><span style="color:#CC0000"><span style="opacity:0.5">0</span></span>
	46	+</pre></div>

tests/frontend/source_checkout.py

---

...	...	@@ -28,12 +28,27 @@ def generate_remote_import_element(input_path, output_path):
28	28
29	29
30	30	@pytest.mark.datafiles(DATA_DIR)
31		-def test_source_checkout(datafiles, cli):
	31	+@pytest.mark.parametrize(
	32	+ "from_workspace,guess_element",
	33	+ [("workspace", "guess"), ("workspace", "no-guess"), ("no-workspace", "no-guess")]
	34	+)
	35	+def test_source_checkout(datafiles, cli, tmpdir_factory, from_workspace, guess_element):
	36	+ tmpdir = tmpdir_factory.mktemp("")
32	37	project = os.path.join(datafiles.dirname, datafiles.basename)
33	38	checkout = os.path.join(cli.directory, 'source-checkout')
34	39	target = 'checkout-deps.bst'
	40	+ workspace = os.path.join(str(tmpdir), 'workspace')
	41	+ elm_cmd = [target] if guess_element == "no-guess" else []
35	42
36		- result = cli.run(project=project, args=['source-checkout', target, '--deps', 'none', checkout])
	43	+ if from_workspace == "workspace":
	44	+ ws_cmd = ['-C', workspace]
	45	+ result = cli.run(project=project, args=["workspace", "open", "--directory", workspace, target])
	46	+ result.assert_success()
	47	+ else:
	48	+ ws_cmd = []
	49	+
	50	+ args = ws_cmd + ['source-checkout', '--deps', 'none'] + elm_cmd + [checkout]
	51	+ result = cli.run(project=project, args=args)
37	52	result.assert_success()
38	53
39	54	assert os.path.exists(os.path.join(checkout, 'checkout-deps', 'etc', 'buildstream', 'config'))

tests/frontend/workspace.py

---

...	...	@@ -31,6 +31,7 @@ import shutil
31	31	import subprocess
32	32	from ruamel.yaml.comments import CommentedSet
33	33	from tests.testutils import cli, create_repo, ALL_REPO_KINDS, wait_for_cache_granularity
	34	+from tests.testutils import create_artifact_share
34	35
35	36	from buildstream import _yaml
36	37	from buildstream._exceptions import ErrorDomain, LoadError, LoadErrorReason
...	...	@@ -615,9 +616,15 @@ def test_list(cli, tmpdir, datafiles):
615	616	@pytest.mark.datafiles(DATA_DIR)
616	617	@pytest.mark.parametrize("kind", repo_kinds)
617	618	@pytest.mark.parametrize("strict", [("strict"), ("non-strict")])
618		-def test_build(cli, tmpdir, datafiles, kind, strict):
	619	+@pytest.mark.parametrize("call_from,guess_element", [
	620	+ ("project", "no-guess"), ("workspace", "guess"), ("workspace", "no-guess")
	621	+])
	622	+def test_build(cli, tmpdir_factory, datafiles, kind, strict, call_from, guess_element):
	623	+ tmpdir = tmpdir_factory.mktemp('')
619	624	element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, kind, False)
620	625	checkout = os.path.join(str(tmpdir), 'checkout')
	626	+ args_dir = ['-C', workspace] if call_from == "workspace" else []
	627	+ args_elm = [element_name] if guess_element == "no-guess" else []
621	628
622	629	# Modify workspace
623	630	shutil.rmtree(os.path.join(workspace, 'usr', 'bin'))
...	...	@@ -640,15 +647,14 @@ def test_build(cli, tmpdir, datafiles, kind, strict):
640	647	# Build modified workspace
641	648	assert cli.get_element_state(project, element_name) == 'buildable'
642	649	assert cli.get_element_key(project, element_name) == "{:?<64}".format('')
643		- result = cli.run(project=project, args=['build', element_name])
	650	+ result = cli.run(project=project, args=args_dir + ['build'] + args_elm)
644	651	result.assert_success()
645	652	assert cli.get_element_state(project, element_name) == 'cached'
646	653	assert cli.get_element_key(project, element_name) != "{:?<64}".format('')
647	654
648	655	# Checkout the result
649		- result = cli.run(project=project, args=[
650		- 'checkout', element_name, checkout
651		- ])
	656	+ result = cli.run(project=project,
	657	+ args=args_dir + ['checkout'] + args_elm + [checkout])
652	658	result.assert_success()
653	659
654	660	# Check that the pony.conf from the modified workspace exists
...	...	@@ -1055,3 +1061,144 @@ def test_multiple_failed_builds(cli, tmpdir, datafiles):
1055	1061	result = cli.run(project=project, args=["build", element_name])
1056	1062	assert "BUG" not in result.stderr
1057	1063	assert cli.get_element_state(project, element_name) != "cached"
	1064	+
	1065	+
	1066	+@pytest.mark.datafiles(DATA_DIR)
	1067	+@pytest.mark.parametrize("guess_element", [("guess"), ("no-guess")])
	1068	+def test_external_fetch(cli, datafiles, tmpdir_factory, guess_element):
	1069	+ # Fetching from a workspace outside a project doesn't fail horribly
	1070	+ tmpdir = tmpdir_factory.mktemp('')
	1071	+ element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", False)
	1072	+ arg_elm = [element_name] if guess_element == "no-guess" else []
	1073	+
	1074	+ result = cli.run(project=project, args=['-C', workspace, 'fetch'] + arg_elm)
	1075	+ result.assert_success()
	1076	+
	1077	+ # We already fetched it by opening the workspace, but we're also checking
	1078	+ # `bst show` works here
	1079	+ result = cli.run(project=project,
	1080	+ args=['-C', workspace, 'show', '--deps', 'none', '--format', '%{state}'] + arg_elm)
	1081	+ result.assert_success()
	1082	+ assert result.output.strip() == 'buildable'
	1083	+
	1084	+
	1085	+@pytest.mark.datafiles(DATA_DIR)
	1086	+@pytest.mark.parametrize("guess_element", [("guess"), ("no-guess")])
	1087	+def test_external_push_pull(cli, datafiles, tmpdir_factory, guess_element):
	1088	+ # Pushing and pulling to/from an artifact cache works from an external workspace
	1089	+ tmpdir = tmpdir_factory.mktemp('')
	1090	+ element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", False)
	1091	+ args_elm = [element_name] if guess_element == "no-guess" else []
	1092	+
	1093	+ with create_artifact_share(os.path.join(str(tmpdir), 'artifactshare')) as share:
	1094	+ result = cli.run(project=project, args=['-C', workspace, 'build', element_name])
	1095	+ result.assert_success()
	1096	+
	1097	+ cli.configure({
	1098	+ 'artifacts': {'url': share.repo, 'push': True}
	1099	+ })
	1100	+
	1101	+ result = cli.run(project=project, args=['-C', workspace, 'push'] + args_elm)
	1102	+ result.assert_success()
	1103	+
	1104	+ result = cli.run(project=project, args=['-C', workspace, 'pull', '--deps', 'all'] + args_elm)
	1105	+ result.assert_success()
	1106	+
	1107	+
	1108	+@pytest.mark.datafiles(DATA_DIR)
	1109	+@pytest.mark.parametrize("guess_element", [("guess"), ("no-guess")])
	1110	+def test_external_track(cli, datafiles, tmpdir_factory, guess_element):
	1111	+ # Tracking does not get horribly confused
	1112	+ tmpdir = tmpdir_factory.mktemp('')
	1113	+ element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", True)
	1114	+ args_elm = [element_name] if guess_element == "no-guess" else []
	1115	+
	1116	+ # The workspace is necessarily already tracked, so we only care that
	1117	+ # there's no weird errors.
	1118	+ result = cli.run(project=project, args=['-C', workspace, 'track'] + args_elm)
	1119	+ result.assert_success()
	1120	+
	1121	+
	1122	+@pytest.mark.datafiles(DATA_DIR)
	1123	+def test_external_open_other(cli, datafiles, tmpdir_factory):
	1124	+ # >From inside an external workspace, open another workspace
	1125	+ tmpdir1 = tmpdir_factory.mktemp('')
	1126	+ tmpdir2 = tmpdir_factory.mktemp('')
	1127	+ # Making use of the assumption that it's the same project in both invocations of open_workspace
	1128	+ alpha_element, project, alpha_workspace = open_workspace(cli, tmpdir1, datafiles, "git", False, suffix="-alpha")
	1129	+ beta_element, _, beta_workspace = open_workspace(cli, tmpdir2, datafiles, "git", False, suffix="-beta")
	1130	+
	1131	+ # Closing the other element first, because I'm too lazy to create an
	1132	+ # element without opening it
	1133	+ result = cli.run(project=project, args=['workspace', 'close', beta_element])
	1134	+ result.assert_success()
	1135	+
	1136	+ result = cli.run(project=project, args=[
	1137	+ '-C', alpha_workspace, 'workspace', 'open', '--force', '--directory', beta_workspace, beta_element
	1138	+ ])
	1139	+ result.assert_success()
	1140	+
	1141	+
	1142	+@pytest.mark.datafiles(DATA_DIR)
	1143	+def test_external_close_other(cli, datafiles, tmpdir_factory):
	1144	+ # >From inside an external workspace, close the other workspace
	1145	+ tmpdir1 = tmpdir_factory.mktemp('')
	1146	+ tmpdir2 = tmpdir_factory.mktemp('')
	1147	+ # Making use of the assumption that it's the same project in both invocations of open_workspace
	1148	+ alpha_element, project, alpha_workspace = open_workspace(cli, tmpdir1, datafiles, "git", False, suffix="-alpha")
	1149	+ beta_element, _, beta_workspace = open_workspace(cli, tmpdir2, datafiles, "git", False, suffix="-beta")
	1150	+
	1151	+ result = cli.run(project=project, args=['-C', alpha_workspace, 'workspace', 'close', beta_element])
	1152	+ result.assert_success()
	1153	+
	1154	+
	1155	+@pytest.mark.datafiles(DATA_DIR)
	1156	+@pytest.mark.parametrize("guess_element", [("guess"), ("no-guess")])
	1157	+def test_external_close_self(cli, datafiles, tmpdir_factory, guess_element):
	1158	+ # >From inside an external workspace, close it
	1159	+ tmpdir1 = tmpdir_factory.mktemp('')
	1160	+ tmpdir2 = tmpdir_factory.mktemp('')
	1161	+ # Making use of the assumption that it's the same project in both invocations of open_workspace
	1162	+ alpha_element, project, alpha_workspace = open_workspace(cli, tmpdir1, datafiles, "git", False, suffix="-alpha")
	1163	+ beta_element, _, beta_workspace = open_workspace(cli, tmpdir2, datafiles, "git", False, suffix="-beta")
	1164	+ arg_elm = [alpha_element] if guess_element == "no-guess" else []
	1165	+
	1166	+ result = cli.run(project=project, args=['-C', alpha_workspace, 'workspace', 'close'] + arg_elm)
	1167	+ result.assert_success()
	1168	+
	1169	+
	1170	+@pytest.mark.datafiles(DATA_DIR)
	1171	+def test_external_reset_other(cli, datafiles, tmpdir_factory):
	1172	+ tmpdir1 = tmpdir_factory.mktemp('')
	1173	+ tmpdir2 = tmpdir_factory.mktemp('')
	1174	+ # Making use of the assumption that it's the same project in both invocations of open_workspace
	1175	+ alpha_element, project, alpha_workspace = open_workspace(cli, tmpdir1, datafiles, "git", False, suffix="-alpha")
	1176	+ beta_element, _, beta_workspace = open_workspace(cli, tmpdir2, datafiles, "git", False, suffix="-beta")
	1177	+
	1178	+ result = cli.run(project=project, args=['-C', alpha_workspace, 'workspace', 'reset', beta_element])
	1179	+ result.assert_success()
	1180	+
	1181	+
	1182	+@pytest.mark.datafiles(DATA_DIR)
	1183	+@pytest.mark.parametrize("guess_element", [("guess"), ("no-guess")])
	1184	+def test_external_reset_self(cli, datafiles, tmpdir, guess_element):
	1185	+ element, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", False)
	1186	+ arg_elm = [element] if guess_element == "no-guess" else []
	1187	+
	1188	+ # Command succeeds
	1189	+ result = cli.run(project=project, args=['-C', workspace, 'workspace', 'reset'] + arg_elm)
	1190	+ result.assert_success()
	1191	+
	1192	+ # Successive commands still work (i.e. .bstproject.yaml hasn't been deleted)
	1193	+ result = cli.run(project=project, args=['-C', workspace, 'workspace', 'list'])
	1194	+ result.assert_success()
	1195	+
	1196	+
	1197	+@pytest.mark.datafiles(DATA_DIR)
	1198	+def test_external_list(cli, datafiles, tmpdir_factory):
	1199	+ tmpdir = tmpdir_factory.mktemp('')
	1200	+ # Making use of the assumption that it's the same project in both invocations of open_workspace
	1201	+ element, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", False)
	1202	+
	1203	+ result = cli.run(project=project, args=['-C', workspace, 'workspace', 'list'])
	1204	+ result.assert_success()

tests/integration/shell.py

---

...	...	@@ -353,3 +353,39 @@ def test_integration_devices(cli, tmpdir, datafiles):
353	353
354	354	result = execute_shell(cli, project, ["true"], element=element_name)
355	355	assert result.exit_code == 0
	356	+
	357	+
	358	+# Test that a shell can be opened from an external workspace
	359	+@pytest.mark.datafiles(DATA_DIR)
	360	+@pytest.mark.parametrize("build_shell", [("build"), ("nobuild")])
	361	+@pytest.mark.parametrize("guess_element", [("guess"), ("no-guess")])
	362	+@pytest.mark.skipif(IS_LINUX and not HAVE_BWRAP, reason='Only available with bubblewrap on Linux')
	363	+def test_integration_external_workspace(cli, tmpdir_factory, datafiles, build_shell, guess_element):
	364	+ tmpdir = tmpdir_factory.mktemp("")
	365	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	366	+ element_name = 'autotools/amhello.bst'
	367	+ workspace_dir = os.path.join(str(tmpdir), 'workspace')
	368	+
	369	+ if guess_element == "guess":
	370	+ # Mutate the project.conf to use a default shell command
	371	+ project_file = os.path.join(project, 'project.conf')
	372	+ config_text = "shell:\n"\
	373	+ " command: ['true']\n"
	374	+ with open(project_file, 'a') as f:
	375	+ f.write(config_text)
	376	+
	377	+ result = cli.run(project=project, args=[
	378	+ 'workspace', 'open', '--directory', workspace_dir, element_name
	379	+ ])
	380	+ result.assert_success()
	381	+
	382	+ result = cli.run(project=project, args=['-C', workspace_dir, 'build', element_name])
	383	+ result.assert_success()
	384	+
	385	+ command = ['-C', workspace_dir, 'shell']
	386	+ if build_shell == 'build':
	387	+ command.append('--build')
	388	+ if guess_element == "no-guess":
	389	+ command.extend([element_name, '--', 'true'])
	390	+ result = cli.run(project=project, cwd=workspace_dir, args=command)
	391	+ result.assert_success()

tests/sources/git.py

---

...	...	@@ -414,9 +414,18 @@ def test_submodule_track_no_ref_or_track(cli, tmpdir, datafiles):
414	414
415	415	@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
416	416	@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
417		-def test_ref_not_in_track_warn(cli, tmpdir, datafiles):
	417	+@pytest.mark.parametrize("fail", ['warn', 'error'])
	418	+def test_ref_not_in_track(cli, tmpdir, datafiles, fail):
418	419	project = os.path.join(datafiles.dirname, datafiles.basename)
419	420
	421	+ # Make the warning an error if we're testing errors
	422	+ if fail == 'error':
	423	+ project_template = {
	424	+ "name": "foo",
	425	+ "fatal-warnings": [CoreWarnings.REF_NOT_IN_TRACK]
	426	+ }
	427	+ _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	428	+
420	429	# Create the repo from 'repofiles', create a branch without latest commit
421	430	repo = create_repo('git', str(tmpdir))
422	431	ref = repo.create(os.path.join(project, 'repofiles'))
...	...	@@ -435,33 +444,184 @@ def test_ref_not_in_track_warn(cli, tmpdir, datafiles):
435	444	}
436	445	_yaml.dump(element, os.path.join(project, 'target.bst'))
437	446
438		- # Assert the warning is raised as ref is not in branch foo.
439		- # Assert warning not error to the user, when not set as fatal.
440	447	result = cli.run(project=project, args=['build', 'target.bst'])
441		- assert "The ref provided for the element does not exist locally" in result.stderr
	448	+
	449	+ # Assert a warning or an error depending on what we're checking
	450	+ if fail == 'error':
	451	+ result.assert_main_error(ErrorDomain.STREAM, None)
	452	+ result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.REF_NOT_IN_TRACK)
	453	+ else:
	454	+ result.assert_success()
	455	+ assert "ref-not-in-track" in result.stderr
442	456
443	457
444	458	@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
445	459	@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
446		-def test_ref_not_in_track_warn_error(cli, tmpdir, datafiles):
	460	+@pytest.mark.parametrize("fail", ['warn', 'error'])
	461	+def test_unlisted_submodule(cli, tmpdir, datafiles, fail):
447	462	project = os.path.join(datafiles.dirname, datafiles.basename)
448	463
449		- # Add fatal-warnings ref-not-in-track to project.conf
450		- project_template = {
451		- "name": "foo",
452		- "fatal-warnings": [CoreWarnings.REF_NOT_IN_TRACK]
	464	+ # Make the warning an error if we're testing errors
	465	+ if fail == 'error':
	466	+ project_template = {
	467	+ "name": "foo",
	468	+ "fatal-warnings": ['git:unlisted-submodule']
	469	+ }
	470	+ _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	471	+
	472	+ # Create the submodule first from the 'subrepofiles' subdir
	473	+ subrepo = create_repo('git', str(tmpdir), 'subrepo')
	474	+ subrepo.create(os.path.join(project, 'subrepofiles'))
	475	+
	476	+ # Create the repo from 'repofiles' subdir
	477	+ repo = create_repo('git', str(tmpdir))
	478	+ ref = repo.create(os.path.join(project, 'repofiles'))
	479	+
	480	+ # Add a submodule pointing to the one we created
	481	+ ref = repo.add_submodule('subdir', 'file://' + subrepo.repo)
	482	+
	483	+ # Create the source, and delete the explicit configuration
	484	+ # of the submodules.
	485	+ #
	486	+ # We expect this to cause an unlisted submodule warning
	487	+ # after the source has been fetched.
	488	+ #
	489	+ gitsource = repo.source_config(ref=ref)
	490	+ del gitsource['submodules']
	491	+
	492	+ # Write out our test target
	493	+ element = {
	494	+ 'kind': 'import',
	495	+ 'sources': [
	496	+ gitsource
	497	+ ]
453	498	}
	499	+ _yaml.dump(element, os.path.join(project, 'target.bst'))
454	500
455		- _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	501	+ # We will not see the warning or error before the first fetch, because
	502	+ # we don't have the repository yet and so we have no knowledge of
	503	+ # the unlisted submodule.
	504	+ result = cli.run(project=project, args=['show', 'target.bst'])
	505	+ result.assert_success()
	506	+ assert "git:unlisted-submodule" not in result.stderr
456	507
457		- # Create the repo from 'repofiles', create a branch without latest commit
	508	+ # We will notice this directly in fetch, as it will try to fetch
	509	+ # the submodules it discovers as a result of fetching the primary repo.
	510	+ result = cli.run(project=project, args=['fetch', 'target.bst'])
	511	+
	512	+ # Assert a warning or an error depending on what we're checking
	513	+ if fail == 'error':
	514	+ result.assert_main_error(ErrorDomain.STREAM, None)
	515	+ result.assert_task_error(ErrorDomain.PLUGIN, 'git:unlisted-submodule')
	516	+ else:
	517	+ result.assert_success()
	518	+ assert "git:unlisted-submodule" in result.stderr
	519	+
	520	+ # Now that we've fetched it, `bst show` will discover the unlisted submodule too
	521	+ result = cli.run(project=project, args=['show', 'target.bst'])
	522	+
	523	+ # Assert a warning or an error depending on what we're checking
	524	+ if fail == 'error':
	525	+ result.assert_main_error(ErrorDomain.PLUGIN, 'git:unlisted-submodule')
	526	+ else:
	527	+ result.assert_success()
	528	+ assert "git:unlisted-submodule" in result.stderr
	529	+
	530	+
	531	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	532	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	533	+@pytest.mark.parametrize("fail", ['warn', 'error'])
	534	+def test_track_unlisted_submodule(cli, tmpdir, datafiles, fail):
	535	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	536	+
	537	+ # Make the warning an error if we're testing errors
	538	+ if fail == 'error':
	539	+ project_template = {
	540	+ "name": "foo",
	541	+ "fatal-warnings": ['git:unlisted-submodule']
	542	+ }
	543	+ _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	544	+
	545	+ # Create the submodule first from the 'subrepofiles' subdir
	546	+ subrepo = create_repo('git', str(tmpdir), 'subrepo')
	547	+ subrepo.create(os.path.join(project, 'subrepofiles'))
	548	+
	549	+ # Create the repo from 'repofiles' subdir
458	550	repo = create_repo('git', str(tmpdir))
459	551	ref = repo.create(os.path.join(project, 'repofiles'))
460	552
	553	+ # Add a submodule pointing to the one we created, but use
	554	+ # the original ref, let the submodules appear after tracking
	555	+ repo.add_submodule('subdir', 'file://' + subrepo.repo)
	556	+
	557	+ # Create the source, and delete the explicit configuration
	558	+ # of the submodules.
461	559	gitsource = repo.source_config(ref=ref)
	560	+ del gitsource['submodules']
462	561
463		- # Overwrite the track value to the added branch
464		- gitsource['track'] = 'foo'
	562	+ # Write out our test target
	563	+ element = {
	564	+ 'kind': 'import',
	565	+ 'sources': [
	566	+ gitsource
	567	+ ]
	568	+ }
	569	+ _yaml.dump(element, os.path.join(project, 'target.bst'))
	570	+
	571	+ # Fetch the repo, we will not see the warning because we
	572	+ # are still pointing to a ref which predates the submodules
	573	+ result = cli.run(project=project, args=['fetch', 'target.bst'])
	574	+ result.assert_success()
	575	+ assert "git:unlisted-submodule" not in result.stderr
	576	+
	577	+ # We won't get a warning/error when tracking either, the source
	578	+ # has not become Consistency.CACHED so the opportunity to check
	579	+ # for the warning has not yet arisen.
	580	+ result = cli.run(project=project, args=['track', 'target.bst'])
	581	+ result.assert_success()
	582	+ assert "git:unlisted-submodule" not in result.stderr
	583	+
	584	+ # Fetching the repo at the new ref will finally reveal the warning
	585	+ result = cli.run(project=project, args=['fetch', 'target.bst'])
	586	+ if fail == 'error':
	587	+ result.assert_main_error(ErrorDomain.STREAM, None)
	588	+ result.assert_task_error(ErrorDomain.PLUGIN, 'git:unlisted-submodule')
	589	+ else:
	590	+ result.assert_success()
	591	+ assert "git:unlisted-submodule" in result.stderr
	592	+
	593	+
	594	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	595	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	596	+@pytest.mark.parametrize("fail", ['warn', 'error'])
	597	+def test_invalid_submodule(cli, tmpdir, datafiles, fail):
	598	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	599	+
	600	+ # Make the warning an error if we're testing errors
	601	+ if fail == 'error':
	602	+ project_template = {
	603	+ "name": "foo",
	604	+ "fatal-warnings": ['git:invalid-submodule']
	605	+ }
	606	+ _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	607	+
	608	+ # Create the repo from 'repofiles' subdir
	609	+ repo = create_repo('git', str(tmpdir))
	610	+ ref = repo.create(os.path.join(project, 'repofiles'))
	611	+
	612	+ # Create the source without any submodules, and add
	613	+ # an invalid submodule configuration to it.
	614	+ #
	615	+ # We expect this to cause an invalid submodule warning
	616	+ # after the source has been fetched and we know what
	617	+ # the real submodules actually are.
	618	+ #
	619	+ gitsource = repo.source_config(ref=ref)
	620	+ gitsource['submodules'] = {
	621	+ 'subdir': {
	622	+ 'url': 'https://pony.org/repo.git'
	623	+ }
	624	+ }
465	625
466	626	# Write out our test target
467	627	element = {
...	...	@@ -472,11 +632,95 @@ def test_ref_not_in_track_warn_error(cli, tmpdir, datafiles):
472	632	}
473	633	_yaml.dump(element, os.path.join(project, 'target.bst'))
474	634
475		- # Assert that build raises a warning here that is captured
476		- # as plugin error, due to the fatal warning being set
477		- result = cli.run(project=project, args=['build', 'target.bst'])
478		- result.assert_main_error(ErrorDomain.STREAM, None)
479		- result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.REF_NOT_IN_TRACK)
	635	+ # We will not see the warning or error before the first fetch, because
	636	+ # we don't have the repository yet and so we have no knowledge of
	637	+ # the unlisted submodule.
	638	+ result = cli.run(project=project, args=['show', 'target.bst'])
	639	+ result.assert_success()
	640	+ assert "git:invalid-submodule" not in result.stderr
	641	+
	642	+ # We will notice this directly in fetch, as it will try to fetch
	643	+ # the submodules it discovers as a result of fetching the primary repo.
	644	+ result = cli.run(project=project, args=['fetch', 'target.bst'])
	645	+
	646	+ # Assert a warning or an error depending on what we're checking
	647	+ if fail == 'error':
	648	+ result.assert_main_error(ErrorDomain.STREAM, None)
	649	+ result.assert_task_error(ErrorDomain.PLUGIN, 'git:invalid-submodule')
	650	+ else:
	651	+ result.assert_success()
	652	+ assert "git:invalid-submodule" in result.stderr
	653	+
	654	+ # Now that we've fetched it, `bst show` will discover the unlisted submodule too
	655	+ result = cli.run(project=project, args=['show', 'target.bst'])
	656	+
	657	+ # Assert a warning or an error depending on what we're checking
	658	+ if fail == 'error':
	659	+ result.assert_main_error(ErrorDomain.PLUGIN, 'git:invalid-submodule')
	660	+ else:
	661	+ result.assert_success()
	662	+ assert "git:invalid-submodule" in result.stderr
	663	+
	664	+
	665	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	666	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	667	+@pytest.mark.parametrize("fail", ['warn', 'error'])
	668	+def test_track_invalid_submodule(cli, tmpdir, datafiles, fail):
	669	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	670	+
	671	+ # Make the warning an error if we're testing errors
	672	+ if fail == 'error':
	673	+ project_template = {
	674	+ "name": "foo",
	675	+ "fatal-warnings": ['git:invalid-submodule']
	676	+ }
	677	+ _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	678	+
	679	+ # Create the submodule first from the 'subrepofiles' subdir
	680	+ subrepo = create_repo('git', str(tmpdir), 'subrepo')
	681	+ subrepo.create(os.path.join(project, 'subrepofiles'))
	682	+
	683	+ # Create the repo from 'repofiles' subdir
	684	+ repo = create_repo('git', str(tmpdir))
	685	+ ref = repo.create(os.path.join(project, 'repofiles'))
	686	+
	687	+ # Add a submodule pointing to the one we created
	688	+ ref = repo.add_submodule('subdir', 'file://' + subrepo.repo)
	689	+
	690	+ # Add a commit beyond the ref which *removes* the submodule we've added
	691	+ repo.remove_path('subdir')
	692	+
	693	+ # Create the source, this will keep the submodules so initially
	694	+ # the configuration is valid for the ref we're using
	695	+ gitsource = repo.source_config(ref=ref)
	696	+
	697	+ # Write out our test target
	698	+ element = {
	699	+ 'kind': 'import',
	700	+ 'sources': [
	701	+ gitsource
	702	+ ]
	703	+ }
	704	+ _yaml.dump(element, os.path.join(project, 'target.bst'))
	705	+
	706	+ # Fetch the repo, we will not see the warning because we
	707	+ # are still pointing to a ref which predates the submodules
	708	+ result = cli.run(project=project, args=['fetch', 'target.bst'])
	709	+ result.assert_success()
	710	+ assert "git:invalid-submodule" not in result.stderr
	711	+
	712	+ # In this case, we will get the error directly after tracking,
	713	+ # since the new HEAD does not require any submodules which are
	714	+ # not locally cached, the Source will be CACHED directly after
	715	+ # tracking and the validations will occur as a result.
	716	+ #
	717	+ result = cli.run(project=project, args=['track', 'target.bst'])
	718	+ if fail == 'error':
	719	+ result.assert_main_error(ErrorDomain.STREAM, None)
	720	+ result.assert_task_error(ErrorDomain.PLUGIN, 'git:invalid-submodule')
	721	+ else:
	722	+ result.assert_success()
	723	+ assert "git:invalid-submodule" in result.stderr
480	724
481	725
482	726	@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")

tests/testutils/repo/git.py

---

...	...	@@ -76,6 +76,12 @@ class Git(Repo):
76	76	self._run_git('commit', '-m', 'Added the submodule')
77	77	return self.latest_commit()
78	78
	79	+ # This can also be used to a file or a submodule
	80	+ def remove_path(self, path):
	81	+ self._run_git('rm', path)
	82	+ self._run_git('commit', '-m', 'Removing {}'.format(path))
	83	+ return self.latest_commit()
	84	+
79	85	def source_config(self, ref=None, checkout_submodules=None):
80	86	config = {
81	87	'kind': 'git',

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.