[Notes] [Git][BuildStream/buildstream][phil/source-checkout-options] 12 commits: _stream.py: Fix ugly error when opening a workspace using a relative path

[Phil Dawson <gitlab mg gitlab com>]

Title: GitLab

Phil Dawson pushed to branch phil/source-checkout-options at BuildStream / buildstream

Commits:

• e564251b
  by Jonathan Maw at 2018-12-11T16:15:19Z

  _stream.py: Fix ugly error when opening a workspace using a relative path
  

• 7cf83343
  by Jonathan Maw at 2018-12-11T16:15:19Z

  _context.py: Store the directory buildstream was invoked on
  

• a1dee91e
  by Jonathan Maw at 2018-12-11T16:15:19Z

  Make specifying elements optional in bst commands
  
  Known issues:
  * `bst shell` works, but `bst shell COMMANDS...` doesn't, because click
    has no way of separating optional args from variable-length args.
  * `bst checkout` and `bst source-checkout`'s usage strings mark LOCATION
    as an optional argument. Because click gets confused if there's an
    optional argument before a mandatory argument, I had to mark LOCATION
    as optional internally.
  * `bst workspace open` makes no sense with element being optional, so
    I skipped it.
  * `bst workspace close` will probably need to be revisited when multiple
    projects can own one workspace.
  * `bst workspace reset` will happily delete the directory you're
    currently in, requiring you to `cd $PWD` to see the contents of your
    directory.
    I could exclude the top-level directory of the workspace being
    deleted, but it is entirely valid to run workspace commands from deeper
    in the workspace.
  
  This is a part of #222
  

• 9b192015
  by Jonathan Maw at 2018-12-11T16:15:19Z

  tests: Add tests for guessing element names
  
  This is a part of #222
  

• e340371f
  by Jonathan Maw at 2018-12-11T16:15:19Z

  Add NEWS for guessing the element in commands
  
  This is a part of #222
  

• bbf81c3d
  by Jonathan Maw at 2018-12-11T16:15:19Z

  doc: Point out that element is optional when commands are run from the workspace
  
  This is a part of #222
  

• a5a53ddd
  by Jonathan Maw at 2018-12-12T13:44:09Z

  Merge branch 'jonathan/workspace-fragment-guess-element' into 'master'
  
  Guess the element when running commands in a workspace
  
  See merge request BuildStream/buildstream!954

• d55b9e39
  by Phil Dawson at 2018-12-12T13:55:19Z

  Add --tar option to source-checkout command
  
  This commit is part of the work towards #672
  

• 733aab53
  by Phil Dawson at 2018-12-12T13:55:19Z

  Add --force / -f option to source-checkout command
  

• c2efeba0
  by Phil Dawson at 2018-12-12T14:43:40Z

  Add option to source-checkout command to generate build scripts
  

• 85c61894
  by Phil Dawson at 2018-12-12T14:43:40Z

  Remove source bundle command
  
  This is part of the work towards #672
  

• a322d5c0
  by Phil Dawson at 2018-12-12T14:43:40Z

  NEWS: Add entry about the removal of source-bundle command
  

15 changed files:

• NEWS
• buildstream/_context.py
• buildstream/_frontend/app.py
• buildstream/_frontend/cli.py
• buildstream/_stream.py
• doc/sessions/developing.run
• doc/source/developing/workspaces.rst
• + doc/source/sessions-stored/developing-build-after-changes-workspace.html
• doc/source/using_commands.rst
• tests/completions/completions.py
• tests/frontend/help.py
• − tests/frontend/source_bundle.py
• tests/frontend/source_checkout.py
• tests/frontend/workspace.py
• tests/integration/shell.py

Changes:

NEWS

---

...	...	@@ -2,6 +2,12 @@
2	2	buildstream 1.3.1
3	3	=================
4	4
	5	+ o BREAKING CHANGE: The bst source-bundle command has been removed. The
	6	+ functionality it provided has been replaced by the `--include-build-scripts`
	7	+ option of the `bst source-checkout` command. To produce a tarball containing
	8	+ an element's sources and generated build scripts you can do the command
	9	+ `bst source-checkout --include-build-scripts --tar foo.bst some-file.tar`
	10	+
5	11	o BREAKING CHANGE: Default strip-commands have been removed as they are too
6	12	specific. Recommendation if you are building in Linux is to use the
7	13	ones being used in freedesktop-sdk project, for example
...	...	@@ -86,6 +92,11 @@ buildstream 1.3.1
86	92	o Opening a workspace now creates a .bstproject.yaml file that allows buildstream
87	93	commands to be run from a workspace that is not inside a project.
88	94
	95	+ o Specifying an element is now optional for some commands when buildstream is run
	96	+ from inside a workspace - the 'build', 'checkout', 'fetch', 'pull', 'push',
	97	+ 'shell', 'show', 'source-checkout', 'track', 'workspace close' and 'workspace reset'
	98	+ commands are affected.
	99	+
89	100
90	101	=================
91	102	buildstream 1.1.5

buildstream/_context.py

---

...	...	@@ -32,7 +32,7 @@ from ._message import Message, MessageType
32	32	from ._profile import Topics, profile_start, profile_end
33	33	from ._artifactcache import ArtifactCache
34	34	from ._artifactcache.cascache import CASCache
35		-from ._workspaces import Workspaces, WorkspaceProjectCache
	35	+from ._workspaces import Workspaces, WorkspaceProjectCache, WORKSPACE_PROJECT_FILE
36	36	from .plugin import _plugin_lookup
37	37
38	38
...	...	@@ -47,9 +47,12 @@ from .plugin import _plugin_lookup
47	47	# verbosity levels and basically anything pertaining to the context
48	48	# in which BuildStream was invoked.
49	49	#
	50	+# Args:
	51	+# directory (str): The directory that buildstream was invoked in
	52	+#
50	53	class Context():
51	54
52		- def __init__(self):
	55	+ def __init__(self, directory=None):
53	56
54	57	# Filename indicating which configuration file was used, or None for the defaults
55	58	self.config_origin = None
...	...	@@ -148,6 +151,7 @@ class Context():
148	151	self._log_handle = None
149	152	self._log_filename = None
150	153	self._cascache = None
	154	+ self._directory = directory
151	155
152	156	# load()
153	157	#
...	...	@@ -645,6 +649,20 @@ class Context():
645	649	self._cascache = CASCache(self.artifactdir)
646	650	return self._cascache
647	651
	652	+ # guess_element()
	653	+ #
	654	+ # Attempts to interpret which element the user intended to run commands on
	655	+ #
	656	+ # Returns:
	657	+ # (str) The name of the element, or None if no element can be guessed
	658	+ def guess_element(self):
	659	+ workspace_project_dir, _ = utils._search_upward_for_files(self._directory, [WORKSPACE_PROJECT_FILE])
	660	+ if workspace_project_dir:
	661	+ workspace_project = self._workspace_project_cache.get(workspace_project_dir)
	662	+ return workspace_project.get_default_element()
	663	+ else:
	664	+ return None
	665	+
648	666
649	667	# _node_get_option_str()
650	668	#

buildstream/_frontend/app.py

---

...	...	@@ -164,7 +164,7 @@ class App():
164	164	# Load the Context
165	165	#
166	166	try:
167		- self.context = Context()
	167	+ self.context = Context(directory)
168	168	self.context.load(config)
169	169	except BstError as e:
170	170	self._error_exit(e, "Error loading user configuration")

buildstream/_frontend/cli.py

---

...	...	@@ -316,10 +316,15 @@ def build(app, elements, all_, track_, track_save, track_all, track_except, trac
316	316	if track_save:
317	317	click.echo("WARNING: --track-save is deprecated, saving is now unconditional", err=True)
318	318
319		- if track_all:
320		- track_ = elements
321		-
322	319	with app.initialized(session_name="Build"):
	320	+ if not all_ and not elements:
	321	+ guessed_target = app.context.guess_element()
	322	+ if guessed_target:
	323	+ elements = (guessed_target,)
	324	+
	325	+ if track_all:
	326	+ track_ = elements
	327	+
323	328	app.stream.build(elements,
324	329	track_targets=track_,
325	330	track_except=track_except,
...	...	@@ -371,6 +376,11 @@ def fetch(app, elements, deps, track_, except_, track_cross_junctions):
371	376	deps = PipelineSelection.ALL
372	377
373	378	with app.initialized(session_name="Fetch"):
	379	+ if not elements:
	380	+ guessed_target = app.context.guess_element()
	381	+ if guessed_target:
	382	+ elements = (guessed_target,)
	383	+
374	384	app.stream.fetch(elements,
375	385	selection=deps,
376	386	except_targets=except_,
...	...	@@ -407,6 +417,11 @@ def track(app, elements, deps, except_, cross_junctions):
407	417	all: All dependencies of all specified elements
408	418	"""
409	419	with app.initialized(session_name="Track"):
	420	+ if not elements:
	421	+ guessed_target = app.context.guess_element()
	422	+ if guessed_target:
	423	+ elements = (guessed_target,)
	424	+
410	425	# Substitute 'none' for 'redirect' so that element redirections
411	426	# will be done
412	427	if deps == 'none':
...	...	@@ -442,7 +457,13 @@ def pull(app, elements, deps, remote):
442	457	none: No dependencies, just the element itself
443	458	all: All dependencies
444	459	"""
	460	+
445	461	with app.initialized(session_name="Pull"):
	462	+ if not elements:
	463	+ guessed_target = app.context.guess_element()
	464	+ if guessed_target:
	465	+ elements = (guessed_target,)
	466	+
446	467	app.stream.pull(elements, selection=deps, remote=remote)
447	468
448	469
...	...	@@ -475,6 +496,11 @@ def push(app, elements, deps, remote):
475	496	all: All dependencies
476	497	"""
477	498	with app.initialized(session_name="Push"):
	499	+ if not elements:
	500	+ guessed_target = app.context.guess_element()
	501	+ if guessed_target:
	502	+ elements = (guessed_target,)
	503	+
478	504	app.stream.push(elements, selection=deps, remote=remote)
479	505
480	506
...	...	@@ -545,6 +571,11 @@ def show(app, elements, deps, except_, order, format_):
545	571	$'---------- %{name} ----------\\n%{vars}'
546	572	"""
547	573	with app.initialized():
	574	+ if not elements:
	575	+ guessed_target = app.context.guess_element()
	576	+ if guessed_target:
	577	+ elements = (guessed_target,)
	578	+
548	579	dependencies = app.stream.load_selection(elements,
549	580	selection=deps,
550	581	except_targets=except_)
...	...	@@ -573,7 +604,7 @@ def show(app, elements, deps, except_, order, format_):
573	604	help="Mount a file or directory into the sandbox")
574	605	@click.option('--isolate', is_flag=True, default=False,
575	606	help='Create an isolated build sandbox')
576		-@click.argument('element',
	607	+@click.argument('element', required=False,
577	608	type=click.Path(readable=False))
578	609	@click.argument('command', type=click.STRING, nargs=-1)
579	610	@click.pass_obj
...	...	@@ -604,6 +635,11 @@ def shell(app, element, sysroot, mount, isolate, build_, command):
604	635	scope = Scope.RUN
605	636
606	637	with app.initialized():
	638	+ if not element:
	639	+ element = app.context.guess_element()
	640	+ if not element:
	641	+ raise AppError('Missing argument "ELEMENT".')
	642	+
607	643	dependencies = app.stream.load_selection((element,), selection=PipelineSelection.NONE)
608	644	element = dependencies[0]
609	645	prompt = app.shell_prompt(element)
...	...	@@ -641,15 +677,24 @@ def shell(app, element, sysroot, mount, isolate, build_, command):
641	677	help="Create a tarball from the artifact contents instead "
642	678	"of a file tree. If LOCATION is '-', the tarball "
643	679	"will be dumped to the standard output.")
644		-@click.argument('element',
	680	+@click.argument('element', required=False,
645	681	type=click.Path(readable=False))
646		-@click.argument('location', type=click.Path())
	682	+@click.argument('location', type=click.Path(), required=False)
647	683	@click.pass_obj
648	684	def checkout(app, element, location, force, deps, integrate, hardlinks, tar):
649	685	"""Checkout a built artifact to the specified location
650	686	"""
651	687	from ..element import Scope
652	688
	689	+ if not element and not location:
	690	+ click.echo("ERROR: LOCATION is not specified", err=True)
	691	+ sys.exit(-1)
	692	+
	693	+ if element and not location:
	694	+ # Nasty hack to get around click's optional args
	695	+ location = element
	696	+ element = None
	697	+
653	698	if hardlinks and tar:
654	699	click.echo("ERROR: options --hardlinks and --tar conflict", err=True)
655	700	sys.exit(-1)
...	...	@@ -662,6 +707,11 @@ def checkout(app, element, location, force, deps, integrate, hardlinks, tar):
662	707	scope = Scope.NONE
663	708
664	709	with app.initialized():
	710	+ if not element:
	711	+ element = app.context.guess_element()
	712	+ if not element:
	713	+ raise AppError('Missing argument "ELEMENT".')
	714	+
665	715	app.stream.checkout(element,
666	716	location=location,
667	717	force=force,
...	...	@@ -675,6 +725,8 @@ def checkout(app, element, location, force, deps, integrate, hardlinks, tar):
675	725	# Source Checkout Command #
676	726	##################################################################
677	727	@cli.command(name='source-checkout', short_help='Checkout sources for an element')
	728	+@click.option('--force', '-f', default=False, is_flag=True,
	729	+ help="Allow files to be overwritten")
678	730	@click.option('--except', 'except_', multiple=True,
679	731	type=click.Path(readable=False),
680	732	help="Except certain dependencies")
...	...	@@ -683,19 +735,40 @@ def checkout(app, element, location, force, deps, integrate, hardlinks, tar):
683	735	help='The dependencies whose sources to checkout (default: none)')
684	736	@click.option('--fetch', 'fetch_', default=False, is_flag=True,
685	737	help='Fetch elements if they are not fetched')
686		-@click.argument('element',
687		- type=click.Path(readable=False))
688		-@click.argument('location', type=click.Path())
	738	+@click.option('--tar', 'tar', default=False, is_flag=True,
	739	+ help='Create a tarball from the element\'s sources instead of a '
	740	+ 'file tree.')
	741	+@click.option('--include-build-scripts', 'build_scripts', is_flag=True)
	742	+@click.argument('element', required=False, type=click.Path(readable=False))
	743	+@click.argument('location', type=click.Path(), required=False)
689	744	@click.pass_obj
690		-def source_checkout(app, element, location, deps, fetch_, except_):
	745	+def source_checkout(app, element, location, force, deps, fetch_, except_,
	746	+ tar, build_scripts):
691	747	"""Checkout sources of an element to the specified location
692	748	"""
	749	+ if not element and not location:
	750	+ click.echo("ERROR: LOCATION is not specified", err=True)
	751	+ sys.exit(-1)
	752	+
	753	+ if element and not location:
	754	+ # Nasty hack to get around click's optional args
	755	+ location = element
	756	+ element = None
	757	+
693	758	with app.initialized():
	759	+ if not element:
	760	+ element = app.context.guess_element()
	761	+ if not element:
	762	+ raise AppError('Missing argument "ELEMENT".')
	763	+
694	764	app.stream.source_checkout(element,
695	765	location=location,
	766	+ force=force,
696	767	deps=deps,
697	768	fetch=fetch_,
698		- except_targets=except_)
	769	+ except_targets=except_,
	770	+ tar=tar,
	771	+ include_build_scripts=build_scripts)
699	772
700	773
701	774	##################################################################
...	...	@@ -747,11 +820,15 @@ def workspace_open(app, no_checkout, force, track_, directory, elements):
747	820	def workspace_close(app, remove_dir, all_, elements):
748	821	"""Close a workspace"""
749	822
750		- if not (all_ or elements):
751		- click.echo('ERROR: no elements specified', err=True)
752		- sys.exit(-1)
753		-
754	823	with app.initialized():
	824	+ if not (all_ or elements):
	825	+ # NOTE: I may need to revisit this when implementing multiple projects
	826	+ # opening one workspace.
	827	+ element = app.context.guess_element()
	828	+ if element:
	829	+ elements = (element,)
	830	+ else:
	831	+ raise AppError('No elements specified')
755	832
756	833	# Early exit if we specified `all` and there are no workspaces
757	834	if all_ and not app.stream.workspace_exists():
...	...	@@ -808,7 +885,11 @@ def workspace_reset(app, soft, track_, all_, elements):
808	885	with app.initialized():
809	886
810	887	if not (all_ or elements):
811		- raise AppError('No elements specified to reset')
	888	+ element = app.context.guess_element()
	889	+ if element:
	890	+ elements = (element,)
	891	+ else:
	892	+ raise AppError('No elements specified to reset')
812	893
813	894	if all_ and not app.stream.workspace_exists():
814	895	raise AppError("No open workspaces to reset")
...	...	@@ -834,34 +915,3 @@ def workspace_list(app):
834	915
835	916	with app.initialized():
836	917	app.stream.workspace_list()
837		-
838		-
839		-##################################################################
840		-# Source Bundle Command #
841		-##################################################################
842		-@cli.command(name="source-bundle", short_help="Produce a build bundle to be manually executed")
843		-@click.option('--except', 'except_', multiple=True,
844		- type=click.Path(readable=False),
845		- help="Elements to except from the tarball")
846		-@click.option('--compression', default='gz',
847		- type=click.Choice(['none', 'gz', 'bz2', 'xz']),
848		- help="Compress the tar file using the given algorithm.")
849		-@click.option('--track', 'track_', default=False, is_flag=True,
850		- help="Track new source references before bundling")
851		-@click.option('--force', '-f', default=False, is_flag=True,
852		- help="Overwrite an existing tarball")
853		-@click.option('--directory', default=os.getcwd(),
854		- help="The directory to write the tarball to")
855		-@click.argument('element',
856		- type=click.Path(readable=False))
857		-@click.pass_obj
858		-def source_bundle(app, element, force, directory,
859		- track_, compression, except_):
860		- """Produce a source bundle to be manually executed
861		- """
862		- with app.initialized():
863		- app.stream.source_bundle(element, directory,
864		- track_first=track_,
865		- force=force,
866		- compression=compression,
867		- except_targets=except_)

buildstream/_stream.py

---

...	...	@@ -25,8 +25,8 @@ import stat
25	25	import shlex
26	26	import shutil
27	27	import tarfile
28		-from contextlib import contextmanager
29		-from tempfile import TemporaryDirectory
	28	+import tempfile
	29	+from contextlib import contextmanager, suppress
30	30
31	31	from ._exceptions import StreamError, ImplError, BstError, set_last_task_error
32	32	from ._message import Message, MessageType
...	...	@@ -449,11 +449,14 @@ class Stream():
449	449	#
450	450	def source_checkout(self, target, *,
451	451	location=None,
	452	+ force=False,
452	453	deps='none',
453	454	fetch=False,
454		- except_targets=()):
	455	+ except_targets=(),
	456	+ tar=False,
	457	+ include_build_scripts=False):
455	458
456		- self._check_location_writable(location)
	459	+ self._check_location_writable(location, force=force, tar=tar)
457	460
458	461	elements, _ = self._load((target,), (),
459	462	selection=deps,
...	...	@@ -467,7 +470,8 @@ class Stream():
467	470
468	471	# Stage all sources determined by scope
469	472	try:
470		- self._write_element_sources(location, elements)
	473	+ self._source_checkout(elements, location, force, deps,
	474	+ fetch, tar, include_build_scripts)
471	475	except BstError as e:
472	476	raise StreamError("Error while writing sources"
473	477	": '{}'".format(e), detail=e.detail, reason=e.reason) from e
...	...	@@ -544,7 +548,8 @@ class Stream():
544	548	if len(elements) != 1:
545	549	raise StreamError("Exactly one element can be given if --directory is used",
546	550	reason='directory-with-multiple-elements')
547		- expanded_directories = [custom_dir, ]
	551	+ directory = os.path.abspath(custom_dir)
	552	+ expanded_directories = [directory, ]
548	553	else:
549	554	# If this fails it is a bug in what ever calls this, usually cli.py and so can not be tested for via the
550	555	# run bst test mechanism.
...	...	@@ -727,87 +732,6 @@ class Stream():
727	732	'workspaces': workspaces
728	733	})
729	734
730		- # source_bundle()
731		- #
732		- # Create a host buildable tarball bundle for the given target.
733		- #
734		- # Args:
735		- # target (str): The target element to bundle
736		- # directory (str): The directory to output the tarball
737		- # track_first (bool): Track new source references before bundling
738		- # compression (str): The compression type to use
739		- # force (bool): Overwrite an existing tarball
740		- #
741		- def source_bundle(self, target, directory, *,
742		- track_first=False,
743		- force=False,
744		- compression="gz",
745		- except_targets=()):
746		-
747		- if track_first:
748		- track_targets = (target,)
749		- else:
750		- track_targets = ()
751		-
752		- elements, track_elements = self._load((target,), track_targets,
753		- selection=PipelineSelection.ALL,
754		- except_targets=except_targets,
755		- track_selection=PipelineSelection.ALL,
756		- fetch_subprojects=True)
757		-
758		- # source-bundle only supports one target
759		- target = self.targets[0]
760		-
761		- self._message(MessageType.INFO, "Bundling sources for target {}".format(target.name))
762		-
763		- # Find the correct filename for the compression algorithm
764		- tar_location = os.path.join(directory, target.normal_name + ".tar")
765		- if compression != "none":
766		- tar_location += "." + compression
767		-
768		- # Attempt writing a file to generate a good error message
769		- # early
770		- #
771		- # FIXME: A bit hackish
772		- try:
773		- open(tar_location, mode="x")
774		- os.remove(tar_location)
775		- except IOError as e:
776		- raise StreamError("Cannot write to {0}: {1}"
777		- .format(tar_location, e)) from e
778		-
779		- # Fetch and possibly track first
780		- #
781		- self._fetch(elements, track_elements=track_elements)
782		-
783		- # We don't use the scheduler for this as it is almost entirely IO
784		- # bound.
785		-
786		- # Create a temporary directory to build the source tree in
787		- builddir = self._context.builddir
788		- os.makedirs(builddir, exist_ok=True)
789		- prefix = "{}-".format(target.normal_name)
790		-
791		- with TemporaryDirectory(prefix=prefix, dir=builddir) as tempdir:
792		- source_directory = os.path.join(tempdir, 'source')
793		- try:
794		- os.makedirs(source_directory)
795		- except OSError as e:
796		- raise StreamError("Failed to create directory: {}"
797		- .format(e)) from e
798		-
799		- # Any elements that don't implement _write_script
800		- # should not be included in the later stages.
801		- elements = [
802		- element for element in elements
803		- if self._write_element_script(source_directory, element)
804		- ]
805		-
806		- self._write_element_sources(os.path.join(tempdir, "source"), elements)
807		- self._write_build_script(tempdir, elements)
808		- self._collect_sources(tempdir, tar_location,
809		- target.normal_name, compression)
810		-
811	735	# redirect_element_names()
812	736	#
813	737	# Takes a list of element names and returns a list where elements have been
...	...	@@ -1188,6 +1112,54 @@ class Stream():
1188	1112
1189	1113	sandbox_vroot.export_files(directory, can_link=True, can_destroy=True)
1190	1114
	1115	+ # Helper function for source_checkout()
	1116	+ def _source_checkout(self, elements,
	1117	+ location=None,
	1118	+ force=False,
	1119	+ deps='none',
	1120	+ fetch=False,
	1121	+ tar=False,
	1122	+ include_build_scripts=False):
	1123	+ location = os.path.abspath(location)
	1124	+ location_parent = os.path.abspath(os.path.join(location, ".."))
	1125	+
	1126	+ # Stage all our sources in a temporary directory. The this
	1127	+ # directory can be used to either construct a tarball or moved
	1128	+ # to the final desired location.
	1129	+ temp_source_dir = tempfile.TemporaryDirectory(dir=location_parent)
	1130	+ try:
	1131	+ self._write_element_sources(temp_source_dir.name, elements)
	1132	+ if include_build_scripts:
	1133	+ self._write_build_scripts(temp_source_dir.name, elements)
	1134	+ if tar:
	1135	+ self._create_tarball(temp_source_dir.name, location)
	1136	+ else:
	1137	+ self._move_directory(temp_source_dir.name, location, force)
	1138	+ except OSError as e:
	1139	+ raise StreamError("Failed to checkout sources to {}: {}"
	1140	+ .format(location, e)) from e
	1141	+ finally:
	1142	+ with suppress(FileNotFoundError):
	1143	+ temp_source_dir.cleanup()
	1144	+
	1145	+ # Move a directory src to dest. This will work across devices and
	1146	+ # may optionaly overwrite existing files.
	1147	+ def _move_directory(self, src, dest, force=False):
	1148	+ def is_empty_dir(path):
	1149	+ return os.path.isdir(dest) and not os.listdir(dest)
	1150	+
	1151	+ try:
	1152	+ os.rename(src, dest)
	1153	+ return
	1154	+ except OSError:
	1155	+ pass
	1156	+
	1157	+ if force or is_empty_dir(dest):
	1158	+ try:
	1159	+ utils.link_files(src, dest)
	1160	+ except utils.UtilError as e:
	1161	+ raise StreamError("Failed to move directory: {}".format(e)) from e
	1162	+
1191	1163	# Write the element build script to the given directory
1192	1164	def _write_element_script(self, directory, element):
1193	1165	try:
...	...	@@ -1204,8 +1176,28 @@ class Stream():
1204	1176	os.makedirs(element_source_dir)
1205	1177	element._stage_sources_at(element_source_dir, mount_workspaces=False)
1206	1178
	1179	+ # Create a tarball from the content of directory
	1180	+ def _create_tarball(self, directory, tar_name):
	1181	+ try:
	1182	+ with utils.save_file_atomic(tar_name, mode='wb') as f:
	1183	+ # This TarFile does not need to be explicitly closed
	1184	+ # as the underlying file object will be closed be the
	1185	+ # save_file_atomic contect manager
	1186	+ tarball = tarfile.open(fileobj=f, mode='w')
	1187	+ for item in os.listdir(str(directory)):
	1188	+ file_to_add = os.path.join(directory, item)
	1189	+ tarball.add(file_to_add, arcname=item)
	1190	+ except OSError as e:
	1191	+ raise StreamError("Failed to create tar archive: {}".format(e)) from e
	1192	+
	1193	+ # Write all the build_scripts for elements in the directory location
	1194	+ def _write_build_scripts(self, location, elements):
	1195	+ for element in elements:
	1196	+ self._write_element_script(location, element)
	1197	+ self._write_master_build_script(location, elements)
	1198	+
1207	1199	# Write a master build script to the sandbox
1208		- def _write_build_script(self, directory, elements):
	1200	+ def _write_master_build_script(self, directory, elements):
1209	1201
1210	1202	module_string = ""
1211	1203	for element in elements:

doc/sessions/developing.run

---

...	...	@@ -24,6 +24,11 @@ commands:
24	24	output: ../source/sessions/developing-build-after-changes.html
25	25	command: build hello.bst
26	26
	27	+# Rebuild, from the workspace
	28	+- directory: ../examples/developing/workspace_hello
	29	+ output: ../source/sessions/developing-build-after-changes-workspace.html
	30	+ command: build
	31	+
27	32	# Capture shell output with changes
28	33	- directory: ../examples/developing/
29	34	output: ../source/sessions/developing-shell-after-changes.html

doc/source/developing/workspaces.rst

---

...	...	@@ -50,11 +50,16 @@ We can open workspace_hello/hello.c and make the following change:
50	50	.. literalinclude:: ../../examples/developing/update.patch
51	51	:language: diff
52	52
53		-Now, rebuild the hello.bst element
	53	+Now, rebuild the hello.bst element.
54	54
55	55	.. raw:: html
56	56	:file: ../sessions/developing-build-after-changes.html
57	57
	58	+Note that if you run the command from inside the workspace, the element name is optional.
	59	+
	60	+.. raw:: html
	61	+ :file: ../sessions/developing-build-after-changes-workspace.html
	62	+
58	63	Now running the hello command using bst shell:
59	64
60	65	.. raw:: html

doc/source/sessions-stored/developing-build-after-changes-workspace.html

---

	1	+<!--
	2	+ WARNING: This file was generated with bst2html.py
	3	+-->
	4	+<div class="highlight" style="font-size:x-small"><pre>
	5	+<span style="color:#C4A000;font-weight:bold">user@host</span>:<span style="color:#3456A4;font-weight:bold">~/workspace_hello</span>$ bst build
	6	+
	7	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#3465A4"><span style=""><span style="opacity:0.5">START </span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Build
	8	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#3465A4"><span style=""><span style="opacity:0.5">START </span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Loading elements
	9	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#4E9A06"><span style=""><span style="opacity:0.5">SUCCESS</span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Loading elements
	10	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#3465A4"><span style=""><span style="opacity:0.5">START </span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Resolving elements
	11	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#4E9A06"><span style=""><span style="opacity:0.5">SUCCESS</span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Resolving elements
	12	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#3465A4"><span style=""><span style="opacity:0.5">START </span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Resolving cached state
	13	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#4E9A06"><span style=""><span style="opacity:0.5">SUCCESS</span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Resolving cached state
	14	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">--</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#3465A4"><span style=""><span style="opacity:0.5">START </span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Checking sources
	15	+<span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#4E9A06"><span style=""><span style="opacity:0.5">SUCCESS</span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Checking sources
	16	+
	17	+<span style="color:#C4A000"><span style="">BuildStream Version 1.3.0+1032.g41813d3a.dirty
	18	+</span></span><span style="color:#06989A"><span style="opacity:0.5"> Session Start: </span></span><span style="color:#C4A000">Wednesday, 05-12-2018 at 16:47:38</span>
	19	+<span style="color:#06989A"><span style="opacity:0.5"> Project: </span></span><span style="color:#C4A000">developing (/home/user/workspace/buildstream/buildstream/doc/examples/developing)</span>
	20	+<span style="color:#06989A"><span style="opacity:0.5"> Targets: </span></span><span style="color:#C4A000">hello.bst</span>
	21	+
	22	+<span style="color:#C4A000"><span style="">User Configuration
	23	+</span></span><span style="color:#06989A"><span style="opacity:0.5"> Configuration File: </span></span><span style="color:#C4A000">/home/user/workspace/buildstream/buildstream/doc/run-bst-7ocq4_a7/buildstream.conf</span>
	24	+<span style="color:#06989A"><span style="opacity:0.5"> Log Files: </span></span><span style="color:#C4A000">/home/user/workspace/buildstream/buildstream/doc/run-bst-7ocq4_a7/logs</span>
	25	+<span style="color:#06989A"><span style="opacity:0.5"> Source Mirrors: </span></span><span style="color:#C4A000">/home/user/workspace/buildstream/buildstream/doc/run-bst-7ocq4_a7/sources</span>
	26	+<span style="color:#06989A"><span style="opacity:0.5"> Build Area: </span></span><span style="color:#C4A000">/home/user/workspace/buildstream/buildstream/doc/run-bst-7ocq4_a7/build</span>
	27	+<span style="color:#06989A"><span style="opacity:0.5"> Artifact Cache: </span></span><span style="color:#C4A000">/home/user/workspace/buildstream/buildstream/doc/run-bst-7ocq4_a7/artifacts</span>
	28	+<span style="color:#06989A"><span style="opacity:0.5"> Strict Build Plan: </span></span><span style="color:#C4A000">Yes</span>
	29	+<span style="color:#06989A"><span style="opacity:0.5"> Maximum Fetch Tasks: </span></span><span style="color:#C4A000">10</span>
	30	+<span style="color:#06989A"><span style="opacity:0.5"> Maximum Build Tasks: </span></span><span style="color:#C4A000">4</span>
	31	+<span style="color:#06989A"><span style="opacity:0.5"> Maximum Push Tasks: </span></span><span style="color:#C4A000">4</span>
	32	+<span style="color:#06989A"><span style="opacity:0.5"> Maximum Network Retries: </span></span><span style="color:#C4A000">2</span>
	33	+
	34	+<span style="color:#C4A000"><span style="">Pipeline
	35	+</span></span><span style="color:#75507B"> cached</span> <span style="color:#C4A000">9afe69d645f0bee106749bc2101aae16ef437bb51e1b343ef1f16f04f0572efb</span> <span style="color:#3465A4"><span style="">base/alpine.bst</span></span>
	36	+<span style="color:#75507B"> cached</span> <span style="color:#C4A000">19f7c50c7a1db9ae4babe9d1f34f4cdbbf2428827d48673861fd1452d6c7e16b</span> <span style="color:#3465A4"><span style="">base.bst</span></span>
	37	+<span style="color:#75507B"> cached</span> <span style="color:#C4A000">faa419610e7309d36d15926a81a8d75bbc113443c23c8162e63843dd86b5f56a</span> <span style="color:#3465A4"><span style="">hello.bst</span></span> Workspace: /home/user/workspace/buildstream/buildstream/doc/examples/developing/workspace_hello
	38	+<span style="color:#06989A"><span style="opacity:0.5">===============================================================================
	39	+</span></span><span style="color:#06989A"><span style="opacity:0.5">[</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">:</span></span><span style="color:#C4A000">00</span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">][</span></span><span style="color:#06989A"><span style="opacity:0.5">] </span></span><span style="color:#4E9A06"><span style=""><span style="opacity:0.5">SUCCESS</span></span></span><span style="color:#06989A"><span style="opacity:0.5"> </span></span>Build
	40	+
	41	+<span style="color:#C4A000"><span style="">Pipeline Summary
	42	+</span></span><span style="color:#06989A"><span style="opacity:0.5"> Total: </span></span><span style="color:#C4A000">3</span>
	43	+<span style="color:#06989A"><span style="opacity:0.5"> Session: </span></span><span style="color:#C4A000">0</span>
	44	+<span style="color:#06989A"><span style="opacity:0.5"> Fetch Queue: </span></span><span style="color:#C4A000">processed </span><span style="color:#4E9A06">0</span><span style="color:#06989A"><span style="opacity:0.5">, </span></span><span style="color:#C4A000">skipped </span><span style="color:#C4A000">0</span><span style="color:#06989A"><span style="opacity:0.5">, </span></span><span style="color:#C4A000">failed </span><span style="color:#CC0000"><span style="opacity:0.5">0</span></span>
	45	+<span style="color:#06989A"><span style="opacity:0.5"> Build Queue: </span></span><span style="color:#C4A000">processed </span><span style="color:#4E9A06">0</span><span style="color:#06989A"><span style="opacity:0.5">, </span></span><span style="color:#C4A000">skipped </span><span style="color:#C4A000">0</span><span style="color:#06989A"><span style="opacity:0.5">, </span></span><span style="color:#C4A000">failed </span><span style="color:#CC0000"><span style="opacity:0.5">0</span></span>
	46	+</pre></div>

doc/source/using_commands.rst

---

...	...	@@ -86,13 +86,6 @@ project's main directory.
86	86
87	87	----
88	88
89		-.. _invoking_source_bundle:
90		-
91		-.. click:: buildstream._frontend.cli:source_bundle
92		- :prog: bst source bundle
93		-
94		-----
95		-
96	89	.. _invoking_workspace:
97	90
98	91	.. click:: buildstream._frontend.cli:workspace

tests/completions/completions.py

---

...	...	@@ -16,7 +16,6 @@ MAIN_COMMANDS = [
16	16	'shell ',
17	17	'show ',
18	18	'source-checkout ',
19		- 'source-bundle ',
20	19	'track ',
21	20	'workspace '
22	21	]

tests/frontend/help.py

---

...	...	@@ -25,7 +25,6 @@ def test_help_main(cli):
25	25	('push'),
26	26	('shell'),
27	27	('show'),
28		- ('source-bundle'),
29	28	('track'),
30	29	('workspace')
31	30	])

tests/frontend/source_bundle.py deleted

---

1		-#
2		-# Copyright (C) 2018 Bloomberg Finance LP
3		-#
4		-# This program is free software; you can redistribute it and/or
5		-# modify it under the terms of the GNU Lesser General Public
6		-# License as published by the Free Software Foundation; either
7		-# version 2 of the License, or (at your option) any later version.
8		-#
9		-# This library is distributed in the hope that it will be useful,
10		-# but WITHOUT ANY WARRANTY; without even the implied warranty of
11		-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
12		-# Lesser General Public License for more details.
13		-#
14		-# You should have received a copy of the GNU Lesser General Public
15		-# License along with this library. If not, see <http://www.gnu.org/licenses/>.
16		-#
17		-# Authors: Chandan Singh <csingh43 bloomberg net>
18		-#
19		-
20		-import os
21		-import tarfile
22		-
23		-import pytest
24		-
25		-from tests.testutils import cli
26		-
27		-# Project directory
28		-DATA_DIR = os.path.join(
29		- os.path.dirname(os.path.realpath(__file__)),
30		- "project",
31		-)
32		-
33		-
34		-@pytest.mark.datafiles(DATA_DIR)
35		-def test_source_bundle(cli, tmpdir, datafiles):
36		- project_path = os.path.join(datafiles.dirname, datafiles.basename)
37		- element_name = 'source-bundle/source-bundle-hello.bst'
38		- normal_name = 'source-bundle-source-bundle-hello'
39		-
40		- # Verify that we can correctly produce a source-bundle
41		- args = ['source-bundle', element_name, '--directory', str(tmpdir)]
42		- result = cli.run(project=project_path, args=args)
43		- result.assert_success()
44		-
45		- # Verify that the source-bundle contains our sources and a build script
46		- with tarfile.open(os.path.join(str(tmpdir), '{}.tar.gz'.format(normal_name))) as bundle:
47		- assert os.path.join(normal_name, 'source', normal_name, 'llamas.txt') in bundle.getnames()
48		- assert os.path.join(normal_name, 'build.sh') in bundle.getnames()

tests/frontend/source_checkout.py

---

1	1	import os
2	2	import pytest
	3	+import tarfile
	4	+from pathlib import Path
3	5
4	6	from tests.testutils import cli
5	7
...	...	@@ -28,24 +30,66 @@ def generate_remote_import_element(input_path, output_path):
28	30
29	31
30	32	@pytest.mark.datafiles(DATA_DIR)
31		-@pytest.mark.parametrize('with_workspace', [('workspace'), ('no-workspace')])
32		-def test_source_checkout(datafiles, tmpdir_factory, cli, with_workspace):
	33	+@pytest.mark.parametrize(
	34	+ "with_workspace,guess_element",
	35	+ [(True, True), (True, False), (False, False)],
	36	+ ids=["workspace-guess", "workspace-no-guess", "no-workspace-no-guess"]
	37	+)
	38	+def test_source_checkout(datafiles, cli, tmpdir_factory, with_workspace, guess_element):
33	39	tmpdir = tmpdir_factory.mktemp("")
34	40	project = os.path.join(datafiles.dirname, datafiles.basename)
35	41	checkout = os.path.join(cli.directory, 'source-checkout')
36	42	target = 'checkout-deps.bst'
37	43	workspace = os.path.join(str(tmpdir), 'workspace')
	44	+ elm_cmd = [target] if not guess_element else []
38	45
39		- if with_workspace == "workspace":
40		- result = cli.run(project=project, args=['workspace', 'open', '--directory', workspace, target])
	46	+ if with_workspace:
	47	+ ws_cmd = ['-C', workspace]
	48	+ result = cli.run(project=project, args=["workspace", "open", "--directory", workspace, target])
41	49	result.assert_success()
	50	+ else:
	51	+ ws_cmd = []
42	52
43		- result = cli.run(project=project, args=['source-checkout', target, '--deps', 'none', checkout])
	53	+ args = ws_cmd + ['source-checkout', '--deps', 'none'] + elm_cmd + [checkout]
	54	+ result = cli.run(project=project, args=args)
44	55	result.assert_success()
45	56
46	57	assert os.path.exists(os.path.join(checkout, 'checkout-deps', 'etc', 'buildstream', 'config'))
47	58
48	59
	60	+@pytest.mark.datafiles(DATA_DIR)
	61	+@pytest.mark.parametrize('force_flag', ['--force', '-f'])
	62	+def test_source_checkout_force(datafiles, cli, force_flag):
	63	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	64	+ checkout = os.path.join(cli.directory, 'source-checkout')
	65	+ target = 'checkout-deps.bst'
	66	+
	67	+ os.makedirs(os.path.join(checkout, 'some-thing'))
	68	+ # Path(os.path.join(checkout, 'some-file')).touch()
	69	+
	70	+ result = cli.run(project=project, args=['source-checkout', force_flag, target, '--deps', 'none', checkout])
	71	+ result.assert_success()
	72	+
	73	+ assert os.path.exists(os.path.join(checkout, 'checkout-deps', 'etc', 'buildstream', 'config'))
	74	+
	75	+
	76	+@pytest.mark.datafiles(DATA_DIR)
	77	+def test_source_checkout_tar(datafiles, cli):
	78	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	79	+ checkout = os.path.join(cli.directory, 'source-checkout.tar')
	80	+ target = 'checkout-deps.bst'
	81	+
	82	+ result = cli.run(project=project, args=['source-checkout', '--tar', target, '--deps', 'none', checkout])
	83	+ result.assert_success()
	84	+
	85	+ assert os.path.exists(checkout)
	86	+ with tarfile.open(checkout) as tf:
	87	+ expected_content = os.path.join(checkout, 'checkout-deps', 'etc', 'buildstream', 'config')
	88	+ tar_members = [f.name for f in tf]
	89	+ for member in tar_members:
	90	+ assert member in expected_content
	91	+
	92	+
49	93	@pytest.mark.datafiles(DATA_DIR)
50	94	@pytest.mark.parametrize('deps', [('build'), ('none'), ('run'), ('all')])
51	95	def test_source_checkout_deps(datafiles, cli, deps):
...	...	@@ -126,3 +170,38 @@ def test_source_checkout_fetch(datafiles, cli, fetch):
126	170	assert os.path.exists(os.path.join(checkout, 'remote-import-dev', 'pony.h'))
127	171	else:
128	172	result.assert_main_error(ErrorDomain.PIPELINE, 'uncached-sources')
	173	+
	174	+
	175	+@pytest.mark.datafiles(DATA_DIR)
	176	+def test_source_checkout_build_scripts(cli, tmpdir, datafiles):
	177	+ project_path = os.path.join(datafiles.dirname, datafiles.basename)
	178	+ element_name = 'source-bundle/source-bundle-hello.bst'
	179	+ normal_name = 'source-bundle-source-bundle-hello'
	180	+ checkout = os.path.join(str(tmpdir), 'source-checkout')
	181	+
	182	+ args = ['source-checkout', '--include-build-scripts', element_name, checkout]
	183	+ result = cli.run(project=project_path, args=args)
	184	+ result.assert_success()
	185	+
	186	+ # There sould be a script for each element (just one in this case) and a top level build script
	187	+ expected_scripts = ['build.sh', 'build-' + normal_name]
	188	+ for script in expected_scripts:
	189	+ assert script in os.listdir(checkout)
	190	+
	191	+
	192	+@pytest.mark.datafiles(DATA_DIR)
	193	+def test_source_checkout_tar_buildscripts(cli, tmpdir, datafiles):
	194	+ project_path = os.path.join(datafiles.dirname, datafiles.basename)
	195	+ element_name = 'source-bundle/source-bundle-hello.bst'
	196	+ normal_name = 'source-bundle-source-bundle-hello'
	197	+ tar_file = os.path.join(str(tmpdir), 'source-checkout.tar')
	198	+
	199	+ args = ['source-checkout', '--include-build-scripts', '--tar', element_name, tar_file]
	200	+ result = cli.run(project=project_path, args=args)
	201	+ result.assert_success()
	202	+
	203	+ expected_scripts = ['build.sh', 'build-' + normal_name]
	204	+
	205	+ with tarfile.open(tar_file, 'r') as tf:
	206	+ for script in expected_scripts:
	207	+ assert script in tf.getnames()

tests/frontend/workspace.py

---

...	...	@@ -616,12 +616,16 @@ def test_list(cli, tmpdir, datafiles):
616	616	@pytest.mark.datafiles(DATA_DIR)
617	617	@pytest.mark.parametrize("kind", repo_kinds)
618	618	@pytest.mark.parametrize("strict", [("strict"), ("non-strict")])
619		-@pytest.mark.parametrize("call_from", [("project"), ("workspace")])
620		-def test_build(cli, tmpdir_factory, datafiles, kind, strict, call_from):
	619	+@pytest.mark.parametrize(
	620	+ "from_workspace,guess_element",
	621	+ [(False, False), (True, True), (True, False)],
	622	+ ids=["project-no-guess", "workspace-guess", "workspace-no-guess"])
	623	+def test_build(cli, tmpdir_factory, datafiles, kind, strict, from_workspace, guess_element):
621	624	tmpdir = tmpdir_factory.mktemp('')
622	625	element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, kind, False)
623	626	checkout = os.path.join(str(tmpdir), 'checkout')
624		- args_pre = ['-C', workspace] if call_from == "workspace" else []
	627	+ args_dir = ['-C', workspace] if from_workspace else []
	628	+ args_elm = [element_name] if not guess_element else []
625	629
626	630	# Modify workspace
627	631	shutil.rmtree(os.path.join(workspace, 'usr', 'bin'))
...	...	@@ -644,14 +648,14 @@ def test_build(cli, tmpdir_factory, datafiles, kind, strict, call_from):
644	648	# Build modified workspace
645	649	assert cli.get_element_state(project, element_name) == 'buildable'
646	650	assert cli.get_element_key(project, element_name) == "{:?<64}".format('')
647		- result = cli.run(project=project, args=args_pre + ['build', element_name])
	651	+ result = cli.run(project=project, args=args_dir + ['build'] + args_elm)
648	652	result.assert_success()
649	653	assert cli.get_element_state(project, element_name) == 'cached'
650	654	assert cli.get_element_key(project, element_name) != "{:?<64}".format('')
651	655
652	656	# Checkout the result
653	657	result = cli.run(project=project,
654		- args=args_pre + ['checkout', element_name, checkout])
	658	+ args=args_dir + ['checkout'] + args_elm + [checkout])
655	659	result.assert_success()
656	660
657	661	# Check that the pony.conf from the modified workspace exists
...	...	@@ -1062,29 +1066,36 @@ def test_multiple_failed_builds(cli, tmpdir, datafiles):
1062	1066
1063	1067	@pytest.mark.datafiles(DATA_DIR)
1064	1068	@pytest.mark.parametrize('subdir', [True, False], ids=["subdir", "no-subdir"])
1065		-def test_external_fetch(cli, datafiles, tmpdir_factory, subdir):
	1069	+@pytest.mark.parametrize("guess_element", [True, False], ids=["guess", "no-guess"])
	1070	+def test_external_fetch(cli, datafiles, tmpdir_factory, subdir, guess_element):
1066	1071	# Fetching from a workspace outside a project doesn't fail horribly
1067	1072	tmpdir = tmpdir_factory.mktemp('')
1068	1073	element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", False)
	1074	+ arg_elm = [element_name] if not guess_element else []
1069	1075
1070	1076	if subdir:
1071	1077	call_dir = os.path.join(workspace, 'usr')
1072	1078	else:
1073	1079	call_dir = workspace
1074	1080
1075		- result = cli.run(project=project, args=['-C', call_dir, 'fetch', element_name])
	1081	+ result = cli.run(project=project, args=['-C', call_dir, 'fetch'] + arg_elm)
1076	1082	result.assert_success()
1077	1083
1078	1084	# We already fetched it by opening the workspace, but we're also checking
1079	1085	# `bst show` works here
1080		- assert cli.get_element_state(project, element_name) == 'buildable'
	1086	+ result = cli.run(project=project,
	1087	+ args=['-C', call_dir, 'show', '--deps', 'none', '--format', '%{state}'] + arg_elm)
	1088	+ result.assert_success()
	1089	+ assert result.output.strip() == 'buildable'
1081	1090
1082	1091
1083	1092	@pytest.mark.datafiles(DATA_DIR)
1084		-def test_external_push_pull(cli, datafiles, tmpdir_factory):
	1093	+@pytest.mark.parametrize("guess_element", [True, False], ids=["guess", "no-guess"])
	1094	+def test_external_push_pull(cli, datafiles, tmpdir_factory, guess_element):
1085	1095	# Pushing and pulling to/from an artifact cache works from an external workspace
1086	1096	tmpdir = tmpdir_factory.mktemp('')
1087	1097	element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", False)
	1098	+ arg_elm = [element_name] if not guess_element else []
1088	1099
1089	1100	with create_artifact_share(os.path.join(str(tmpdir), 'artifactshare')) as share:
1090	1101	result = cli.run(project=project, args=['-C', workspace, 'build', element_name])
...	...	@@ -1094,22 +1105,24 @@ def test_external_push_pull(cli, datafiles, tmpdir_factory):
1094	1105	'artifacts': {'url': share.repo, 'push': True}
1095	1106	})
1096	1107
1097		- result = cli.run(project=project, args=['-C', workspace, 'push', element_name])
	1108	+ result = cli.run(project=project, args=['-C', workspace, 'push'] + arg_elm)
1098	1109	result.assert_success()
1099	1110
1100		- result = cli.run(project=project, args=['-C', workspace, 'pull', '--deps', 'all', element_name])
	1111	+ result = cli.run(project=project, args=['-C', workspace, 'pull', '--deps', 'all'] + arg_elm)
1101	1112	result.assert_success()
1102	1113
1103	1114
1104	1115	@pytest.mark.datafiles(DATA_DIR)
1105		-def test_external_track(cli, datafiles, tmpdir_factory):
	1116	+@pytest.mark.parametrize("guess_element", [True, False], ids=["guess", "no-guess"])
	1117	+def test_external_track(cli, datafiles, tmpdir_factory, guess_element):
1106	1118	# Tracking does not get horribly confused
1107	1119	tmpdir = tmpdir_factory.mktemp('')
1108	1120	element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", True)
	1121	+ arg_elm = [element_name] if not guess_element else []
1109	1122
1110	1123	# The workspace is necessarily already tracked, so we only care that
1111	1124	# there's no weird errors.
1112		- result = cli.run(project=project, args=['-C', workspace, 'track', element_name])
	1125	+ result = cli.run(project=project, args=['-C', workspace, 'track'] + arg_elm)
1113	1126	result.assert_success()
1114	1127
1115	1128
...	...	@@ -1147,15 +1160,17 @@ def test_external_close_other(cli, datafiles, tmpdir_factory):
1147	1160
1148	1161
1149	1162	@pytest.mark.datafiles(DATA_DIR)
1150		-def test_external_close_self(cli, datafiles, tmpdir_factory):
	1163	+@pytest.mark.parametrize("guess_element", [True, False], ids=["guess", "no-guess"])
	1164	+def test_external_close_self(cli, datafiles, tmpdir_factory, guess_element):
1151	1165	# >From inside an external workspace, close it
1152	1166	tmpdir1 = tmpdir_factory.mktemp('')
1153	1167	tmpdir2 = tmpdir_factory.mktemp('')
1154	1168	# Making use of the assumption that it's the same project in both invocations of open_workspace
1155	1169	alpha_element, project, alpha_workspace = open_workspace(cli, tmpdir1, datafiles, "git", False, suffix="-alpha")
1156	1170	beta_element, _, beta_workspace = open_workspace(cli, tmpdir2, datafiles, "git", False, suffix="-beta")
	1171	+ arg_elm = [alpha_element] if not guess_element else []
1157	1172
1158		- result = cli.run(project=project, args=['-C', alpha_workspace, 'workspace', 'close', alpha_element])
	1173	+ result = cli.run(project=project, args=['-C', alpha_workspace, 'workspace', 'close'] + arg_elm)
1159	1174	result.assert_success()
1160	1175
1161	1176
...	...	@@ -1172,11 +1187,13 @@ def test_external_reset_other(cli, datafiles, tmpdir_factory):
1172	1187
1173	1188
1174	1189	@pytest.mark.datafiles(DATA_DIR)
1175		-def test_external_reset_self(cli, datafiles, tmpdir):
	1190	+@pytest.mark.parametrize("guess_element", [True, False], ids=["guess", "no-guess"])
	1191	+def test_external_reset_self(cli, datafiles, tmpdir, guess_element):
1176	1192	element, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", False)
	1193	+ arg_elm = [element] if not guess_element else []
1177	1194
1178	1195	# Command succeeds
1179		- result = cli.run(project=project, args=['-C', workspace, 'workspace', 'reset', element])
	1196	+ result = cli.run(project=project, args=['-C', workspace, 'workspace', 'reset'] + arg_elm)
1180	1197	result.assert_success()
1181	1198
1182	1199	# Successive commands still work (i.e. .bstproject.yaml hasn't been deleted)

tests/integration/shell.py

---

...	...	@@ -358,13 +358,22 @@ def test_integration_devices(cli, tmpdir, datafiles):
358	358	# Test that a shell can be opened from an external workspace
359	359	@pytest.mark.datafiles(DATA_DIR)
360	360	@pytest.mark.parametrize("build_shell", [("build"), ("nobuild")])
	361	+@pytest.mark.parametrize("guess_element", [True, False], ids=["guess", "no-guess"])
361	362	@pytest.mark.skipif(IS_LINUX and not HAVE_BWRAP, reason='Only available with bubblewrap on Linux')
362		-def test_integration_external_workspace(cli, tmpdir_factory, datafiles, build_shell):
	363	+def test_integration_external_workspace(cli, tmpdir_factory, datafiles, build_shell, guess_element):
363	364	tmpdir = tmpdir_factory.mktemp("")
364	365	project = os.path.join(datafiles.dirname, datafiles.basename)
365	366	element_name = 'autotools/amhello.bst'
366	367	workspace_dir = os.path.join(str(tmpdir), 'workspace')
367	368
	369	+ if guess_element:
	370	+ # Mutate the project.conf to use a default shell command
	371	+ project_file = os.path.join(project, 'project.conf')
	372	+ config_text = "shell:\n"\
	373	+ " command: ['true']\n"
	374	+ with open(project_file, 'a') as f:
	375	+ f.write(config_text)
	376	+
368	377	result = cli.run(project=project, args=[
369	378	'workspace', 'open', '--directory', workspace_dir, element_name
370	379	])
...	...	@@ -373,9 +382,10 @@ def test_integration_external_workspace(cli, tmpdir_factory, datafiles, build_sh
373	382	result = cli.run(project=project, args=['-C', workspace_dir, 'build', element_name])
374	383	result.assert_success()
375	384
376		- command = ['shell']
	385	+ command = ['-C', workspace_dir, 'shell']
377	386	if build_shell == 'build':
378	387	command.append('--build')
379		- command.extend([element_name, '--', 'true'])
	388	+ if not guess_element:
	389	+ command.extend([element_name, '--', 'true'])
380	390	result = cli.run(project=project, cwd=workspace_dir, args=command)
381	391	result.assert_success()

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.