[Notes] [Git][BuildStream/buildstream][jonathan/debug-remote-failed-builds] 14 commits: Rename element_enums.py -> types.py

[Jonathan Maw <gitlab mg gitlab com>]

Title: GitLab

Jonathan Maw pushed to branch jonathan/debug-remote-failed-builds at BuildStream / buildstream

Commits:

• 4f0bfb4a
  by Tristan Van Berkom at 2018-10-11T15:37:51Z

  Rename element_enums.py -> types.py
  
  This will be the place to store low level data types used
  throughout the core, for now this includes public and private
  types.
  

• 9dc10cc7
  by Tristan Van Berkom at 2018-10-11T15:45:36Z

  doc/source/core_framework.rst: Include the foundation types in the API docs.
  

• efc0d4cc
  by Tristan Van Berkom at 2018-10-11T15:48:44Z

  types.py: Moved Consistency definition here from source.py
  
  And slightly touch up the doc strings for Consistency and Scope.
  

• a0712ead
  by Tristan Van Berkom at 2018-10-11T16:23:54Z

  Merge branch 'tristan/refactor-types-api' into 'master'
  
  Refactor types api
  
  See merge request BuildStream/buildstream!870

• ce55b9a0
  by Tiago Gomes at 2018-10-12T10:15:46Z

  doc: updates considering website being live now
  
  * Add a link to the website on the main page.
  * Remove install instructions as they are now on the website.
  * Remove Resources section as that information can be found at the
    website, and also looks bad.
  * Move artifact server setup from the no longer existing Install section
    to the Using section.
  

• 10b092e1
  by Tristan Van Berkom at 2018-10-12T10:21:27Z

  doc/source/additional_docker.rst: Fix link to refer to website
  
  The docker guide, which is part of the install guide, has moved
  to the website.
  

• 1c7ba9a0
  by Tristan Van Berkom at 2018-10-12T10:24:31Z

  doc/source/index.rst: Moved references to the website below the simple ToC.
  

• 3738dd06
  by Tristan Van Berkom at 2018-10-12T10:49:39Z

  Merge branch 'tristan/remove-install-guide' into 'master'
  
  Remove install guide
  
  See merge request BuildStream/buildstream!872

• 9acc0bab
  by Jonathan Maw at 2018-10-12T16:18:05Z

  element.py: Always clean up the rootdir
  
  We shouldn't need it to persist now that we cache failed build dirs.
  
  This change breaks the test
  `tests/integration/shell.py::test_sysroot_workspace_visible`.
  I can no longer see a use-case for this test.
  AIUI, it tested that the failed build sysroot stored in the builddir has
  the workspace's files in, despite the workspace being unmounted.
  I believe this behaviour is made redundant by cached buildtrees.
  
  This fixes part of #539
  

• 3cd09f53
  by Jonathan Maw at 2018-10-12T16:18:05Z

  Element: Use cached buildtree in build shells and failure shells
  
  This includes changes in app.py:
  * Interactive failure shell no longer uses the failed build sysroot,
    defaulting to the cached build tree.
  * Always allow producing a failure shell.
  
  Changes in element.py are:
  * Errors caused by building don't store the failed build sysroot.
  * When staging sources, will stage the element's cached build tree if it
    exists.
  
  This fixes part of #539
  

• 5bbf1ce0
  by Jonathan Maw at 2018-10-12T16:18:05Z

  tests: Add test that cached build trees are staged in build shells
  
  This is related to #539
  

• 3b021a3d
  by Jonathan Maw at 2018-10-12T16:18:05Z

  sandbox.py: Remove redundant Sandbox.__directory
  

• 6e8c09cd
  by Jonathan Maw at 2018-10-12T16:18:05Z

  element: Make "--sysroot" take a bare directory
  
  i.e. instead of taking a directory that must contain "root" and
  "scratch", and treating "root" as the root, use the directory directly.
  
  In element.py:
  * __sandbox takes the `bare_sandbox` arg, to pass into the sandbox's constructor
  
  In sandbox.py:
  * If bare_sandbox, `_root` is the passed-in directory, and `__scratch`
    is None.
  * Trying to use `__scratch` when bare_sandbox is True is a bug.
  
  In _mount.py:
  * Don't get the value of `__scratch` if it's not needed.
  
  This is part of #539
  

• 8ac61194
  by Jonathan Maw at 2018-10-12T16:18:05Z

  tests: Add tests for 'bst shell --sysroot'
  
  This is related to #539
  

23 changed files:

• buildstream/__init__.py
• buildstream/_artifactcache/artifactcache.py
• buildstream/_frontend/app.py
• buildstream/element.py
• buildstream/sandbox/_mount.py
• buildstream/sandbox/sandbox.py
• buildstream/source.py
• buildstream/element_enums.py → buildstream/types.py
• doc/source/additional_docker.rst
• doc/source/core_framework.rst
• doc/source/index.rst
• − doc/source/install_docker.rst
• − doc/source/install_linux_distro.rst
• − doc/source/install_source.rst
• − doc/source/install_versions.rst
• − doc/source/main_install.rst
• doc/source/main_using.rst
• − doc/source/release-badge.rst
• − doc/source/snapshot-badge.rst
• doc/source/install_artifacts.rst → doc/source/using_configuring_artifact_server.rst
• + tests/integration/build-tree.py
• + tests/integration/project/elements/build-shell/buildtree.bst
• tests/integration/shell.py

Changes:

buildstream/__init__.py

---

...	...	@@ -28,9 +28,9 @@ if "_BST_COMPLETION" not in os.environ:
28	28
29	29	from .utils import UtilError, ProgramNotFoundError
30	30	from .sandbox import Sandbox, SandboxFlags
	31	+ from .types import Scope, Consistency
31	32	from .plugin import Plugin
32		- from .source import Source, SourceError, Consistency, SourceFetcher
	33	+ from .source import Source, SourceError, SourceFetcher
33	34	from .element import Element, ElementError
34		- from .element_enums import Scope
35	35	from .buildelement import BuildElement
36	36	from .scriptelement import ScriptElement

buildstream/_artifactcache/artifactcache.py

---

...	...	@@ -21,7 +21,7 @@ import os
21	21	import string
22	22	from collections import Mapping, namedtuple
23	23
24		-from ..element_enums import _KeyStrength
	24	+from ..types import _KeyStrength
25	25	from .._exceptions import ArtifactError, ImplError, LoadError, LoadErrorReason
26	26	from .._message import Message, MessageType
27	27	from .. import utils

buildstream/_frontend/app.py

---

...	...	@@ -564,18 +564,15 @@ class App():
564	564	" (c)ontinue - Continue queueing jobs as much as possible\n" +
565	565	" (q)uit - Exit after all ongoing jobs complete\n" +
566	566	" (t)erminate - Terminate any ongoing jobs and exit\n" +
567		- " (r)etry - Retry this job\n")
	567	+ " (r)etry - Retry this job\n" +
	568	+ " (s)hell - Drop into a shell in the failed build sandbox\n")
568	569	if failure.logfile:
569	570	summary += " (l)og - View the full log file\n"
570		- if failure.sandbox:
571		- summary += " (s)hell - Drop into a shell in the failed build sandbox\n"
572	571	summary += "\nPressing ^C will terminate jobs and exit\n"
573	572
574		- choices = ['continue', 'quit', 'terminate', 'retry']
	573	+ choices = ['continue', 'quit', 'terminate', 'retry', 'shell']
575	574	if failure.logfile:
576	575	choices += ['log']
577		- if failure.sandbox:
578		- choices += ['shell']
579	576
580	577	choice = ''
581	578	while choice not in ['continue', 'quit', 'terminate', 'retry']:
...	...	@@ -595,10 +592,10 @@ class App():
595	592	# Handle choices which you can come back from
596	593	#
597	594	if choice == 'shell':
598		- click.echo("\nDropping into an interactive shell in the failed build sandbox\n", err=True)
	595	+ click.echo("\nDropping into an interactive shell in the failed build tree\n", err=True)
599	596	try:
600	597	prompt = self.shell_prompt(element)
601		- self.stream.shell(element, Scope.BUILD, prompt, directory=failure.sandbox, isolate=True)
	598	+ self.stream.shell(element, Scope.BUILD, prompt, isolate=True)
602	599	except BstError as e:
603	600	click.echo("Error while attempting to create interactive shell: {}".format(e), err=True)
604	601	elif choice == 'log':

buildstream/element.py

---

...	...	@@ -86,7 +86,7 @@ from ._variables import Variables
86	86	from ._versions import BST_CORE_ARTIFACT_VERSION
87	87	from ._exceptions import BstError, LoadError, LoadErrorReason, ImplError, ErrorDomain
88	88	from .utils import UtilError
89		-from . import Plugin, Consistency
	89	+from . import Plugin, Consistency, Scope
90	90	from . import SandboxFlags
91	91	from . import utils
92	92	from . import _cachekey
...	...	@@ -96,11 +96,11 @@ from ._platform import Platform
96	96	from .plugin import CoreWarnings
97	97	from .sandbox._config import SandboxConfig
98	98	from .sandbox._sandboxremote import SandboxRemote
	99	+from .types import _KeyStrength
99	100
100	101	from .storage.directory import Directory
101	102	from .storage._filebaseddirectory import FileBasedDirectory
102	103	from .storage.directory import VirtualDirectoryError
103		-from .element_enums import _KeyStrength, Scope
104	104
105	105
106	106	class ElementError(BstError):
...	...	@@ -1317,7 +1317,9 @@ class Element(Plugin):
1317	1317	@contextmanager
1318	1318	def _prepare_sandbox(self, scope, directory, deps='run', integrate=True):
1319	1319	# bst shell and bst checkout require a local sandbox.
1320		- with self.__sandbox(directory, config=self.__sandbox_config, allow_remote=False) as sandbox:
	1320	+ bare_directory = True if directory else False
	1321	+ with self.__sandbox(directory, config=self.__sandbox_config, allow_remote=False,
	1322	+ bare_directory=True if directory else False) as sandbox:
1321	1323
1322	1324	# Configure always comes first, and we need it.
1323	1325	self.configure_sandbox(sandbox)
...	...	@@ -1384,6 +1386,7 @@ class Element(Plugin):
1384	1386	# the same filing system as the rest of our cache.
1385	1387	temp_staging_location = os.path.join(self._get_context().artifactdir, "staging_temp")
1386	1388	temp_staging_directory = tempfile.mkdtemp(prefix=temp_staging_location)
	1389	+ import_dir = temp_staging_directory
1387	1390
1388	1391	try:
1389	1392	workspace = self._get_workspace()
...	...	@@ -1394,12 +1397,16 @@ class Element(Plugin):
1394	1397	with self.timed_activity("Staging local files at {}"
1395	1398	.format(workspace.get_absolute_path())):
1396	1399	workspace.stage(temp_staging_directory)
	1400	+ elif self._cached():
	1401	+ # We have a cached buildtree to use, instead
	1402	+ artifact_base, _ = self.__extract()
	1403	+ import_dir = os.path.join(artifact_base, 'buildtree')
1397	1404	else:
1398	1405	# No workspace, stage directly
1399	1406	for source in self.sources():
1400	1407	source._stage(temp_staging_directory)
1401	1408
1402		- vdirectory.import_files(temp_staging_directory)
	1409	+ vdirectory.import_files(import_dir)
1403	1410
1404	1411	finally:
1405	1412	# Staging may produce directories with less than 'rwx' permissions
...	...	@@ -1565,10 +1572,6 @@ class Element(Plugin):
1565	1572	collect = self.assemble(sandbox)
1566	1573	self.__set_build_result(success=True, description="succeeded")
1567	1574	except BstError as e:
1568		- # If an error occurred assembling an element in a sandbox,
1569		- # then tack on the sandbox directory to the error
1570		- e.sandbox = rootdir
1571		-
1572	1575	# If there is a workspace open on this element, it will have
1573	1576	# been mounted for sandbox invocations instead of being staged.
1574	1577	#
...	...	@@ -1682,8 +1685,8 @@ class Element(Plugin):
1682	1685	"unable to collect artifact contents"
1683	1686	.format(collect))
1684	1687
1685		- # Finally cleanup the build dir
1686		- cleanup_rootdir()
	1688	+ # Finally cleanup the build dir
	1689	+ cleanup_rootdir()
1687	1690
1688	1691	return artifact_size
1689	1692
...	...	@@ -2151,12 +2154,14 @@ class Element(Plugin):
2151	2154	# stderr (fileobject): The stream for stderr for the sandbox
2152	2155	# config (SandboxConfig): The SandboxConfig object
2153	2156	# allow_remote (bool): Whether the sandbox is allowed to be remote
	2157	+ # bare_directory (bool): Whether the directory is bare i.e. doesn't have
	2158	+ # a separate 'root' subdir
2154	2159	#
2155	2160	# Yields:
2156	2161	# (Sandbox): A usable sandbox
2157	2162	#
2158	2163	@contextmanager
2159		- def __sandbox(self, directory, stdout=None, stderr=None, config=None, allow_remote=True):
	2164	+ def __sandbox(self, directory, stdout=None, stderr=None, config=None, allow_remote=True, bare_directory=False):
2160	2165	context = self._get_context()
2161	2166	project = self._get_project()
2162	2167	platform = Platform.get_platform()
...	...	@@ -2187,6 +2192,7 @@ class Element(Plugin):
2187	2192	stdout=stdout,
2188	2193	stderr=stderr,
2189	2194	config=config,
	2195	+ bare_directory=bare_directory,
2190	2196	allow_real_directory=not self.BST_VIRTUAL_DIRECTORY)
2191	2197	yield sandbox
2192	2198
...	...	@@ -2196,7 +2202,7 @@ class Element(Plugin):
2196	2202
2197	2203	# Recursive contextmanager...
2198	2204	with self.__sandbox(rootdir, stdout=stdout, stderr=stderr, config=config,
2199		- allow_remote=allow_remote) as sandbox:
	2205	+ allow_remote=allow_remote, bare_directory=False) as sandbox:
2200	2206	yield sandbox
2201	2207
2202	2208	# Cleanup the build dir

buildstream/sandbox/_mount.py

---

...	...	@@ -31,7 +31,6 @@ from .._fuse import SafeHardlinks
31	31	#
32	32	class Mount():
33	33	def __init__(self, sandbox, mount_point, safe_hardlinks, fuse_mount_options={}):
34		- scratch_directory = sandbox._get_scratch_directory()
35	34	# Getting _get_underlying_directory() here is acceptable as
36	35	# we're part of the sandbox code. This will fail if our
37	36	# directory is CAS-based.
...	...	@@ -51,6 +50,7 @@ class Mount():
51	50	# a regular mount point within the parent's redirected mount.
52	51	#
53	52	if self.safe_hardlinks:
	53	+ scratch_directory = sandbox._get_scratch_directory()
54	54	# Redirected mount
55	55	self.mount_origin = os.path.join(root_directory, mount_point.lstrip(os.sep))
56	56	self.mount_base = os.path.join(scratch_directory, utils.url_directory_name(mount_point))

buildstream/sandbox/sandbox.py

---

...	...	@@ -98,16 +98,23 @@ class Sandbox():
98	98	self.__config = kwargs['config']
99	99	self.__stdout = kwargs['stdout']
100	100	self.__stderr = kwargs['stderr']
	101	+ self.__bare_directory = kwargs['bare_directory']
101	102
102	103	# Setup the directories. Root and output_directory should be
103	104	# available to subclasses, hence being single-underscore. The
104	105	# others are private to this class.
105		- self._root = os.path.join(directory, 'root')
	106	+ # If the directory is bare, it probably doesn't need scratch
	107	+ if self.__bare_directory:
	108	+ self._root = directory
	109	+ self.__scratch = None
	110	+ os.makedirs(self._root, exist_ok=True)
	111	+ else:
	112	+ self._root = os.path.join(directory, 'root')
	113	+ self.__scratch = os.path.join(directory, 'scratch')
	114	+ for directory_ in [self._root, self.__scratch]:
	115	+ os.makedirs(directory_, exist_ok=True)
	116	+
106	117	self._output_directory = None
107		- self.__directory = directory
108		- self.__scratch = os.path.join(self.__directory, 'scratch')
109		- for directory_ in [self._root, self.__scratch]:
110		- os.makedirs(directory_, exist_ok=True)
111	118	self._vdir = None
112	119
113	120	# This is set if anyone requests access to the underlying
...	...	@@ -332,6 +339,7 @@ class Sandbox():
332	339	# Returns:
333	340	# (str): The sandbox scratch directory
334	341	def _get_scratch_directory(self):
	342	+ assert not self.__bare_directory, "Scratch is not going to work with bare directories"
335	343	return self.__scratch
336	344
337	345	# _get_output()

buildstream/source.py

---

...	...	@@ -145,35 +145,12 @@ import os
145	145	from collections import Mapping
146	146	from contextlib import contextmanager
147	147
148		-from . import Plugin
	148	+from . import Plugin, Consistency
149	149	from . import _yaml, utils
150	150	from ._exceptions import BstError, ImplError, ErrorDomain
151	151	from ._projectrefs import ProjectRefStorage
152	152
153	153
154		-class Consistency():
155		- INCONSISTENT = 0
156		- """Inconsistent
157		-
158		- Inconsistent sources have no explicit reference set. They cannot
159		- produce a cache key, be fetched or staged. They can only be tracked.
160		- """
161		-
162		- RESOLVED = 1
163		- """Resolved
164		-
165		- Resolved sources have a reference and can produce a cache key and
166		- be fetched, however they cannot be staged.
167		- """
168		-
169		- CACHED = 2
170		- """Cached
171		-
172		- Cached sources have a reference which is present in the local
173		- source cache. Only cached sources can be staged.
174		- """
175		-
176		-
177	154	class SourceError(BstError):
178	155	"""This exception should be raised by :class:`.Source` implementations
179	156	to report errors to the user.

buildstream/element_enums.py → buildstream/types.py

---

...	...	@@ -19,31 +19,19 @@
19	19	# Jim MacArthur <jim macarthur codethink co uk>
20	20
21	21	"""
22		-Element - Globally visible enumerations
23		-=======================================
	22	+Foundation types
	23	+================
24	24
25	25	"""
26	26
27	27	from enum import Enum
28	28
29	29
30		-# _KeyStrength():
31		-#
32		-# Strength of cache key
33		-#
34		-class _KeyStrength(Enum):
35		-
36		- # Includes strong cache keys of all build dependencies and their
37		- # runtime dependencies.
38		- STRONG = 1
39		-
40		- # Includes names of direct build dependencies but does not include
41		- # cache keys of dependencies.
42		- WEAK = 2
43		-
44		-
45	30	class Scope(Enum):
46		- """Types of scope for a given element"""
	31	+ """Defines the scope of dependencies to include for a given element
	32	+ when iterating over the dependency graph in APIs like
	33	+ :func:`Element.dependencies() <buildstream.element.Element.dependencies>`
	34	+ """
47	35
48	36	ALL = 1
49	37	"""All elements which the given element depends on, following
...	...	@@ -59,3 +47,44 @@ class Scope(Enum):
59	47	"""All elements required for running the element. Including the element
60	48	itself.
61	49	"""
	50	+
	51	+
	52	+class Consistency():
	53	+ """Defines the various consistency states of a :class:`.Source`.
	54	+ """
	55	+
	56	+ INCONSISTENT = 0
	57	+ """Inconsistent
	58	+
	59	+ Inconsistent sources have no explicit reference set. They cannot
	60	+ produce a cache key, be fetched or staged. They can only be tracked.
	61	+ """
	62	+
	63	+ RESOLVED = 1
	64	+ """Resolved
	65	+
	66	+ Resolved sources have a reference and can produce a cache key and
	67	+ be fetched, however they cannot be staged.
	68	+ """
	69	+
	70	+ CACHED = 2
	71	+ """Cached
	72	+
	73	+ Cached sources have a reference which is present in the local
	74	+ source cache. Only cached sources can be staged.
	75	+ """
	76	+
	77	+
	78	+# _KeyStrength():
	79	+#
	80	+# Strength of cache key
	81	+#
	82	+class _KeyStrength(Enum):
	83	+
	84	+ # Includes strong cache keys of all build dependencies and their
	85	+ # runtime dependencies.
	86	+ STRONG = 1
	87	+
	88	+ # Includes names of direct build dependencies but does not include
	89	+ # cache keys of dependencies.
	90	+ WEAK = 2

doc/source/additional_docker.rst

---

...	...	@@ -4,19 +4,18 @@
4	4
5	5	BuildStream and Docker
6	6	======================
7		-
8	7	BuildStream integrates with Docker in multiple ways. Here are some ways in
9	8	which these integrations work.
10	9
	10	+
11	11	Run BuildStream inside Docker
12	12	-----------------------------
	13	+Refer to the `BuildStream inside Docker <https://buildstream.build/docker_install.html>`_
	14	+documentation for instructions on how to run BuildStream as a Docker container.
13	15
14		-Refer to the :ref:`BuildStream inside Docker <docker>` documentation for
15		-instructions on how to run BuildStream as a Docker container.
16	16
17	17	Generate Docker images
18	18	----------------------
19		-
20	19	The
21	20	`bst-docker-import script <https://gitlab.com/BuildStream/buildstream/blob/master/contrib/bst-docker-import>`_
22	21	can be used to generate a Docker image from built artifacts.

doc/source/core_framework.rst

---

...	...	@@ -12,6 +12,7 @@ useful for working on BuildStream itself.
12	12	.. toctree::
13	13	:maxdepth: 1
14	14
	15	+ buildstream.types
15	16	buildstream.plugin
16	17	buildstream.source
17	18	buildstream.element

doc/source/index.rst

---

...	...	@@ -13,20 +13,13 @@ They begin with a basic introduction to BuildStream, background
13	13	information on basic concepts, and a guide to the BuildStream command line interface.
14	14	Later sections provide detailed information on BuildStream internals.
15	15
16		-
17	16	.. toctree::
18	17	:maxdepth: 1
19	18
20	19	main_about
21		- main_install
22	20	main_using
23	21	main_core
24	22	CONTRIBUTING
25	23
26		-
27		-Resources
28		----------
29		-* GitLab repository: https://gitlab.com/BuildStream/buildstream
30		-* Bug Tracking: https://gitlab.com/BuildStream/buildstream/issues
31		-* Mailing list: https://mail.gnome.org/mailman/listinfo/buildstream-list
32		-* IRC Channel: irc://irc.gnome.org/#buildstream
	24	+For any other information, including `how to install BuildStream <https://buildstream.build/install.html>`_,
	25	+refer to `the BuildStream website <https://buildstream.build>`_.

doc/source/install_docker.rst deleted

---

1		-
2		-
3		-.. _docker:
4		-
5		-BuildStream inside Docker
6		--------------------------
7		-If your system cannot provide the base system requirements for BuildStream, then it is possible to run buildstream within a Docker image.
8		-
9		-The BuildStream project provides
10		-`Docker images <https://hub.docker.com/r/buildstream/buildstream-fedora>`_
11		-containing BuildStream and its dependencies.
12		-This gives you an easy way to get started using BuildStream on any Unix-like
13		-platform where Docker is available, including Mac OS X.
14		-
15		-We recommend using the
16		-`bst-here wrapper script <https://gitlab.com/BuildStream/buildstream/blob/master/contrib/bst-here>`_
17		-which automates the necessary container setup. You can download it and make
18		-it executable like this:
19		-
20		-.. code:: bash
21		-
22		- mkdir -p ~/.local/bin
23		- curl --get https://gitlab.com/BuildStream/buildstream/raw/master/contrib/bst-here > ~/.local/bin/bst-here
24		- chmod +x ~/.local/bin/bst-here
25		-
26		-Check if ``~/.local/bin`` appears in your PATH environment variable -- if it
27		-doesn't, you should
28		-`edit your ~/.profile so that it does <https://stackoverflow.com/questions/14637979/>`_.
29		-
30		-Once the script is available in your PATH, you can run ``bst-here`` to open a
31		-shell session inside a new container based off the latest version of the
32		-buildstream-fedora Docker image. The current working directory will be mounted
33		-inside the container at ``/src``.
34		-
35		-You can also run individual BuildStream commands as ``bst-here COMMAND``. For
36		-example: ``bst-here show systems/my-system.bst``. Note that BuildStream won't
37		-be able to integrate with Bash tab-completion if you invoke it in this way.
38		-
39		-Two Docker volumes are set up by the ``bst-here`` script:
40		-
41		- * ``buildstream-cache --`` mounted at ``~/.cache/buildstream``
42		- * ``buildstream-config --`` mounted at ``~/.config/``
43		-
44		-These are necessary so that your BuildStream cache and configuration files
45		-persist between invocations of ``bst-here``.

doc/source/install_linux_distro.rst deleted

---

1		-
2		-
3		-.. _install_linux_distro:
4		-
5		-Installing from distro packages
6		-===============================
7		-BuildStream is available on some linux distributions, here are
8		-some install instructions for the linux distributions which
9		-have packaged BuildStream.
10		-
11		-
12		-Arch Linux
13		-----------
14		-Packages for Arch exist in `AUR <https://wiki.archlinux.org/index.php/Arch_User_Repository#Installing_packages>`_.
15		-Two different package versions are available:
16		-
17		-* Latest release: `buildstream <https://aur.archlinux.org/packages/buildstream>`_
18		-* Latest development snapshot: `buildstream-git <https://aur.archlinux.org/packages/buildstream-git>`_
19		-
20		-
21		-Fedora
22		-------
23		-BuildStream is not yet in the official Fedora repositories, but you can
24		-install it from a Copr::
25		-
26		- sudo dnf copr enable bochecha/buildstream
27		- sudo dnf install buildstream
28		-
29		-Optionally, install the ``buildstream-docs`` package to have the BuildStream
30		-documentation in Devhelp or GNOME Builder.

doc/source/install_source.rst deleted

---

1		-
2		-
3		-Installing from source
4		-======================
5		-Until BuildStream is available in :ref:`your distro <install_linux_distro>`, you will
6		-need to install it yourself from source.
7		-
8		-
9		-Installing dependencies
10		------------------------
11		-Before installing BuildStream from source, it is necessary to first install
12		-the system dependencies. Below are some linux distribution specific instructions
13		-for installing these dependencies.
14		-
15		-BuildStream requires the following base system requirements:
16		-
17		-* python3 >= 3.5
18		-* bubblewrap >= 0.1.2
19		-* fuse2
20		-
21		-BuildStream also depends on the host tools for the :mod:`Source <buildstream.source>` plugins.
22		-Refer to the respective :ref:`source plugin <plugins_sources>` documentation for host tool
23		-requirements of specific plugins.
24		-
25		-The default plugins with extra host dependencies are:
26		-
27		-* bzr
28		-* deb
29		-* git
30		-* ostree
31		-* patch
32		-* pip
33		-* tar
34		-
35		-If you intend to push built artifacts to a remote artifact server,
36		-which requires special permissions, you will also need:
37		-
38		-* ssh
39		-
40		-
41		-Arch Linux
42		-~~~~~~~~~~
43		-Install the dependencies with::
44		-
45		- sudo pacman -S \
46		- python fuse2 bubblewrap \
47		- python-pip
48		-
49		-For the default plugins::
50		-
51		- sudo pacman -S \
52		- bzr git lzip ostree patch python-gobject
53		-
54		-
55		-The package *python-arpy* is required by the deb source plugin. This is not
56		-obtainable via `pacman`, you must get *python-arpy* from AUR:
57		-https://aur.archlinux.org/packages/python-arpy/
58		-
59		-To install::
60		-
61		- wget https://aur.archlinux.org/cgit/aur.git/snapshot/python-arpy.tar.gz
62		- tar -xvf python-arpy.tar.gz
63		- cd python-arpy
64		- makepkg -si
65		-
66		-
67		-Debian
68		-~~~~~~
69		-Install the dependencies with::
70		-
71		- sudo apt-get install \
72		- python3 fuse bubblewrap \
73		- python3-pip python3-dev
74		-
75		-For the default plugins:
76		-
77		-
78		-Stretch
79		-+++++++
80		-With stretch, you first need to ensure that you have the backports repository
81		-setup as described `here <https://backports.debian.org/Instructions/>`_
82		-
83		-By adding the following line to your sources.list::
84		-
85		- deb http://deb.debian.org/debian stretch-backports main
86		-
87		-And then running::
88		-
89		- sudo apt update
90		-
91		-At this point you should be able to get the system requirements for the default plugins with::
92		-
93		- sudo apt install \
94		- bzr git lzip patch python3-arpy python3-gi
95		- sudo apt install -t stretch-backports \
96		- gir1.2-ostree-1.0 ostree
97		-
98		-
99		-Buster or Sid
100		-+++++++++++++
101		-For debian unstable or testing, only the following line should be enough
102		-to get the system requirements for the default plugins installed::
103		-
104		- sudo apt-get install \
105		- lzip gir1.2-ostree-1.0 git bzr ostree patch python3-arpy python3-gi
106		-
107		-
108		-Fedora
109		-~~~~~~
110		-For recent fedora systems, the following line should get you the system
111		-requirements you need::
112		-
113		- dnf install -y \
114		- python3 fuse bubblewrap \
115		- python3-pip python3-devel
116		-
117		-For the default plugins::
118		-
119		- dnf install -y \
120		- bzr git lzip patch ostree python3-gobject
121		- pip3 install --user arpy
122		-
123		-
124		-Ubuntu
125		-~~~~~~
126		-
127		-
128		-Ubuntu 18.04 LTS or later
129		-+++++++++++++++++++++++++
130		-Install the dependencies with::
131		-
132		- sudo apt install \
133		- python3 fuse bubblewrap \
134		- python3-pip python3-dev
135		-
136		-For the default plugins::
137		-
138		- sudo apt install \
139		- bzr gir1.2-ostree-1.0 git lzip ostree patch python3-arpy python3-gi
140		-
141		-
142		-Ubuntu 16.04 LTS
143		-++++++++++++++++
144		-On Ubuntu 16.04, neither `bubblewrap <https://github.com/projectatomic/bubblewrap/>`_
145		-or `ostree <https://github.com/ostreedev/ostree>`_ are available in the official repositories.
146		-You will need to install them in whichever way you see fit. Refer the the upstream documentation
147		-for advice on this.
148		-
149		-
150		-Installing
151		-----------
152		-Once you have the base system dependencies, you can install the BuildStream
153		-python package as a regular user.
154		-
155		-
156		-Installing from PyPI (recommended)
157		-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
158		-Since we only ever publish :ref:`release versions <install_semantic_versioning>` on
159		-PyPI, it is currently recommended to use this installation path. This will
160		-ensure that you always have the latest recommended version of BuildStream that
161		-we recommend.
162		-
163		-To install from PyPI, you will additionally require:
164		-
165		-* pip for python3 (only required for setup)
166		-* Python 3 development libraries and headers
167		-
168		-Simply run the following command::
169		-
170		- pip3 install --user BuildStream
171		-
172		-This will install latest stable version of BuildStream and its pure python
173		-dependencies into your user's homedir in ``~/.local``.
174		-
175		-Keep following the instructions below to ensure that the ``bst``
176		-command is in your ``PATH`` and to enable bash completions for it.
177		-
178		-.. note::
179		-
180		- If you want a specific version of BuildStream, you can install it using
181		- ``pip install --user BuildStream==<version-number>``
182		-
183		-
184		-Upgrading from PyPI
185		-+++++++++++++++++++
186		-Once you have already installed BuildStream from PyPI, you can later update
187		-to the latest recommended version like so::
188		-
189		- pip install --user --upgrade BuildStream
190		-
191		-
192		-.. _install_git_checkout:
193		-
194		-Installing from a git checkout
195		-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
196		-To install directly from the `git repository <https://gitlab.com/BuildStream/buildstream.git>`_
197		-using python's ``pip`` package manager, you will additionally require:
198		-
199		-* pip for python3 (only required for setup)
200		-* Python 3 development libraries and headers
201		-* git (to checkout BuildStream)
202		-
203		-Before installing, please check the existing tags in the git repository
204		-and determine which version you want to install, and whether you want
205		-to install an official release version (recommended), or a development snapshot
206		-to help us out testing the bleeding edge of development. Follow the
207		-:ref:`semantic versioning guide <install_semantic_versioning>` to determine
208		-which tag you intend to install.
209		-
210		-Run the following commands::
211		-
212		- git clone https://gitlab.com/BuildStream/buildstream.git
213		- cd buildstream
214		- git checkout <desired release tag>
215		- pip3 install --user -e .
216		-
217		-This will install buildstream's pure python dependencies into
218		-your user's homedir in ``~/.local`` and will run BuildStream directly
219		-from the git checkout directory.
220		-
221		-Keep following the instructions below to ensure that the ``bst``
222		-command is in your ``PATH`` and to enable bash completions for it.
223		-
224		-.. note::
225		-
226		- We recommend the ``-e`` option because you can upgrade your
227		- installation by simply updating the checked out git repository.
228		-
229		- If you want a full installation that is not linked to your
230		- git checkout, just omit the ``-e`` option from the above commands.
231		-
232		-
233		-Upgrading from a git checkout
234		-+++++++++++++++++++++++++++++
235		-If you installed BuildStream from a local git checkout using ``-e`` option, all
236		-you need to do to upgrade BuildStream is to update your local git checkout::
237		-
238		- cd /path/to/buildstream
239		- git pull --rebase
240		-
241		-If you did not specify the ``-e`` option at install time or the dependancies
242		-have changed, you will need to cleanly reinstall BuildStream::
243		-
244		- pip3 uninstall buildstream
245		- cd /path/to/buildstream
246		- git pull --rebase
247		- pip3 install --user .
248		-
249		-.. note::
250		-
251		- If BuildStream has added any dependencies since the last upgrade,
252		- you will need to uninstall and reinstall to ensure those dependencies
253		- are met, regardless of whether you have used the ``-e`` option at
254		- install time.
255		-
256		-
257		-Post install setup
258		-------------------
259		-After having installed from source using any of the above methods, some
260		-setup will be required to use BuildStream.
261		-
262		-
263		-Adjust PATH
264		-~~~~~~~~~~~
265		-Since BuildStream is now installed under your local user's install directories,
266		-you need to ensure that ``PATH`` is adjusted.
267		-
268		-A regular way to do this is to add the following line to the end of your ``~/.bashrc``::
269		-
270		- export PATH="${PATH}:${HOME}/.local/bin"
271		-
272		-.. note::
273		-
274		- You will have to restart your terminal in order for these changes to take effect.
275		-
276		-
277		-Bash completions
278		-~~~~~~~~~~~~~~~~
279		-Bash completions are supported by sourcing the ``buildstream/data/bst``
280		-script found in the BuildStream repository. On many systems this script
281		-can be installed into a completions directory but when installing BuildStream
282		-without a package manager this is not an option.
283		-
284		-To enable completions for an installation of BuildStream you
285		-installed yourself from git, just append the script verbatim
286		-to your ``~/.bash_completion``:
287		-
288		-.. literalinclude:: ../../buildstream/data/bst
289		- :language: yaml

doc/source/install_versions.rst deleted

---

1		-
2		-
3		-.. _install_semantic_versioning:
4		-
5		-Semantic Versioning
6		-===================
7		-BuildStream follows the Semantic Versioning Convention `(SemVer) <https://semver.org/>`_,
8		-and uses even minor point numbers to denote releases intended for users while
9		-odd minor point numbers represent development snapshops.
10		-
11		-For example, for a given version number ``X.Y.Z``
12		- * The ``X.<even number>.*`` versions are releases intended for users.
13		- * The ``X.<odd number>.*`` versions are development spanshots intended for testing.
14		-
15		-If you are :ref:`installing from git <install_git_checkout>`, please look for the latest
16		-tag to ensure you're getting the latest release.
17		-
18		-* Latest release:
19		-
20		- .. include:: release-badge.rst
21		-
22		-* Latest development snapshot:
23		-
24		- .. include:: snapshot-badge.rst

doc/source/main_install.rst deleted

---

1		-
2		-
3		-.. _install:
4		-
5		-Install
6		-=======
7		-
8		-.. include:: release-badge.rst
9		-
10		-.. include:: snapshot-badge.rst
11		-
12		-This section provides instructions for installing BuildStream and its
13		-companion artifact server on various platforms, along with any installation
14		-related materials.
15		-
16		-.. note::
17		-
18		- BuildStream is currently only supported natively on Linux. Users of Unix-like
19		- systems where Docker is available can still use BuildStream by following the
20		- :ref:`Docker install guide <docker>`
21		-
22		-.. toctree::
23		- :maxdepth: 1
24		-
25		- install_source
26		- install_linux_distro
27		- install_docker
28		- install_artifacts
29		- install_versions

doc/source/main_using.rst

---

...	...	@@ -15,3 +15,4 @@ guides and information on user preferences and configuration.
15	15	using_examples
16	16	using_config
17	17	using_commands
	18	+ using_configuring_artifact_server

doc/source/release-badge.rst deleted

---

1		-
2		-.. Use this file to include the badge in the documentation, but not in
3		- the README.rst or gitlab rendered materials, that doesnt work.
4		-
5		- This is partly a workaround for a sphinx issue, we will be able
6		- to avoid the raw html once this is implemented in sphinx:
7		-
8		- https://github.com/sphinx-doc/sphinx/issues/2240
9		-
10		- Using the <object> tag instead of the <img> tag which sphinx generates
11		- allows the svg to be "interactive", for us this basically means that
12		- the link we encode in the badge svg is used, rather than static urls
13		- which need to be used around the <img> tag.
14		-
15		- WARNING: The custom CSS on the style tag will need to change if we
16		- change the theme, so that the <object> tag behaves similar
17		- to how the <img> tag is themed by the style sheets.
18		-
19		-.. raw:: html
20		-
21		- <a class="reference external image-reference">
22		- <object style="margin-bottom:24px;vertical-align:middle"
23		- data=""
24		- type="image/svg+xml"/>
25		- </object>
26		- </a>

doc/source/snapshot-badge.rst deleted

---

1		-
2		-.. Use this file to include the badge in the documentation, but not in
3		- the README.rst or gitlab rendered materials, that doesnt work.
4		-
5		- This is partly a workaround for a sphinx issue, we will be able
6		- to avoid the raw html once this is implemented in sphinx:
7		-
8		- https://github.com/sphinx-doc/sphinx/issues/2240
9		-
10		- Using the <object> tag instead of the <img> tag which sphinx generates
11		- allows the svg to be "interactive", for us this basically means that
12		- the link we encode in the badge svg is used, rather than static urls
13		- which need to be used around the <img> tag.
14		-
15		- WARNING: The custom CSS on the style tag will need to change if we
16		- change the theme, so that the <object> tag behaves similar
17		- to how the <img> tag is themed by the style sheets.
18		-
19		-.. raw:: html
20		-
21		- <a class="reference external image-reference">
22		- <object style="margin-bottom:24px;vertical-align:middle"
23		- data=""
24		- type="image/svg+xml"/>
25		- </object>
26		- </a>

doc/source/install_artifacts.rst → doc/source/using_configuring_artifact_server.rst

---

...	...	@@ -2,8 +2,8 @@
2	2
3	3	.. _artifacts:
4	4
5		-Installing an artifact server
6		-=============================
	5	+Configuring Artifact Server
	6	+===========================
7	7	BuildStream caches the results of builds in a local artifact cache, and will
8	8	avoid building an element if there is a suitable build already present in the
9	9	local artifact cache.
...	...	@@ -72,7 +72,7 @@ Installing the server
72	72	~~~~~~~~~~~~~~~~~~~~~
73	73	You will also need to install BuildStream on the artifact server in order
74	74	to receive uploaded artifacts over ssh. Follow the instructions for installing
75		-BuildStream :ref:`here <install>`
	75	+BuildStream `here <https://buildstream.build/install.html>`_.
76	76
77	77	When installing BuildStream on the artifact server, it must be installed
78	78	in a system wide location, with ``pip3 install .`` in the BuildStream

tests/integration/build-tree.py

---

	1	+import os
	2	+import pytest
	3	+import shutil
	4	+
	5	+from tests.testutils import cli, cli_integration, create_artifact_share
	6	+
	7	+
	8	+pytestmark = pytest.mark.integration
	9	+
	10	+
	11	+DATA_DIR = os.path.join(
	12	+ os.path.dirname(os.path.realpath(__file__)),
	13	+ "project"
	14	+)
	15	+
	16	+
	17	+@pytest.mark.datafiles(DATA_DIR)
	18	+def test_buildtree_staged(cli_integration, tmpdir, datafiles):
	19	+ # i.e. tests that cached build trees are staged by `bst shell --build`
	20	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	21	+ element_name = 'build-shell/buildtree.bst'
	22	+
	23	+ res = cli_integration.run(project=project, args=['build', element_name])
	24	+ res.assert_success()
	25	+
	26	+ res = cli_integration.run(project=project, args=[
	27	+ 'shell', '--build', element_name, '--', 'grep', '-q', 'Hi', 'test'
	28	+ ])
	29	+ res.assert_success()
	30	+
	31	+
	32	+# Check that build shells work when pulled from a remote cache
	33	+# This is to roughly simulate remote execution
	34	+@pytest.mark.datafiles(DATA_DIR)
	35	+def test_buildtree_pulled(cli, tmpdir, datafiles):
	36	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	37	+ element_name = 'build-shell/buildtree.bst'
	38	+
	39	+ with create_artifact_share(os.path.join(str(tmpdir), 'artifactshare')) as share:
	40	+ # Build the element to push it to cache
	41	+ cli.configure({
	42	+ 'artifacts': {'url': share.repo, 'push': True}
	43	+ })
	44	+ result = cli.run(project=project, args=['build', element_name])
	45	+ result.assert_success()
	46	+ assert cli.get_element_state(project, element_name) == 'cached'
	47	+
	48	+ # Discard the cache
	49	+ cli.configure({
	50	+ 'artifacts': {'url': share.repo, 'push': True},
	51	+ 'artifactdir': os.path.join(cli.directory, 'artifacts2')
	52	+ })
	53	+ assert cli.get_element_state(project, element_name) != 'cached'
	54	+
	55	+ # Pull from cache
	56	+ result = cli.run(project=project, args=['pull', '--deps', 'all', element_name])
	57	+ result.assert_success()
	58	+
	59	+ # Check it's using the cached build tree
	60	+ res = cli.run(project=project, args=[
	61	+ 'shell', '--build', element_name, '--', 'grep', '-q', 'Hi', 'test'
	62	+ ])
	63	+ res.assert_success()

tests/integration/project/elements/build-shell/buildtree.bst

---

	1	+kind: manual
	2	+description: |
	3	+ Puts a file in the build tree so that build tree caching and staging can be tested.
	4	+
	5	+depends:
	6	+ - filename: base.bst
	7	+ type: build
	8	+
	9	+config:
	10	+ build-commands:
	11	+ - "echo 'Hi' > %{build-root}/test"

tests/integration/shell.py

---

...	...	@@ -302,46 +302,33 @@ def test_workspace_visible(cli, tmpdir, datafiles):
302	302	assert result.output == workspace_hello
303	303
304	304
305		-# Test that we can see the workspace files in a shell
306		-@pytest.mark.integration
	305	+# Test that '--sysroot' works
307	306	@pytest.mark.datafiles(DATA_DIR)
308		-def test_sysroot_workspace_visible(cli, tmpdir, datafiles):
	307	+def test_sysroot(cli, tmpdir, datafiles):
309	308	project = os.path.join(datafiles.dirname, datafiles.basename)
310		- workspace = os.path.join(cli.directory, 'workspace')
311		- element_name = 'workspace/workspace-mount-fail.bst'
312		-
313		- # Open a workspace on our build failing element
314		- #
315		- res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
316		- assert res.exit_code == 0
317		-
318		- # Ensure the dependencies of our build failing element are built
319		- result = cli.run(project=project, args=['build', element_name])
320		- result.assert_main_error(ErrorDomain.STREAM, None)
321		-
322		- # Discover the sysroot of the failed build directory, after one
323		- # failed build, there should be only one directory there.
324		- #
325		- build_base = os.path.join(cli.directory, 'build')
326		- build_dirs = os.listdir(path=build_base)
327		- assert len(build_dirs) == 1
328		- build_dir = os.path.join(build_base, build_dirs[0])
329		-
330		- # Obtain a copy of the hello.c content from the workspace
331		- #
332		- workspace_hello_path = os.path.join(cli.directory, 'workspace', 'hello.c')
333		- assert os.path.exists(workspace_hello_path)
334		- with open(workspace_hello_path, 'r') as f:
335		- workspace_hello = f.read()
336		-
337		- # Cat the hello.c file from a bst shell command, and assert
338		- # that we got the same content here
339		- #
340		- result = cli.run(project=project, args=[
341		- 'shell', '--build', '--sysroot', build_dir, element_name, '--', 'cat', 'hello.c'
	309	+ base_element = "base/base-alpine.bst"
	310	+ # test element only needs to be something lightweight for this test
	311	+ test_element = "script/script.bst"
	312	+ checkout_dir = os.path.join(str(tmpdir), 'alpine-sysroot')
	313	+ test_file = 'hello'
	314	+
	315	+ # Build and check out a sysroot
	316	+ res = cli.run(project=project, args=['build', base_element])
	317	+ res.assert_success()
	318	+ res = cli.run(project=project, args=['checkout', base_element, checkout_dir])
	319	+ res.assert_success()
	320	+
	321	+ # Mutate the sysroot
	322	+ test_path = os.path.join(checkout_dir, test_file)
	323	+ with open(test_path, 'w') as f:
	324	+ f.write('hello\n')
	325	+
	326	+ # Shell into the sysroot and check the test file exists
	327	+ res = cli.run(project=project, args=[
	328	+ 'shell', '--build', '--sysroot', checkout_dir, test_element, '--',
	329	+ 'grep', '-q', 'hello', '/' + test_file
342	330	])
343		- assert result.exit_code == 0
344		- assert result.output == workspace_hello
	331	+ res.assert_success()
345	332
346	333
347	334	# Test system integration commands can access devices in /dev

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.