[Notes] [Git][BuildStream/buildstream][jonathan/debug-remote-failed-builds] 11 commits: doc: Add new document about Docker integrations

[Jonathan Maw <gitlab mg gitlab com>]

Title: GitLab

Jonathan Maw pushed to branch jonathan/debug-remote-failed-builds at BuildStream / buildstream

Commits:

• cda1354c
  by Chandan Singh at 2018-10-10T00:17:51Z

  doc: Add new document about Docker integrations
  
  Add a new section under "Additional writings" section about how
  BuildStream integrates with Docker. At present, this section includes
  only two sections:
  
  - "Run BuildStream inside Docker", that refers to existing instructions
  for doing so
  - "Generate Docker images", that adds instructions for using the newly
  added `contrib/bst-docker-import` script
  
  In future, this document can be extended to include the following
  things:
  
  - Importing Docker sources via `docker` source plugin - if/when the
  Docker source plguin is moved from bst-external to BuildStream proper
  - A Docker sandbox, if/when that is added
  

• dd472d95
  by Chandan Singh at 2018-10-10T00:18:17Z

  NEWS: Add entry about contrib/bst-docker-import
  
  As suggested in
  https://gitlab.com/BuildStream/buildstream/merge_requests/857#note_106809743,
  add a NEWS entry for `contrib/bst-docker-import` that was added
  recently.
  

• 35ac26a7
  by Chandan Singh at 2018-10-10T00:59:17Z

  Merge branch 'chandan/bst-and-docker' into 'master'
  
  Add documentation and NEWS entry about bst-docker-import
  
  See merge request BuildStream/buildstream!864

• 7d96333f
  by Laurence Urhegyi at 2018-10-10T19:24:14Z

  README: Update to add link to website
  

• 39492db8
  by devcurmudgeon at 2018-10-10T19:24:14Z

  Merge branch 'patch-2' into 'master'
  
  README: Update to add link to website
  
  See merge request BuildStream/buildstream!859

• 41966d38
  by Jonathan Maw at 2018-10-11T14:43:32Z

  element.py: Always clean up the rootdir
  
  We shouldn't need it to persist now that we cache failed build dirs.
  
  This change breaks the test
  `tests/integration/shell.py::test_sysroot_workspace_visible`.
  I can no longer see a use-case for this test.
  AIUI, it tested that the failed build sysroot stored in the builddir has
  the workspace's files in, despite the workspace being unmounted.
  I believe this behaviour is made redundant by cached buildtrees.
  
  This fixes part of #539
  

• a7d1e4c8
  by Jonathan Maw at 2018-10-11T14:43:32Z

  Element: Use cached buildtree in build shells and failure shells
  
  This includes changes in app.py:
  * Interactive failure shell no longer uses the failed build sysroot,
    defaulting to the cached build tree.
  * Always allow producing a failure shell.
  
  Changes in element.py are:
  * Errors caused by building don't store the failed build sysroot.
  * When staging sources, will stage the element's cached build tree if it
    exists.
  
  This fixes part of #539
  

• e77e2755
  by Jonathan Maw at 2018-10-11T14:43:32Z

  tests: Add test that cached build trees are staged in build shells
  
  This is related to #539
  

• 033a3b73
  by Jonathan Maw at 2018-10-11T14:43:32Z

  sandbox.py: Remove redundant Sandbox.__directory
  

• aebc8b94
  by Jonathan Maw at 2018-10-11T14:43:32Z

  element: Make "--sysroot" take a bare directory
  
  i.e. instead of taking a directory that must contain "root" and
  "scratch", and treating "root" as the root, use the directory directly.
  
  In element.py:
  * __sandbox takes the `bare_sandbox` arg, to pass into the sandbox's constructor
  
  In sandbox.py:
  * If bare_sandbox, `_root` is the passed-in directory, and `__scratch`
    is None.
  * Trying to use `__scratch` when bare_sandbox is True is a bug.
  
  In _mount.py:
  * Don't get the value of `__scratch` if it's not needed.
  
  This is part of #539
  

• 1ab82354
  by Jonathan Maw at 2018-10-11T14:43:32Z

  tests: Add tests for 'bst shell --sysroot'
  
  This is related to #539
  

11 changed files:

• NEWS
• README.rst
• buildstream/_frontend/app.py
• buildstream/element.py
• buildstream/sandbox/_mount.py
• buildstream/sandbox/sandbox.py
• + doc/source/additional_docker.rst
• doc/source/core_additional.rst
• + tests/integration/build-tree.py
• + tests/integration/project/elements/build-shell/buildtree.bst
• tests/integration/shell.py

Changes:

NEWS

---

...	...	@@ -24,6 +24,9 @@ buildstream 1.3.1
24	24	o Add new `pip` source plugin for downloading python packages using pip,
25	25	based on requirements files from previous sources.
26	26
	27	+ o Generate Docker images from built artifacts using
	28	+ `contrib/bst-docker-import` script.
	29	+
27	30
28	31	=================
29	32	buildstream 1.1.5

README.rst

---

...	...	@@ -19,7 +19,8 @@ About
19	19
20	20	What is BuildStream?
21	21	====================
22		-BuildStream is a Free Software tool for building/integrating software stacks.
	22	+`BuildStream <https://buildstream.build>`_ is a Free Software tool for
	23	+building/integrating software stacks.
23	24	It takes inspiration, lessons and use-cases from various projects including
24	25	OBS, Reproducible Builds, Yocto, Baserock, Buildroot, Aboriginal, GNOME Continuous,
25	26	JHBuild, Flatpak Builder and Android repo.

buildstream/_frontend/app.py

---

...	...	@@ -564,18 +564,15 @@ class App():
564	564	" (c)ontinue - Continue queueing jobs as much as possible\n" +
565	565	" (q)uit - Exit after all ongoing jobs complete\n" +
566	566	" (t)erminate - Terminate any ongoing jobs and exit\n" +
567		- " (r)etry - Retry this job\n")
	567	+ " (r)etry - Retry this job\n" +
	568	+ " (s)hell - Drop into a shell in the failed build sandbox\n")
568	569	if failure.logfile:
569	570	summary += " (l)og - View the full log file\n"
570		- if failure.sandbox:
571		- summary += " (s)hell - Drop into a shell in the failed build sandbox\n"
572	571	summary += "\nPressing ^C will terminate jobs and exit\n"
573	572
574		- choices = ['continue', 'quit', 'terminate', 'retry']
	573	+ choices = ['continue', 'quit', 'terminate', 'retry', 'shell']
575	574	if failure.logfile:
576	575	choices += ['log']
577		- if failure.sandbox:
578		- choices += ['shell']
579	576
580	577	choice = ''
581	578	while choice not in ['continue', 'quit', 'terminate', 'retry']:
...	...	@@ -595,10 +592,10 @@ class App():
595	592	# Handle choices which you can come back from
596	593	#
597	594	if choice == 'shell':
598		- click.echo("\nDropping into an interactive shell in the failed build sandbox\n", err=True)
	595	+ click.echo("\nDropping into an interactive shell in the failed build tree\n", err=True)
599	596	try:
600	597	prompt = self.shell_prompt(element)
601		- self.stream.shell(element, Scope.BUILD, prompt, directory=failure.sandbox, isolate=True)
	598	+ self.stream.shell(element, Scope.BUILD, prompt, isolate=True)
602	599	except BstError as e:
603	600	click.echo("Error while attempting to create interactive shell: {}".format(e), err=True)
604	601	elif choice == 'log':

buildstream/element.py

---

...	...	@@ -1317,7 +1317,9 @@ class Element(Plugin):
1317	1317	@contextmanager
1318	1318	def _prepare_sandbox(self, scope, directory, deps='run', integrate=True):
1319	1319	# bst shell and bst checkout require a local sandbox.
1320		- with self.__sandbox(directory, config=self.__sandbox_config, allow_remote=False) as sandbox:
	1320	+ bare_directory = True if directory else False
	1321	+ with self.__sandbox(directory, config=self.__sandbox_config, allow_remote=False,
	1322	+ bare_directory=True if directory else False) as sandbox:
1321	1323
1322	1324	# Configure always comes first, and we need it.
1323	1325	self.configure_sandbox(sandbox)
...	...	@@ -1384,6 +1386,7 @@ class Element(Plugin):
1384	1386	# the same filing system as the rest of our cache.
1385	1387	temp_staging_location = os.path.join(self._get_context().artifactdir, "staging_temp")
1386	1388	temp_staging_directory = tempfile.mkdtemp(prefix=temp_staging_location)
	1389	+ import_dir = temp_staging_directory
1387	1390
1388	1391	try:
1389	1392	workspace = self._get_workspace()
...	...	@@ -1394,12 +1397,16 @@ class Element(Plugin):
1394	1397	with self.timed_activity("Staging local files at {}"
1395	1398	.format(workspace.get_absolute_path())):
1396	1399	workspace.stage(temp_staging_directory)
	1400	+ elif self._cached():
	1401	+ # We have a cached buildtree to use, instead
	1402	+ artifact_base, _ = self.__extract()
	1403	+ import_dir = os.path.join(artifact_base, 'buildtree')
1397	1404	else:
1398	1405	# No workspace, stage directly
1399	1406	for source in self.sources():
1400	1407	source._stage(temp_staging_directory)
1401	1408
1402		- vdirectory.import_files(temp_staging_directory)
	1409	+ vdirectory.import_files(import_dir)
1403	1410
1404	1411	finally:
1405	1412	# Staging may produce directories with less than 'rwx' permissions
...	...	@@ -1565,10 +1572,6 @@ class Element(Plugin):
1565	1572	collect = self.assemble(sandbox)
1566	1573	self.__set_build_result(success=True, description="succeeded")
1567	1574	except BstError as e:
1568		- # If an error occurred assembling an element in a sandbox,
1569		- # then tack on the sandbox directory to the error
1570		- e.sandbox = rootdir
1571		-
1572	1575	# If there is a workspace open on this element, it will have
1573	1576	# been mounted for sandbox invocations instead of being staged.
1574	1577	#
...	...	@@ -1682,8 +1685,8 @@ class Element(Plugin):
1682	1685	"unable to collect artifact contents"
1683	1686	.format(collect))
1684	1687
1685		- # Finally cleanup the build dir
1686		- cleanup_rootdir()
	1688	+ # Finally cleanup the build dir
	1689	+ cleanup_rootdir()
1687	1690
1688	1691	return artifact_size
1689	1692
...	...	@@ -2151,12 +2154,14 @@ class Element(Plugin):
2151	2154	# stderr (fileobject): The stream for stderr for the sandbox
2152	2155	# config (SandboxConfig): The SandboxConfig object
2153	2156	# allow_remote (bool): Whether the sandbox is allowed to be remote
	2157	+ # bare_directory (bool): Whether the directory is bare i.e. doesn't have
	2158	+ # a separate 'root' subdir
2154	2159	#
2155	2160	# Yields:
2156	2161	# (Sandbox): A usable sandbox
2157	2162	#
2158	2163	@contextmanager
2159		- def __sandbox(self, directory, stdout=None, stderr=None, config=None, allow_remote=True):
	2164	+ def __sandbox(self, directory, stdout=None, stderr=None, config=None, allow_remote=True, bare_directory=False):
2160	2165	context = self._get_context()
2161	2166	project = self._get_project()
2162	2167	platform = Platform.get_platform()
...	...	@@ -2187,6 +2192,7 @@ class Element(Plugin):
2187	2192	stdout=stdout,
2188	2193	stderr=stderr,
2189	2194	config=config,
	2195	+ bare_directory=bare_directory,
2190	2196	allow_real_directory=not self.BST_VIRTUAL_DIRECTORY)
2191	2197	yield sandbox
2192	2198
...	...	@@ -2196,7 +2202,7 @@ class Element(Plugin):
2196	2202
2197	2203	# Recursive contextmanager...
2198	2204	with self.__sandbox(rootdir, stdout=stdout, stderr=stderr, config=config,
2199		- allow_remote=allow_remote) as sandbox:
	2205	+ allow_remote=allow_remote, bare_directory=False) as sandbox:
2200	2206	yield sandbox
2201	2207
2202	2208	# Cleanup the build dir

buildstream/sandbox/_mount.py

---

...	...	@@ -31,7 +31,6 @@ from .._fuse import SafeHardlinks
31	31	#
32	32	class Mount():
33	33	def __init__(self, sandbox, mount_point, safe_hardlinks, fuse_mount_options={}):
34		- scratch_directory = sandbox._get_scratch_directory()
35	34	# Getting _get_underlying_directory() here is acceptable as
36	35	# we're part of the sandbox code. This will fail if our
37	36	# directory is CAS-based.
...	...	@@ -51,6 +50,7 @@ class Mount():
51	50	# a regular mount point within the parent's redirected mount.
52	51	#
53	52	if self.safe_hardlinks:
	53	+ scratch_directory = sandbox._get_scratch_directory()
54	54	# Redirected mount
55	55	self.mount_origin = os.path.join(root_directory, mount_point.lstrip(os.sep))
56	56	self.mount_base = os.path.join(scratch_directory, utils.url_directory_name(mount_point))

buildstream/sandbox/sandbox.py

---

...	...	@@ -98,16 +98,23 @@ class Sandbox():
98	98	self.__config = kwargs['config']
99	99	self.__stdout = kwargs['stdout']
100	100	self.__stderr = kwargs['stderr']
	101	+ self.__bare_directory = kwargs['bare_directory']
101	102
102	103	# Setup the directories. Root and output_directory should be
103	104	# available to subclasses, hence being single-underscore. The
104	105	# others are private to this class.
105		- self._root = os.path.join(directory, 'root')
	106	+ # If the directory is bare, it probably doesn't need scratch
	107	+ if self.__bare_directory:
	108	+ self._root = directory
	109	+ self.__scratch = None
	110	+ os.makedirs(self._root, exist_ok=True)
	111	+ else:
	112	+ self._root = os.path.join(directory, 'root')
	113	+ self.__scratch = os.path.join(directory, 'scratch')
	114	+ for directory_ in [self._root, self.__scratch]:
	115	+ os.makedirs(directory_, exist_ok=True)
	116	+
106	117	self._output_directory = None
107		- self.__directory = directory
108		- self.__scratch = os.path.join(self.__directory, 'scratch')
109		- for directory_ in [self._root, self.__scratch]:
110		- os.makedirs(directory_, exist_ok=True)
111	118	self._vdir = None
112	119
113	120	# This is set if anyone requests access to the underlying
...	...	@@ -332,6 +339,7 @@ class Sandbox():
332	339	# Returns:
333	340	# (str): The sandbox scratch directory
334	341	def _get_scratch_directory(self):
	342	+ assert not self.__bare_directory, "Scratch is not going to work with bare directories"
335	343	return self.__scratch
336	344
337	345	# _get_output()

doc/source/additional_docker.rst

---

	1	+
	2	+.. _bst_and_docker:
	3	+
	4	+
	5	+BuildStream and Docker
	6	+======================
	7	+
	8	+BuildStream integrates with Docker in multiple ways. Here are some ways in
	9	+which these integrations work.
	10	+
	11	+Run BuildStream inside Docker
	12	+-----------------------------
	13	+
	14	+Refer to the :ref:`BuildStream inside Docker <docker>` documentation for
	15	+instructions on how to run BuildStream as a Docker container.
	16	+
	17	+Generate Docker images
	18	+----------------------
	19	+
	20	+The
	21	+`bst-docker-import script <https://gitlab.com/BuildStream/buildstream/blob/master/contrib/bst-docker-import>`_
	22	+can be used to generate a Docker image from built artifacts.
	23	+
	24	+You can download it and make it executable like this:
	25	+
	26	+.. code:: bash
	27	+
	28	+ mkdir -p ~/.local/bin
	29	+ curl --get https://gitlab.com/BuildStream/buildstream/raw/master/contrib/bst-docker-import > ~/.local/bin/bst-docker-import
	30	+ chmod +x ~/.local/bin/bst-docker-import
	31	+
	32	+Check if ``~/.local/bin`` appears in your PATH environment variable -- if it
	33	+doesn't, you should
	34	+`edit your ~/.profile so that it does <https://stackoverflow.com/questions/14637979/>`_.
	35	+
	36	+Once the script is available in your PATH and assuming you have Docker
	37	+installed, you can start using the ``bst-docker-import`` script. Here is a
	38	+minimal example to generate an image called ``bst-hello`` from an element
	39	+called ``hello.bst`` assuming it is already built:
	40	+
	41	+.. code:: bash
	42	+
	43	+ bst-docker-import -t bst-hello hello.bst
	44	+
	45	+This script can also be used if you are running BuildStream inside Docker. In
	46	+this case, you will need to supply the command that you are using to run
	47	+BuildStream using the ``-c`` option. If you are using the
	48	+`bst-here wrapper script <https://gitlab.com/BuildStream/buildstream/blob/master/contrib/bst-here>`_,
	49	+you can achieve the same results as the above example like this:
	50	+
	51	+.. code:: bash
	52	+
	53	+ bst-docker-import -c bst-here -t bst-hello hello.bst

doc/source/core_additional.rst

---

...	...	@@ -8,3 +8,4 @@ Additional writings
8	8
9	9	additional_cachekeys
10	10	additional_sandboxing
	11	+ additional_docker

tests/integration/build-tree.py

---

	1	+import os
	2	+import pytest
	3	+
	4	+from tests.testutils import cli_integration as cli
	5	+
	6	+
	7	+pytestmark = pytest.mark.integration
	8	+
	9	+
	10	+DATA_DIR = os.path.join(
	11	+ os.path.dirname(os.path.realpath(__file__)),
	12	+ "project"
	13	+)
	14	+
	15	+
	16	+@pytest.mark.datafiles(DATA_DIR)
	17	+def test_buildtree_staged(cli, tmpdir, datafiles):
	18	+ # i.e. tests that cached build trees are staged by `bst shell --build`
	19	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	20	+ element_path = os.path.join(project, 'elements')
	21	+ element_name = 'build-shell/buildtree.bst'
	22	+
	23	+ res = cli.run(project=project, args=['build', element_name])
	24	+ res.assert_success()
	25	+
	26	+ res = cli.run(project=project, args=['shell', '--build', element_name, '--', 'grep', '-q', 'Hi', 'test'])
	27	+ res.assert_success()

tests/integration/project/elements/build-shell/buildtree.bst

---

	1	+kind: manual
	2	+description: |
	3	+ Puts a file in the build tree so that build tree caching and staging can be tested.
	4	+
	5	+depends:
	6	+ - filename: base.bst
	7	+ type: build
	8	+
	9	+config:
	10	+ build-commands:
	11	+ - "echo 'Hi' > %{build-root}/test"

tests/integration/shell.py

---

...	...	@@ -302,46 +302,33 @@ def test_workspace_visible(cli, tmpdir, datafiles):
302	302	assert result.output == workspace_hello
303	303
304	304
305		-# Test that we can see the workspace files in a shell
306		-@pytest.mark.integration
	305	+# Test that '--sysroot' works
307	306	@pytest.mark.datafiles(DATA_DIR)
308		-def test_sysroot_workspace_visible(cli, tmpdir, datafiles):
	307	+def test_sysroot(cli, tmpdir, datafiles):
309	308	project = os.path.join(datafiles.dirname, datafiles.basename)
310		- workspace = os.path.join(cli.directory, 'workspace')
311		- element_name = 'workspace/workspace-mount-fail.bst'
312		-
313		- # Open a workspace on our build failing element
314		- #
315		- res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
316		- assert res.exit_code == 0
317		-
318		- # Ensure the dependencies of our build failing element are built
319		- result = cli.run(project=project, args=['build', element_name])
320		- result.assert_main_error(ErrorDomain.STREAM, None)
321		-
322		- # Discover the sysroot of the failed build directory, after one
323		- # failed build, there should be only one directory there.
324		- #
325		- build_base = os.path.join(cli.directory, 'build')
326		- build_dirs = os.listdir(path=build_base)
327		- assert len(build_dirs) == 1
328		- build_dir = os.path.join(build_base, build_dirs[0])
329		-
330		- # Obtain a copy of the hello.c content from the workspace
331		- #
332		- workspace_hello_path = os.path.join(cli.directory, 'workspace', 'hello.c')
333		- assert os.path.exists(workspace_hello_path)
334		- with open(workspace_hello_path, 'r') as f:
335		- workspace_hello = f.read()
336		-
337		- # Cat the hello.c file from a bst shell command, and assert
338		- # that we got the same content here
339		- #
340		- result = cli.run(project=project, args=[
341		- 'shell', '--build', '--sysroot', build_dir, element_name, '--', 'cat', 'hello.c'
	309	+ base_element = "base/base-alpine.bst"
	310	+ # test element only needs to be something lightweight for this test
	311	+ test_element = "script/script.bst"
	312	+ checkout_dir = os.path.join(str(tmpdir), 'alpine-sysroot')
	313	+ test_file = 'hello'
	314	+
	315	+ # Build and check out a sysroot
	316	+ res = cli.run(project=project, args=['build', base_element])
	317	+ res.assert_success()
	318	+ res = cli.run(project=project, args=['checkout', base_element, checkout_dir])
	319	+ res.assert_success()
	320	+
	321	+ # Mutate the sysroot
	322	+ test_path = os.path.join(checkout_dir, test_file)
	323	+ with open(test_path, 'w') as f:
	324	+ f.write('hello\n')
	325	+
	326	+ # Shell into the sysroot and check the test file exists
	327	+ res = cli.run(project=project, args=[
	328	+ 'shell', '--build', '--sysroot', checkout_dir, test_element, '--',
	329	+ 'grep', '-q', 'hello', '/' + test_file
342	330	])
343		- assert result.exit_code == 0
344		- assert result.output == workspace_hello
	331	+ res.assert_success()
345	332
346	333
347	334	# Test system integration commands can access devices in /dev

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.