Jonathan Maw pushed to branch jonathan/debug-remote-failed-builds at BuildStream / buildstream
Commits:
-
c31ed138
by Tristan Van Berkom at 2018-10-14T15:24:53Z
-
b39d87b4
by Jürg Billeter at 2018-10-15T11:07:05Z
-
4e3ec89e
by Jürg Billeter at 2018-10-15T11:48:16Z
-
1f12658d
by Jonathan Maw at 2018-10-15T12:52:00Z
-
c73de158
by Jonathan Maw at 2018-10-15T12:52:00Z
-
03f5c33e
by Jonathan Maw at 2018-10-15T12:52:00Z
-
025488c0
by Jonathan Maw at 2018-10-15T12:52:00Z
-
7b178788
by Jonathan Maw at 2018-10-15T12:52:00Z
-
fd8d3d25
by Jonathan Maw at 2018-10-15T12:52:00Z
-
88ed3990
by Jonathan Maw at 2018-10-15T12:52:00Z
-
9827a94f
by Jonathan Maw at 2018-10-15T12:52:01Z
13 changed files:
- CONTRIBUTING.rst
- NEWS
- buildstream/_artifactcache/cascache.py
- buildstream/_frontend/app.py
- buildstream/element.py
- buildstream/sandbox/_mount.py
- buildstream/sandbox/sandbox.py
- doc/source/main_using.rst
- doc/source/using_examples.rst
- doc/source/using_tutorial.rst
- + tests/integration/build-tree.py
- + tests/integration/project/elements/build-shell/buildtree.bst
- tests/integration/shell.py
Changes:
... | ... | @@ -1147,7 +1147,7 @@ To actually regenerate the code:: |
1147 | 1147 |
Documenting
|
1148 | 1148 |
-----------
|
1149 | 1149 |
BuildStream starts out as a documented project from day one and uses
|
1150 |
-sphinx to document itself.
|
|
1150 |
+`sphinx <www.sphinx-doc.org>`_ to document itself.
|
|
1151 | 1151 |
|
1152 | 1152 |
This section discusses formatting policies for editing files in the
|
1153 | 1153 |
``doc/source`` directory, and describes the details of how the docs are
|
... | ... | @@ -1194,10 +1194,8 @@ The BuildStream documentation style is as follows: |
1194 | 1194 |
the target, e.g.: ``:ref:`Link text <anchor_name>```.
|
1195 | 1195 |
Note that the "_" prefix is not used when referring to the target.
|
1196 | 1196 |
|
1197 |
-Useful links:
|
|
1198 |
- |
|
1199 |
-For further information, please see the `Sphinx Documentation
|
|
1200 |
-<http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_.
|
|
1197 |
+For further information about using the reStructuredText with sphinx, please see the
|
|
1198 |
+`Sphinx Documentation <http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html>`_.
|
|
1201 | 1199 |
|
1202 | 1200 |
|
1203 | 1201 |
Building Docs
|
... | ... | @@ -1266,10 +1264,101 @@ the ``man/`` subdirectory, which will be automatically included |
1266 | 1264 |
in the buildstream distribution.
|
1267 | 1265 |
|
1268 | 1266 |
|
1269 |
-Documentation Examples
|
|
1270 |
-~~~~~~~~~~~~~~~~~~~~~~
|
|
1271 |
-The examples section of the documentation contains a series of standalone
|
|
1272 |
-examples, here are the criteria for an example addition.
|
|
1267 |
+User guide
|
|
1268 |
+~~~~~~~~~~
|
|
1269 |
+The :ref:`user guide <using>` is comprised of free form documentation
|
|
1270 |
+in manually written ``.rst`` files and is split up into a few sections,
|
|
1271 |
+of main interest are the :ref:`tutorial <tutorial>` and the
|
|
1272 |
+:ref:`examples <examples>`.
|
|
1273 |
+ |
|
1274 |
+The distinction of the two categories of user guides is important to
|
|
1275 |
+understand too.
|
|
1276 |
+ |
|
1277 |
+* **Tutorial**
|
|
1278 |
+ |
|
1279 |
+ The tutorial is structured as a series of exercises which start with
|
|
1280 |
+ the most basic concepts and build upon the previous chapters in order
|
|
1281 |
+ to arrive at a basic understanding of how to create BuildStream projects.
|
|
1282 |
+ |
|
1283 |
+ This series of examples should be easy enough to complete in a matter
|
|
1284 |
+ of a few hours for a new user, and should provide just enough insight to
|
|
1285 |
+ get the user started in creating their own projects.
|
|
1286 |
+ |
|
1287 |
+ Going through the tutorial step by step should also result in the user
|
|
1288 |
+ becoming proficient enough with the reference manual to get by on their own.
|
|
1289 |
+ |
|
1290 |
+* **Examples**
|
|
1291 |
+ |
|
1292 |
+ These exist to demonstrate how to accomplish more advanced tasks which
|
|
1293 |
+ are not always obvious and discoverable.
|
|
1294 |
+ |
|
1295 |
+ Alternatively, these also demonstrate elegant and recommended ways of
|
|
1296 |
+ accomplishing some tasks which could be done in various ways.
|
|
1297 |
+ |
|
1298 |
+ |
|
1299 |
+Guidelines
|
|
1300 |
+''''''''''
|
|
1301 |
+Here are some general guidelines for adding new free form documentation
|
|
1302 |
+to the user guide.
|
|
1303 |
+ |
|
1304 |
+* **Focus on a single subject**
|
|
1305 |
+ |
|
1306 |
+ It is important to stay focused on a single subject and avoid getting
|
|
1307 |
+ into tangential material when creating a new entry, so that the articles
|
|
1308 |
+ remain concise and the user is not distracted by unrelated subject material.
|
|
1309 |
+ |
|
1310 |
+ A single tutorial chapter or example should not introduce any additional
|
|
1311 |
+ subject material than the material being added for the given example.
|
|
1312 |
+ |
|
1313 |
+* **Reuse existing sample project elements**
|
|
1314 |
+ |
|
1315 |
+ To help avoid distracting from the topic at hand, it is always preferable to
|
|
1316 |
+ reuse the same project sample material from other examples and only deviate
|
|
1317 |
+ slightly to demonstrate the new material, than to create completely new projects.
|
|
1318 |
+ |
|
1319 |
+ This helps us remain focused on a single topic at a time, and reduces the amount
|
|
1320 |
+ of unrelated material the reader needs to learn in order to digest the new
|
|
1321 |
+ example.
|
|
1322 |
+ |
|
1323 |
+* **Don't be redundant**
|
|
1324 |
+ |
|
1325 |
+ When something has already been explained in the tutorial or in another example,
|
|
1326 |
+ it is best to simply refer to the other user guide entry in a new example.
|
|
1327 |
+ |
|
1328 |
+ Always prefer to link to the tutorial if an explanation exists in the tutorial,
|
|
1329 |
+ rather than linking to another example, where possible.
|
|
1330 |
+ |
|
1331 |
+* **Link into the reference manual at every opportunity**
|
|
1332 |
+ |
|
1333 |
+ The format and plugin API is 100% documented at all times. Whenever discussing
|
|
1334 |
+ anything about the format or plugin API, always do so while providing a link
|
|
1335 |
+ into the more terse reference material.
|
|
1336 |
+ |
|
1337 |
+ We don't want users to have to search for the material themselves, and we also
|
|
1338 |
+ want the user to become proficient at navigating the reference material over
|
|
1339 |
+ time.
|
|
1340 |
+ |
|
1341 |
+* **Use concise terminology**
|
|
1342 |
+ |
|
1343 |
+ As developers, we tend to come up with code names for features we develop, and
|
|
1344 |
+ then end up documenting a new feature in an example.
|
|
1345 |
+ |
|
1346 |
+ Never use a code name or shorthand to refer to a feature in the user guide, instead
|
|
1347 |
+ always use fully qualified sentences outlining very explicitly what we are doing
|
|
1348 |
+ in the example, or what the example is for in the case of a title.
|
|
1349 |
+ |
|
1350 |
+ We need to be considerate that the audience of our user guide is probably a
|
|
1351 |
+ proficient developer or integrator, but has no idea what we might have decided
|
|
1352 |
+ to name a given activity.
|
|
1353 |
+ |
|
1354 |
+ |
|
1355 |
+Structure of an example
|
|
1356 |
+'''''''''''''''''''''''
|
|
1357 |
+The :ref:`tutorial <tutorial>` and the :ref:`examples <examples>` sections
|
|
1358 |
+of the documentation contain a series of sample projects, each chapter in
|
|
1359 |
+the tutoral, or standalone example uses a sample project.
|
|
1360 |
+ |
|
1361 |
+Here is the the structure for adding new examples and tutorial chapters.
|
|
1273 | 1362 |
|
1274 | 1363 |
* The example has a ``${name}``.
|
1275 | 1364 |
|
... | ... | @@ -1280,11 +1369,17 @@ examples, here are the criteria for an example addition. |
1280 | 1369 |
* The example has a documentation component.
|
1281 | 1370 |
|
1282 | 1371 |
* This is added at ``doc/source/examples/${name}.rst``
|
1283 |
- * A reference to ``examples/${name}`` is added to the toctree in ``doc/source/examples.rst``
|
|
1372 |
+ * An entry for ``examples/${name}`` is added to the toctree in ``doc/source/using_examples.rst``
|
|
1284 | 1373 |
* This documentation discusses the project elements declared in the project and may
|
1285 | 1374 |
provide some BuildStream command examples.
|
1286 | 1375 |
* This documentation links out to the reference manual at every opportunity.
|
1287 | 1376 |
|
1377 |
+ .. note::
|
|
1378 |
+ |
|
1379 |
+ In the case of a tutorial chapter, the ``.rst`` file is added in at
|
|
1380 |
+ ``doc/source/tutorial/${name}.rst`` and an entry for ``tutorial/${name}``
|
|
1381 |
+ is added to ``doc/source/using_tutorial.rst``.
|
|
1382 |
+ |
|
1288 | 1383 |
* The example has a CI test component.
|
1289 | 1384 |
|
1290 | 1385 |
* This is an integration test added at ``tests/examples/${name}``.
|
... | ... | @@ -27,6 +27,14 @@ buildstream 1.3.1 |
27 | 27 |
o Generate Docker images from built artifacts using
|
28 | 28 |
`contrib/bst-docker-import` script.
|
29 | 29 |
|
30 |
+ o Creating a build shell through the interactive mode or `bst shell
|
|
31 |
+ --build` will now use the cached build tree. It is now *easier* to debug
|
|
32 |
+ local build failures, and *possible* to debug remote build failures.
|
|
33 |
+ |
|
34 |
+ o `bst shell --sysroot` now takes any directory that contains a sysroot,
|
|
35 |
+ instead of just a specially-formatted build-root with a `root` and `scratch`
|
|
36 |
+ subdirectory.
|
|
37 |
+ |
|
30 | 38 |
|
31 | 39 |
=================
|
32 | 40 |
buildstream 1.1.5
|
... | ... | @@ -365,7 +365,10 @@ class CASCache(ArtifactCache): |
365 | 365 |
Raises: ArtifactError if no push remotes are configured.
|
366 | 366 |
"""
|
367 | 367 |
|
368 |
- push_remotes = [r for r in self._remotes[project] if r.spec.push]
|
|
368 |
+ if self._has_push_remotes:
|
|
369 |
+ push_remotes = [r for r in self._remotes[project] if r.spec.push]
|
|
370 |
+ else:
|
|
371 |
+ push_remotes = []
|
|
369 | 372 |
|
370 | 373 |
if not push_remotes:
|
371 | 374 |
raise ArtifactError("CASCache: push_directory was called, but no remote artifact " +
|
... | ... | @@ -564,18 +564,15 @@ class App(): |
564 | 564 |
" (c)ontinue - Continue queueing jobs as much as possible\n" +
|
565 | 565 |
" (q)uit - Exit after all ongoing jobs complete\n" +
|
566 | 566 |
" (t)erminate - Terminate any ongoing jobs and exit\n" +
|
567 |
- " (r)etry - Retry this job\n")
|
|
567 |
+ " (r)etry - Retry this job\n" +
|
|
568 |
+ " (s)hell - Drop into a shell in the failed build sandbox\n")
|
|
568 | 569 |
if failure.logfile:
|
569 | 570 |
summary += " (l)og - View the full log file\n"
|
570 |
- if failure.sandbox:
|
|
571 |
- summary += " (s)hell - Drop into a shell in the failed build sandbox\n"
|
|
572 | 571 |
summary += "\nPressing ^C will terminate jobs and exit\n"
|
573 | 572 |
|
574 |
- choices = ['continue', 'quit', 'terminate', 'retry']
|
|
573 |
+ choices = ['continue', 'quit', 'terminate', 'retry', 'shell']
|
|
575 | 574 |
if failure.logfile:
|
576 | 575 |
choices += ['log']
|
577 |
- if failure.sandbox:
|
|
578 |
- choices += ['shell']
|
|
579 | 576 |
|
580 | 577 |
choice = ''
|
581 | 578 |
while choice not in ['continue', 'quit', 'terminate', 'retry']:
|
... | ... | @@ -595,10 +592,10 @@ class App(): |
595 | 592 |
# Handle choices which you can come back from
|
596 | 593 |
#
|
597 | 594 |
if choice == 'shell':
|
598 |
- click.echo("\nDropping into an interactive shell in the failed build sandbox\n", err=True)
|
|
595 |
+ click.echo("\nDropping into an interactive shell in the failed build tree\n", err=True)
|
|
599 | 596 |
try:
|
600 | 597 |
prompt = self.shell_prompt(element)
|
601 |
- self.stream.shell(element, Scope.BUILD, prompt, directory=failure.sandbox, isolate=True)
|
|
598 |
+ self.stream.shell(element, Scope.BUILD, prompt, isolate=True)
|
|
602 | 599 |
except BstError as e:
|
603 | 600 |
click.echo("Error while attempting to create interactive shell: {}".format(e), err=True)
|
604 | 601 |
elif choice == 'log':
|
... | ... | @@ -1317,7 +1317,9 @@ class Element(Plugin): |
1317 | 1317 |
@contextmanager
|
1318 | 1318 |
def _prepare_sandbox(self, scope, directory, deps='run', integrate=True):
|
1319 | 1319 |
# bst shell and bst checkout require a local sandbox.
|
1320 |
- with self.__sandbox(directory, config=self.__sandbox_config, allow_remote=False) as sandbox:
|
|
1320 |
+ bare_directory = True if directory else False
|
|
1321 |
+ with self.__sandbox(directory, config=self.__sandbox_config, allow_remote=False,
|
|
1322 |
+ bare_directory=True if directory else False) as sandbox:
|
|
1321 | 1323 |
|
1322 | 1324 |
# Configure always comes first, and we need it.
|
1323 | 1325 |
self.configure_sandbox(sandbox)
|
... | ... | @@ -1384,6 +1386,7 @@ class Element(Plugin): |
1384 | 1386 |
# the same filing system as the rest of our cache.
|
1385 | 1387 |
temp_staging_location = os.path.join(self._get_context().artifactdir, "staging_temp")
|
1386 | 1388 |
temp_staging_directory = tempfile.mkdtemp(prefix=temp_staging_location)
|
1389 |
+ import_dir = temp_staging_directory
|
|
1387 | 1390 |
|
1388 | 1391 |
try:
|
1389 | 1392 |
workspace = self._get_workspace()
|
... | ... | @@ -1394,12 +1397,16 @@ class Element(Plugin): |
1394 | 1397 |
with self.timed_activity("Staging local files at {}"
|
1395 | 1398 |
.format(workspace.get_absolute_path())):
|
1396 | 1399 |
workspace.stage(temp_staging_directory)
|
1400 |
+ elif self._cached():
|
|
1401 |
+ # We have a cached buildtree to use, instead
|
|
1402 |
+ artifact_base, _ = self.__extract()
|
|
1403 |
+ import_dir = os.path.join(artifact_base, 'buildtree')
|
|
1397 | 1404 |
else:
|
1398 | 1405 |
# No workspace, stage directly
|
1399 | 1406 |
for source in self.sources():
|
1400 | 1407 |
source._stage(temp_staging_directory)
|
1401 | 1408 |
|
1402 |
- vdirectory.import_files(temp_staging_directory)
|
|
1409 |
+ vdirectory.import_files(import_dir)
|
|
1403 | 1410 |
|
1404 | 1411 |
finally:
|
1405 | 1412 |
# Staging may produce directories with less than 'rwx' permissions
|
... | ... | @@ -1565,10 +1572,6 @@ class Element(Plugin): |
1565 | 1572 |
collect = self.assemble(sandbox)
|
1566 | 1573 |
self.__set_build_result(success=True, description="succeeded")
|
1567 | 1574 |
except BstError as e:
|
1568 |
- # If an error occurred assembling an element in a sandbox,
|
|
1569 |
- # then tack on the sandbox directory to the error
|
|
1570 |
- e.sandbox = rootdir
|
|
1571 |
- |
|
1572 | 1575 |
# If there is a workspace open on this element, it will have
|
1573 | 1576 |
# been mounted for sandbox invocations instead of being staged.
|
1574 | 1577 |
#
|
... | ... | @@ -1682,8 +1685,8 @@ class Element(Plugin): |
1682 | 1685 |
"unable to collect artifact contents"
|
1683 | 1686 |
.format(collect))
|
1684 | 1687 |
|
1685 |
- # Finally cleanup the build dir
|
|
1686 |
- cleanup_rootdir()
|
|
1688 |
+ # Finally cleanup the build dir
|
|
1689 |
+ cleanup_rootdir()
|
|
1687 | 1690 |
|
1688 | 1691 |
return artifact_size
|
1689 | 1692 |
|
... | ... | @@ -2151,12 +2154,14 @@ class Element(Plugin): |
2151 | 2154 |
# stderr (fileobject): The stream for stderr for the sandbox
|
2152 | 2155 |
# config (SandboxConfig): The SandboxConfig object
|
2153 | 2156 |
# allow_remote (bool): Whether the sandbox is allowed to be remote
|
2157 |
+ # bare_directory (bool): Whether the directory is bare i.e. doesn't have
|
|
2158 |
+ # a separate 'root' subdir
|
|
2154 | 2159 |
#
|
2155 | 2160 |
# Yields:
|
2156 | 2161 |
# (Sandbox): A usable sandbox
|
2157 | 2162 |
#
|
2158 | 2163 |
@contextmanager
|
2159 |
- def __sandbox(self, directory, stdout=None, stderr=None, config=None, allow_remote=True):
|
|
2164 |
+ def __sandbox(self, directory, stdout=None, stderr=None, config=None, allow_remote=True, bare_directory=False):
|
|
2160 | 2165 |
context = self._get_context()
|
2161 | 2166 |
project = self._get_project()
|
2162 | 2167 |
platform = Platform.get_platform()
|
... | ... | @@ -2187,6 +2192,7 @@ class Element(Plugin): |
2187 | 2192 |
stdout=stdout,
|
2188 | 2193 |
stderr=stderr,
|
2189 | 2194 |
config=config,
|
2195 |
+ bare_directory=bare_directory,
|
|
2190 | 2196 |
allow_real_directory=not self.BST_VIRTUAL_DIRECTORY)
|
2191 | 2197 |
yield sandbox
|
2192 | 2198 |
|
... | ... | @@ -2196,7 +2202,7 @@ class Element(Plugin): |
2196 | 2202 |
|
2197 | 2203 |
# Recursive contextmanager...
|
2198 | 2204 |
with self.__sandbox(rootdir, stdout=stdout, stderr=stderr, config=config,
|
2199 |
- allow_remote=allow_remote) as sandbox:
|
|
2205 |
+ allow_remote=allow_remote, bare_directory=False) as sandbox:
|
|
2200 | 2206 |
yield sandbox
|
2201 | 2207 |
|
2202 | 2208 |
# Cleanup the build dir
|
... | ... | @@ -31,7 +31,6 @@ from .._fuse import SafeHardlinks |
31 | 31 |
#
|
32 | 32 |
class Mount():
|
33 | 33 |
def __init__(self, sandbox, mount_point, safe_hardlinks, fuse_mount_options={}):
|
34 |
- scratch_directory = sandbox._get_scratch_directory()
|
|
35 | 34 |
# Getting _get_underlying_directory() here is acceptable as
|
36 | 35 |
# we're part of the sandbox code. This will fail if our
|
37 | 36 |
# directory is CAS-based.
|
... | ... | @@ -51,6 +50,7 @@ class Mount(): |
51 | 50 |
# a regular mount point within the parent's redirected mount.
|
52 | 51 |
#
|
53 | 52 |
if self.safe_hardlinks:
|
53 |
+ scratch_directory = sandbox._get_scratch_directory()
|
|
54 | 54 |
# Redirected mount
|
55 | 55 |
self.mount_origin = os.path.join(root_directory, mount_point.lstrip(os.sep))
|
56 | 56 |
self.mount_base = os.path.join(scratch_directory, utils.url_directory_name(mount_point))
|
... | ... | @@ -98,16 +98,23 @@ class Sandbox(): |
98 | 98 |
self.__config = kwargs['config']
|
99 | 99 |
self.__stdout = kwargs['stdout']
|
100 | 100 |
self.__stderr = kwargs['stderr']
|
101 |
+ self.__bare_directory = kwargs['bare_directory']
|
|
101 | 102 |
|
102 | 103 |
# Setup the directories. Root and output_directory should be
|
103 | 104 |
# available to subclasses, hence being single-underscore. The
|
104 | 105 |
# others are private to this class.
|
105 |
- self._root = os.path.join(directory, 'root')
|
|
106 |
+ # If the directory is bare, it probably doesn't need scratch
|
|
107 |
+ if self.__bare_directory:
|
|
108 |
+ self._root = directory
|
|
109 |
+ self.__scratch = None
|
|
110 |
+ os.makedirs(self._root, exist_ok=True)
|
|
111 |
+ else:
|
|
112 |
+ self._root = os.path.join(directory, 'root')
|
|
113 |
+ self.__scratch = os.path.join(directory, 'scratch')
|
|
114 |
+ for directory_ in [self._root, self.__scratch]:
|
|
115 |
+ os.makedirs(directory_, exist_ok=True)
|
|
116 |
+ |
|
106 | 117 |
self._output_directory = None
|
107 |
- self.__directory = directory
|
|
108 |
- self.__scratch = os.path.join(self.__directory, 'scratch')
|
|
109 |
- for directory_ in [self._root, self.__scratch]:
|
|
110 |
- os.makedirs(directory_, exist_ok=True)
|
|
111 | 118 |
self._vdir = None
|
112 | 119 |
|
113 | 120 |
# This is set if anyone requests access to the underlying
|
... | ... | @@ -332,6 +339,7 @@ class Sandbox(): |
332 | 339 |
# Returns:
|
333 | 340 |
# (str): The sandbox scratch directory
|
334 | 341 |
def _get_scratch_directory(self):
|
342 |
+ assert not self.__bare_directory, "Scratch is not going to work with bare directories"
|
|
335 | 343 |
return self.__scratch
|
336 | 344 |
|
337 | 345 |
# _get_output()
|
1 | 1 |
|
2 | 2 |
|
3 |
+.. _using:
|
|
4 |
+ |
|
3 | 5 |
Using
|
4 | 6 |
=====
|
5 | 7 |
This section includes user facing documentation including tutorials,
|
1 | 1 |
|
2 |
+ |
|
3 |
+.. _examples:
|
|
4 |
+ |
|
2 | 5 |
Examples
|
3 | 6 |
========
|
4 | 7 |
This page contains documentation for real examples of BuildStream projects,
|
1 | 1 |
|
2 |
+ |
|
3 |
+.. _tutorial:
|
|
4 |
+ |
|
2 | 5 |
Tutorial
|
3 | 6 |
========
|
4 | 7 |
This is a step by step walkthrough meant help the user quickly get
|
1 |
+import os
|
|
2 |
+import pytest
|
|
3 |
+import shutil
|
|
4 |
+ |
|
5 |
+from tests.testutils import cli, cli_integration, create_artifact_share
|
|
6 |
+ |
|
7 |
+ |
|
8 |
+pytestmark = pytest.mark.integration
|
|
9 |
+ |
|
10 |
+ |
|
11 |
+DATA_DIR = os.path.join(
|
|
12 |
+ os.path.dirname(os.path.realpath(__file__)),
|
|
13 |
+ "project"
|
|
14 |
+)
|
|
15 |
+ |
|
16 |
+ |
|
17 |
+@pytest.mark.datafiles(DATA_DIR)
|
|
18 |
+def test_buildtree_staged(cli_integration, tmpdir, datafiles):
|
|
19 |
+ # i.e. tests that cached build trees are staged by `bst shell --build`
|
|
20 |
+ project = os.path.join(datafiles.dirname, datafiles.basename)
|
|
21 |
+ element_name = 'build-shell/buildtree.bst'
|
|
22 |
+ |
|
23 |
+ res = cli_integration.run(project=project, args=['build', element_name])
|
|
24 |
+ res.assert_success()
|
|
25 |
+ |
|
26 |
+ res = cli_integration.run(project=project, args=[
|
|
27 |
+ 'shell', '--build', element_name, '--', 'grep', '-q', 'Hi', 'test'
|
|
28 |
+ ])
|
|
29 |
+ res.assert_success()
|
|
30 |
+ |
|
31 |
+ |
|
32 |
+# Check that build shells work when pulled from a remote cache
|
|
33 |
+# This is to roughly simulate remote execution
|
|
34 |
+@pytest.mark.datafiles(DATA_DIR)
|
|
35 |
+def test_buildtree_pulled(cli, tmpdir, datafiles):
|
|
36 |
+ project = os.path.join(datafiles.dirname, datafiles.basename)
|
|
37 |
+ element_name = 'build-shell/buildtree.bst'
|
|
38 |
+ |
|
39 |
+ with create_artifact_share(os.path.join(str(tmpdir), 'artifactshare')) as share:
|
|
40 |
+ # Build the element to push it to cache
|
|
41 |
+ cli.configure({
|
|
42 |
+ 'artifacts': {'url': share.repo, 'push': True}
|
|
43 |
+ })
|
|
44 |
+ result = cli.run(project=project, args=['build', element_name])
|
|
45 |
+ result.assert_success()
|
|
46 |
+ assert cli.get_element_state(project, element_name) == 'cached'
|
|
47 |
+ |
|
48 |
+ # Discard the cache
|
|
49 |
+ cli.configure({
|
|
50 |
+ 'artifacts': {'url': share.repo, 'push': True},
|
|
51 |
+ 'artifactdir': os.path.join(cli.directory, 'artifacts2')
|
|
52 |
+ })
|
|
53 |
+ assert cli.get_element_state(project, element_name) != 'cached'
|
|
54 |
+ |
|
55 |
+ # Pull from cache
|
|
56 |
+ result = cli.run(project=project, args=['pull', '--deps', 'all', element_name])
|
|
57 |
+ result.assert_success()
|
|
58 |
+ |
|
59 |
+ # Check it's using the cached build tree
|
|
60 |
+ res = cli.run(project=project, args=[
|
|
61 |
+ 'shell', '--build', element_name, '--', 'grep', '-q', 'Hi', 'test'
|
|
62 |
+ ])
|
|
63 |
+ res.assert_success()
|
1 |
+kind: manual
|
|
2 |
+description: |
|
|
3 |
+ Puts a file in the build tree so that build tree caching and staging can be tested.
|
|
4 |
+ |
|
5 |
+depends:
|
|
6 |
+ - filename: base.bst
|
|
7 |
+ type: build
|
|
8 |
+ |
|
9 |
+config:
|
|
10 |
+ build-commands:
|
|
11 |
+ - "echo 'Hi' > %{build-root}/test"
|
... | ... | @@ -302,46 +302,33 @@ def test_workspace_visible(cli, tmpdir, datafiles): |
302 | 302 |
assert result.output == workspace_hello
|
303 | 303 |
|
304 | 304 |
|
305 |
-# Test that we can see the workspace files in a shell
|
|
306 |
-@pytest.mark.integration
|
|
305 |
+# Test that '--sysroot' works
|
|
307 | 306 |
@pytest.mark.datafiles(DATA_DIR)
|
308 |
-def test_sysroot_workspace_visible(cli, tmpdir, datafiles):
|
|
307 |
+def test_sysroot(cli, tmpdir, datafiles):
|
|
309 | 308 |
project = os.path.join(datafiles.dirname, datafiles.basename)
|
310 |
- workspace = os.path.join(cli.directory, 'workspace')
|
|
311 |
- element_name = 'workspace/workspace-mount-fail.bst'
|
|
312 |
- |
|
313 |
- # Open a workspace on our build failing element
|
|
314 |
- #
|
|
315 |
- res = cli.run(project=project, args=['workspace', 'open', element_name, workspace])
|
|
316 |
- assert res.exit_code == 0
|
|
317 |
- |
|
318 |
- # Ensure the dependencies of our build failing element are built
|
|
319 |
- result = cli.run(project=project, args=['build', element_name])
|
|
320 |
- result.assert_main_error(ErrorDomain.STREAM, None)
|
|
321 |
- |
|
322 |
- # Discover the sysroot of the failed build directory, after one
|
|
323 |
- # failed build, there should be only one directory there.
|
|
324 |
- #
|
|
325 |
- build_base = os.path.join(cli.directory, 'build')
|
|
326 |
- build_dirs = os.listdir(path=build_base)
|
|
327 |
- assert len(build_dirs) == 1
|
|
328 |
- build_dir = os.path.join(build_base, build_dirs[0])
|
|
329 |
- |
|
330 |
- # Obtain a copy of the hello.c content from the workspace
|
|
331 |
- #
|
|
332 |
- workspace_hello_path = os.path.join(cli.directory, 'workspace', 'hello.c')
|
|
333 |
- assert os.path.exists(workspace_hello_path)
|
|
334 |
- with open(workspace_hello_path, 'r') as f:
|
|
335 |
- workspace_hello = f.read()
|
|
336 |
- |
|
337 |
- # Cat the hello.c file from a bst shell command, and assert
|
|
338 |
- # that we got the same content here
|
|
339 |
- #
|
|
340 |
- result = cli.run(project=project, args=[
|
|
341 |
- 'shell', '--build', '--sysroot', build_dir, element_name, '--', 'cat', 'hello.c'
|
|
309 |
+ base_element = "base/base-alpine.bst"
|
|
310 |
+ # test element only needs to be something lightweight for this test
|
|
311 |
+ test_element = "script/script.bst"
|
|
312 |
+ checkout_dir = os.path.join(str(tmpdir), 'alpine-sysroot')
|
|
313 |
+ test_file = 'hello'
|
|
314 |
+ |
|
315 |
+ # Build and check out a sysroot
|
|
316 |
+ res = cli.run(project=project, args=['build', base_element])
|
|
317 |
+ res.assert_success()
|
|
318 |
+ res = cli.run(project=project, args=['checkout', base_element, checkout_dir])
|
|
319 |
+ res.assert_success()
|
|
320 |
+ |
|
321 |
+ # Mutate the sysroot
|
|
322 |
+ test_path = os.path.join(checkout_dir, test_file)
|
|
323 |
+ with open(test_path, 'w') as f:
|
|
324 |
+ f.write('hello\n')
|
|
325 |
+ |
|
326 |
+ # Shell into the sysroot and check the test file exists
|
|
327 |
+ res = cli.run(project=project, args=[
|
|
328 |
+ 'shell', '--build', '--sysroot', checkout_dir, test_element, '--',
|
|
329 |
+ 'grep', '-q', 'hello', '/' + test_file
|
|
342 | 330 |
])
|
343 |
- assert result.exit_code == 0
|
|
344 |
- assert result.output == workspace_hello
|
|
331 |
+ res.assert_success()
|
|
345 | 332 |
|
346 | 333 |
|
347 | 334 |
# Test system integration commands can access devices in /dev
|