[Notes] [Git][BuildStream/buildstream][bschubert/dont-cache-errors-from-host-tools] 15 commits: Fix cache corruption by scripts when layout and integration commands are used

[Benjamin Schubert <gitlab mg gitlab com>]

Title: GitLab

Benjamin Schubert pushed to branch bschubert/dont-cache-errors-from-host-tools at BuildStream / buildstream

Commits:

• 6ccfab0b
  by Valentin David at 2018-11-07T16:28:47Z

  Fix cache corruption by scripts when layout and integration commands are used
  
  Root directory was marked as a non-artifact mount, so not using
  SafeHardLink.  However integration commands executed with write access
  to the root directory.
  
  Fixes #749
  

• f24e20e9
  by richardmaw-codethink at 2018-11-07T16:59:50Z

  Merge branch 'valentindavid/script-artifact-corruption' into 'master'
  
  Fix cache corruption by scripts when layout and integration commands are used
  
  Closes #749
  
  See merge request BuildStream/buildstream!934

• e28f668a
  by Valentin David at 2018-11-08T09:34:11Z

  Fix infinite recursion in default strip debug command
  
  Fixes #645.
  

• 83d15350
  by Valentin David at 2018-11-08T10:01:39Z

  Merge branch 'valentindavid/fix-recursive-strip-debug' into 'master'
  
  Fix infinite recursion in default strip debug command
  
  Closes #645
  
  See merge request BuildStream/buildstream!918

• 9f0e12f1
  by Benjamin Schubert at 2018-11-08T10:21:12Z

  Move bwrap checks in platform/linux.py
  
  Remove the bwraps checks from _site.py and put them in platform.linux
  which is the only place where they are run.
  
  This allows the removal of a double level of caching, making
  reasoning about tests easier
  

• cf2e0059
  by Benjamin Schubert at 2018-11-08T10:21:12Z

  conftest.py: Ensure platform is not maintained between tests
  
  This removes the `_instance` on the platform object that we use
  for caching and not recreating the object everytime at the start
  of every test.
  
  This is to ensure our tests share the least amount of state.
  
  The performance penalty is from 5 to 10% accross the whole test suite.
  The readings were done 5 times for each before and after the change
  and on the same computer.
  

• c51ba01b
  by Benjamin Schubert at 2018-11-08T10:21:12Z

  Test that helpful messages are raised when missing dependencies
  
  This adds a `reason` to the SandboxEror thrown in sandboxdummy
  to be able to understand where the error comes from
  

• f116b9b7
  by Benjamin Schubert at 2018-11-08T11:07:12Z

  Merge branch 'bschubert/tests-no-chroot-linux' into 'master'
  
  test behavior when there is no sandbox on Linux
  
  Closes #696 and #736
  
  See merge request BuildStream/buildstream!921

• cc2e6ae5
  by Valentin David at 2018-11-08T12:41:36Z

  Fix bug with root mounted as non-artifact in script plugin.
  
  The issue was introduced by 6ccfab0b1b25990e406446d5cbe5aee83a5e158a.
  

• e578a89f
  by Valentin David at 2018-11-08T13:11:10Z

  Merge branch 'valentindavid/fix-script-install-root' into 'master'
  
  Fix bug with root mounted as non-artifact in script plugin.
  
  Closes #756 and #749
  
  See merge request BuildStream/buildstream!941

• c0a8bb66
  by Angelos Evripiotis at 2018-11-08T15:49:16Z

  contributing: WIP again to be kind to reviewers
  
  As someone coming from GitHub to GitLab, I was pleasantly surprised by
  the 'filter by WIP status' option.
  
  To make sure we get the most out of it, add a guideline to keep the
  filter clean, to reduce reviewer burden.
  

• f7231e90
  by Angelos Evripiotis at 2018-11-08T15:49:16Z

  contributing: non-WIP should always be landable
  
  I was recently surprised that we don't prefer addressing review comments
  in 'fixup!' commits.
  
  Coming from GitHub, I've found that fixup commits make it easier for
  reviewers to see what has changed since their last review. The idea is
  to use '--autosquash' before landing to clean up the history again.
  
  It's a pleasant surprise that it's easy to keep track of what changed in
  merge-requests between pushes, so we can always keep the history clean.
  
  Document this, so folks like me can see the light sooner.
  

• f7643440
  by Angelos Evripiotis at 2018-11-08T15:49:16Z

  contributing: emphasise the 'why' in commits
  
  Explain why it's useful to mention the decisions made for a change and
  provide links for background, e.g. issue numbers.
  
  My general experience of git histories is that folks don't do enough
  explaining, so it's great we have a contributing section for it.
  
  I've found it's easier for folks to stick to rules when they know the
  practical reasons for them, so provide some.
  

• dd5e7b04
  by Angelos Evripiotis at 2018-11-08T16:17:04Z

  Merge branch 'aevri/contributing_gitlab' into 'master'
  
  Add more to GitLab-relevant parts of contributing
  
  See merge request BuildStream/buildstream!935

• 3e22708a
  by Benjamin Schubert at 2018-11-08T18:15:59Z

  Don't cache sandbox errors
  
  Sandbox errors (like missing host tools) are dependent on the host
  system and rarely on what is actually done.
  
  It is therefore better to not cache them as they are subject to
  change between two runs.
  
  Also add test to ensure sandbox failure are not cached
  

25 changed files:

• CONTRIBUTING.rst
• buildstream/_platform/linux.py
• buildstream/_site.py
• buildstream/data/projectconfig.yaml
• buildstream/element.py
• buildstream/sandbox/_sandboxdummy.py
• buildstream/scriptelement.py
• conftest.py
• tests/cachekey/project/elements/build1.expected
• tests/cachekey/project/elements/build2.expected
• tests/cachekey/project/target.expected
• tests/integration/cachedfail.py
• + tests/integration/project/elements/script/corruption-2.bst
• + tests/integration/project/elements/script/corruption-image.bst
• + tests/integration/project/elements/script/corruption-integration.bst
• + tests/integration/project/elements/script/corruption.bst
• + tests/integration/project/elements/script/marked-tmpdir.bst
• + tests/integration/project/elements/script/no-tmpdir.bst
• + tests/integration/project/elements/script/tmpdir.bst
• + tests/integration/project/files/canary
• tests/integration/script.py
• + tests/sandboxes/missing-dependencies/elements/base.bst
• + tests/sandboxes/missing-dependencies/files/base/bin/sh
• + tests/sandboxes/missing-dependencies/project.conf
• + tests/sandboxes/missing_dependencies.py

Changes:

CONTRIBUTING.rst

---

...	...	@@ -97,7 +97,13 @@ a new merge request. You can also `create a merge request for an existing branch
97	97	You may open merge requests for the branches you create before you are ready
98	98	to have them reviewed and considered for inclusion if you like. Until your merge
99	99	request is ready for review, the merge request title must be prefixed with the
100		-``WIP:`` identifier.
	100	+``WIP:`` identifier. GitLab `treats this specially
	101	+<https://docs.gitlab.com/ee/user/project/merge_requests/work_in_progress_merge_requests.html>`_,
	102	+which helps reviewers.
	103	+
	104	+Consider marking a merge request as WIP again if you are taking a while to
	105	+address a review point. This signals that the next action is on you, and it
	106	+won't appear in a reviewer's search for non-WIP merge requests to review.
101	107
102	108
103	109	Organized commits
...	...	@@ -122,6 +128,12 @@ If a commit in your branch modifies behavior such that a test must also
122	128	be changed to match the new behavior, then the tests should be updated
123	129	with the same commit, so that every commit passes its own tests.
124	130
	131	+These principles apply whenever a branch is non-WIP. So for example, don't push
	132	+'fixup!' commits when addressing review comments, instead amend the commits
	133	+directly before pushing. GitLab has `good support
	134	+<https://docs.gitlab.com/ee/user/project/merge_requests/versions.html>`_ for
	135	+diffing between pushes, so 'fixup!' commits are not necessary for reviewers.
	136	+
125	137
126	138	Commit messages
127	139	~~~~~~~~~~~~~~~
...	...	@@ -144,6 +156,16 @@ number must be referenced in the commit message.
144	156
145	157	Fixes #123
146	158
	159	+Note that the 'why' of a change is as important as the 'what'.
	160	+
	161	+When reviewing this, folks can suggest better alternatives when they know the
	162	+'why'. Perhaps there are other ways to avoid an error when things are not
	163	+frobnicated.
	164	+
	165	+When folks modify this code, there may be uncertainty around whether the foos
	166	+should always be frobnicated. The comments, the commit message, and issue #123
	167	+should shed some light on that.
	168	+
147	169	In the case that you have a commit which necessarily modifies multiple
148	170	components, then the summary line should still mention generally what
149	171	changed (if possible), followed by a colon and a brief summary.

buildstream/_platform/linux.py

---

...	...	@@ -18,9 +18,9 @@
18	18	# Tristan Maat <tristan maat codethink co uk>
19	19
20	20	import os
	21	+import shutil
21	22	import subprocess
22	23
23		-from .. import _site
24	24	from .. import utils
25	25	from ..sandbox import SandboxDummy
26	26
...	...	@@ -37,12 +37,19 @@ class Linux(Platform):
37	37	self._gid = os.getegid()
38	38
39	39	self._have_fuse = os.path.exists("/dev/fuse")
40		- self._bwrap_exists = _site.check_bwrap_version(0, 0, 0)
41		- self._have_good_bwrap = _site.check_bwrap_version(0, 1, 2)
42	40
43		- self._local_sandbox_available = self._have_fuse and self._have_good_bwrap
	41	+ bwrap_version = self._get_bwrap_version()
44	42
45		- self._die_with_parent_available = _site.check_bwrap_version(0, 1, 8)
	43	+ if bwrap_version is None:
	44	+ self._bwrap_exists = False
	45	+ self._have_good_bwrap = False
	46	+ self._die_with_parent_available = False
	47	+ else:
	48	+ self._bwrap_exists = True
	49	+ self._have_good_bwrap = (0, 1, 2) <= bwrap_version
	50	+ self._die_with_parent_available = (0, 1, 8) <= bwrap_version
	51	+
	52	+ self._local_sandbox_available = self._have_fuse and self._have_good_bwrap
46	53
47	54	if self._local_sandbox_available:
48	55	self._user_ns_available = self._check_user_ns_available()
...	...	@@ -112,3 +119,21 @@ class Linux(Platform):
112	119	output = ''
113	120
114	121	return output == 'root'
	122	+
	123	+ def _get_bwrap_version(self):
	124	+ # Get the current bwrap version
	125	+ #
	126	+ # returns None if no bwrap was found
	127	+ # otherwise returns a tuple of 3 int: major, minor, patch
	128	+ bwrap_path = shutil.which('bwrap')
	129	+
	130	+ if not bwrap_path:
	131	+ return None
	132	+
	133	+ cmd = [bwrap_path, "--version"]
	134	+ try:
	135	+ version = str(subprocess.check_output(cmd).split()[1], "utf-8")
	136	+ except subprocess.CalledProcessError:
	137	+ return None
	138	+
	139	+ return tuple(int(x) for x in version.split("."))

buildstream/_site.py

---

...	...	@@ -18,8 +18,6 @@
18	18	# Tristan Van Berkom <tristan vanberkom codethink co uk>
19	19
20	20	import os
21		-import shutil
22		-import subprocess
23	21
24	22	#
25	23	# Private module declaring some info about where the buildstream
...	...	@@ -46,44 +44,3 @@ build_all_template = os.path.join(root, 'data', 'build-all.sh.in')
46	44
47	45	# Module building script template
48	46	build_module_template = os.path.join(root, 'data', 'build-module.sh.in')
49		-
50		-# Cached bwrap version
51		-_bwrap_major = None
52		-_bwrap_minor = None
53		-_bwrap_patch = None
54		-
55		-
56		-# check_bwrap_version()
57		-#
58		-# Checks the version of installed bwrap against the requested version
59		-#
60		-# Args:
61		-# major (int): The required major version
62		-# minor (int): The required minor version
63		-# patch (int): The required patch level
64		-#
65		-# Returns:
66		-# (bool): Whether installed bwrap meets the requirements
67		-#
68		-def check_bwrap_version(major, minor, patch):
69		- # pylint: disable=global-statement
70		-
71		- global _bwrap_major
72		- global _bwrap_minor
73		- global _bwrap_patch
74		-
75		- # Parse bwrap version and save into cache, if not already cached
76		- if _bwrap_major is None:
77		- bwrap_path = shutil.which('bwrap')
78		- if not bwrap_path:
79		- return False
80		- cmd = [bwrap_path, "--version"]
81		- try:
82		- version = str(subprocess.check_output(cmd).split()[1], "utf-8")
83		- except subprocess.CalledProcessError:
84		- # Failure trying to run bubblewrap
85		- return False
86		- _bwrap_major, _bwrap_minor, _bwrap_patch = map(int, version.split("."))
87		-
88		- # Check whether the installed version meets the requirements
89		- return (_bwrap_major, _bwrap_minor, _bwrap_patch) >= (major, minor, patch)

buildstream/data/projectconfig.yaml

---

...	...	@@ -62,6 +62,11 @@ variables:
62	62	-o -name '*.cmxs' -o -name '*.node' ')' \
63	63	-exec sh -ec \
64	64	'read -n4 hdr <"$1" # check for elf header
	65	+ case "$1" in
	66	+ %{install-root}%{debugdir}/*)
	67	+ exit 0
	68	+ ;;
	69	+ esac
65	70	if [ "$hdr" != "$(printf \\x7fELF)" ]; then
66	71	exit 0
67	72	fi

buildstream/element.py

---

...	...	@@ -85,7 +85,8 @@ import shutil
85	85	from . import _yaml
86	86	from ._variables import Variables
87	87	from ._versions import BST_CORE_ARTIFACT_VERSION
88		-from ._exceptions import BstError, LoadError, LoadErrorReason, ImplError, ErrorDomain
	88	+from ._exceptions import BstError, LoadError, LoadErrorReason, ImplError, \
	89	+ ErrorDomain, SandboxError
89	90	from .utils import UtilError
90	91	from . import Plugin, Consistency, Scope
91	92	from . import SandboxFlags
...	...	@@ -1554,6 +1555,8 @@ class Element(Plugin):
1554	1555
1555	1556	# Call the abstract plugin methods
1556	1557	collect = None
	1558	+ save_artifacts = True
	1559	+
1557	1560	try:
1558	1561	# Step 1 - Configure
1559	1562	self.configure_sandbox(sandbox)
...	...	@@ -1565,6 +1568,9 @@ class Element(Plugin):
1565	1568	collect = self.assemble(sandbox) # pylint: disable=assignment-from-no-return
1566	1569	self.__set_build_result(success=True, description="succeeded")
1567	1570	except BstError as e:
	1571	+ if isinstance(e, SandboxError):
	1572	+ save_artifacts = False
	1573	+
1568	1574	# Shelling into a sandbox is useful to debug this error
1569	1575	e.sandbox = True
1570	1576
...	...	@@ -1592,100 +1598,108 @@ class Element(Plugin):
1592	1598	self.__set_build_result(success=False, description=str(e), detail=e.detail)
1593	1599	raise
1594	1600	finally:
1595		- if collect is not None:
1596		- try:
1597		- sandbox_vroot = sandbox.get_virtual_directory()
1598		- collectvdir = sandbox_vroot.descend(collect.lstrip(os.sep).split(os.sep))
1599		- except VirtualDirectoryError:
1600		- # No collect directory existed
1601		- collectvdir = None
1602		-
1603		- # Create artifact directory structure
1604		- assembledir = os.path.join(rootdir, 'artifact')
1605		- filesdir = os.path.join(assembledir, 'files')
1606		- logsdir = os.path.join(assembledir, 'logs')
1607		- metadir = os.path.join(assembledir, 'meta')
1608		- buildtreedir = os.path.join(assembledir, 'buildtree')
1609		- os.mkdir(assembledir)
1610		- if collect is not None and collectvdir is not None:
1611		- os.mkdir(filesdir)
1612		- os.mkdir(logsdir)
1613		- os.mkdir(metadir)
1614		- os.mkdir(buildtreedir)
1615		-
1616		- # Hard link files from collect dir to files directory
1617		- if collect is not None and collectvdir is not None:
1618		- collectvdir.export_files(filesdir, can_link=True)
1619		-
1620		- try:
1621		- sandbox_vroot = sandbox.get_virtual_directory()
1622		- sandbox_build_dir = sandbox_vroot.descend(
1623		- self.get_variable('build-root').lstrip(os.sep).split(os.sep))
1624		- # Hard link files from build-root dir to buildtreedir directory
1625		- sandbox_build_dir.export_files(buildtreedir)
1626		- except VirtualDirectoryError:
1627		- # Directory could not be found. Pre-virtual
1628		- # directory behaviour was to continue silently
1629		- # if the directory could not be found.
1630		- pass
1631		-
1632		- # Copy build log
1633		- log_filename = context.get_log_filename()
1634		- self._build_log_path = os.path.join(logsdir, 'build.log')
1635		- if log_filename:
1636		- shutil.copyfile(log_filename, self._build_log_path)
1637		-
1638		- # Store public data
1639		- _yaml.dump(_yaml.node_sanitize(self.__dynamic_public), os.path.join(metadir, 'public.yaml'))
1640		-
1641		- # Store result
1642		- build_result_dict = {"success": self.__build_result[0], "description": self.__build_result[1]}
1643		- if self.__build_result[2] is not None:
1644		- build_result_dict["detail"] = self.__build_result[2]
1645		- _yaml.dump(build_result_dict, os.path.join(metadir, 'build-result.yaml'))
1646		-
1647		- # ensure we have cache keys
1648		- self._assemble_done()
1649		-
1650		- # Store keys.yaml
1651		- _yaml.dump(_yaml.node_sanitize({
1652		- 'strong': self._get_cache_key(),
1653		- 'weak': self._get_cache_key(_KeyStrength.WEAK),
1654		- }), os.path.join(metadir, 'keys.yaml'))
1655		-
1656		- # Store dependencies.yaml
1657		- _yaml.dump(_yaml.node_sanitize({
1658		- e.name: e._get_cache_key() for e in self.dependencies(Scope.BUILD)
1659		- }), os.path.join(metadir, 'dependencies.yaml'))
1660		-
1661		- # Store workspaced.yaml
1662		- _yaml.dump(_yaml.node_sanitize({
1663		- 'workspaced': True if self._get_workspace() else False
1664		- }), os.path.join(metadir, 'workspaced.yaml'))
1665		-
1666		- # Store workspaced-dependencies.yaml
1667		- _yaml.dump(_yaml.node_sanitize({
1668		- 'workspaced-dependencies': [
1669		- e.name for e in self.dependencies(Scope.BUILD)
1670		- if e._get_workspace()
1671		- ]
1672		- }), os.path.join(metadir, 'workspaced-dependencies.yaml'))
1673		-
1674		- with self.timed_activity("Caching artifact"):
1675		- artifact_size = utils._get_dir_size(assembledir)
1676		- self.__artifacts.commit(self, assembledir, self.__get_cache_keys_for_commit())
1677		-
1678		- if collect is not None and collectvdir is None:
1679		- raise ElementError(
1680		- "Directory '{}' was not found inside the sandbox, "
1681		- "unable to collect artifact contents"
1682		- .format(collect))
	1601	+ if save_artifacts:
	1602	+ artifact_size = self._cache_artifact(rootdir, sandbox, context, collect)
	1603	+ else:
	1604	+ artifact_size = 0
1683	1605
1684	1606	# Finally cleanup the build dir
1685	1607	cleanup_rootdir()
1686	1608
1687	1609	return artifact_size
1688	1610
	1611	+ def _cache_artifact(self, rootdir, sandbox, context, collect):
	1612	+ if collect is not None:
	1613	+ try:
	1614	+ sandbox_vroot = sandbox.get_virtual_directory()
	1615	+ collectvdir = sandbox_vroot.descend(collect.lstrip(os.sep).split(os.sep))
	1616	+ except VirtualDirectoryError:
	1617	+ # No collect directory existed
	1618	+ collectvdir = None
	1619	+
	1620	+ # Create artifact directory structure
	1621	+ assembledir = os.path.join(rootdir, 'artifact')
	1622	+ filesdir = os.path.join(assembledir, 'files')
	1623	+ logsdir = os.path.join(assembledir, 'logs')
	1624	+ metadir = os.path.join(assembledir, 'meta')
	1625	+ buildtreedir = os.path.join(assembledir, 'buildtree')
	1626	+ os.mkdir(assembledir)
	1627	+ if collect is not None and collectvdir is not None:
	1628	+ os.mkdir(filesdir)
	1629	+ os.mkdir(logsdir)
	1630	+ os.mkdir(metadir)
	1631	+ os.mkdir(buildtreedir)
	1632	+
	1633	+ # Hard link files from collect dir to files directory
	1634	+ if collect is not None and collectvdir is not None:
	1635	+ collectvdir.export_files(filesdir, can_link=True)
	1636	+
	1637	+ try:
	1638	+ sandbox_vroot = sandbox.get_virtual_directory()
	1639	+ sandbox_build_dir = sandbox_vroot.descend(
	1640	+ self.get_variable('build-root').lstrip(os.sep).split(os.sep))
	1641	+ # Hard link files from build-root dir to buildtreedir directory
	1642	+ sandbox_build_dir.export_files(buildtreedir)
	1643	+ except VirtualDirectoryError:
	1644	+ # Directory could not be found. Pre-virtual
	1645	+ # directory behaviour was to continue silently
	1646	+ # if the directory could not be found.
	1647	+ pass
	1648	+
	1649	+ # Copy build log
	1650	+ log_filename = context.get_log_filename()
	1651	+ self._build_log_path = os.path.join(logsdir, 'build.log')
	1652	+ if log_filename:
	1653	+ shutil.copyfile(log_filename, self._build_log_path)
	1654	+
	1655	+ # Store public data
	1656	+ _yaml.dump(_yaml.node_sanitize(self.__dynamic_public), os.path.join(metadir, 'public.yaml'))
	1657	+
	1658	+ # Store result
	1659	+ build_result_dict = {"success": self.__build_result[0], "description": self.__build_result[1]}
	1660	+ if self.__build_result[2] is not None:
	1661	+ build_result_dict["detail"] = self.__build_result[2]
	1662	+ _yaml.dump(build_result_dict, os.path.join(metadir, 'build-result.yaml'))
	1663	+
	1664	+ # ensure we have cache keys
	1665	+ self._assemble_done()
	1666	+
	1667	+ # Store keys.yaml
	1668	+ _yaml.dump(_yaml.node_sanitize({
	1669	+ 'strong': self._get_cache_key(),
	1670	+ 'weak': self._get_cache_key(_KeyStrength.WEAK),
	1671	+ }), os.path.join(metadir, 'keys.yaml'))
	1672	+
	1673	+ # Store dependencies.yaml
	1674	+ _yaml.dump(_yaml.node_sanitize({
	1675	+ e.name: e._get_cache_key() for e in self.dependencies(Scope.BUILD)
	1676	+ }), os.path.join(metadir, 'dependencies.yaml'))
	1677	+
	1678	+ # Store workspaced.yaml
	1679	+ _yaml.dump(_yaml.node_sanitize({
	1680	+ 'workspaced': True if self._get_workspace() else False
	1681	+ }), os.path.join(metadir, 'workspaced.yaml'))
	1682	+
	1683	+ # Store workspaced-dependencies.yaml
	1684	+ _yaml.dump(_yaml.node_sanitize({
	1685	+ 'workspaced-dependencies': [
	1686	+ e.name for e in self.dependencies(Scope.BUILD)
	1687	+ if e._get_workspace()
	1688	+ ]
	1689	+ }), os.path.join(metadir, 'workspaced-dependencies.yaml'))
	1690	+
	1691	+ with self.timed_activity("Caching artifact"):
	1692	+ artifact_size = utils._get_dir_size(assembledir)
	1693	+ self.__artifacts.commit(self, assembledir, self.__get_cache_keys_for_commit())
	1694	+
	1695	+ if collect is not None and collectvdir is None:
	1696	+ raise ElementError(
	1697	+ "Directory '{}' was not found inside the sandbox, "
	1698	+ "unable to collect artifact contents"
	1699	+ .format(collect))
	1700	+
	1701	+ return artifact_size
	1702	+
1689	1703	def _get_build_log(self):
1690	1704	return self._build_log_path
1691	1705

buildstream/sandbox/_sandboxdummy.py

---

...	...	@@ -42,4 +42,5 @@ class SandboxDummy(Sandbox):
42	42	"'{}'".format(command[0]),
43	43	reason='missing-command')
44	44
45		- raise SandboxError("This platform does not support local builds: {}".format(self._reason))
	45	+ raise SandboxError("This platform does not support local builds: {}".format(self._reason),
	46	+ reason="unavailable-local-sandbox")

buildstream/scriptelement.py

---

...	...	@@ -201,16 +201,20 @@ class ScriptElement(Element):
201	201	# Setup environment
202	202	sandbox.set_environment(self.get_environment())
203	203
	204	+ # Tell the sandbox to mount the install root
	205	+ directories = {self.__install_root: False}
	206	+
204	207	# Mark the artifact directories in the layout
205	208	for item in self.__layout:
206		- if item['destination'] != '/':
207		- if item['element']:
208		- sandbox.mark_directory(item['destination'], artifact=True)
209		- else:
210		- sandbox.mark_directory(item['destination'])
211		-
212		- # Tell the sandbox to mount the install root
213		- sandbox.mark_directory(self.__install_root)
	209	+ destination = item['destination']
	210	+ was_artifact = directories.get(destination, False)
	211	+ directories[destination] = item['element'] or was_artifact
	212	+
	213	+ for directory, artifact in directories.items():
	214	+ # Root does not need to be marked as it is always mounted
	215	+ # with artifact (unless explicitly marked non-artifact)
	216	+ if directory != '/':
	217	+ sandbox.mark_directory(directory, artifact=artifact)
214	218
215	219	def stage(self, sandbox):
216	220

conftest.py

---

...	...	@@ -23,6 +23,8 @@ import shutil
23	23
24	24	import pytest
25	25
	26	+from buildstream._platform.platform import Platform
	27	+
26	28
27	29	def pytest_addoption(parser):
28	30	parser.addoption('--integration', action='store_true', default=False,
...	...	@@ -52,3 +54,8 @@ def integration_cache(request):
52	54	shutil.rmtree(os.path.join(cache_dir, 'artifacts'))
53	55	except FileNotFoundError:
54	56	pass
	57	+
	58	+
	59	+@pytest.fixture(autouse=True)
	60	+def clean_platform_cache():
	61	+ Platform._instance = None

tests/cachekey/project/elements/build1.expected

---

1		-dd5e29baefb84f68eb4abac3a1befc332077ec4c97bb2572e57f3ca98ba46707
		\ No newline at end of file
	1	+ce0ddf7126d45d14f5ec1a525337c39ec8ddbbe4b0ec2ef51bae777619ed39bb
		\ No newline at end of file

tests/cachekey/project/elements/build2.expected

---

1		-99d80454cce44645597c885800edf0bf254d1c3606d869f2ccdd5043ec7685cb
		\ No newline at end of file
	1	+5e2a48dbeae43f6bab84071dbd02345a3aa32a473c189645ab26f3d5d6cfe547
		\ No newline at end of file

tests/cachekey/project/target.expected

---

1		-29a1252ec30dd6ae73c772381f0eb417e3874c75710d08be819f5715dcaa942b
		\ No newline at end of file
	1	+125d9e7dcf4f49e5f80d85b7f144b43ed43186064afc2e596e57f26cce679cf5
		\ No newline at end of file

tests/integration/cachedfail.py

---

...	...	@@ -158,3 +158,37 @@ def test_push_cached_fail(cli, tmpdir, datafiles, on_error):
158	158	assert cli.get_element_state(project, 'element.bst') == 'failed'
159	159	# This element should have been pushed to the remote
160	160	assert share.has_artifact('test', 'element.bst', cli.get_element_key(project, 'element.bst'))
	161	+
	162	+
	163	+@pytest.mark.skipif(not IS_LINUX, reason='Only available on linux')
	164	+@pytest.mark.datafiles(DATA_DIR)
	165	+def test_host_tools_errors_are_not_cached(cli, tmpdir, datafiles):
	166	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	167	+ element_path = os.path.join(project, 'elements', 'element.bst')
	168	+
	169	+ # Write out our test target
	170	+ element = {
	171	+ 'kind': 'script',
	172	+ 'depends': [
	173	+ {
	174	+ 'filename': 'base.bst',
	175	+ 'type': 'build',
	176	+ },
	177	+ ],
	178	+ 'config': {
	179	+ 'commands': [
	180	+ 'true',
	181	+ ],
	182	+ },
	183	+ }
	184	+ _yaml.dump(element, element_path)
	185	+
	186	+ # Build without access to host tools, this will fail
	187	+ result1 = cli.run(project=project, args=['build', 'element.bst'], env={'PATH': ''})
	188	+ result1.assert_task_error(ErrorDomain.SANDBOX, 'unavailable-local-sandbox')
	189	+
	190	+ Platform._instance = None
	191	+ _site._bwrap_major = _site._bwrap_minor = _site._bwrap_patch = None
	192	+ # When rebuilding, this should work
	193	+ result2 = cli.run(project=project, args=['build', 'element.bst'])
	194	+ result2.assert_success()

tests/integration/project/elements/script/corruption-2.bst

---

	1	+kind: script
	2	+
	3	+depends:
	4	+- filename: base.bst
	5	+ type: build
	6	+- filename: script/corruption-image.bst
	7	+ type: build
	8	+
	9	+config:
	10	+ commands:
	11	+ - echo smashed >>/canary

tests/integration/project/elements/script/corruption-image.bst

---

	1	+kind: import
	2	+sources:
	3	+- kind: local
	4	+ path: files/canary

tests/integration/project/elements/script/corruption-integration.bst

---

	1	+kind: stack
	2	+
	3	+public:
	4	+ bst:
	5	+ integration-commands:
	6	+ - echo smashed >>/canary
	7	+

tests/integration/project/elements/script/corruption.bst

---

	1	+kind: script
	2	+
	3	+depends:
	4	+- filename: base.bst
	5	+ type: build
	6	+- filename: script/corruption-image.bst
	7	+ type: build
	8	+- filename: script/corruption-integration.bst
	9	+ type: build
	10	+
	11	+variables:
	12	+ install-root: "/"
	13	+
	14	+config:
	15	+ layout:
	16	+ - element: base.bst
	17	+ destination: "/"
	18	+ - element: script/corruption-image.bst
	19	+ destination: "/"
	20	+ - element: script/corruption-integration.bst
	21	+ destination: "/"

tests/integration/project/elements/script/marked-tmpdir.bst

---

	1	+kind: compose
	2	+
	3	+depends:
	4	+- filename: base.bst
	5	+ type: build
	6	+
	7	+public:
	8	+ bst:
	9	+ split-rules:
	10	+ remove:
	11	+ - "/tmp/**"
	12	+ - "/tmp"

tests/integration/project/elements/script/no-tmpdir.bst

---

	1	+kind: filter
	2	+
	3	+depends:
	4	+- filename: script/marked-tmpdir.bst
	5	+ type: build
	6	+
	7	+config:
	8	+ exclude:
	9	+ - remove
	10	+ include-orphans: True
	11	+
	12	+

tests/integration/project/elements/script/tmpdir.bst

---

	1	+kind: script
	2	+
	3	+depends:
	4	+- filename: script/no-tmpdir.bst
	5	+ type: build
	6	+
	7	+config:
	8	+ commands:
	9	+ - |
	10	+ mkdir -p /tmp/blah

tests/integration/project/files/canary

---

1

+alive

tests/integration/script.py

---

...	...	@@ -155,3 +155,70 @@ def test_script_layout(cli, tmpdir, datafiles):
155	155	text = f.read()
156	156
157	157	assert text == "Hi\n"
	158	+
	159	+
	160	+@pytest.mark.datafiles(DATA_DIR)
	161	+def test_regression_cache_corruption(cli, tmpdir, datafiles):
	162	+ project = str(datafiles)
	163	+ checkout_original = os.path.join(cli.directory, 'checkout-original')
	164	+ checkout_after = os.path.join(cli.directory, 'checkout-after')
	165	+ element_name = 'script/corruption.bst'
	166	+ canary_element_name = 'script/corruption-image.bst'
	167	+
	168	+ res = cli.run(project=project, args=['build', canary_element_name])
	169	+ assert res.exit_code == 0
	170	+
	171	+ res = cli.run(project=project, args=['checkout', canary_element_name,
	172	+ checkout_original])
	173	+ assert res.exit_code == 0
	174	+
	175	+ with open(os.path.join(checkout_original, 'canary')) as f:
	176	+ assert f.read() == 'alive\n'
	177	+
	178	+ res = cli.run(project=project, args=['build', element_name])
	179	+ assert res.exit_code == 0
	180	+
	181	+ res = cli.run(project=project, args=['checkout', canary_element_name,
	182	+ checkout_after])
	183	+ assert res.exit_code == 0
	184	+
	185	+ with open(os.path.join(checkout_after, 'canary')) as f:
	186	+ assert f.read() == 'alive\n'
	187	+
	188	+
	189	+@pytest.mark.datafiles(DATA_DIR)
	190	+def test_regression_tmpdir(cli, tmpdir, datafiles):
	191	+ project = str(datafiles)
	192	+ element_name = 'script/tmpdir.bst'
	193	+
	194	+ res = cli.run(project=project, args=['build', element_name])
	195	+ assert res.exit_code == 0
	196	+
	197	+
	198	+@pytest.mark.datafiles(DATA_DIR)
	199	+def test_regression_cache_corruption_2(cli, tmpdir, datafiles):
	200	+ project = str(datafiles)
	201	+ checkout_original = os.path.join(cli.directory, 'checkout-original')
	202	+ checkout_after = os.path.join(cli.directory, 'checkout-after')
	203	+ element_name = 'script/corruption-2.bst'
	204	+ canary_element_name = 'script/corruption-image.bst'
	205	+
	206	+ res = cli.run(project=project, args=['build', canary_element_name])
	207	+ assert res.exit_code == 0
	208	+
	209	+ res = cli.run(project=project, args=['checkout', canary_element_name,
	210	+ checkout_original])
	211	+ assert res.exit_code == 0
	212	+
	213	+ with open(os.path.join(checkout_original, 'canary')) as f:
	214	+ assert f.read() == 'alive\n'
	215	+
	216	+ res = cli.run(project=project, args=['build', element_name])
	217	+ assert res.exit_code == 0
	218	+
	219	+ res = cli.run(project=project, args=['checkout', canary_element_name,
	220	+ checkout_after])
	221	+ assert res.exit_code == 0
	222	+
	223	+ with open(os.path.join(checkout_after, 'canary')) as f:
	224	+ assert f.read() == 'alive\n'

tests/sandboxes/missing-dependencies/elements/base.bst

---

	1	+kind: import
	2	+sources:
	3	+- kind: local
	4	+ path: files/base/

tests/sandboxes/missing-dependencies/files/base/bin/sh

---

1

+# This is the original bash

tests/sandboxes/missing-dependencies/project.conf

---

	1	+# Project config for missing dependencies test
	2	+name: test
	3	+
	4	+element-path: elements

tests/sandboxes/missing_dependencies.py

---

	1	+import os
	2	+import pytest
	3	+from tests.testutils import cli
	4	+from tests.testutils.site import IS_LINUX
	5	+
	6	+from buildstream import _yaml
	7	+from buildstream._exceptions import ErrorDomain
	8	+
	9	+
	10	+# Project directory
	11	+DATA_DIR = os.path.join(
	12	+ os.path.dirname(os.path.realpath(__file__)),
	13	+ "missing-dependencies",
	14	+)
	15	+
	16	+
	17	+@pytest.mark.skipif(not IS_LINUX, reason='Only available on Linux')
	18	+@pytest.mark.datafiles(DATA_DIR)
	19	+def test_missing_brwap_has_nice_error_message(cli, datafiles):
	20	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	21	+ element_path = os.path.join(project, 'elements', 'element.bst')
	22	+
	23	+ # Write out our test target
	24	+ element = {
	25	+ 'kind': 'script',
	26	+ 'depends': [
	27	+ {
	28	+ 'filename': 'base.bst',
	29	+ 'type': 'build',
	30	+ },
	31	+ ],
	32	+ 'config': {
	33	+ 'commands': [
	34	+ 'false',
	35	+ ],
	36	+ },
	37	+ }
	38	+ _yaml.dump(element, element_path)
	39	+
	40	+ # Build without access to host tools, this should fail with a nice error
	41	+ result = cli.run(
	42	+ project=project, args=['build', 'element.bst'], env={'PATH': ''})
	43	+ result.assert_task_error(ErrorDomain.SANDBOX, 'unavailable-local-sandbox')
	44	+ assert "not found" in result.stderr
	45	+
	46	+
	47	+@pytest.mark.skipif(not IS_LINUX, reason='Only available on Linux')
	48	+@pytest.mark.datafiles(DATA_DIR)
	49	+def test_old_brwap_has_nice_error_message(cli, datafiles, tmp_path):
	50	+ bwrap = tmp_path.joinpath('bin/bwrap')
	51	+ bwrap.parent.mkdir()
	52	+ with bwrap.open('w') as fp:
	53	+ fp.write('''
	54	+ #!/bin/sh
	55	+ echo bubblewrap 0.0.1
	56	+ '''.strip())
	57	+
	58	+ bwrap.chmod(0o755)
	59	+
	60	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	61	+ element_path = os.path.join(project, 'elements', 'element3.bst')
	62	+
	63	+ # Write out our test target
	64	+ element = {
	65	+ 'kind': 'script',
	66	+ 'depends': [
	67	+ {
	68	+ 'filename': 'base.bst',
	69	+ 'type': 'build',
	70	+ },
	71	+ ],
	72	+ 'config': {
	73	+ 'commands': [
	74	+ 'false',
	75	+ ],
	76	+ },
	77	+ }
	78	+ _yaml.dump(element, element_path)
	79	+
	80	+ # Build without access to host tools, this should fail with a nice error
	81	+ result = cli.run(
	82	+ project=project,
	83	+ args=['--debug', '--verbose', 'build', 'element3.bst'],
	84	+ env={'PATH': str(tmp_path.joinpath('bin'))})
	85	+ result.assert_task_error(ErrorDomain.SANDBOX, 'unavailable-local-sandbox')
	86	+ assert "too old" in result.stderr

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.