[Notes] [Git][BuildStream/buildstream][valentindavid/silverblue-path] 11 commits: Added `set_resource_limits()` to platform

[Tiago Gomes <gitlab mg gitlab com>]

Title: GitLab

Tiago Gomes pushed to branch valentindavid/silverblue-path at BuildStream / buildstream

Commits:

• fcf37f9c
  by knownexus at 2018-09-27T14:22:09Z

  Added `set_resource_limits()` to platform
  
  This has been moved from app.py
  As it will have different functionality depending on platform
  Once the Darwin (MacOS) platform is added
  
  Removed `resource.setrlimit()` functionality from app.py
  Added `resource.setrlimit()` functionality to platform.py as function
  

• 80ebdd2c
  by knownexus at 2018-09-27T14:22:09Z

  Added FUSE check, as WSL does not support FUSE yet
  
  _platform/linux.py: Add FUSE check
  And Ensure exception is not raised when performing sandbox availability check
  Without this check, WSL would error upon finding FUSE missing
  This must be bypassed to allow remote execution
  

• 5e81573b
  by knownexus at 2018-09-27T14:22:09Z

  Max-jobs is now defined on a platform basis
  
  This is due to Darwin (MacOS) having a different Max-Jobs limit
  _platform/platform.py: Adding max-jobs call
  
  _project.py: redirected 'max-jobs' to point at the platform
  

• 3c93fe97
  by knownexus at 2018-09-27T14:22:09Z

  Adding Dummy sandbox
  
  This is to allow platforms that do not support sandboxing
  To error in a controlled/known way
  

• 0f3ef369
  by knownexus at 2018-09-27T14:22:24Z

  Adding darwin.py (MacOS) platform
  
  Adding functionality to recognise Darwin as a platform in plaform.py
  

• b78ae767
  by knownexus at 2018-09-27T14:29:29Z

  utils.py: Reworked safe_remove
  
  Non-Linux platforms don't return EISDIR when attempting to unlink a
  directory.
  
  Stopped safe_remove attempting to unlink dirs
  Previously safe_remove would attempt to unlink a path
  Before attempting to remove it if it was a dir
  
  Now it checks for a dir before that step
  

• ab5e78b4
  by James Ennis at 2018-09-27T14:29:32Z

  Import sandbox subclass when instantiating sandbox
  
  This patch was required due to not being able to execute `bst` on
  a Linux machine without FUSE available.
  

• 261f65ca
  by Jürg Billeter at 2018-09-27T14:54:47Z

  Merge branch 'mac_fixes' into 'master'
  
  Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412"
  
  See merge request BuildStream/buildstream!726

• d2ea3069
  by Richard Maw at 2018-09-27T15:16:57Z

  tests: Add regression test for pushing failed builds
  
  This test ensures that failed builds are pushed when set to on-error=quit.
  
  Closes https://gitlab.com/BuildStream/buildstream/issues/534
  

• c128429a
  by richardmaw-codethink at 2018-09-27T15:42:18Z

  Merge branch 'richardmaw/push-failed-build-regression' into 'master'
  
  tests: Add regression test for pushing failed builds
  
  Closes #534
  
  See merge request BuildStream/buildstream!824

• e91eb38b
  by Valentin David at 2018-09-27T16:32:57Z

  Fix outside-of-project check when project path is not canonical.
  
  The issue happens on Silverblue where /home is a symlink to /var/home.
  With this element-path is something like
  /var/home/user/project/elements, when the project path is
  /home/usr/project. Comparing canonical paths solves the issue.
  
  Fixes #673
  

12 changed files:

• buildstream/_frontend/app.py
• + buildstream/_platform/darwin.py
• buildstream/_platform/linux.py
• buildstream/_platform/platform.py
• buildstream/_platform/unix.py
• buildstream/_project.py
• buildstream/_yaml.py
• buildstream/sandbox/__init__.py
• + buildstream/sandbox/_sandboxdummy.py
• buildstream/utils.py
• tests/format/project.py
• tests/integration/cachedfail.py

Changes:

buildstream/_frontend/app.py

---

...	...	@@ -115,14 +115,6 @@ class App():
115	115	else:
116	116	self.colors = False
117	117
118		- # Increase the soft limit for open file descriptors to the maximum.
119		- # SafeHardlinks FUSE needs to hold file descriptors for all processes in the sandbox.
120		- # Avoid hitting the limit too quickly.
121		- limits = resource.getrlimit(resource.RLIMIT_NOFILE)
122		- if limits[0] != limits[1]:
123		- # Set soft limit to hard limit
124		- resource.setrlimit(resource.RLIMIT_NOFILE, (limits[1], limits[1]))
125		-
126	118	# create()
127	119	#
128	120	# Should be used instead of the regular constructor.

buildstream/_platform/darwin.py

---

	1	+#
	2	+# Copyright (C) 2017 Codethink Limited
	3	+# Copyright (C) 2018 Bloomberg Finance LP
	4	+#
	5	+# This program is free software; you can redistribute it and/or
	6	+# modify it under the terms of the GNU Lesser General Public
	7	+# License as published by the Free Software Foundation; either
	8	+# version 2 of the License, or (at your option) any later version.
	9	+#
	10	+# This library is distributed in the hope that it will be useful,
	11	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	12	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	13	+# Lesser General Public License for more details.
	14	+#
	15	+# You should have received a copy of the GNU Lesser General Public
	16	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	17	+
	18	+import os
	19	+import resource
	20	+
	21	+from .._exceptions import PlatformError
	22	+from ..sandbox import SandboxDummy
	23	+
	24	+from . import Platform
	25	+
	26	+
	27	+class Darwin(Platform):
	28	+
	29	+ # This value comes from OPEN_MAX in syslimits.h
	30	+ OPEN_MAX = 10240
	31	+
	32	+ def __init__(self):
	33	+
	34	+ super().__init__()
	35	+
	36	+ def create_sandbox(self, *args, **kwargs):
	37	+ return SandboxDummy(*args, **kwargs)
	38	+
	39	+ def check_sandbox_config(self, config):
	40	+ # Accept all sandbox configs as it's irrelevant with the dummy sandbox (no Sandbox.run).
	41	+ return True
	42	+
	43	+ def get_cpu_count(self, cap=None):
	44	+ if cap < os.cpu_count():
	45	+ return cap
	46	+ else:
	47	+ return os.cpu_count()
	48	+
	49	+ def set_resource_limits(self, soft_limit=OPEN_MAX, hard_limit=None):
	50	+ super().set_resource_limits(soft_limit)

buildstream/_platform/linux.py

---

...	...	@@ -23,7 +23,7 @@ import subprocess
23	23	from .. import _site
24	24	from .. import utils
25	25	from .._message import Message, MessageType
26		-from ..sandbox import SandboxBwrap
	26	+from ..sandbox import SandboxDummy
27	27
28	28	from . import Platform
29	29
...	...	@@ -38,13 +38,21 @@ class Linux(Platform):
38	38	self._gid = os.getegid()
39	39
40	40	self._die_with_parent_available = _site.check_bwrap_version(0, 1, 8)
41		- self._user_ns_available = self._check_user_ns_available()
	41	+
	42	+ if self._local_sandbox_available():
	43	+ self._user_ns_available = self._check_user_ns_available()
	44	+ else:
	45	+ self._user_ns_available = False
42	46
43	47	def create_sandbox(self, *args, **kwargs):
44		- # Inform the bubblewrap sandbox as to whether it can use user namespaces or not
45		- kwargs['user_ns_available'] = self._user_ns_available
46		- kwargs['die_with_parent_available'] = self._die_with_parent_available
47		- return SandboxBwrap(*args, **kwargs)
	48	+ if not self._local_sandbox_available():
	49	+ return SandboxDummy(*args, **kwargs)
	50	+ else:
	51	+ from ..sandbox._sandboxbwrap import SandboxBwrap
	52	+ # Inform the bubblewrap sandbox as to whether it can use user namespaces or not
	53	+ kwargs['user_ns_available'] = self._user_ns_available
	54	+ kwargs['die_with_parent_available'] = self._die_with_parent_available
	55	+ return SandboxBwrap(*args, **kwargs)
48	56
49	57	def check_sandbox_config(self, config):
50	58	if self._user_ns_available:
...	...	@@ -58,8 +66,13 @@ class Linux(Platform):
58	66	################################################
59	67	# Private Methods #
60	68	################################################
61		- def _check_user_ns_available(self):
	69	+ def _local_sandbox_available(self):
	70	+ try:
	71	+ return os.path.exists(utils.get_host_tool('bwrap')) and os.path.exists('/dev/fuse')
	72	+ except utils.ProgramNotFoundError:
	73	+ return False
62	74
	75	+ def _check_user_ns_available(self):
63	76	# Here, lets check if bwrap is able to create user namespaces,
64	77	# issue a warning if it's not available, and save the state
65	78	# locally so that we can inform the sandbox to not try it

buildstream/_platform/platform.py

---

...	...	@@ -19,6 +19,7 @@
19	19
20	20	import os
21	21	import sys
	22	+import resource
22	23
23	24	from .._exceptions import PlatformError, ImplError
24	25
...	...	@@ -32,23 +33,26 @@ class Platform():
32	33	# sandbox factory as well as platform helpers.
33	34	#
34	35	def __init__(self):
35		- pass
	36	+ self.set_resource_limits()
36	37
37	38	@classmethod
38	39	def _create_instance(cls):
39		- if sys.platform.startswith('linux'):
40		- backend = 'linux'
41		- else:
42		- backend = 'unix'
43		-
44	40	# Meant for testing purposes and therefore hidden in the
45	41	# deepest corners of the source code. Try not to abuse this,
46	42	# please?
47	43	if os.getenv('BST_FORCE_BACKEND'):
48	44	backend = os.getenv('BST_FORCE_BACKEND')
	45	+ elif sys.platform.startswith('linux'):
	46	+ backend = 'linux'
	47	+ elif sys.platform.startswith('darwin'):
	48	+ backend = 'darwin'
	49	+ else:
	50	+ backend = 'unix'
49	51
50	52	if backend == 'linux':
51	53	from .linux import Linux as PlatformImpl
	54	+ elif backend == 'darwin':
	55	+ from .darwin import Darwin as PlatformImpl
52	56	elif backend == 'unix':
53	57	from .unix import Unix as PlatformImpl
54	58	else:
...	...	@@ -62,6 +66,9 @@ class Platform():
62	66	cls._create_instance()
63	67	return cls._instance
64	68
	69	+ def get_cpu_count(self, cap=None):
	70	+ return min(len(os.sched_getaffinity(0)), cap)
	71	+
65	72	##################################################################
66	73	# Sandbox functions #
67	74	##################################################################
...	...	@@ -84,3 +91,15 @@ class Platform():
84	91	def check_sandbox_config(self, config):
85	92	raise ImplError("Platform {platform} does not implement check_sandbox_config()"
86	93	.format(platform=type(self).__name__))
	94	+
	95	+ def set_resource_limits(self, soft_limit=None, hard_limit=None):
	96	+ # Need to set resources for _frontend/app.py as this is dependent on the platform
	97	+ # SafeHardlinks FUSE needs to hold file descriptors for all processes in the sandbox.
	98	+ # Avoid hitting the limit too quickly.
	99	+ limits = resource.getrlimit(resource.RLIMIT_NOFILE)
	100	+ if limits[0] != limits[1]:
	101	+ if soft_limit is None:
	102	+ soft_limit = limits[1]
	103	+ if hard_limit is None:
	104	+ hard_limit = limits[1]
	105	+ resource.setrlimit(resource.RLIMIT_NOFILE, (soft_limit, hard_limit))

buildstream/_platform/unix.py

---

...	...	@@ -20,7 +20,6 @@
20	20	import os
21	21
22	22	from .._exceptions import PlatformError
23		-from ..sandbox import SandboxChroot
24	23
25	24	from . import Platform
26	25
...	...	@@ -39,6 +38,7 @@ class Unix(Platform):
39	38	raise PlatformError("Root privileges are required to run without bubblewrap.")
40	39
41	40	def create_sandbox(self, *args, **kwargs):
	41	+ from ..sandbox._sandboxchroot import SandboxChroot
42	42	return SandboxChroot(*args, **kwargs)
43	43
44	44	def check_sandbox_config(self, config):

buildstream/_project.py

---

...	...	@@ -38,6 +38,7 @@ from ._loader import Loader
38	38	from .element import Element
39	39	from ._message import Message, MessageType
40	40	from ._includes import Includes
	41	+from ._platform import Platform
41	42
42	43
43	44	# Project Configuration file
...	...	@@ -617,7 +618,8 @@ class Project():
617	618	# Based on some testing (mainly on AWS), maximum effective
618	619	# max-jobs value seems to be around 8-10 if we have enough cores
619	620	# users should set values based on workload and build infrastructure
620		- output.base_variables['max-jobs'] = str(min(len(os.sched_getaffinity(0)), 8))
	621	+ platform = Platform.get_platform()
	622	+ output.base_variables['max-jobs'] = str(platform.get_cpu_count(8))
621	623
622	624	# Export options into variables, if that was requested
623	625	output.options.export_variables(output.base_variables)

buildstream/_yaml.py

---

...	...	@@ -467,7 +467,7 @@ def node_get_project_path(node, key, project_dir, *,
467	467	"{}: Specified path '{}' does not exist"
468	468	.format(provenance, path_str))
469	469
470		- is_inside = project_dir_path in full_resolved_path.parents or (
	470	+ is_inside = project_dir_path.resolve() in full_resolved_path.parents or (
471	471	full_resolved_path == project_dir_path)
472	472
473	473	if path.is_absolute() or not is_inside:

buildstream/sandbox/__init__.py

---

...	...	@@ -18,6 +18,5 @@
18	18	# Tristan Maat <tristan maat codethink co uk>
19	19
20	20	from .sandbox import Sandbox, SandboxFlags
21		-from ._sandboxchroot import SandboxChroot
22		-from ._sandboxbwrap import SandboxBwrap
23	21	from ._sandboxremote import SandboxRemote
	22	+from ._sandboxdummy import SandboxDummy

buildstream/sandbox/_sandboxdummy.py

---

	1	+#
	2	+# Copyright (C) 2017 Codethink Limited
	3	+#
	4	+# This program is free software; you can redistribute it and/or
	5	+# modify it under the terms of the GNU Lesser General Public
	6	+# License as published by the Free Software Foundation; either
	7	+# version 2 of the License, or (at your option) any later version.
	8	+#
	9	+# This library is distributed in the hope that it will be useful,
	10	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	11	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	12	+# Lesser General Public License for more details.
	13	+#
	14	+# You should have received a copy of the GNU Lesser General Public
	15	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	16	+#
	17	+# Authors:
	18	+
	19	+from .._exceptions import SandboxError
	20	+from . import Sandbox
	21	+
	22	+
	23	+class SandboxDummy(Sandbox):
	24	+ def __init__(self, *args, **kwargs):
	25	+ super().__init__(*args, **kwargs)
	26	+
	27	+ def run(self, command, flags, *, cwd=None, env=None):
	28	+
	29	+ # Fallback to the sandbox default settings for
	30	+ # the cwd and env.
	31	+ #
	32	+ cwd = self._get_work_directory(cwd=cwd)
	33	+ env = self._get_environment(cwd=cwd, env=env)
	34	+
	35	+ if not self._has_command(command[0], env):
	36	+ raise SandboxError("Staged artifacts do not provide command "
	37	+ "'{}'".format(command[0]),
	38	+ reason='missing-command')
	39	+
	40	+ raise SandboxError("This platform does not support local builds")

buildstream/utils.py

---

...	...	@@ -35,6 +35,7 @@ import tempfile
35	35	import itertools
36	36	import functools
37	37	from contextlib import contextmanager
	38	+from stat import S_ISDIR
38	39
39	40	import psutil
40	41
...	...	@@ -328,27 +329,25 @@ def safe_remove(path):
328	329	Raises:
329	330	UtilError: In the case of unexpected system call failures
330	331	"""
331		- if os.path.lexists(path):
332		-
333		- # Try to remove anything that is in the way, but issue
334		- # a warning instead if it removes a non empty directory
335		- try:
	332	+ try:
	333	+ if S_ISDIR(os.lstat(path).st_mode):
	334	+ os.rmdir(path)
	335	+ else:
336	336	os.unlink(path)
337		- except OSError as e:
338		- if e.errno != errno.EISDIR:
339		- raise UtilError("Failed to remove '{}': {}"
340		- .format(path, e))
341		-
342		- try:
343		- os.rmdir(path)
344		- except OSError as e:
345		- if e.errno == errno.ENOTEMPTY:
346		- return False
347		- else:
348		- raise UtilError("Failed to remove '{}': {}"
349		- .format(path, e))
350	337
351		- return True
	338	+ # File removed/unlinked successfully
	339	+ return True
	340	+
	341	+ except OSError as e:
	342	+ if e.errno == errno.ENOTEMPTY:
	343	+ # Path is non-empty directory
	344	+ return False
	345	+ elif e.errno == errno.ENOENT:
	346	+ # Path does not exist
	347	+ return True
	348	+
	349	+ raise UtilError("Failed to remove '{}': {}"
	350	+ .format(path, e))
352	351
353	352
354	353	def copy_files(src, dest, *, files=None, ignore_missing=False, report_written=False):

tests/format/project.py

---

...	...	@@ -188,3 +188,15 @@ def test_project_refs_options(cli, datafiles):
188	188
189	189	# Assert that the cache keys are different
190	190	assert result1.output != result2.output
	191	+
	192	+
	193	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'element-path'))
	194	+def test_element_path_project_path_contains_symlinks(cli, datafiles, tmpdir):
	195	+ real_project = str(datafiles)
	196	+ linked_project = os.path.join(str(tmpdir), 'linked')
	197	+ os.symlink(real_project, linked_project)
	198	+ os.makedirs(os.path.join(real_project, 'elements'), exist_ok=True)
	199	+ with open(os.path.join(real_project, 'elements', 'element.bst'), 'w') as f:
	200	+ f.write("kind: manual\n")
	201	+ result = cli.run(project=linked_project, args=['show', 'element.bst'])
	202	+ result.assert_success()

tests/integration/cachedfail.py

---

...	...	@@ -121,7 +121,7 @@ def test_build_depend_on_cached_fail(cli, tmpdir, datafiles):
121	121
122	122	@pytest.mark.skipif(not IS_LINUX, reason='Only available on linux')
123	123	@pytest.mark.datafiles(DATA_DIR)
124		-@pytest.mark.parametrize("on_error", ("continue",))
	124	+@pytest.mark.parametrize("on_error", ("continue", "quit"))
125	125	def test_push_cached_fail(cli, tmpdir, datafiles, on_error):
126	126	project = os.path.join(datafiles.dirname, datafiles.basename)
127	127	element_path = os.path.join(project, 'elements', 'element.bst')

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.