[Notes] [Git][BuildStream/buildstream][mac_fixes] 11 commits: source/install_source.rst: pip plugin depends on host pip

[Phillip Smyth <gitlab mg gitlab com>]

Title: GitLab

Phillip Smyth pushed to branch mac_fixes at BuildStream / buildstream

Commits:

• 6f925bcb
  by Javier Jardón at 2018-09-13T07:38:48Z

  source/install_source.rst: pip plugin depends on host pip
  

• c6155f8d
  by Javier Jardón at 2018-09-13T08:11:54Z

  Merge branch 'jjardon/pip_dependency' into 'master'
  
  source/install_source.rst: pip plugin depends on host pip
  
  See merge request BuildStream/buildstream!791

• 19838a07
  by Chandan Singh at 2018-09-13T10:58:38Z

  README.rst: Add status badges for PyPI release and Python versions
  
  The first badge will work fine right away while the second badge will
  show "not found" until a release is made after merging this branch:
  https://gitlab.com/BuildStream/buildstream/merge_requests/718.
  

• 3b81d451
  by Chandan Singh at 2018-09-13T12:27:59Z

  Merge branch 'chandan/pypi-badge' into 'master'
  
  README.rst: Add status badges for PyPI release and Python versions
  
  See merge request BuildStream/buildstream!719

• 40f20f25
  by knownexus at 2018-09-13T13:09:44Z

  Moved `resource.getrlimit()`
  
  Removed `resource.getrlimit()` functionality from app.py
  Added `resource.getrlimit()` functionality to platform.py as function
  Called new function from __init__.py of linux.py and unix.py
  

• 0a3ae83a
  by knownexus at 2018-09-13T13:09:44Z

  Add FUSE check to linux.py
  
  This needed to be implemented as Windows WSL
  Does not currently support FUSE
  

• 3bb13520
  by knownexus at 2018-09-13T13:09:44Z

  Moving max-jobs call to Platform
  

• 3699693c
  by knownexus at 2018-09-13T13:09:44Z

  Move Cas call to Platform
  

• 89b58b07
  by knownexus at 2018-09-13T13:09:44Z

  Adding darwin.py platform
  
  Adding functionality to recognise Darwin as a platform in plaform.py
  

• deef9bd5
  by knownexus at 2018-09-13T13:09:44Z

  Stopped safe_remove attempting to unlink dirs
  
  Previously safe_remove would attempt to unlink a path
  Before attempting to remove it if it was a dir
  
  Now it checks for a dir before that step
  

• d9627c5b
  by knownexus at 2018-09-14T10:12:52Z

  Adding Dummy sandbox and nolocal platform
  

11 changed files:

• README.rst
• buildstream/_frontend/app.py
• + buildstream/_platform/darwin.py
• buildstream/_platform/linux.py
• buildstream/_platform/platform.py
• buildstream/_platform/unix.py
• buildstream/_project.py
• buildstream/sandbox/__init__.py
• + buildstream/sandbox/_dummysandbox.py
• buildstream/utils.py
• doc/source/install_source.rst

Changes:

README.rst

---

...	...	@@ -13,6 +13,9 @@ About
13	13	.. image:: https://gitlab.com/BuildStream/buildstream/badges/master/coverage.svg?job=coverage
14	14	:target: https://gitlab.com/BuildStream/buildstream/commits/master
15	15
	16	+.. image:: https://img.shields.io/pypi/v/BuildStream.svg
	17	+ :target: https://pypi.org/project/BuildStream
	18	+
16	19
17	20	What is BuildStream?
18	21	====================

buildstream/_frontend/app.py

---

...	...	@@ -116,14 +116,6 @@ class App():
116	116	else:
117	117	self.colors = False
118	118
119		- # Increase the soft limit for open file descriptors to the maximum.
120		- # SafeHardlinks FUSE needs to hold file descriptors for all processes in the sandbox.
121		- # Avoid hitting the limit too quickly.
122		- limits = resource.getrlimit(resource.RLIMIT_NOFILE)
123		- if limits[0] != limits[1]:
124		- # Set soft limit to hard limit
125		- resource.setrlimit(resource.RLIMIT_NOFILE, (limits[1], limits[1]))
126		-
127	119	# create()
128	120	#
129	121	# Should be used instead of the regular constructor.

buildstream/_platform/darwin.py

---

	1	+#
	2	+# Copyright (C) 2017 Codethink Limited
	3	+# Copyright (C) 2018 Bloomberg Finance LP
	4	+#
	5	+# This program is free software; you can redistribute it and/or
	6	+# modify it under the terms of the GNU Lesser General Public
	7	+# License as published by the Free Software Foundation; either
	8	+# version 2 of the License, or (at your option) any later version.
	9	+#
	10	+# This library is distributed in the hope that it will be useful,
	11	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	12	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	13	+# Lesser General Public License for more details.
	14	+#
	15	+# You should have received a copy of the GNU Lesser General Public
	16	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	17	+
	18	+import os
	19	+import resource
	20	+
	21	+from .._exceptions import PlatformError
	22	+from ..sandbox import SandboxChroot, DummySandbox
	23	+
	24	+from . import Platform
	25	+
	26	+
	27	+class Darwin(Platform):
	28	+
	29	+ # This value comes from OPEN_MAX in syslimits.h
	30	+ OPEN_MAX = 10240
	31	+
	32	+ def __init__(self, context):
	33	+
	34	+ super().__init__(context)
	35	+
	36	+ @property
	37	+ def artifactcache(self):
	38	+ return self._artifact_cache
	39	+
	40	+ def create_sandbox(self, *args, **kwargs):
	41	+ return DummySandbox(*args, **kwargs)
	42	+
	43	+ def get_cpu_count(self, cap=None):
	44	+ if cap < os.cpu_count():
	45	+ return cap
	46	+ else:
	47	+ return os.cpu_count()
	48	+
	49	+ def set_resources(self, soft_limit=OPEN_MAX, hard_limit=None):
	50	+ super().set_resources(soft_limit)

buildstream/_platform/linux.py

---

...	...	@@ -17,13 +17,14 @@
17	17	# Authors:
18	18	# Tristan Maat <tristan maat codethink co uk>
19	19
	20	+import os
20	21	import subprocess
21	22
22	23	from .. import _site
23	24	from .. import utils
24	25	from .._artifactcache.cascache import CASCache
25	26	from .._message import Message, MessageType
26		-from ..sandbox import SandboxBwrap
	27	+from ..sandbox import SandboxBwrap, DummySandbox
27	28
28	29	from . import Platform
29	30
...	...	@@ -35,7 +36,11 @@ class Linux(Platform):
35	36	super().__init__(context)
36	37
37	38	self._die_with_parent_available = _site.check_bwrap_version(0, 1, 8)
38		- self._user_ns_available = self._check_user_ns_available(context)
	39	+
	40	+ if self._local_sandbox_available():
	41	+ self._user_ns_available = self._check_user_ns_available(context)
	42	+ else:
	43	+ self._user_ns_available = False
39	44	self._artifact_cache = CASCache(context, enable_push=self._user_ns_available)
40	45
41	46	@property
...	...	@@ -43,16 +48,21 @@ class Linux(Platform):
43	48	return self._artifact_cache
44	49
45	50	def create_sandbox(self, *args, **kwargs):
46		- # Inform the bubblewrap sandbox as to whether it can use user namespaces or not
47		- kwargs['user_ns_available'] = self._user_ns_available
48		- kwargs['die_with_parent_available'] = self._die_with_parent_available
49		- return SandboxBwrap(*args, **kwargs)
	51	+ if not self._local_sandbox_available():
	52	+ return DummySandbox(*args, **kwargs)
	53	+ else:
	54	+ # Inform the bubblewrap sandbox as to whether it can use user namespaces or not
	55	+ kwargs['user_ns_available'] = self._user_ns_available
	56	+ kwargs['die_with_parent_available'] = self._die_with_parent_available
	57	+ return SandboxBwrap(*args, **kwargs)
50	58
51	59	################################################
52	60	# Private Methods #
53	61	################################################
54		- def _check_user_ns_available(self, context):
	62	+ def _local_sandbox_available(self):
	63	+ return os.path.exists(utils.get_host_tool('bwrap')) and os.path.exists('/dev/fuse')
55	64
	65	+ def _check_user_ns_available(self, context):
56	66	# Here, lets check if bwrap is able to create user namespaces,
57	67	# issue a warning if it's not available, and save the state
58	68	# locally so that we can inform the sandbox to not try it

buildstream/_platform/platform.py

---

...	...	@@ -19,8 +19,10 @@
19	19
20	20	import os
21	21	import sys
	22	+import resource
22	23
23	24	from .._exceptions import PlatformError, ImplError
	25	+from .._artifactcache.cascache import CASCache
24	26
25	27
26	28	class Platform():
...	...	@@ -37,22 +39,30 @@ class Platform():
37	39	#
38	40	def __init__(self, context):
39	41	self.context = context
	42	+ self.set_resources()
	43	+ self._artifact_cache = CASCache(context)
40	44
41	45	@classmethod
42	46	def create_instance(cls, *args, **kwargs):
43		- if sys.platform.startswith('linux'):
44		- backend = 'linux'
45		- else:
46		- backend = 'unix'
47	47
48	48	# Meant for testing purposes and therefore hidden in the
49	49	# deepest corners of the source code. Try not to abuse this,
50	50	# please?
51	51	if os.getenv('BST_FORCE_BACKEND'):
52	52	backend = os.getenv('BST_FORCE_BACKEND')
	53	+ elif sys.platform.startswith('linux'):
	54	+ backend = 'linux'
	55	+ elif sys.platform.startswith('darwin'):
	56	+ backend = 'darwin'
	57	+ elif not (os.path.exists(utils.get_host_tool('bwrap')) and os.path.exists('/dev/fuse')):
	58	+ backend = 'no_local'
	59	+ else:
	60	+ backend = 'unix'
53	61
54	62	if backend == 'linux':
55	63	from .linux import Linux as PlatformImpl
	64	+ elif backend == 'darwin':
	65	+ from .darwin import Darwin as PlatformImpl
56	66	elif backend == 'unix':
57	67	from .unix import Unix as PlatformImpl
58	68	else:
...	...	@@ -66,6 +76,9 @@ class Platform():
66	76	raise PlatformError("Platform needs to be initialized first")
67	77	return cls._instance
68	78
	79	+ def get_cpu_count(self, cap=None):
	80	+ return min(len(os.sched_getaffinity(0)), cap)
	81	+
69	82	##################################################################
70	83	# Platform properties #
71	84	##################################################################
...	...	@@ -92,3 +105,15 @@ class Platform():
92	105	def create_sandbox(self, *args, **kwargs):
93	106	raise ImplError("Platform {platform} does not implement create_sandbox()"
94	107	.format(platform=type(self).__name__))
	108	+
	109	+ def set_resources(self, soft_limit=None, hard_limit=None):
	110	+ # Need to set resources for _frontend/app.py as this is dependent on the platform
	111	+ # SafeHardlinks FUSE needs to hold file descriptors for all processes in the sandbox.
	112	+ # Avoid hitting the limit too quickly.
	113	+ limits = resource.getrlimit(resource.RLIMIT_NOFILE)
	114	+ if limits[0] != limits[1]:
	115	+ if soft_limit is None:
	116	+ soft_limit = limits[1]
	117	+ if hard_limit is None:
	118	+ hard_limit = limits[1]
	119	+ resource.setrlimit(resource.RLIMIT_NOFILE, (soft_limit, hard_limit))

buildstream/_platform/unix.py

---

...	...	@@ -31,7 +31,6 @@ class Unix(Platform):
31	31	def __init__(self, context):
32	32
33	33	super().__init__(context)
34		- self._artifact_cache = CASCache(context)
35	34
36	35	# Not necessarily 100% reliable, but we want to fail early.
37	36	if os.geteuid() != 0:

buildstream/_project.py

---

...	...	@@ -38,6 +38,7 @@ from ._loader import Loader
38	38	from .element import Element
39	39	from ._message import Message, MessageType
40	40	from ._includes import Includes
	41	+from ._platform import Platform
41	42
42	43
43	44	# Project Configuration file
...	...	@@ -611,7 +612,8 @@ class Project():
611	612	# Based on some testing (mainly on AWS), maximum effective
612	613	# max-jobs value seems to be around 8-10 if we have enough cores
613	614	# users should set values based on workload and build infrastructure
614		- output.base_variables['max-jobs'] = str(min(len(os.sched_getaffinity(0)), 8))
	615	+ platform = Platform.get_platform()
	616	+ output.base_variables['max-jobs'] = str(platform.get_cpu_count(8))
615	617
616	618	# Export options into variables, if that was requested
617	619	output.options.export_variables(output.base_variables)

buildstream/sandbox/__init__.py

---

...	...	@@ -21,3 +21,4 @@ from .sandbox import Sandbox, SandboxFlags
21	21	from ._sandboxchroot import SandboxChroot
22	22	from ._sandboxbwrap import SandboxBwrap
23	23	from ._sandboxremote import SandboxRemote
	24	+from ._dummysandbox import DummySandbox

buildstream/sandbox/_dummysandbox.py

---

	1	+#
	2	+# Copyright (C) 2017 Codethink Limited
	3	+#
	4	+# This program is free software; you can redistribute it and/or
	5	+# modify it under the terms of the GNU Lesser General Public
	6	+# License as published by the Free Software Foundation; either
	7	+# version 2 of the License, or (at your option) any later version.
	8	+#
	9	+# This library is distributed in the hope that it will be useful,
	10	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	11	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	12	+# Lesser General Public License for more details.
	13	+#
	14	+# You should have received a copy of the GNU Lesser General Public
	15	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	16	+#
	17	+# Authors:
	18	+
	19	+from .._exceptions import SandboxError
	20	+from . import Sandbox
	21	+
	22	+
	23	+class DummySandbox(Sandbox):
	24	+ def __init__(self, *args, **kwargs):
	25	+ super().__init__(*args, **kwargs)
	26	+
	27	+ uid = self._get_config().build_uid
	28	+ gid = self._get_config().build_gid
	29	+ if uid != 0 or gid != 0:
	30	+ raise SandboxError("Chroot sandboxes cannot specify a non-root uid/gid "
	31	+ "({},{} were supplied via config)".format(uid, gid))
	32	+
	33	+ self.mount_map = None
	34	+
	35	+ def run(self, command, flags, *, cwd=None, env=None):
	36	+
	37	+ # Default settings
	38	+ if cwd is None:
	39	+ cwd = self._get_work_directory()
	40	+
	41	+ if cwd is None:
	42	+ cwd = '/'
	43	+
	44	+ if env is None:
	45	+ env = self._get_environment()
	46	+
	47	+ # Naive getcwd implementations can break when bind-mounts to different
	48	+ # paths on the same filesystem are present. Letting the command know
	49	+ # what directory it is in makes it unnecessary to call the faulty
	50	+ # getcwd.
	51	+ env['PWD'] = cwd
	52	+
	53	+ if not self._has_command(command[0], env):
	54	+ raise SandboxError("Staged artifacts do not provide command "
	55	+ "'{}'".format(command[0]),
	56	+ reason='missing-command')
	57	+
	58	+ raise SandboxError("This platform does not support local builds")

buildstream/utils.py

---

...	...	@@ -35,6 +35,7 @@ import tempfile
35	35	import itertools
36	36	import functools
37	37	from contextlib import contextmanager
	38	+from stat import S_ISDIR
38	39
39	40	import psutil
40	41
...	...	@@ -328,26 +329,28 @@ def safe_remove(path):
328	329	Raises:
329	330	UtilError: In the case of unexpected system call failures
330	331	"""
	332	+ # return True if path does not exist
331	333	if os.path.lexists(path):
	334	+ # Check if path is a directory
	335	+ if not S_ISDIR(os.lstat(path).st_mode):
	336	+ # If path is not a directory, try to unlink
	337	+ try:
	338	+ os.unlink(path)
	339	+ return True
	340	+ except OSError as e:
	341	+ raise UtilError("Failed to remove '{}': {}"
	342	+ .format(path, e))
332	343
333		- # Try to remove anything that is in the way, but issue
334		- # a warning instead if it removes a non empty directory
	344	+ # If path is a directory, try to remove
335	345	try:
336		- os.unlink(path)
	346	+ os.rmdir(path)
337	347	except OSError as e:
338		- if e.errno != errno.EISDIR:
	348	+ if e.errno == errno.ENOTEMPTY:
	349	+ return False
	350	+ else:
339	351	raise UtilError("Failed to remove '{}': {}"
340	352	.format(path, e))
341	353
342		- try:
343		- os.rmdir(path)
344		- except OSError as e:
345		- if e.errno == errno.ENOTEMPTY:
346		- return False
347		- else:
348		- raise UtilError("Failed to remove '{}': {}"
349		- .format(path, e))
350		-
351	354	return True
352	355
353	356

doc/source/install_source.rst

---

...	...	@@ -29,6 +29,7 @@ The default plugins with extra host dependencies are:
29	29	* git
30	30	* ostree
31	31	* patch
	32	+* pip
32	33	* tar
33	34
34	35	If you intend to push built artifacts to a remote artifact server,

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.