[Notes] [Git][BuildStream/buildstream][mac_fixes] 11 commits: git.py: Make `ref` human readable

[Jürg Billeter <gitlab mg gitlab com>]

Title: GitLab

Jürg Billeter pushed to branch mac_fixes at BuildStream / buildstream

Commits:

• 900e8900
  by Jürg Billeter at 2018-09-27T13:12:34Z

  git.py: Make `ref` human readable
  
  Use `git describe` to prefix `ref` with closest tag, if available.
  

• 107269c1
  by Jürg Billeter at 2018-09-27T13:12:34Z

  testutils/repo/git.py: Add add_tag() method
  

• 8060ad8c
  by Jürg Billeter at 2018-09-27T13:12:34Z

  tests/sources/git.py: Add track and fetch test with and without tag
  

• ab1cb672
  by Jürg Billeter at 2018-09-27T13:46:41Z

  Merge branch 'juerg/git-describe' into 'master'
  
  git.py: Make `ref` human readable
  
  See merge request BuildStream/buildstream!291

• fcf37f9c
  by knownexus at 2018-09-27T14:22:09Z

  Added `set_resource_limits()` to platform
  
  This has been moved from app.py
  As it will have different functionality depending on platform
  Once the Darwin (MacOS) platform is added
  
  Removed `resource.setrlimit()` functionality from app.py
  Added `resource.setrlimit()` functionality to platform.py as function
  

• 80ebdd2c
  by knownexus at 2018-09-27T14:22:09Z

  Added FUSE check, as WSL does not support FUSE yet
  
  _platform/linux.py: Add FUSE check
  And Ensure exception is not raised when performing sandbox availability check
  Without this check, WSL would error upon finding FUSE missing
  This must be bypassed to allow remote execution
  

• 5e81573b
  by knownexus at 2018-09-27T14:22:09Z

  Max-jobs is now defined on a platform basis
  
  This is due to Darwin (MacOS) having a different Max-Jobs limit
  _platform/platform.py: Adding max-jobs call
  
  _project.py: redirected 'max-jobs' to point at the platform
  

• 3c93fe97
  by knownexus at 2018-09-27T14:22:09Z

  Adding Dummy sandbox
  
  This is to allow platforms that do not support sandboxing
  To error in a controlled/known way
  

• 0f3ef369
  by knownexus at 2018-09-27T14:22:24Z

  Adding darwin.py (MacOS) platform
  
  Adding functionality to recognise Darwin as a platform in plaform.py
  

• b78ae767
  by knownexus at 2018-09-27T14:29:29Z

  utils.py: Reworked safe_remove
  
  Non-Linux platforms don't return EISDIR when attempting to unlink a
  directory.
  
  Stopped safe_remove attempting to unlink dirs
  Previously safe_remove would attempt to unlink a path
  Before attempting to remove it if it was a dir
  
  Now it checks for a dir before that step
  

• ab5e78b4
  by James Ennis at 2018-09-27T14:29:32Z

  Import sandbox subclass when instantiating sandbox
  
  This patch was required due to not being able to execute `bst` on
  a Linux machine without FUSE available.
  

12 changed files:

• buildstream/_frontend/app.py
• + buildstream/_platform/darwin.py
• buildstream/_platform/linux.py
• buildstream/_platform/platform.py
• buildstream/_platform/unix.py
• buildstream/_project.py
• buildstream/plugins/sources/git.py
• buildstream/sandbox/__init__.py
• + buildstream/sandbox/_sandboxdummy.py
• buildstream/utils.py
• tests/sources/git.py
• tests/testutils/repo/git.py

Changes:

buildstream/_frontend/app.py

---

...	...	@@ -115,14 +115,6 @@ class App():
115	115	else:
116	116	self.colors = False
117	117
118		- # Increase the soft limit for open file descriptors to the maximum.
119		- # SafeHardlinks FUSE needs to hold file descriptors for all processes in the sandbox.
120		- # Avoid hitting the limit too quickly.
121		- limits = resource.getrlimit(resource.RLIMIT_NOFILE)
122		- if limits[0] != limits[1]:
123		- # Set soft limit to hard limit
124		- resource.setrlimit(resource.RLIMIT_NOFILE, (limits[1], limits[1]))
125		-
126	118	# create()
127	119	#
128	120	# Should be used instead of the regular constructor.

buildstream/_platform/darwin.py

---

	1	+#
	2	+# Copyright (C) 2017 Codethink Limited
	3	+# Copyright (C) 2018 Bloomberg Finance LP
	4	+#
	5	+# This program is free software; you can redistribute it and/or
	6	+# modify it under the terms of the GNU Lesser General Public
	7	+# License as published by the Free Software Foundation; either
	8	+# version 2 of the License, or (at your option) any later version.
	9	+#
	10	+# This library is distributed in the hope that it will be useful,
	11	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	12	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	13	+# Lesser General Public License for more details.
	14	+#
	15	+# You should have received a copy of the GNU Lesser General Public
	16	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	17	+
	18	+import os
	19	+import resource
	20	+
	21	+from .._exceptions import PlatformError
	22	+from ..sandbox import SandboxDummy
	23	+
	24	+from . import Platform
	25	+
	26	+
	27	+class Darwin(Platform):
	28	+
	29	+ # This value comes from OPEN_MAX in syslimits.h
	30	+ OPEN_MAX = 10240
	31	+
	32	+ def __init__(self):
	33	+
	34	+ super().__init__()
	35	+
	36	+ def create_sandbox(self, *args, **kwargs):
	37	+ return SandboxDummy(*args, **kwargs)
	38	+
	39	+ def check_sandbox_config(self, config):
	40	+ # Accept all sandbox configs as it's irrelevant with the dummy sandbox (no Sandbox.run).
	41	+ return True
	42	+
	43	+ def get_cpu_count(self, cap=None):
	44	+ if cap < os.cpu_count():
	45	+ return cap
	46	+ else:
	47	+ return os.cpu_count()
	48	+
	49	+ def set_resource_limits(self, soft_limit=OPEN_MAX, hard_limit=None):
	50	+ super().set_resource_limits(soft_limit)

buildstream/_platform/linux.py

---

...	...	@@ -23,7 +23,7 @@ import subprocess
23	23	from .. import _site
24	24	from .. import utils
25	25	from .._message import Message, MessageType
26		-from ..sandbox import SandboxBwrap
	26	+from ..sandbox import SandboxDummy
27	27
28	28	from . import Platform
29	29
...	...	@@ -38,13 +38,21 @@ class Linux(Platform):
38	38	self._gid = os.getegid()
39	39
40	40	self._die_with_parent_available = _site.check_bwrap_version(0, 1, 8)
41		- self._user_ns_available = self._check_user_ns_available()
	41	+
	42	+ if self._local_sandbox_available():
	43	+ self._user_ns_available = self._check_user_ns_available()
	44	+ else:
	45	+ self._user_ns_available = False
42	46
43	47	def create_sandbox(self, *args, **kwargs):
44		- # Inform the bubblewrap sandbox as to whether it can use user namespaces or not
45		- kwargs['user_ns_available'] = self._user_ns_available
46		- kwargs['die_with_parent_available'] = self._die_with_parent_available
47		- return SandboxBwrap(*args, **kwargs)
	48	+ if not self._local_sandbox_available():
	49	+ return SandboxDummy(*args, **kwargs)
	50	+ else:
	51	+ from ..sandbox._sandboxbwrap import SandboxBwrap
	52	+ # Inform the bubblewrap sandbox as to whether it can use user namespaces or not
	53	+ kwargs['user_ns_available'] = self._user_ns_available
	54	+ kwargs['die_with_parent_available'] = self._die_with_parent_available
	55	+ return SandboxBwrap(*args, **kwargs)
48	56
49	57	def check_sandbox_config(self, config):
50	58	if self._user_ns_available:
...	...	@@ -58,8 +66,13 @@ class Linux(Platform):
58	66	################################################
59	67	# Private Methods #
60	68	################################################
61		- def _check_user_ns_available(self):
	69	+ def _local_sandbox_available(self):
	70	+ try:
	71	+ return os.path.exists(utils.get_host_tool('bwrap')) and os.path.exists('/dev/fuse')
	72	+ except utils.ProgramNotFoundError:
	73	+ return False
62	74
	75	+ def _check_user_ns_available(self):
63	76	# Here, lets check if bwrap is able to create user namespaces,
64	77	# issue a warning if it's not available, and save the state
65	78	# locally so that we can inform the sandbox to not try it

buildstream/_platform/platform.py

---

...	...	@@ -19,6 +19,7 @@
19	19
20	20	import os
21	21	import sys
	22	+import resource
22	23
23	24	from .._exceptions import PlatformError, ImplError
24	25
...	...	@@ -32,23 +33,26 @@ class Platform():
32	33	# sandbox factory as well as platform helpers.
33	34	#
34	35	def __init__(self):
35		- pass
	36	+ self.set_resource_limits()
36	37
37	38	@classmethod
38	39	def _create_instance(cls):
39		- if sys.platform.startswith('linux'):
40		- backend = 'linux'
41		- else:
42		- backend = 'unix'
43		-
44	40	# Meant for testing purposes and therefore hidden in the
45	41	# deepest corners of the source code. Try not to abuse this,
46	42	# please?
47	43	if os.getenv('BST_FORCE_BACKEND'):
48	44	backend = os.getenv('BST_FORCE_BACKEND')
	45	+ elif sys.platform.startswith('linux'):
	46	+ backend = 'linux'
	47	+ elif sys.platform.startswith('darwin'):
	48	+ backend = 'darwin'
	49	+ else:
	50	+ backend = 'unix'
49	51
50	52	if backend == 'linux':
51	53	from .linux import Linux as PlatformImpl
	54	+ elif backend == 'darwin':
	55	+ from .darwin import Darwin as PlatformImpl
52	56	elif backend == 'unix':
53	57	from .unix import Unix as PlatformImpl
54	58	else:
...	...	@@ -62,6 +66,9 @@ class Platform():
62	66	cls._create_instance()
63	67	return cls._instance
64	68
	69	+ def get_cpu_count(self, cap=None):
	70	+ return min(len(os.sched_getaffinity(0)), cap)
	71	+
65	72	##################################################################
66	73	# Sandbox functions #
67	74	##################################################################
...	...	@@ -84,3 +91,15 @@ class Platform():
84	91	def check_sandbox_config(self, config):
85	92	raise ImplError("Platform {platform} does not implement check_sandbox_config()"
86	93	.format(platform=type(self).__name__))
	94	+
	95	+ def set_resource_limits(self, soft_limit=None, hard_limit=None):
	96	+ # Need to set resources for _frontend/app.py as this is dependent on the platform
	97	+ # SafeHardlinks FUSE needs to hold file descriptors for all processes in the sandbox.
	98	+ # Avoid hitting the limit too quickly.
	99	+ limits = resource.getrlimit(resource.RLIMIT_NOFILE)
	100	+ if limits[0] != limits[1]:
	101	+ if soft_limit is None:
	102	+ soft_limit = limits[1]
	103	+ if hard_limit is None:
	104	+ hard_limit = limits[1]
	105	+ resource.setrlimit(resource.RLIMIT_NOFILE, (soft_limit, hard_limit))

buildstream/_platform/unix.py

---

...	...	@@ -20,7 +20,6 @@
20	20	import os
21	21
22	22	from .._exceptions import PlatformError
23		-from ..sandbox import SandboxChroot
24	23
25	24	from . import Platform
26	25
...	...	@@ -39,6 +38,7 @@ class Unix(Platform):
39	38	raise PlatformError("Root privileges are required to run without bubblewrap.")
40	39
41	40	def create_sandbox(self, *args, **kwargs):
	41	+ from ..sandbox._sandboxchroot import SandboxChroot
42	42	return SandboxChroot(*args, **kwargs)
43	43
44	44	def check_sandbox_config(self, config):

buildstream/_project.py

---

...	...	@@ -38,6 +38,7 @@ from ._loader import Loader
38	38	from .element import Element
39	39	from ._message import Message, MessageType
40	40	from ._includes import Includes
	41	+from ._platform import Platform
41	42
42	43
43	44	# Project Configuration file
...	...	@@ -617,7 +618,8 @@ class Project():
617	618	# Based on some testing (mainly on AWS), maximum effective
618	619	# max-jobs value seems to be around 8-10 if we have enough cores
619	620	# users should set values based on workload and build infrastructure
620		- output.base_variables['max-jobs'] = str(min(len(os.sched_getaffinity(0)), 8))
	621	+ platform = Platform.get_platform()
	622	+ output.base_variables['max-jobs'] = str(platform.get_cpu_count(8))
621	623
622	624	# Export options into variables, if that was requested
623	625	output.options.export_variables(output.base_variables)

buildstream/plugins/sources/git.py

---

...	...	@@ -43,6 +43,12 @@ git - stage files from a git repository
43	43	# will be used to update the 'ref' when refreshing the pipeline.
44	44	track: master
45	45
	46	+ # Optionally specify the ref format used for tracking.
	47	+ # The default is 'sha1' for the raw commit hash.
	48	+ # If you specify 'git-describe', the commit hash will be prefixed
	49	+ # with the closest tag.
	50	+ ref-format: sha1
	51	+
46	52	# Specify the commit ref, this must be specified in order to
47	53	# checkout sources and build, but can be automatically updated
48	54	# if the 'track' attribute was specified.
...	...	@@ -205,7 +211,18 @@ class GitMirror(SourceFetcher):
205	211	[self.source.host_git, 'rev-parse', tracking],
206	212	fail="Unable to find commit for specified branch name '{}'".format(tracking),
207	213	cwd=self.mirror)
208		- return output.rstrip('\n')
	214	+ ref = output.rstrip('\n')
	215	+
	216	+ if self.source.ref_format == 'git-describe':
	217	+ # Prefix the ref with the closest tag, if available,
	218	+ # to make the ref human readable
	219	+ exit_code, output = self.source.check_output(
	220	+ [self.source.host_git, 'describe', '--tags', '--abbrev=40', '--long', ref],
	221	+ cwd=self.mirror)
	222	+ if exit_code == 0:
	223	+ ref = output.rstrip('\n')
	224	+
	225	+ return ref
209	226
210	227	def stage(self, directory, track=None):
211	228	fullpath = os.path.join(directory, self.path)
...	...	@@ -341,13 +358,18 @@ class GitSource(Source):
341	358	def configure(self, node):
342	359	ref = self.node_get_member(node, str, 'ref', None)
343	360
344		- config_keys = ['url', 'track', 'ref', 'submodules', 'checkout-submodules']
	361	+ config_keys = ['url', 'track', 'ref', 'submodules', 'checkout-submodules', 'ref-format']
345	362	self.node_validate(node, config_keys + Source.COMMON_CONFIG_KEYS)
346	363
347	364	self.original_url = self.node_get_member(node, str, 'url')
348	365	self.mirror = GitMirror(self, '', self.original_url, ref, primary=True)
349	366	self.tracking = self.node_get_member(node, str, 'track', None)
350	367
	368	+ self.ref_format = self.node_get_member(node, str, 'ref-format', 'sha1')
	369	+ if self.ref_format not in ['sha1', 'git-describe']:
	370	+ provenance = self.node_provenance(node, member_name='ref-format')
	371	+ raise SourceError("{}: Unexpected value for ref-format: {}".format(provenance, self.ref_format))
	372	+
351	373	# At this point we now know if the source has a ref and/or a track.
352	374	# If it is missing both then we will be unable to track or build.
353	375	if self.mirror.ref is None and self.tracking is None:

buildstream/sandbox/__init__.py

---

...	...	@@ -18,6 +18,5 @@
18	18	# Tristan Maat <tristan maat codethink co uk>
19	19
20	20	from .sandbox import Sandbox, SandboxFlags
21		-from ._sandboxchroot import SandboxChroot
22		-from ._sandboxbwrap import SandboxBwrap
23	21	from ._sandboxremote import SandboxRemote
	22	+from ._sandboxdummy import SandboxDummy

buildstream/sandbox/_sandboxdummy.py

---

	1	+#
	2	+# Copyright (C) 2017 Codethink Limited
	3	+#
	4	+# This program is free software; you can redistribute it and/or
	5	+# modify it under the terms of the GNU Lesser General Public
	6	+# License as published by the Free Software Foundation; either
	7	+# version 2 of the License, or (at your option) any later version.
	8	+#
	9	+# This library is distributed in the hope that it will be useful,
	10	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	11	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	12	+# Lesser General Public License for more details.
	13	+#
	14	+# You should have received a copy of the GNU Lesser General Public
	15	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	16	+#
	17	+# Authors:
	18	+
	19	+from .._exceptions import SandboxError
	20	+from . import Sandbox
	21	+
	22	+
	23	+class SandboxDummy(Sandbox):
	24	+ def __init__(self, *args, **kwargs):
	25	+ super().__init__(*args, **kwargs)
	26	+
	27	+ def run(self, command, flags, *, cwd=None, env=None):
	28	+
	29	+ # Fallback to the sandbox default settings for
	30	+ # the cwd and env.
	31	+ #
	32	+ cwd = self._get_work_directory(cwd=cwd)
	33	+ env = self._get_environment(cwd=cwd, env=env)
	34	+
	35	+ if not self._has_command(command[0], env):
	36	+ raise SandboxError("Staged artifacts do not provide command "
	37	+ "'{}'".format(command[0]),
	38	+ reason='missing-command')
	39	+
	40	+ raise SandboxError("This platform does not support local builds")

buildstream/utils.py

---

...	...	@@ -35,6 +35,7 @@ import tempfile
35	35	import itertools
36	36	import functools
37	37	from contextlib import contextmanager
	38	+from stat import S_ISDIR
38	39
39	40	import psutil
40	41
...	...	@@ -328,27 +329,25 @@ def safe_remove(path):
328	329	Raises:
329	330	UtilError: In the case of unexpected system call failures
330	331	"""
331		- if os.path.lexists(path):
332		-
333		- # Try to remove anything that is in the way, but issue
334		- # a warning instead if it removes a non empty directory
335		- try:
	332	+ try:
	333	+ if S_ISDIR(os.lstat(path).st_mode):
	334	+ os.rmdir(path)
	335	+ else:
336	336	os.unlink(path)
337		- except OSError as e:
338		- if e.errno != errno.EISDIR:
339		- raise UtilError("Failed to remove '{}': {}"
340		- .format(path, e))
341		-
342		- try:
343		- os.rmdir(path)
344		- except OSError as e:
345		- if e.errno == errno.ENOTEMPTY:
346		- return False
347		- else:
348		- raise UtilError("Failed to remove '{}': {}"
349		- .format(path, e))
350	337
351		- return True
	338	+ # File removed/unlinked successfully
	339	+ return True
	340	+
	341	+ except OSError as e:
	342	+ if e.errno == errno.ENOTEMPTY:
	343	+ # Path is non-empty directory
	344	+ return False
	345	+ elif e.errno == errno.ENOENT:
	346	+ # Path does not exist
	347	+ return True
	348	+
	349	+ raise UtilError("Failed to remove '{}': {}"
	350	+ .format(path, e))
352	351
353	352
354	353	def copy_files(src, dest, *, files=None, ignore_missing=False, report_written=False):

tests/sources/git.py

---

...	...	@@ -476,3 +476,50 @@ def test_ref_not_in_track_warn_error(cli, tmpdir, datafiles):
476	476	result = cli.run(project=project, args=['build', 'target.bst'])
477	477	result.assert_main_error(ErrorDomain.STREAM, None)
478	478	result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.REF_NOT_IN_TRACK)
	479	+
	480	+
	481	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	482	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	483	+@pytest.mark.parametrize("ref_format", ['sha1', 'git-describe'])
	484	+@pytest.mark.parametrize("tag,extra_commit", [(False, False), (True, False), (True, True)])
	485	+def test_track_fetch(cli, tmpdir, datafiles, ref_format, tag, extra_commit):
	486	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	487	+
	488	+ # Create the repo from 'repofiles' subdir
	489	+ repo = create_repo('git', str(tmpdir))
	490	+ ref = repo.create(os.path.join(project, 'repofiles'))
	491	+ if tag:
	492	+ repo.add_tag('tag')
	493	+ if extra_commit:
	494	+ repo.add_commit()
	495	+
	496	+ # Write out our test target
	497	+ element = {
	498	+ 'kind': 'import',
	499	+ 'sources': [
	500	+ repo.source_config()
	501	+ ]
	502	+ }
	503	+ element['sources'][0]['ref-format'] = ref_format
	504	+ element_path = os.path.join(project, 'target.bst')
	505	+ _yaml.dump(element, element_path)
	506	+
	507	+ # Track it
	508	+ result = cli.run(project=project, args=['track', 'target.bst'])
	509	+ result.assert_success()
	510	+
	511	+ element = _yaml.load(element_path)
	512	+ new_ref = element['sources'][0]['ref']
	513	+
	514	+ if ref_format == 'git-describe' and tag:
	515	+ # Check and strip prefix
	516	+ prefix = 'tag-{}-g'.format(0 if not extra_commit else 1)
	517	+ assert new_ref.startswith(prefix)
	518	+ new_ref = new_ref[len(prefix):]
	519	+
	520	+ # 40 chars for SHA-1
	521	+ assert len(new_ref) == 40
	522	+
	523	+ # Fetch it
	524	+ result = cli.run(project=project, args=['fetch', 'target.bst'])
	525	+ result.assert_success()

tests/testutils/repo/git.py

---

...	...	@@ -42,6 +42,9 @@ class Git(Repo):
42	42	self._run_git('commit', '-m', 'Initial commit')
43	43	return self.latest_commit()
44	44
	45	+ def add_tag(self, tag):
	46	+ self._run_git('tag', tag)
	47	+
45	48	def add_commit(self):
46	49	self._run_git('commit', '--allow-empty', '-m', 'Additional commit')
47	50	return self.latest_commit()

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.