[Notes] [Git][BuildStream/buildstream][tpollard/483] 22 commits: setup.py: Add more metadata useful for PyPI

[Tom Pollard <gitlab mg gitlab com>]

Title: GitLab

Tom Pollard pushed to branch tpollard/483 at BuildStream / buildstream

Commits:

• 323a5403
  by Chandan Singh at 2018-08-29T12:13:28Z

  setup.py: Add more metadata useful for PyPI
  
  - setup.py: List useful links using `project_urls`
  
  - setup.py: Specify minumum python vresion using `python_requires`
  
    `python_requires` is the new way of specifying requirements for python
    version, as documented @
    https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires.
  
    This will also make this requirement appear nicely on the PyPI project
    page.
  
  - setup.py: Add project classifiers
  
    PyPI will use these classifiers to categorize projects while searching
    or browsing. Full list of classifiers can be found at
    https://pypi.org/classifiers.
  

• 5fc7b5ac
  by Tristan Van Berkom at 2018-08-29T12:37:08Z

  Merge branch 'chandan/setup.py-metadata' into 'master'
  
  setup.py: Add more metadata useful for PyPI
  
  See merge request BuildStream/buildstream!718

• 42ad937d
  by Josh Smith at 2018-08-29T12:38:15Z

  Prevent ValueError on URLs without an alias
  
  Fixes #597
  

• aa3a33b3
  by Tristan Van Berkom at 2018-08-29T13:03:58Z

  Merge branch 'Qinusty/597-non-alias-url-fix' into 'master'
  
  Fix ValueError traceback for URL without alias or URI scheme
  
  Closes #597
  
  See merge request BuildStream/buildstream!700

• 2a2a79de
  by Valentin David at 2018-08-29T14:11:48Z

  Disable round-tripping when element is not modified
  

• 2df7607c
  by Tristan Van Berkom at 2018-08-29T14:43:55Z

  Merge branch 'valentindavid/roundtripping_only_when_modified' into 'master'
  
  Disable round-tripping when element is not modified
  
  See merge request BuildStream/buildstream!748

• c146dde5
  by Benjamin Schubert at 2018-08-29T16:00:24Z

  Log not-found objects in the cache as SKIPPED
  

• 19e87afb
  by Benjamin Schubert at 2018-08-29T16:01:12Z

  Add tests that not-found objects in cache are shown as SKIPPED
  

• 9cc5817f
  by Qinusty at 2018-08-29T16:25:48Z

  Merge branch 'bschubert/log-missed-cache' into 'master'
  
  Log not-found objects in the cache as SKIPPED
  
  See merge request BuildStream/buildstream!729

• 3c8646e3
  by Josh Smith at 2018-08-29T16:34:22Z

  Add cyclic check within variable resolution
  
  This aims to address #600, this will raise an exception when a resolved
  variable contains a reference to the variable.
  

• d41c42d6
  by Josh Smith at 2018-08-29T16:34:22Z

  Add tests for cyclic variables check
  
  Note: This modifies the docker containers used for testing to supply the
  pytest-timeout package.
  

• c7b72c30
  by Qinusty at 2018-08-29T16:58:48Z

  Merge branch 'Qinusty/600-recursive-variables' into 'master'
  
  Add cyclic check within variable resolution
  
  See merge request BuildStream/buildstream!712

• d28e1353
  by Tristan Van Berkom at 2018-08-30T06:02:41Z

  tests/plugins/filter.py: Don't run redundant tests
  
  There is no reason that the filter element codepaths can behave
  differently depending on the Source implementation used in the test,
  as the Source implementation does not have any filter specific
  virtual methods.
  
  Removing the redundant tests and just performing these tests with the git source.
  

• e77ae07a
  by Tristan Van Berkom at 2018-08-30T06:36:41Z

  Merge branch 'tristan/reduce-filter-tests' into 'master'
  
  tests/plugins/filter.py: Don't run redundant tests
  
  See merge request BuildStream/buildstream!753

• 9ea40e2c
  by Tristan Van Berkom at 2018-08-30T06:38:13Z

  tests/frontend/mirror.py: Reenable test_mirror_fetch_upstream_absent[ostree]
  
  This test was skipped because of issue #538, but #538 was fixed
  and the test was still not reenabled.
  

• 8aec1102
  by Tristan Van Berkom at 2018-08-30T07:06:17Z

  Merge branch 'tristan/538-reenable-ostree-test' into 'master'
  
  tests/frontend/mirror.py: Reenable test_mirror_fetch_upstream_absent[ostree]
  
  See merge request BuildStream/buildstream!755

• fcc17c82
  by Tristan Van Berkom at 2018-08-30T08:44:34Z

  source.py: Stylistic changes in Source.__do_fetch()
  
  Added some comments to make the flow easier to follow, and
  removed an annoying 'success' variabled in favor of a for / else
  loop statement.
  

• 60ad044a
  by Tristan Van Berkom at 2018-08-30T08:44:34Z

  source.py: Move Source.mark_download_url() to the public methods.
  
  This was sitting in the section for abstract methods, but this
  is most definitely not an abstract method to be implemented by
  Sources.
  

• c339f917
  by Tristan Van Berkom at 2018-08-30T08:44:34Z

  plugins/source/git.py: Fix formatting of url in tracking
  
  This was displaying the aliased URL which is pretty useless,
  use the translated URL for the timed activity.
  

• 26565f11
  by Tristan Van Berkom at 2018-08-30T09:12:28Z

  Merge branch 'tristan/source-mirroring-changes' into 'master'
  
  Minor code changes revolving around source mirroring
  
  See merge request BuildStream/buildstream!758

• ce85bbde
  by Tom Pollard at 2018-08-30T10:27:48Z

  plugins/git.py: Warn if ref is not in given track
  
  Add a helper function assert_ref_in_track to git.py GitMirror()
  which is used when staging & initing the source workspace. It
  checks the element's ref exists in the track (branch/tag) if it
  has been specified, raising a warning if necessary. The warning makes
  use of the warning token 'REF_NOT_IN_TRACK' from the configurable
  CoreWarnings. If the element has been tracked with bst, it is assumed
  that the value of ref exists in the track as it was generated from it
  & as such is not asserted.
  

• 097b8ca5
  by Tom Pollard at 2018-08-30T10:27:48Z

  tests/sources/git.py: Add tests for REF_NOT_IN_TRACK
  
  Add tests that cover assert_ref_in_track & the configurable
  CoreWarnings REF_NOT_IN_TRACK warnings token.
  

17 changed files:

• .gitlab-ci.yml
• buildstream/_artifactcache/cascache.py
• buildstream/_exceptions.py
• buildstream/_variables.py
• buildstream/plugins/sources/git.py
• buildstream/source.py
• buildstream/utils.py
• dev-requirements.txt
• setup.py
• tests/format/variables.py
• + tests/format/variables/cyclic_variables/cyclic.bst
• + tests/format/variables/cyclic_variables/project.conf
• tests/frontend/mirror.py
• tests/frontend/pull.py
• tests/plugins/filter.py
• tests/sources/git.py
• tests/testutils/runcli.py

Changes:

.gitlab-ci.yml

---

...	...	@@ -84,25 +84,25 @@ source_dist:
84	84	- coverage-linux/
85	85
86	86	tests-debian-9:
87		- image: buildstream/testsuite-debian:9-master-114-4cab18e3
	87	+ image: buildstream/testsuite-debian:9-master-117-aa3a33b3
88	88	<<: *linux-tests
89	89
90	90	tests-fedora-27:
91		- image: buildstream/testsuite-fedora:27-master-114-4cab18e3
	91	+ image: buildstream/testsuite-fedora:27-master-117-aa3a33b3
92	92	<<: *linux-tests
93	93
94	94	tests-fedora-28:
95		- image: buildstream/testsuite-fedora:28-master-114-4cab18e3
	95	+ image: buildstream/testsuite-fedora:28-master-117-aa3a33b3
96	96	<<: *linux-tests
97	97
98	98	tests-ubuntu-18.04:
99		- image: buildstream/testsuite-ubuntu:18.04-master-114-4cab18e3
	99	+ image: buildstream/testsuite-ubuntu:18.04-master-117-aa3a33b3
100	100	<<: *linux-tests
101	101
102	102	tests-unix:
103	103	# Use fedora here, to a) run a test on fedora and b) ensure that we
104	104	# can get rid of ostree - this is not possible with debian-8
105		- image: buildstream/testsuite-fedora:27-master-114-4cab18e3
	105	+ image: buildstream/testsuite-fedora:27-master-117-aa3a33b3
106	106	stage: test
107	107	variables:
108	108	BST_FORCE_BACKEND: "unix"

buildstream/_artifactcache/cascache.py

---

...	...	@@ -249,6 +249,13 @@ class CASCache(ArtifactCache):
249	249	if e.code() != grpc.StatusCode.NOT_FOUND:
250	250	raise ArtifactError("Failed to pull artifact {}: {}".format(
251	251	element._get_brief_display_key(), e)) from e
	252	+ else:
	253	+ self.context.message(Message(
	254	+ None,
	255	+ MessageType.SKIPPED,
	256	+ "Remote ({}) does not have {} cached".format(
	257	+ remote.spec.url, element._get_brief_display_key())
	258	+ ))
252	259
253	260	return False
254	261

buildstream/_exceptions.py

---

...	...	@@ -217,6 +217,9 @@ class LoadErrorReason(Enum):
217	217	# A recursive include has been encountered.
218	218	RECURSIVE_INCLUDE = 21
219	219
	220	+ # A recursive variable has been encountered
	221	+ RECURSIVE_VARIABLE = 22
	222	+
220	223
221	224	# LoadError
222	225	#

buildstream/_variables.py

---

...	...	@@ -61,7 +61,7 @@ class Variables():
61	61	# LoadError, if the string contains unresolved variable references.
62	62	#
63	63	def subst(self, string):
64		- substitute, unmatched = self._subst(string, self.variables)
	64	+ substitute, unmatched, _ = self._subst(string, self.variables)
65	65	unmatched = list(set(unmatched))
66	66	if unmatched:
67	67	if len(unmatched) == 1:
...	...	@@ -82,6 +82,7 @@ class Variables():
82	82	def subst_callback(match):
83	83	nonlocal variables
84	84	nonlocal unmatched
	85	+ nonlocal matched
85	86
86	87	token = match.group(0)
87	88	varname = match.group(1)
...	...	@@ -91,6 +92,7 @@ class Variables():
91	92	# We have to check if the inner string has variables
92	93	# and return unmatches for those
93	94	unmatched += re.findall(_VARIABLE_MATCH, value)
	95	+ matched += [varname]
94	96	else:
95	97	# Return unmodified token
96	98	unmatched += [varname]
...	...	@@ -98,10 +100,11 @@ class Variables():
98	100
99	101	return value
100	102
	103	+ matched = []
101	104	unmatched = []
102	105	replacement = re.sub(_VARIABLE_MATCH, subst_callback, string)
103	106
104		- return (replacement, unmatched)
	107	+ return (replacement, unmatched, matched)
105	108
106	109	# Variable resolving code
107	110	#
...	...	@@ -131,7 +134,15 @@ class Variables():
131	134	# Ensure stringness of the value before substitution
132	135	value = _yaml.node_get(variables, str, key)
133	136
134		- resolved_var, item_unmatched = self._subst(value, variables)
	137	+ resolved_var, item_unmatched, matched = self._subst(value, variables)
	138	+
	139	+ if _wrap_variable(key) in resolved_var:
	140	+ referenced_through = find_recursive_variable(key, matched, variables)
	141	+ raise LoadError(LoadErrorReason.RECURSIVE_VARIABLE,
	142	+ "{}: ".format(_yaml.node_get_provenance(variables, key)) +
	143	+ ("Variable '{}' expands to contain a reference to itself. " +
	144	+ "Perhaps '{}' contains '{}").format(key, referenced_through, _wrap_variable(key)))
	145	+
135	146	resolved[key] = resolved_var
136	147	unmatched += item_unmatched
137	148
...	...	@@ -168,8 +179,21 @@ class Variables():
168	179	# Helper function to fetch information about the node referring to a variable
169	180	#
170	181	def _find_references(self, varname):
171		- fullname = '%{' + varname + '}'
	182	+ fullname = _wrap_variable(varname)
172	183	for key, value in _yaml.node_items(self.original):
173	184	if fullname in value:
174	185	provenance = _yaml.node_get_provenance(self.original, key)
175	186	yield (key, provenance)
	187	+
	188	+
	189	+def find_recursive_variable(variable, matched_variables, all_vars):
	190	+ matched_values = (_yaml.node_get(all_vars, str, key) for key in matched_variables)
	191	+ for key, value in zip(matched_variables, matched_values):
	192	+ if _wrap_variable(variable) in value:
	193	+ return key
	194	+ else:
	195	+ return None
	196	+
	197	+
	198	+def _wrap_variable(var):
	199	+ return "%{" + var + "}"

buildstream/plugins/sources/git.py

---

...	...	@@ -74,6 +74,9 @@ This plugin provides the following configurable warnings:
74	74
75	75	- 'git:inconsistent-submodule' - A submodule was found to be missing from the underlying git repository.
76	76
	77	+This plugin also utilises the following configurable core plugin warnings:
	78	+
	79	+- 'ref-not-in-track' - The provided ref was not found in the provided track in the element's git repository.
77	80	"""
78	81
79	82	import os
...	...	@@ -87,6 +90,7 @@ from configparser import RawConfigParser
87	90
88	91	from buildstream import Source, SourceError, Consistency, SourceFetcher
89	92	from buildstream import utils
	93	+from buildstream.plugin import CoreWarnings
90	94
91	95	GIT_MODULES = '.gitmodules'
92	96
...	...	@@ -199,7 +203,7 @@ class GitMirror(SourceFetcher):
199	203	cwd=self.mirror)
200	204	return output.rstrip('\n')
201	205
202		- def stage(self, directory):
	206	+ def stage(self, directory, track=None):
203	207	fullpath = os.path.join(directory, self.path)
204	208
205	209	# Using --shared here avoids copying the objects into the checkout, in any
...	...	@@ -213,10 +217,14 @@ class GitMirror(SourceFetcher):
213	217	fail="Failed to checkout git ref {}".format(self.ref),
214	218	cwd=fullpath)
215	219
	220	+ # Check that the user specified ref exists in the track if provided & not already tracked
	221	+ if track:
	222	+ self.assert_ref_in_track(fullpath, track)
	223	+
216	224	# Remove .git dir
217	225	shutil.rmtree(os.path.join(fullpath, ".git"))
218	226
219		- def init_workspace(self, directory):
	227	+ def init_workspace(self, directory, track=None):
220	228	fullpath = os.path.join(directory, self.path)
221	229	url = self.source.translate_url(self.url)
222	230
...	...	@@ -232,6 +240,10 @@ class GitMirror(SourceFetcher):
232	240	fail="Failed to checkout git ref {}".format(self.ref),
233	241	cwd=fullpath)
234	242
	243	+ # Check that the user specified ref exists in the track if provided & not already tracked
	244	+ if track:
	245	+ self.assert_ref_in_track(fullpath, track)
	246	+
235	247	# List the submodules (path/url tuples) present at the given ref of this repo
236	248	def submodule_list(self):
237	249	modules = "{}:{}".format(self.ref, GIT_MODULES)
...	...	@@ -296,6 +308,28 @@ class GitMirror(SourceFetcher):
296	308
297	309	return None
298	310
	311	+ # Assert that ref exists in track, if track has been specified.
	312	+ def assert_ref_in_track(self, fullpath, track):
	313	+ _, branch = self.source.check_output([self.source.host_git, 'branch', '--list', track,
	314	+ '--contains', self.ref],
	315	+ cwd=fullpath,)
	316	+ if branch:
	317	+ return True
	318	+ else:
	319	+ _, tag = self.source.check_output([self.source.host_git, 'tag', '--list', track,
	320	+ '--contains', self.ref],
	321	+ cwd=fullpath,)
	322	+ if tag:
	323	+ return True
	324	+
	325	+ detail = "The ref provided for the element does not exist locally in the provided track branch / tag " + \
	326	+ "'{}'.\nYou may wish to track the element to update the ref from '{}' ".format(track, track) + \
	327	+ "with `bst track`,\nor examine the upstream at '{}' for the specific ref.".format(self.url)
	328	+
	329	+ self.source.warn("{}: expected ref '{}' was not found in given track '{}' for staged repository: '{}'\n"
	330	+ .format(self.source, self.ref, track, self.url),
	331	+ detail=detail, warning_token=CoreWarnings.REF_NOT_IN_TRACK)
	332	+
299	333
300	334	class GitSource(Source):
301	335	# pylint: disable=attribute-defined-outside-init
...	...	@@ -333,6 +367,7 @@ class GitSource(Source):
333	367	self.submodule_checkout_overrides[path] = checkout
334	368
335	369	self.mark_download_url(self.original_url)
	370	+ self.tracked = False
336	371
337	372	def preflight(self):
338	373	# Check if git is installed, get the binary at the same time
...	...	@@ -387,8 +422,10 @@ class GitSource(Source):
387	422	detail=detail, reason="track-attempt-no-track")
388	423	return None
389	424
	425	+ # Resolve the URL for the message
	426	+ resolved_url = self.translate_url(self.mirror.url)
390	427	with self.timed_activity("Tracking {} from {}"
391		- .format(self.tracking, self.mirror.url),
	428	+ .format(self.tracking, resolved_url),
392	429	silent_nested=True):
393	430	self.mirror.ensure()
394	431	self.mirror._fetch()
...	...	@@ -396,6 +433,8 @@ class GitSource(Source):
396	433	# Update self.mirror.ref and node.ref from the self.tracking branch
397	434	ret = self.mirror.latest_commit(self.tracking)
398	435
	436	+ # Set tracked attribute, parameter for if self.mirror.assert_ref_in_track is needed
	437	+ self.tracked = True
399	438	return ret
400	439
401	440	def init_workspace(self, directory):
...	...	@@ -403,7 +442,7 @@ class GitSource(Source):
403	442	self.refresh_submodules()
404	443
405	444	with self.timed_activity('Setting up workspace "{}"'.format(directory), silent_nested=True):
406		- self.mirror.init_workspace(directory)
	445	+ self.mirror.init_workspace(directory, track=(self.tracking if not self.tracked else None))
407	446	for mirror in self.submodules:
408	447	mirror.init_workspace(directory)
409	448
...	...	@@ -419,7 +458,7 @@ class GitSource(Source):
419	458	# Stage the main repo in the specified directory
420	459	#
421	460	with self.timed_activity("Staging {}".format(self.mirror.url), silent_nested=True):
422		- self.mirror.stage(directory)
	461	+ self.mirror.stage(directory, track=(self.tracking if not self.tracked else None))
423	462	for mirror in self.submodules:
424	463	if mirror.path in self.submodule_checkout_overrides:
425	464	checkout = self.submodule_checkout_overrides[mirror.path]

buildstream/source.py

---

...	...	@@ -219,10 +219,7 @@ class SourceFetcher():
219	219	Args:
220	220	url (str): The url used to download.
221	221	"""
222		- # Not guaranteed to be a valid alias yet.
223		- # Ensuring it's a valid alias currently happens in Project.get_alias_uris
224		- alias, _ = url.split(utils._ALIAS_SEPARATOR, 1)
225		- self.__alias = alias
	222	+ self.__alias = _extract_alias(url)
226	223
227	224	#############################################################
228	225	# Private Methods used in BuildStream #
...	...	@@ -448,20 +445,6 @@ class Source(Plugin):
448	445	"""
449	446	self.stage(directory)
450	447
451		- def mark_download_url(self, url):
452		- """Identifies the URL that this Source uses to download
453		-
454		- This must be called during :func:`~buildstream.plugin.Plugin.configure` if
455		- :func:`~buildstream.source.Source.translate_url` is not called.
456		-
457		- Args:
458		- url (str): The url used to download
459		-
460		- *Since: 1.2*
461		- """
462		- alias, _ = url.split(utils._ALIAS_SEPARATOR, 1)
463		- self.__expected_alias = alias
464		-
465	448	def get_source_fetchers(self):
466	449	"""Get the objects that are used for fetching
467	450
...	...	@@ -525,12 +508,24 @@ class Source(Plugin):
525	508	else:
526	509	# Sneakily store the alias if it hasn't already been stored
527	510	if not self.__expected_alias and url and utils._ALIAS_SEPARATOR in url:
528		- url_alias, _ = url.split(utils._ALIAS_SEPARATOR, 1)
529		- self.__expected_alias = url_alias
	511	+ self.mark_download_url(url)
530	512
531	513	project = self._get_project()
532	514	return project.translate_url(url, first_pass=self.__first_pass)
533	515
	516	+ def mark_download_url(self, url):
	517	+ """Identifies the URL that this Source uses to download
	518	+
	519	+ This must be called during :func:`~buildstream.plugin.Plugin.configure` if
	520	+ :func:`~buildstream.source.Source.translate_url` is not called.
	521	+
	522	+ Args:
	523	+ url (str): The url used to download
	524	+
	525	+ *Since: 1.2*
	526	+ """
	527	+ self.__expected_alias = _extract_alias(url)
	528	+
534	529	def get_project_directory(self):
535	530	"""Fetch the project base directory
536	531
...	...	@@ -774,7 +769,8 @@ class Source(Plugin):
774	769	#
775	770	# Step 2 - Set the ref in memory, and determine changed state
776	771	#
777		- changed = self._set_ref(new_ref, node)
	772	+ if not self._set_ref(new_ref, node):
	773	+ return False
778	774
779	775	def do_save_refs(refs):
780	776	try:
...	...	@@ -811,7 +807,7 @@ class Source(Plugin):
811	807	.format(provenance.filename.shortname),
812	808	reason="tracking-junction-fragment")
813	809
814		- return changed
	810	+ return True
815	811
816	812	# Wrapper for track()
817	813	#
...	...	@@ -869,10 +865,12 @@ class Source(Plugin):
869	865	def __do_fetch(self, **kwargs):
870	866	project = self._get_project()
871	867	source_fetchers = self.get_source_fetchers()
	868	+
	869	+ # Use the source fetchers if they are provided
	870	+ #
872	871	if source_fetchers:
873	872	for fetcher in source_fetchers:
874	873	alias = fetcher._get_alias()
875		- success = False
876	874	for uri in project.get_alias_uris(alias, first_pass=self.__first_pass):
877	875	try:
878	876	fetcher.fetch(uri)
...	...	@@ -881,10 +879,16 @@ class Source(Plugin):
881	879	except BstError as e:
882	880	last_error = e
883	881	continue
884		- success = True
	882	+
	883	+ # No error, we're done with this fetcher
885	884	break
886		- if not success:
	885	+
	886	+ else:
	887	+ # No break occurred, raise the last detected error
887	888	raise last_error
	889	+
	890	+ # Default codepath is to reinstantiate the Source
	891	+ #
888	892	else:
889	893	alias = self._get_alias()
890	894	if self.__first_pass:
...	...	@@ -908,18 +912,22 @@ class Source(Plugin):
908	912	except BstError as e:
909	913	last_error = e
910	914	continue
	915	+
	916	+ # No error, we're done here
911	917	return
	918	+
	919	+ # Re raise the last detected error
912	920	raise last_error
913	921
914	922	# Tries to call track for every mirror, stopping once it succeeds
915	923	def __do_track(self, **kwargs):
916	924	project = self._get_project()
917		- # If there are no mirrors, or no aliases to replace, there's nothing to do here.
918	925	alias = self._get_alias()
919	926	if self.__first_pass:
920	927	mirrors = project.first_pass_config.mirrors
921	928	else:
922	929	mirrors = project.config.mirrors
	930	+ # If there are no mirrors, or no aliases to replace, there's nothing to do here.
923	931	if not mirrors or not alias:
924	932	return self.track(**kwargs)
925	933
...	...	@@ -988,3 +996,11 @@ class Source(Plugin):
988	996
989	997	if src.get_consistency() == Consistency.RESOLVED:
990	998	src._fetch(previous_sources[0:index])
	999	+
	1000	+
	1001	+def _extract_alias(url):
	1002	+ parts = url.split(utils._ALIAS_SEPARATOR, 1)
	1003	+ if len(parts) > 1 and not parts[0].lower() in utils._URI_SCHEMES:
	1004	+ return parts[0]
	1005	+ else:
	1006	+ return ""

buildstream/utils.py

---

...	...	@@ -47,6 +47,7 @@ _magic_timestamp = calendar.timegm([2011, 11, 11, 11, 11, 11])
47	47
48	48	# The separator we use for user specified aliases
49	49	_ALIAS_SEPARATOR = ':'
	50	+_URI_SCHEMES = ["http", "https", "ftp", "file", "git", "sftp", "ssh"]
50	51
51	52
52	53	class UtilError(BstError):

dev-requirements.txt

---

...	...	@@ -8,3 +8,4 @@ pytest-env
8	8	pytest-pep8
9	9	pytest-pylint
10	10	pytest-xdist
	11	+pytest-timeout

setup.py

---

...	...	@@ -25,7 +25,14 @@ import subprocess
25	25	import sys
26	26	import versioneer
27	27
28		-if sys.version_info[0] != 3 or sys.version_info[1] < 5:
	28	+
	29	+##################################################################
	30	+# Python requirements
	31	+##################################################################
	32	+REQUIRED_PYTHON_MAJOR = 3
	33	+REQUIRED_PYTHON_MINOR = 5
	34	+
	35	+if sys.version_info[0] != REQUIRED_PYTHON_MAJOR or sys.version_info[1] < REQUIRED_PYTHON_MINOR:
29	36	print("BuildStream requires Python >= 3.5")
30	37	sys.exit(1)
31	38
...	...	@@ -242,11 +249,28 @@ setup(name='BuildStream',
242	249
243	250	author='BuildStream Developers',
244	251	author_email='buildstream-list gnome org',
	252	+ classifiers=[
	253	+ 'Environment :: Console',
	254	+ 'Intended Audience :: Developers',
	255	+ 'License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+)',
	256	+ 'Operating System :: POSIX',
	257	+ 'Programming Language :: Python :: 3',
	258	+ 'Programming Language :: Python :: 3.5',
	259	+ 'Programming Language :: Python :: 3.6',
	260	+ 'Programming Language :: Python :: 3.7',
	261	+ 'Topic :: Software Development :: Build Tools'
	262	+ ],
245	263	description='A framework for modelling build pipelines in YAML',
246	264	license='LGPL',
247	265	long_description=long_description,
248	266	long_description_content_type='text/x-rst; charset=UTF-8',
249	267	url='https://gitlab.com/BuildStream/buildstream',
	268	+ project_urls={
	269	+ 'Documentation': 'https://buildstream.gitlab.io/buildstream/',
	270	+ 'Tracker': 'https://gitlab.com/BuildStream/buildstream/issues',
	271	+ 'Mailing List': 'https://mail.gnome.org/mailman/listinfo/buildstream-list'
	272	+ },
	273	+ python_requires='~={}.{}'.format(REQUIRED_PYTHON_MAJOR, REQUIRED_PYTHON_MINOR),
250	274	packages=find_packages(exclude=('tests', 'tests.*')),
251	275	package_data={'buildstream': ['plugins/*/*.py', 'plugins/*/*.yaml',
252	276	'data/*.yaml', 'data/*.sh.in']},

tests/format/variables.py

---

1	1	import os
2	2	import pytest
	3	+import sys
3	4	from buildstream import _yaml
4	5	from buildstream._exceptions import ErrorDomain, LoadErrorReason
5	6	from tests.testutils.runcli import cli
...	...	@@ -72,3 +73,20 @@ def test_missing_variable(cli, datafiles, tmpdir):
72	73	])
73	74	result.assert_main_error(ErrorDomain.LOAD,
74	75	LoadErrorReason.UNRESOLVED_VARIABLE)
	76	+
	77	+
	78	+@pytest.mark.timeout(3, method="signal")
	79	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'cyclic_variables'))
	80	+def test_cyclic_variables(cli, datafiles):
	81	+ print_warning("Performing cyclic test, if this test times out it will " +
	82	+ "exit the test sequence")
	83	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	84	+ result = cli.run(project=project, silent=True, args=[
	85	+ "build", "cyclic.bst"
	86	+ ])
	87	+ result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.RECURSIVE_VARIABLE)
	88	+
	89	+
	90	+def print_warning(msg):
	91	+ RED, END = "\033[91m", "\033[0m"
	92	+ print(("\n{}{}{}").format(RED, msg, END), file=sys.stderr)

tests/format/variables/cyclic_variables/cyclic.bst

---

	1	+kind: manual
	2	+
	3	+variables:
	4	+ a: "%{prefix}/a"
	5	+ prefix: "%{a}/some_prefix/"
		\ No newline at end of file

tests/format/variables/cyclic_variables/project.conf

---

1

+name: test

tests/frontend/mirror.py

---

...	...	@@ -142,10 +142,6 @@ def test_mirror_fetch(cli, tmpdir, datafiles, kind):
142	142	@pytest.mark.datafiles(DATA_DIR)
143	143	@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
144	144	def test_mirror_fetch_upstream_absent(cli, tmpdir, datafiles, kind):
145		- if kind == 'ostree':
146		- # FIXME: Mirroring fallback fails with ostree
147		- pytest.skip("Bug #538 - ostree mirror fallback breaks assertion")
148		-
149	145	bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
150	146	dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
151	147	upstream_repodir = os.path.join(str(tmpdir), 'upstream')

tests/frontend/pull.py

---

...	...	@@ -338,3 +338,22 @@ def test_pull_missing_blob(cli, tmpdir, datafiles):
338	338
339	339	# Assert that no artifacts were pulled
340	340	assert len(result.get_pulled_elements()) == 0
	341	+
	342	+
	343	+@pytest.mark.datafiles(DATA_DIR)
	344	+def test_pull_missing_notifies_user(caplog, cli, tmpdir, datafiles):
	345	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	346	+ caplog.set_level(1)
	347	+
	348	+ with create_artifact_share(os.path.join(str(tmpdir), 'artifactshare')) as share:
	349	+
	350	+ cli.configure({
	351	+ 'artifacts': {'url': share.repo}
	352	+ })
	353	+ result = cli.run(project=project, args=['build', 'target.bst'])
	354	+
	355	+ result.assert_success()
	356	+ assert not result.get_pulled_elements(), \
	357	+ "No elements should have been pulled since the cache was empty"
	358	+
	359	+ assert "SKIPPED Remote ({}) does not have".format(share.repo) in result.stderr

tests/plugins/filter.py

---

...	...	@@ -174,9 +174,8 @@ def test_filter_workspace_reset(datafiles, cli, tmpdir):
174	174
175	175
176	176	@pytest.mark.datafiles(os.path.join(DATA_DIR, 'basic'))
177		-@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS if kind not in ("patch", "local")])
178		-def test_filter_track(datafiles, cli, tmpdir, kind):
179		- repo = create_repo(kind, str(tmpdir))
	177	+def test_filter_track(datafiles, cli, tmpdir):
	178	+ repo = create_repo('git', str(tmpdir))
180	179	ref = repo.create(os.path.join(str(datafiles), "files"))
181	180	elements_dir = os.path.join(str(tmpdir), "elements")
182	181	project = str(tmpdir)
...	...	@@ -228,9 +227,8 @@ def test_filter_track(datafiles, cli, tmpdir, kind):
228	227
229	228
230	229	@pytest.mark.datafiles(os.path.join(DATA_DIR, 'basic'))
231		-@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS if kind not in ("patch", "local")])
232		-def test_filter_track_excepted(datafiles, cli, tmpdir, kind):
233		- repo = create_repo(kind, str(tmpdir))
	230	+def test_filter_track_excepted(datafiles, cli, tmpdir):
	231	+ repo = create_repo('git', str(tmpdir))
234	232	ref = repo.create(os.path.join(str(datafiles), "files"))
235	233	elements_dir = os.path.join(str(tmpdir), "elements")
236	234	project = str(tmpdir)
...	...	@@ -282,9 +280,8 @@ def test_filter_track_excepted(datafiles, cli, tmpdir, kind):
282	280
283	281
284	282	@pytest.mark.datafiles(os.path.join(DATA_DIR, 'basic'))
285		-@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS if kind not in ("patch", "local")])
286		-def test_filter_track_multi_to_one(datafiles, cli, tmpdir, kind):
287		- repo = create_repo(kind, str(tmpdir))
	283	+def test_filter_track_multi_to_one(datafiles, cli, tmpdir):
	284	+ repo = create_repo('git', str(tmpdir))
288	285	ref = repo.create(os.path.join(str(datafiles), "files"))
289	286	elements_dir = os.path.join(str(tmpdir), "elements")
290	287	project = str(tmpdir)
...	...	@@ -336,9 +333,8 @@ def test_filter_track_multi_to_one(datafiles, cli, tmpdir, kind):
336	333
337	334
338	335	@pytest.mark.datafiles(os.path.join(DATA_DIR, 'basic'))
339		-@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS if kind not in ("patch", "local")])
340		-def test_filter_track_multi(datafiles, cli, tmpdir, kind):
341		- repo = create_repo(kind, str(tmpdir))
	336	+def test_filter_track_multi(datafiles, cli, tmpdir):
	337	+ repo = create_repo('git', str(tmpdir))
342	338	ref = repo.create(os.path.join(str(datafiles), "files"))
343	339	elements_dir = os.path.join(str(tmpdir), "elements")
344	340	project = str(tmpdir)
...	...	@@ -398,9 +394,8 @@ def test_filter_track_multi(datafiles, cli, tmpdir, kind):
398	394
399	395
400	396	@pytest.mark.datafiles(os.path.join(DATA_DIR, 'basic'))
401		-@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS if kind not in ("patch", "local")])
402		-def test_filter_track_multi_exclude(datafiles, cli, tmpdir, kind):
403		- repo = create_repo(kind, str(tmpdir))
	397	+def test_filter_track_multi_exclude(datafiles, cli, tmpdir):
	398	+ repo = create_repo('git', str(tmpdir))
404	399	ref = repo.create(os.path.join(str(datafiles), "files"))
405	400	elements_dir = os.path.join(str(tmpdir), "elements")
406	401	project = str(tmpdir)

tests/sources/git.py

---

...	...	@@ -25,6 +25,7 @@ import pytest
25	25
26	26	from buildstream._exceptions import ErrorDomain
27	27	from buildstream import _yaml
	28	+from buildstream.plugin import CoreWarnings
28	29
29	30	from tests.testutils import cli, create_repo
30	31	from tests.testutils.site import HAVE_GIT
...	...	@@ -408,3 +409,70 @@ def test_submodule_track_no_ref_or_track(cli, tmpdir, datafiles):
408	409	result = cli.run(project=project, args=['show', 'target.bst'])
409	410	result.assert_main_error(ErrorDomain.SOURCE, "missing-track-and-ref")
410	411	result.assert_task_error(None, None)
	412	+
	413	+
	414	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	415	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	416	+def test_ref_not_in_track_warn(cli, tmpdir, datafiles):
	417	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	418	+
	419	+ # Create the repo from 'repofiles', create a branch without latest commit
	420	+ repo = create_repo('git', str(tmpdir))
	421	+ ref = repo.create(os.path.join(project, 'repofiles'))
	422	+
	423	+ gitsource = repo.source_config(ref=ref)
	424	+
	425	+ # Overwrite the track value to the added branch
	426	+ gitsource['track'] = 'foo'
	427	+
	428	+ # Write out our test target
	429	+ element = {
	430	+ 'kind': 'import',
	431	+ 'sources': [
	432	+ gitsource
	433	+ ]
	434	+ }
	435	+ _yaml.dump(element, os.path.join(project, 'target.bst'))
	436	+
	437	+ # Assert the warning is raised as ref is not in branch foo.
	438	+ # Assert warning not error to the user, when not set as fatal.
	439	+ result = cli.run(project=project, args=['build', 'target.bst'])
	440	+ assert "The ref provided for the element does not exist locally" in result.stderr
	441	+
	442	+
	443	+@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
	444	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'template'))
	445	+def test_ref_not_in_track_warn_error(cli, tmpdir, datafiles):
	446	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	447	+
	448	+ # Add fatal-warnings ref-not-in-track to project.conf
	449	+ project_template = {
	450	+ "name": "foo",
	451	+ "fatal-warnings": [CoreWarnings.REF_NOT_IN_TRACK]
	452	+ }
	453	+
	454	+ _yaml.dump(project_template, os.path.join(project, 'project.conf'))
	455	+
	456	+ # Create the repo from 'repofiles', create a branch without latest commit
	457	+ repo = create_repo('git', str(tmpdir))
	458	+ ref = repo.create(os.path.join(project, 'repofiles'))
	459	+
	460	+ gitsource = repo.source_config(ref=ref)
	461	+
	462	+ # Overwrite the track value to the added branch
	463	+ gitsource['track'] = 'foo'
	464	+
	465	+ # Write out our test target
	466	+ element = {
	467	+ 'kind': 'import',
	468	+ 'sources': [
	469	+ gitsource
	470	+ ]
	471	+ }
	472	+ _yaml.dump(element, os.path.join(project, 'target.bst'))
	473	+
	474	+ # Assert that build raises a warning here that is captured
	475	+ # as plugin error, due to the fatal warning being set
	476	+ result = cli.run(project=project, args=['build', 'target.bst'])
	477	+ result.assert_main_error(ErrorDomain.STREAM, None)
	478	+ result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.REF_NOT_IN_TRACK)

tests/testutils/runcli.py

---

...	...	@@ -94,14 +94,28 @@ class Result():
94	94	# error_reason (any): The reason field of the error which occurred
95	95	# fail_message (str): An optional message to override the automatic
96	96	# assertion error messages
	97	+ # debug (bool): If true, prints information regarding the exit state of the result()
97	98	# Raises:
98	99	# (AssertionError): If any of the assertions fail
99	100	#
100	101	def assert_main_error(self,
101	102	error_domain,
102	103	error_reason,
103		- fail_message=''):
104		-
	104	+ fail_message='',
	105	+ *, debug=False):
	106	+ if debug:
	107	+ print(
	108	+ """
	109	+ Exit code: {}
	110	+ Exception: {}
	111	+ Domain: {}
	112	+ Reason: {}
	113	+ """.format(
	114	+ self.exit_code,
	115	+ self.exception,
	116	+ self.exception.domain,
	117	+ self.exception.reason
	118	+ ))
105	119	assert self.exit_code == -1, fail_message
106	120	assert self.exc is not None, fail_message
107	121	assert self.exception is not None, fail_message

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.