[Notes] [Git][BuildStream/buildstream][coldtom/collections] 11 commits: tests/frontend/buildcheckout.py: Fix bad filename issue for Windows

[Tristan Van Berkom <gitlab mg gitlab com>]

Title: GitLab

Tristan Van Berkom pushed to branch coldtom/collections at BuildStream / buildstream

Commits:

• ca547f19
  by Chandan Singh at 2019-01-03T19:34:49Z

  tests/frontend/buildcheckout.py: Fix bad filename issue for Windows
  
  In BuildStream/buildstream!1028, we added a test specifically to test
  that BuildStream correctly raises an warning when the name of an element
  contains characters that are invalid on Windows. Unfortunately, we
  didn't see it coming that it would make it impossible to checkout this
  branch on Windows.
  
  Fix it by generating this file, only if we are not running on Windows.
  
  * tests/testutils/site.py: Add `IS_WINDOWS` check
  * tests/frontend/buildcheckout.py: Generate file with invalid filename
    on the fly
  * Remove tests/frontend/project/elements/invalid-chars.
  
  Fixes #842.
  
  Note that this may still cause issues on WSL when running tests on a
  shared filesystem, but that seems to be a generic issue on WSL with
  `os.rename`.
  

• 01c3761c
  by Chandan Singh at 2019-01-03T20:03:32Z

  Merge branch 'chandan/fix-bad-filename-mini-disaster' into 'master'
  
  tests/frontend/buildcheckout.py: Fix bad filename issue for Windows
  
  Closes #842
  
  See merge request BuildStream/buildstream!1032

• 7850e4e2
  by Tristan Van Berkom at 2019-01-03T20:55:00Z

  setup.cfg: Specify the pycodestyle configurations separately from pytest configuration
  

• 3c9d592d
  by Tristan Van Berkom at 2019-01-03T20:55:00Z

  setup.cfg: Don't lint by default through setup.py test invocations
  

• feba48e7
  by Tristan Van Berkom at 2019-01-03T20:55:00Z

  tox.ini: Added new 'lint' environment to lint separately
  

• 9caa0ea0
  by Tristan Van Berkom at 2019-01-03T20:55:00Z

  .gitlab-ci.yml: Update the CI to run the linter separately through tox
  
  This should save us some cycles in CI, and also allow the
  developer to more conveniently lint separately from testing.
  

• 91c8b634
  by Tristan Van Berkom at 2019-01-03T21:04:49Z

  CONTRIBUTING.rst: Adding instructions to run the linter separately
  

• 892bf99e
  by Tristan Van Berkom at 2019-01-03T21:31:04Z

  tools/requirements.{in,txt}: Fix warning about duplicate jinja
  
  Apparently you need to spell Jinja2 with a capital J, otherwise
  we find it listed twice in the resulting requirements.txt files.
  

• 7c681c42
  by Tristan Van Berkom at 2019-01-03T21:32:40Z

  tools/dev-requirements.{in,txt}: Removed requirements on pylint/codestyle pytest plugins
  
  We no longer run the linters through pytest, so we don't need these
  plugins anymore.
  

• d6689ed2
  by Tristan Van Berkom at 2019-01-03T22:09:02Z

  Merge branch 'tristan/toxic-linting' into 'master'
  
  Run the linter separately from the tests
  
  See merge request BuildStream/buildstream!1033

• e15278d7
  by Thomas Coldrick at 2019-01-03T22:14:01Z

  Use collections.abc for Mapping, Iterable
  
  In _yaml.py and _frontend/complete.py we were getting pylint warnings
  for using collections.Mapping and collections.Iterable, which are
  abstract classes now provided from collections.abc. This patch just
  uses the classes from the right place.
  

13 changed files:

• .gitlab-ci.yml
• CONTRIBUTING.rst
• buildstream/_frontend/complete.py
• buildstream/_yaml.py
• setup.cfg
• tests/frontend/buildcheckout.py
• − tests/frontend/project/elements/invalid-chars|<>-in-name.bst
• tests/testutils/site.py
• tools/dev-requirements.in
• tools/dev-requirements.txt
• tools/requirements.in
• tools/requirements.txt
• tox.ini

Changes:

.gitlab-ci.yml

---

...	...	@@ -151,6 +151,18 @@ tests-fedora-missing-deps:
151	151
152	152	- ${TEST_COMMAND}
153	153
	154	+# Lint separately from testing
	155	+lint:
	156	+ stage: test
	157	+
	158	+ before_script:
	159	+ # Diagnostics
	160	+ - python3 --version
	161	+
	162	+ script:
	163	+ - tox -e lint
	164	+ except:
	165	+ - schedules
154	166
155	167	# Automatically build documentation for every commit, we want to know
156	168	# if building documentation fails even if we're not deploying it.

CONTRIBUTING.rst

---

...	...	@@ -1498,6 +1498,18 @@ option when running tox::
1498	1498
1499	1499	tox -e py37
1500	1500
	1501	+Linting is performed separately from testing. In order to run the linting step which
	1502	+consists of running the ``pycodestyle`` and ``pylint`` tools, run the following::
	1503	+
	1504	+ tox -e lint
	1505	+
	1506	+.. tip::
	1507	+
	1508	+ The project specific pylint and pycodestyle configurations are stored in the
	1509	+ toplevel buildstream directory in the ``.pylintrc`` file and ``setup.cfg`` files
	1510	+ respectively. These configurations can be interesting to use with IDEs and
	1511	+ other developer tooling.
	1512	+
1501	1513	The output of all failing tests will always be printed in the summary, but
1502	1514	if you want to observe the stdout and stderr generated by a passing test,
1503	1515	you can pass the ``-s`` option to pytest as such::
...	...	@@ -1532,15 +1544,6 @@ output. To run them, you can use::
1532	1544
1533	1545	tox -- --integration
1534	1546
1535		-By default, buildstream also runs pylint on all files. Should you want
1536		-to run just pylint (these checks are a lot faster), you can do so
1537		-with::
1538		-
1539		- tox -- -m pylint
1540		-
1541		-Alternatively, any IDE plugin that uses pytest should automatically
1542		-detect the ``.pylintrc`` in the project's root directory.
1543		-
1544	1547	In case BuildStream's dependencies were updated since you last ran the
1545	1548	tests, you might see some errors like
1546	1549	``pytest: error: unrecognized arguments: --codestyle``. If this happens, you

buildstream/_frontend/complete.py

---

...	...	@@ -31,7 +31,7 @@
31	31	# (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
32	32	# OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
33	33	#
34		-import collections
	34	+import collections.abc
35	35	import copy
36	36	import os
37	37
...	...	@@ -218,7 +218,7 @@ def is_incomplete_argument(current_params, cmd_param):
218	218	return True
219	219	if cmd_param.nargs == -1:
220	220	return True
221		- if isinstance(current_param_values, collections.Iterable) \
	221	+ if isinstance(current_param_values, collections.abc.Iterable) \
222	222	and cmd_param.nargs > 1 and len(current_param_values) < cmd_param.nargs:
223	223	return True
224	224	return False

buildstream/_yaml.py

---

...	...	@@ -287,7 +287,7 @@ def node_decorate_dict(filename, target, source, toplevel):
287	287	provenance.members[key] = member
288	288
289	289	target_value = target.get(key)
290		- if isinstance(value, collections.Mapping):
	290	+ if isinstance(value, collections.abc.Mapping):
291	291	node_decorate_dict(filename, target_value, value, toplevel)
292	292	elif isinstance(value, list):
293	293	member.elements = node_decorate_list(filename, target_value, value, toplevel)
...	...	@@ -302,7 +302,7 @@ def node_decorate_list(filename, target, source, toplevel):
302	302	target_item = target[idx]
303	303	element = ElementProvenance(filename, source, idx, toplevel)
304	304
305		- if isinstance(item, collections.Mapping):
	305	+ if isinstance(item, collections.abc.Mapping):
306	306	node_decorate_dict(filename, target_item, item, toplevel)
307	307	elif isinstance(item, list):
308	308	element.elements = node_decorate_list(filename, target_item, item, toplevel)
...	...	@@ -578,7 +578,7 @@ def is_ruamel_str(value):
578	578	#
579	579	def is_composite_list(node):
580	580
581		- if isinstance(node, collections.Mapping):
	581	+ if isinstance(node, collections.abc.Mapping):
582	582	has_directives = False
583	583	has_keys = False
584	584
...	...	@@ -847,7 +847,7 @@ def composite_dict(target, source, path=None):
847	847
848	848	target_value = target.get(key)
849	849
850		- if isinstance(source_value, collections.Mapping):
	850	+ if isinstance(source_value, collections.abc.Mapping):
851	851
852	852	# Handle creating new dicts on target side
853	853	if target_value is None:
...	...	@@ -862,7 +862,7 @@ def composite_dict(target, source, path=None):
862	862	# Add a new provenance member element to the containing dict
863	863	target_provenance.members[key] = source_provenance.members[key]
864	864
865		- if not isinstance(target_value, collections.Mapping):
	865	+ if not isinstance(target_value, collections.abc.Mapping):
866	866	raise CompositeTypeError(thispath, type(target_value), type(source_value))
867	867
868	868	# Recurse into matching dictionary
...	...	@@ -923,7 +923,7 @@ RoundTripRepresenter.add_representer(SanitizedDict,
923	923	#
924	924	def node_sanitize(node):
925	925
926		- if isinstance(node, collections.Mapping):
	926	+ if isinstance(node, collections.abc.Mapping):
927	927
928	928	result = SanitizedDict()
929	929
...	...	@@ -1067,7 +1067,7 @@ class ChainMap(collections.ChainMap):
1067	1067	def node_chain_copy(source):
1068	1068	copy = ChainMap({}, source)
1069	1069	for key, value in source.items():
1070		- if isinstance(value, collections.Mapping):
	1070	+ if isinstance(value, collections.abc.Mapping):
1071	1071	copy[key] = node_chain_copy(value)
1072	1072	elif isinstance(value, list):
1073	1073	copy[key] = list_chain_copy(value)
...	...	@@ -1080,7 +1080,7 @@ def node_chain_copy(source):
1080	1080	def list_chain_copy(source):
1081	1081	copy = []
1082	1082	for item in source:
1083		- if isinstance(item, collections.Mapping):
	1083	+ if isinstance(item, collections.abc.Mapping):
1084	1084	copy.append(node_chain_copy(item))
1085	1085	elif isinstance(item, list):
1086	1086	copy.append(list_chain_copy(item))
...	...	@@ -1095,7 +1095,7 @@ def list_chain_copy(source):
1095	1095	def node_copy(source):
1096	1096	copy = {}
1097	1097	for key, value in source.items():
1098		- if isinstance(value, collections.Mapping):
	1098	+ if isinstance(value, collections.abc.Mapping):
1099	1099	copy[key] = node_copy(value)
1100	1100	elif isinstance(value, list):
1101	1101	copy[key] = list_copy(value)
...	...	@@ -1112,7 +1112,7 @@ def node_copy(source):
1112	1112	def list_copy(source):
1113	1113	copy = []
1114	1114	for item in source:
1115		- if isinstance(item, collections.Mapping):
	1115	+ if isinstance(item, collections.abc.Mapping):
1116	1116	copy.append(node_copy(item))
1117	1117	elif isinstance(item, list):
1118	1118	copy.append(list_copy(item))
...	...	@@ -1147,7 +1147,7 @@ def node_final_assertions(node):
1147	1147	raise LoadError(LoadErrorReason.TRAILING_LIST_DIRECTIVE,
1148	1148	"{}: Attempt to override non-existing list".format(provenance))
1149	1149
1150		- if isinstance(value, collections.Mapping):
	1150	+ if isinstance(value, collections.abc.Mapping):
1151	1151	node_final_assertions(value)
1152	1152	elif isinstance(value, list):
1153	1153	list_final_assertions(value)
...	...	@@ -1155,7 +1155,7 @@ def node_final_assertions(node):
1155	1155
1156	1156	def list_final_assertions(values):
1157	1157	for value in values:
1158		- if isinstance(value, collections.Mapping):
	1158	+ if isinstance(value, collections.abc.Mapping):
1159	1159	node_final_assertions(value)
1160	1160	elif isinstance(value, list):
1161	1161	list_final_assertions(value)

setup.cfg

---

...	...	@@ -11,11 +11,13 @@ parentdir_prefix = BuildStream-
11	11	test=pytest
12	12
13	13	[tool:pytest]
14		-addopts = --verbose --basetemp ./tmp --codestyle --pylint --pylint-rcfile=.pylintrc --cov=buildstream --cov-config .coveragerc --durations=20
	14	+addopts = --verbose --basetemp ./tmp --cov=buildstream --cov-config .coveragerc --durations=20
15	15	norecursedirs = tests/integration/project integration-cache tmp __pycache__ .eggs
16	16	python_files = tests/*/*.py
17		-codestyle_max_line_length = 119
18		-codestyle_ignore = E129 E125 W504 W605
19		-codestyle_exclude = doc/source/conf.py buildstream/_fuse/fuse.py buildstream/_protos/**/*py
20	17	env =
21	18	D:BST_TEST_SUITE=True
	19	+
	20	+[pycodestyle]
	21	+max-line-length = 119
	22	+ignore = E129,E125,W504,W605
	23	+exclude = .git/**,.tox/**,doc/source/conf.py,buildstream/_fuse/fuse.py,buildstream/_protos/**/*py

tests/frontend/buildcheckout.py

---

...	...	@@ -3,6 +3,7 @@ import tarfile
3	3	import hashlib
4	4	import pytest
5	5	from tests.testutils import cli, create_repo, ALL_REPO_KINDS, generate_junction
	6	+from tests.testutils.site import IS_WINDOWS
6	7
7	8	from buildstream import _yaml
8	9	from buildstream._exceptions import ErrorDomain, LoadErrorReason
...	...	@@ -85,16 +86,37 @@ def test_build_invalid_suffix_dep(datafiles, cli, strict, hardlinks):
85	86	result.assert_main_error(ErrorDomain.LOAD, "bad-element-suffix")
86	87
87	88
	89	+@pytest.mark.skipif(IS_WINDOWS, reason='Not available on Windows')
88	90	@pytest.mark.datafiles(DATA_DIR)
89	91	def test_build_invalid_filename_chars(datafiles, cli):
90	92	project = os.path.join(datafiles.dirname, datafiles.basename)
91		- result = cli.run(project=project, args=strict_args(['build', 'invalid-chars|<>-in-name.bst'], 'non-strict'))
	93	+ element_name = 'invalid-chars|<>-in-name.bst'
	94	+
	95	+ # The name of this file contains characters that are not allowed by
	96	+ # BuildStream, using it should raise a warning.
	97	+ element = {
	98	+ 'kind': 'stack',
	99	+ }
	100	+ _yaml.dump(element, os.path.join(project, 'elements', element_name))
	101	+
	102	+ result = cli.run(project=project, args=strict_args(['build', element_name], 'non-strict'))
92	103	result.assert_main_error(ErrorDomain.LOAD, "bad-characters-in-name")
93	104
94	105
	106	+@pytest.mark.skipif(IS_WINDOWS, reason='Not available on Windows')
95	107	@pytest.mark.datafiles(DATA_DIR)
96	108	def test_build_invalid_filename_chars_dep(datafiles, cli):
97	109	project = os.path.join(datafiles.dirname, datafiles.basename)
	110	+ element_name = 'invalid-chars|<>-in-name.bst'
	111	+
	112	+ # The name of this file contains characters that are not allowed by
	113	+ # BuildStream, and is listed as a dependency of 'invalid-chars-in-dep.bst'.
	114	+ # This should also raise a warning.
	115	+ element = {
	116	+ 'kind': 'stack',
	117	+ }
	118	+ _yaml.dump(element, os.path.join(project, 'elements', element_name))
	119	+
98	120	result = cli.run(project=project, args=strict_args(['build', 'invalid-chars-in-dep.bst'], 'non-strict'))
99	121	result.assert_main_error(ErrorDomain.LOAD, "bad-characters-in-name")
100	122

tests/frontend/project/elements/invalid-chars|<>-in-name.bst deleted

---

1		-kind: stack
2		-description: |
3		- The name of this files contains characters that are not allowed by
4		- BuildStream, using it should raise a warning.

tests/testutils/site.py

---

...	...	@@ -52,5 +52,6 @@ except ImportError:
52	52	HAVE_ARPY = False
53	53
54	54	IS_LINUX = os.getenv('BST_FORCE_BACKEND', sys.platform).startswith('linux')
	55	+IS_WINDOWS = (os.name == 'nt')
55	56
56	57	MACHINE_ARCH = Platform.get_host_arch()

tools/dev-requirements.in

---

1	1	coverage == 4.4.0
2	2	pylint
	3	+pycodestyle
3	4	pytest >= 3.9
4		-pytest-codestyle >= 1.4.0
5	5	pytest-cov >= 2.5.0
6	6	pytest-datafiles >= 2.0
7	7	pytest-env
8		-pytest-pylint
9	8	pytest-xdist
10	9	pytest-timeout
11	10	pyftpdlib

tools/dev-requirements.txt

---

1	1	coverage==4.4
2	2	pylint==2.2.2
	3	+pycodestyle==2.4.0
3	4	pytest==4.0.2
4		-pytest-codestyle==1.4.0
5	5	pytest-cov==2.6.0
6	6	pytest-datafiles==2.0
7	7	pytest-env==0.6.2
8		-pytest-pylint==0.13.0
9	8	pytest-xdist==1.25.0
10	9	pytest-timeout==1.3.3
11	10	pyftpdlib==1.5.4
...	...	@@ -21,8 +20,6 @@ mccabe==0.6.1
21	20	more-itertools==5.0.0
22	21	pluggy==0.8.0
23	22	py==1.7.0
24		-pycodestyle==2.4.0
25	23	pytest-forked==0.2
26	24	six==1.12.0
27		-typed-ast==1.1.1
28	25	wrapt==1.10.11

tools/requirements.in

---

1	1	Click
2	2	grpcio >= 1.10
3		-jinja2 >= 2.10
	3	+Jinja2 >= 2.10
4	4	pluginbase
5	5	protobuf >= 3.5
6	6	psutil

tools/requirements.txt

---

1	1	Click==7.0
2	2	grpcio==1.17.1
	3	+Jinja2==2.10
3	4	pluginbase==0.7
4	5	protobuf==3.6.1
5	6	psutil==5.4.8
...	...	@@ -13,6 +14,5 @@ psutil==5.4.8
13	14	ruamel.yaml==0.15.51
14	15	setuptools==39.0.1
15	16	## The following requirements were added by pip freeze:
16		-Jinja2==2.10
17	17	MarkupSafe==1.1.0
18	18	six==1.12.0

tox.ini

---

...	...	@@ -13,6 +13,15 @@ passenv =
13	13	GI_TYPELIB_PATH
14	14	INTEGRATION_CACHE
15	15
	16	+[testenv:lint]
	17	+commands =
	18	+ pycodestyle
	19	+ pylint buildstream
	20	+deps =
	21	+ -rtools/requirements.txt
	22	+ -rtools/dev-requirements.txt
	23	+ -rtools/plugin-requirements.txt
	24	+
16	25	[testenv:docs]
17	26	commands =
18	27	make -C doc

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.