[Notes] [Git][BuildStream/buildstream][phil/add-ubuntu-ci-job] 4 commits: Add remote source plugin

[Phil Dawson <gitlab mg gitlab com>]

Title: GitLab

Phil Dawson pushed to branch phil/add-ubuntu-ci-job at BuildStream / buildstream

Commits:

• c049d021
  by Ed Baunton at 2018-07-25T19:43:51Z

  Add remote source plugin
  
  Add a plugin that supports downloading files verbatim from a source with
  an optional overridable filename and destination directory. Bumps bst
  format version to 10.
  
  Fixes #163
  

• bd1196ef
  by Ed Baunton at 2018-07-26T13:22:52Z

  Provide explicit reason for SourceError exception on path
  
  When the user provides a path for the filename parameter, provide a reason
  

• 9a46c16f
  by Ed Baunton at 2018-07-26T14:06:52Z

  Merge branch 'edbaunton/remote-source' into 'master'
  
  Add remote source plugin
  
  Closes #163
  
  See merge request BuildStream/buildstream!541

• abbcec0a
  by Phil Dawson at 2018-07-26T14:13:28Z

  .gitlab-ci-yml: Add ubuntu 18 test
  

17 changed files:

• .gitlab-ci.yml
• buildstream/_versions.py
• + buildstream/plugins/sources/remote.py
• doc/source/core_plugins.rst
• + tests/sources/remote.py
• + tests/sources/remote/missing-file/target.bst
• + tests/sources/remote/no-ref/file
• + tests/sources/remote/no-ref/target.bst
• + tests/sources/remote/path-in-filename/dir/file
• + tests/sources/remote/path-in-filename/target.bst
• + tests/sources/remote/single-file-custom-name/dir/file
• + tests/sources/remote/single-file-custom-name/target.bst
• + tests/sources/remote/single-file/dir/file
• + tests/sources/remote/single-file/target.bst
• + tests/sources/remote/unique-keys/dir/file
• + tests/sources/remote/unique-keys/target-custom.bst
• + tests/sources/remote/unique-keys/target.bst

Changes:

.gitlab-ci.yml

---

...	...	@@ -88,6 +88,10 @@ source_dist:
88	88	paths:
89	89	- coverage-linux/
90	90
	91	+tests-ubuntu-18.04:
	92	+ image: buildstream/testsuite-ubuntu-18:18-master-96-91d87e3c
	93	+ <<: *linux-tests
	94	+
91	95	tests-debian-9:
92	96	image: buildstream/testsuite-debian:9-master-102-9067e269
93	97	<<: *linux-tests

buildstream/_versions.py

---

...	...	@@ -23,7 +23,7 @@
23	23	# This version is bumped whenever enhancements are made
24	24	# to the `project.conf` format or the core element format.
25	25	#
26		-BST_FORMAT_VERSION = 9
	26	+BST_FORMAT_VERSION = 10
27	27
28	28
29	29	# The base BuildStream artifact version

buildstream/plugins/sources/remote.py

---

	1	+#
	2	+# Copyright Bloomberg Finance LP
	3	+#
	4	+# This program is free software; you can redistribute it and/or
	5	+# modify it under the terms of the GNU Lesser General Public
	6	+# License as published by the Free Software Foundation; either
	7	+# version 2 of the License, or (at your option) any later version.
	8	+#
	9	+# This library is distributed in the hope that it will be useful,
	10	+# but WITHOUT ANY WARRANTY; without even the implied warranty of
	11	+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
	12	+# Lesser General Public License for more details.
	13	+#
	14	+# You should have received a copy of the GNU Lesser General Public
	15	+# License along with this library. If not, see <http://www.gnu.org/licenses/>.
	16	+#
	17	+# Authors:
	18	+# Ed Baunton <ebaunton1 bloomberg net>
	19	+
	20	+"""
	21	+remote - stage files from remote urls
	22	+=====================================
	23	+
	24	+**Usage:**
	25	+
	26	+.. code:: yaml
	27	+
	28	+ # Specify the remote source kind
	29	+ kind: remote
	30	+
	31	+ # Optionally specify a relative staging directory
	32	+ # directory: path/to/stage
	33	+
	34	+ # Optionally specify a relative staging filename.
	35	+ # If not specified, the basename of the url will be used.
	36	+ # filename: customfilename
	37	+
	38	+ # Specify the url. Using an alias defined in your project
	39	+ # configuration is encouraged. 'bst track' will update the
	40	+ # sha256sum in 'ref' to the downloaded file's sha256sum.
	41	+ url: upstream:foo
	42	+
	43	+ # Specify the ref. It's a sha256sum of the file you download.
	44	+ ref: 6c9f6f68a131ec6381da82f2bff978083ed7f4f7991d931bfa767b7965ebc94b
	45	+
	46	+.. note::
	47	+
	48	+ The ``remote`` plugin is available since :ref:`format version 10 <project_format_version>`
	49	+
	50	+"""
	51	+import os
	52	+from buildstream import SourceError, utils
	53	+from ._downloadablefilesource import DownloadableFileSource
	54	+
	55	+
	56	+class RemoteSource(DownloadableFileSource):
	57	+ # pylint: disable=attribute-defined-outside-init
	58	+
	59	+ def configure(self, node):
	60	+ super().configure(node)
	61	+
	62	+ self.filename = self.node_get_member(node, str, 'filename', os.path.basename(self.url))
	63	+
	64	+ if os.sep in self.filename:
	65	+ raise SourceError('{}: filename parameter cannot contain directories'.format(self),
	66	+ reason="filename-contains-directory")
	67	+ self.node_validate(node, DownloadableFileSource.COMMON_CONFIG_KEYS + ['filename'])
	68	+
	69	+ def get_unique_key(self):
	70	+ return super().get_unique_key() + [self.filename]
	71	+
	72	+ def stage(self, directory):
	73	+ # Same as in local plugin, don't use hardlinks to stage sources, they
	74	+ # are not write protected in the sandbox.
	75	+ dest = os.path.join(directory, self.filename)
	76	+ with self.timed_activity("Staging remote file to {}".format(dest)):
	77	+ utils.safe_copy(self._get_mirror_file(), dest)
	78	+
	79	+
	80	+def setup():
	81	+ return RemoteSource

doc/source/core_plugins.rst

---

...	...	@@ -50,6 +50,7 @@ Sources
50	50	:maxdepth: 1
51	51
52	52	sources/local
	53	+ sources/remote
53	54	sources/tar
54	55	sources/zip
55	56	sources/git

tests/sources/remote.py

---

	1	+import os
	2	+import pytest
	3	+
	4	+from buildstream._exceptions import ErrorDomain
	5	+from buildstream import _yaml
	6	+from tests.testutils import cli
	7	+
	8	+DATA_DIR = os.path.join(
	9	+ os.path.dirname(os.path.realpath(__file__)),
	10	+ 'remote',
	11	+)
	12	+
	13	+
	14	+def generate_project(project_dir, tmpdir):
	15	+ project_file = os.path.join(project_dir, "project.conf")
	16	+ _yaml.dump({
	17	+ 'name': 'foo',
	18	+ 'aliases': {
	19	+ 'tmpdir': "file:///" + str(tmpdir)
	20	+ }
	21	+ }, project_file)
	22	+
	23	+
	24	+# Test that without ref, consistency is set appropriately.
	25	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'no-ref'))
	26	+def test_no_ref(cli, tmpdir, datafiles):
	27	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	28	+ generate_project(project, tmpdir)
	29	+ assert cli.get_element_state(project, 'target.bst') == 'no reference'
	30	+
	31	+
	32	+# Here we are doing a fetch on a file that doesn't exist. target.bst
	33	+# refers to 'file' but that file is not present.
	34	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'missing-file'))
	35	+def test_missing_file(cli, tmpdir, datafiles):
	36	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	37	+ generate_project(project, tmpdir)
	38	+
	39	+ # Try to fetch it
	40	+ result = cli.run(project=project, args=[
	41	+ 'fetch', 'target.bst'
	42	+ ])
	43	+
	44	+ result.assert_main_error(ErrorDomain.STREAM, None)
	45	+ result.assert_task_error(ErrorDomain.SOURCE, None)
	46	+
	47	+
	48	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'path-in-filename'))
	49	+def test_path_in_filename(cli, tmpdir, datafiles):
	50	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	51	+ generate_project(project, tmpdir)
	52	+
	53	+ # Try to fetch it
	54	+ result = cli.run(project=project, args=[
	55	+ 'fetch', 'target.bst'
	56	+ ])
	57	+
	58	+ # The bst file has a / in the filename param
	59	+ result.assert_main_error(ErrorDomain.SOURCE, "filename-contains-directory")
	60	+
	61	+
	62	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'single-file'))
	63	+def test_simple_file_build(cli, tmpdir, datafiles):
	64	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	65	+ generate_project(project, tmpdir)
	66	+ checkoutdir = os.path.join(str(tmpdir), "checkout")
	67	+
	68	+ # Try to fetch it
	69	+ result = cli.run(project=project, args=[
	70	+ 'fetch', 'target.bst'
	71	+ ])
	72	+ result.assert_success()
	73	+
	74	+ result = cli.run(project=project, args=[
	75	+ 'build', 'target.bst'
	76	+ ])
	77	+ result.assert_success()
	78	+
	79	+ result = cli.run(project=project, args=[
	80	+ 'checkout', 'target.bst', checkoutdir
	81	+ ])
	82	+ result.assert_success()
	83	+ # Note that the url of the file in target.bst is actually /dir/file
	84	+ # but this tests confirms we take the basename
	85	+ assert(os.path.exists(os.path.join(checkoutdir, 'file')))
	86	+
	87	+
	88	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'single-file-custom-name'))
	89	+def test_simple_file_custom_name_build(cli, tmpdir, datafiles):
	90	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	91	+ generate_project(project, tmpdir)
	92	+ checkoutdir = os.path.join(str(tmpdir), "checkout")
	93	+
	94	+ # Try to fetch it
	95	+ result = cli.run(project=project, args=[
	96	+ 'fetch', 'target.bst'
	97	+ ])
	98	+ result.assert_success()
	99	+
	100	+ result = cli.run(project=project, args=[
	101	+ 'build', 'target.bst'
	102	+ ])
	103	+ result.assert_success()
	104	+
	105	+ result = cli.run(project=project, args=[
	106	+ 'checkout', 'target.bst', checkoutdir
	107	+ ])
	108	+ result.assert_success()
	109	+ assert(not os.path.exists(os.path.join(checkoutdir, 'file')))
	110	+ assert(os.path.exists(os.path.join(checkoutdir, 'custom-file')))
	111	+
	112	+
	113	+@pytest.mark.datafiles(os.path.join(DATA_DIR, 'unique-keys'))
	114	+def test_unique_key(cli, tmpdir, datafiles):
	115	+ '''This test confirms that the 'filename' parameter is honoured when it comes
	116	+ to generating a cache key for the source.
	117	+ '''
	118	+ project = os.path.join(datafiles.dirname, datafiles.basename)
	119	+ generate_project(project, tmpdir)
	120	+ assert cli.get_element_state(project, 'target.bst') == "fetch needed"
	121	+ assert cli.get_element_state(project, 'target-custom.bst') == "fetch needed"
	122	+ # Try to fetch it
	123	+ result = cli.run(project=project, args=[
	124	+ 'fetch', 'target.bst'
	125	+ ])
	126	+
	127	+ # We should download the file only once
	128	+ assert cli.get_element_state(project, 'target.bst') == 'buildable'
	129	+ assert cli.get_element_state(project, 'target-custom.bst') == 'buildable'
	130	+
	131	+ # But the cache key is different because the 'filename' is different.
	132	+ assert cli.get_element_key(project, 'target.bst') != \
	133	+ cli.get_element_key(project, 'target-custom.bst')

tests/sources/remote/missing-file/target.bst

---

	1	+kind: autotools
	2	+description: The kind of this element is irrelevant.
	3	+sources:
	4	+- kind: remote
	5	+ url: tmpdir:/file
	6	+ ref: abcdef
	7	+ filename: filename

tests/sources/remote/no-ref/file

---

1

+filecontent

tests/sources/remote/no-ref/target.bst

---

	1	+kind: autotools
	2	+description: The kind of this element is irrelevant.
	3	+sources:
	4	+- kind: remote
	5	+ url: tmpdir:/file

tests/sources/remote/path-in-filename/dir/file

---

tests/sources/remote/path-in-filename/target.bst

---

	1	+kind: import
	2	+description: test
	3	+sources:
	4	+- kind: remote
	5	+ url: tmpdir:/dir/file
	6	+ ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
	7	+ filename: path/to/file

tests/sources/remote/single-file-custom-name/dir/file

---

tests/sources/remote/single-file-custom-name/target.bst

---

	1	+kind: import
	2	+description: test
	3	+sources:
	4	+- kind: remote
	5	+ url: tmpdir:/dir/file
	6	+ ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
	7	+ filename: custom-file

tests/sources/remote/single-file/dir/file

---

tests/sources/remote/single-file/target.bst

---

	1	+kind: import
	2	+description: test
	3	+sources:
	4	+- kind: remote
	5	+ url: tmpdir:/dir/file
	6	+ ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

tests/sources/remote/unique-keys/dir/file

---

tests/sources/remote/unique-keys/target-custom.bst

---

	1	+kind: import
	2	+description: test
	3	+sources:
	4	+- kind: remote
	5	+ url: tmpdir:/dir/file
	6	+ ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
	7	+ filename: some-custom-file

tests/sources/remote/unique-keys/target.bst

---

	1	+kind: import
	2	+description: test
	3	+sources:
	4	+- kind: remote
	5	+ url: tmpdir:/dir/file
	6	+ ref: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.