[Notes] [Git][BuildStream/buildstream][bschubert/pipeline] 11 commits: _frontend/cli.py: Reinstate support for guessing targets

[Benjamin Schubert <gitlab mg gitlab com>]

Title: GitLab

Benjamin Schubert pushed to branch bschubert/pipeline at BuildStream / buildstream

Commits:

• cf0e7d26
  by Jürg Billeter at 2019-01-04T07:07:59Z

  _frontend/cli.py: Reinstate support for guessing targets
  
  Moving fetch and track to the source command group accidentally dropped
  the support for guessing targets for these commands when invoked from a
  workspace directory. This brings it back.
  

• 0751fc8f
  by Jürg Billeter at 2019-01-04T11:27:58Z

  Merge branch 'juerg/source-guess-target' into 'master'
  
  _frontend/cli.py: Reinstate support for guessing targets
  
  See merge request BuildStream/buildstream!1036

• ebd965fe
  by Chandan Singh at 2019-01-04T15:21:35Z

  Rename "tools" directory to "requirements"
  
  This seems like a better name for the directory, as it more closely
  describes the purpose of its contents.
  

• bb712886
  by Chandan Singh at 2019-01-04T15:21:35Z

  Add Makefile to update requirements files
  

• b9792837
  by Chandan Singh at 2019-01-04T15:21:35Z

  CONTRIBUTING.rst: Add instructions to update requirements files
  
  Split the "The MANIFEST.in and setup.py" section in two:
  "Managing data files" and "Updating BuildStream's Python dependencies".
  
  Briefly explain the layout of `requirements` directory and add
  instructions to use the Makefile added in the last commit.
  

• 45692335
  by Tristan Van Berkom at 2019-01-04T15:50:14Z

  Merge branch 'chandan/update-requirements-one-liner' into 'master'
  
  Add Makefile to update requirements files
  
  See merge request BuildStream/buildstream!1035

• f42dcb54
  by Chandan Singh at 2019-01-04T17:16:43Z

  .gitlab-ci.yml: Remove prepare stage
  
  As we now run tests using `tox`, we don't need to worry about manually
  packing and unpacking BuildStream. So, we can remove the preapre stage
  entirely.
  
  Update `coverage` and nightly jobs to appropriately cope with this
  change. Both these jobs now install all runtime dependencies from
  requirements files.
  

• ee29f207
  by Chandan Singh at 2019-01-04T17:41:02Z

  Merge branch 'chandan/no-sdist' into 'master'
  
  .gitlab-ci.yml: Remove prepare stage
  
  See merge request BuildStream/buildstream!1037

• afd8dc84
  by Benjamin Schubert at 2019-01-04T22:59:29Z

  Don't compute status twice for jobs that are ready when queuing.
  
  Previously, we would put all jobs in the wait queue, if they were
  not skipped. However, if a job is ready, its state will not change
  in the future, and it is therefore useless to check its state
  right after again.
  
  This introduces a new `ready_queue`, which contains those jobs that
  are then sent first.
  

• 2e3b77c8
  by Benjamin Schubert at 2019-01-04T23:01:27Z

  buildqueue: only update state of direct reverse dependencies
  
  Previously, we would update the state of all elements in the pipeline,
  everytime an element in the queue would change state.
  
  This is inefficient. We know that the only impacted elements will
  be the direct build reverse dependencies. We can therefore, by recording
  the direct build reverse dependencies, only update those and reduce
  the time we spend updating states.
  

• 0004acf4
  by Benjamin Schubert at 2019-01-04T23:03:14Z

  WIP: notes
  

18 changed files:

• .gitlab-ci.yml
• CONTRIBUTING.rst
• MANIFEST.in
• buildstream/_frontend/cli.py
• buildstream/_scheduler/queues/buildqueue.py
• buildstream/_scheduler/queues/fetchqueue.py
• buildstream/_scheduler/queues/pullqueue.py
• buildstream/_scheduler/queues/queue.py
• buildstream/element.py
• + requirements/Makefile
• tools/dev-requirements.in → requirements/dev-requirements.in
• tools/dev-requirements.txt → requirements/dev-requirements.txt
• tools/plugin-requirements.in → requirements/plugin-requirements.in
• tools/plugin-requirements.txt → requirements/plugin-requirements.txt
• tools/requirements.in → requirements/requirements.in
• tools/requirements.txt → requirements/requirements.txt
• setup.py
• tox.ini

Changes:

.gitlab-ci.yml

---

...	...	@@ -6,7 +6,6 @@ cache:
6	6	- cache/
7	7
8	8	stages:
9		- - prepare
10	9	- test
11	10	- post
12	11
...	...	@@ -15,41 +14,6 @@ variables:
15	14	INTEGRATION_CACHE: "${CI_PROJECT_DIR}/cache/integration-cache"
16	15	TEST_COMMAND: "tox -- --color=yes --integration"
17	16
18		-#####################################################
19		-# Prepare stage #
20		-#####################################################
21		-
22		-# Create a source distribution
23		-#
24		-source_dist:
25		- stage: prepare
26		- script:
27		-
28		- # Generate the source distribution tarball
29		- #
30		- - python3 setup.py sdist
31		- - tar -ztf dist/*
32		- - tarball=$(cd dist && echo $(ls *))
33		-
34		- # Verify that the source distribution tarball can be installed correctly
35		- #
36		- - pip3 install dist/*.tar.gz
37		- - bst --version
38		-
39		- # unpack tarball as `dist/buildstream` directory
40		- - |
41		- cat > dist/unpack.sh << EOF
42		- #!/bin/sh
43		- tar -zxf ${tarball}
44		- mv ${tarball%.tar.gz} buildstream
45		- EOF
46		-
47		- # Make our helpers executable
48		- - chmod +x dist/unpack.sh
49		- artifacts:
50		- paths:
51		- - dist/
52		-
53	17
54	18	#####################################################
55	19	# Test stage #
...	...	@@ -72,8 +36,7 @@ source_dist:
72	36	- useradd -Um buildstream
73	37	- chown -R buildstream:buildstream .
74	38
75		- # Run the tests from the source distribution, We run as a simple
76		- # user to test for permission issues
	39	+ # Run the tests as a simple user to test for permission issues
77	40	- su buildstream -c "${TEST_COMMAND}"
78	41
79	42	after_script:
...	...	@@ -192,7 +155,8 @@ docs:
192	155	scheduler:
193	156	fetchers: 2
194	157	EOF
195		- - (cd dist && ./unpack.sh && cd buildstream && pip3 install .)
	158	+ - pip3 install -r requirements/requirements.txt -r requirements/plugin-requirements.txt
	159	+ - pip3 install --no-index .
196	160	- pip3 install --user -e ${BST_EXT_URL}@${BST_EXT_REF}#egg=bst_ext
197	161	- git clone https://gitlab.com/freedesktop-sdk/freedesktop-sdk.git
198	162	- git -C freedesktop-sdk checkout ${FD_SDK_REF}
...	...	@@ -275,13 +239,12 @@ coverage:
275	239	stage: post
276	240	coverage: '/TOTAL +\d+ +\d+ +(\d+\.\d+)%/'
277	241	script:
278		- - cd dist && ./unpack.sh && cd buildstream
279		- - pip3 install -r tools/requirements.txt -r tools/dev-requirements.txt
	242	+ - pip3 install -r requirements/requirements.txt -r requirements/dev-requirements.txt
280	243	- pip3 install --no-index .
281	244	- mkdir report
282	245	- cd report
283		- - cp ../../../coverage-unix/coverage.* .
284		- - cp ../../../coverage-linux/coverage.* .
	246	+ - cp ../coverage-unix/coverage.* .
	247	+ - cp ../coverage-linux/coverage.* .
285	248	- ls coverage.*
286	249	- coverage combine --rcfile=../.coveragerc -a coverage.*
287	250	- coverage report --rcfile=../.coveragerc -m
...	...	@@ -290,7 +253,6 @@ coverage:
290	253	- tests-fedora-27
291	254	- tests-fedora-28
292	255	- tests-unix
293		- - source_dist
294	256	except:
295	257	- schedules
296	258
...	...	@@ -299,7 +261,6 @@ coverage:
299	261	pages:
300	262	stage: post
301	263	dependencies:
302		- - source_dist
303	264	- docs
304	265	variables:
305	266	ACME_DIR: public/.well-known/acme-challenge

CONTRIBUTING.rst

---

...	...	@@ -1736,10 +1736,8 @@ obtain profiles::
1736	1736	ForceCommand BST_PROFILE=artifact-receive cd /tmp && bst-artifact-receive --pull-url https://example.com/ /home/artifacts/artifacts
1737	1737
1738	1738
1739		-The MANIFEST.in and setup.py
1740		-----------------------------
1741		-When adding a dependency to BuildStream, it's important to update the setup.py accordingly.
1742		-
	1739	+Managing data files
	1740	+-------------------
1743	1741	When adding data files which need to be discovered at runtime by BuildStream, update setup.py accordingly.
1744	1742
1745	1743	When adding data files for the purpose of docs or tests, or anything that is not covered by
...	...	@@ -1749,3 +1747,23 @@ At any time, running the following command to create a source distribution shoul
1749	1747	creating a tarball which contains everything we want it to include::
1750	1748
1751	1749	./setup.py sdist
	1750	+
	1751	+
	1752	+Updating BuildStream's Python dependencies
	1753	+------------------------------------------
	1754	+BuildStream's Python dependencies are listed in multiple
	1755	+`requirements files <https://pip.readthedocs.io/en/latest/reference/pip_install/#requirements-file-format>`
	1756	+present in the ``requirements`` directory.
	1757	+
	1758	+All ``.txt`` files in this directory are generated from the corresponding
	1759	+``.in`` file, and each ``.in`` file represents a set of dependencies. For
	1760	+example, ``requirements.in`` contains all runtime dependencies of BuildStream.
	1761	+``requirements.txt`` is generated from it, and contains pinned versions of all
	1762	+runtime dependencies (including transitive dependencies) of BuildStream.
	1763	+
	1764	+When adding a new dependency to BuildStream, or updating existing dependencies,
	1765	+it is important to update the appropriate requirements file accordingly. After
	1766	+changing the ``.in`` file, run the following to update the matching ``.txt``
	1767	+file::
	1768	+
	1769	+ make -C requirements

MANIFEST.in

---

...	...	@@ -32,12 +32,12 @@ include .pylintrc
32	32	recursive-include buildstream/_protos *.proto
33	33
34	34	# Requirements files
35		-include tools/requirements.in
36		-include tools/requirements.txt
37		-include tools/dev-requirements.in
38		-include tools/dev-requirements.txt
39		-include tools/plugin-requirements.in
40		-include tools/plugin-requirements.txt
	35	+include requirements/requirements.in
	36	+include requirements/requirements.txt
	37	+include requirements/dev-requirements.in
	38	+include requirements/dev-requirements.txt
	39	+include requirements/plugin-requirements.in
	40	+include requirements/plugin-requirements.txt
41	41
42	42	# Versioneer
43	43	include versioneer.py

buildstream/_frontend/cli.py

---

...	...	@@ -719,6 +719,11 @@ def source_fetch(app, elements, deps, track_, except_, track_cross_junctions):
719	719	deps = PipelineSelection.ALL
720	720
721	721	with app.initialized(session_name="Fetch"):
	722	+ if not elements:
	723	+ guessed_target = app.context.guess_element()
	724	+ if guessed_target:
	725	+ elements = (guessed_target,)
	726	+
722	727	app.stream.fetch(elements,
723	728	selection=deps,
724	729	except_targets=except_,
...	...	@@ -755,6 +760,11 @@ def source_track(app, elements, deps, except_, cross_junctions):
755	760	all: All dependencies of all specified elements
756	761	"""
757	762	with app.initialized(session_name="Track"):
	763	+ if not elements:
	764	+ guessed_target = app.context.guess_element()
	765	+ if guessed_target:
	766	+ elements = (guessed_target,)
	767	+
758	768	# Substitute 'none' for 'redirect' so that element redirections
759	769	# will be done
760	770	if deps == 'none':

buildstream/_scheduler/queues/buildqueue.py

---

...	...	@@ -71,9 +71,6 @@ class BuildQueue(Queue):
71	71	return element._assemble()
72	72
73	73	def status(self, element):
74		- # state of dependencies may have changed, recalculate element state
75		- element._update_state()
76		-
77	74	if not element._is_required():
78	75	# Artifact is not currently required but it may be requested later.
79	76	# Keep it in the queue.
...	...	@@ -109,6 +106,10 @@ class BuildQueue(Queue):
109	106	# Inform element in main process that assembly is done
110	107	element._assemble_done()
111	108
	109	+ # Update the state of all reverse dependencies
	110	+ for reverse_dependency in element.reverse_build_dependencies:
	111	+ reverse_dependency._update_state()
	112	+
112	113	# This has to be done after _assemble_done, such that the
113	114	# element may register its cache key as required
114	115	#

buildstream/_scheduler/queues/fetchqueue.py

---

...	...	@@ -44,6 +44,7 @@ class FetchQueue(Queue):
44	44
45	45	def status(self, element):
46	46	# state of dependencies may have changed, recalculate element state
	47	+ # FIXME: we should never be doing this
47	48	element._update_state()
48	49
49	50	if not element._is_required():

buildstream/_scheduler/queues/pullqueue.py

---

...	...	@@ -39,6 +39,7 @@ class PullQueue(Queue):
39	39
40	40	def status(self, element):
41	41	# state of dependencies may have changed, recalculate element state
	42	+ # FIXME: this should never be needed
42	43	element._update_state()
43	44
44	45	if not element._is_required():

buildstream/_scheduler/queues/queue.py

---

...	...	@@ -73,6 +73,7 @@ class Queue():
73	73	#
74	74	self._scheduler = scheduler
75	75	self._wait_queue = deque()
	76	+ self._ready_queue = deque()
76	77	self._done_queue = deque()
77	78	self._max_retries = 0
78	79
...	...	@@ -166,14 +167,17 @@ class Queue():
166	167	complete_cb=self._job_done,
167	168	max_retries=self._max_retries)
168	169
169		- # Place skipped elements directly on the done queue
170		- jobs = [create_job(elt) for elt in elts]
171		- skip = [job for job in jobs if self.status(job.element) == QueueStatus.SKIP]
172		- wait = [job for job in jobs if job not in skip]
	170	+ for element in elts:
	171	+ status = self.status(element)
	172	+ job = create_job(element)
173	173
174		- self._wait_queue.extend(wait)
175		- self._done_queue.extend(skip)
176		- self.skipped_elements.extend(skip)
	174	+ if status == QueueStatus.SKIP:
	175	+ self._done_queue.append(job)
	176	+ self.skipped_elements.append(job)
	177	+ elif status == QueueStatus.READY:
	178	+ self._ready_queue.append(job)
	179	+ else:
	180	+ self._wait_queue.append(job)
177	181
178	182	# dequeue()
179	183	#
...	...	@@ -219,7 +223,8 @@ class Queue():
219	223	#
220	224	def pop_ready_jobs(self):
221	225	unready = []
222		- ready = []
	226	+ ready = self._ready_queue.copy()
	227	+ self._ready_queue.clear()
223	228
224	229	while self._wait_queue:
225	230	job = self._wait_queue.popleft()

buildstream/element.py

---

...	...	@@ -197,6 +197,7 @@ class Element(Plugin):
197	197
198	198	self.__runtime_dependencies = [] # Direct runtime dependency Elements
199	199	self.__build_dependencies = [] # Direct build dependency Elements
	200	+ self.reverse_build_dependencies = [] # Direct reverse build dependency Elements
200	201	self.__sources = [] # List of Sources
201	202	self.__weak_cache_key = None # Our cached weak cache key
202	203	self.__strict_cache_key = None # Our cached cache key for strict builds
...	...	@@ -930,6 +931,7 @@ class Element(Plugin):
930	931	for meta_dep in meta.build_dependencies:
931	932	dependency = Element._new_from_meta(meta_dep)
932	933	element.__build_dependencies.append(dependency)
	934	+ dependency.reverse_build_dependencies.append(element)
933	935
934	936	return element
935	937
...	...	@@ -1446,6 +1448,7 @@ class Element(Plugin):
1446	1448	# This unblocks pull/fetch/build.
1447	1449	#
1448	1450	def _set_required(self):
	1451	+ # FIXME: this should enqueue stuff in the queue, it should not be here by default
1449	1452	if self.__required:
1450	1453	# Already done
1451	1454	return
...	...	@@ -1456,6 +1459,7 @@ class Element(Plugin):
1456	1459	for dep in self.dependencies(Scope.RUN, recurse=False):
1457	1460	dep._set_required()
1458	1461
	1462	+ # FIXME: this should not be done at all here
1459	1463	self._update_state()
1460	1464
1461	1465	# _is_required():

requirements/Makefile

---

	1	+# Makefile for updating BuildStream's requirements files.
	2	+#
	3	+
	4	+REQUIREMENTS_IN := $(wildcard *.in)
	5	+REQUIREMENTS_TXT := $(REQUIREMENTS_IN:.in=.txt)
	6	+PYTHON := python3
	7	+VENV := $(PYTHON) -m venv
	8	+
	9	+VENV_PIP = $(VENVDIR)/bin/pip
	10	+
	11	+
	12	+.PHONY: all
	13	+
	14	+all: $(REQUIREMENTS_TXT)
	15	+
	16	+%.txt: %.in
	17	+ $(eval VENVDIR := $(shell mktemp -d $(CURDIR)/.bst-venv.XXXXXX))
	18	+ $(VENV) $(VENVDIR)
	19	+ $(VENV_PIP) install -r $^
	20	+ $(VENV_PIP) freeze -r $^ > $@
	21	+ rm -rf $(VENVDIR)

tools/dev-requirements.in → requirements/dev-requirements.in

---

tools/dev-requirements.txt → requirements/dev-requirements.txt

---

tools/plugin-requirements.in → requirements/plugin-requirements.in

---

tools/plugin-requirements.txt → requirements/plugin-requirements.txt

---

tools/requirements.in → requirements/requirements.in

---

tools/requirements.txt → requirements/requirements.txt

---

setup.py

---

...	...	@@ -270,10 +270,10 @@ def get_cmdclass():
270	270	#####################################################
271	271	# Gather requirements #
272	272	#####################################################
273		-with open('tools/dev-requirements.in') as dev_reqs:
	273	+with open('requirements/dev-requirements.in') as dev_reqs:
274	274	dev_requires = dev_reqs.read().splitlines()
275	275
276		-with open('tools/requirements.in') as install_reqs:
	276	+with open('requirements/requirements.in') as install_reqs:
277	277	install_requires = install_reqs.read().splitlines()
278	278
279	279	#####################################################

tox.ini

---

...	...	@@ -5,9 +5,9 @@ skip_missing_interpreters = true
5	5	[testenv]
6	6	commands = pytest {posargs}
7	7	deps =
8		- -rtools/requirements.txt
9		- -rtools/dev-requirements.txt
10		- -rtools/plugin-requirements.txt
	8	+ -rrequirements/requirements.txt
	9	+ -rrequirements/dev-requirements.txt
	10	+ -rrequirements/plugin-requirements.txt
11	11	passenv =
12	12	BST_FORCE_BACKEND
13	13	GI_TYPELIB_PATH
...	...	@@ -18,9 +18,9 @@ commands =
18	18	pycodestyle
19	19	pylint buildstream
20	20	deps =
21		- -rtools/requirements.txt
22		- -rtools/dev-requirements.txt
23		- -rtools/plugin-requirements.txt
	21	+ -rrequirements/requirements.txt
	22	+ -rrequirements/dev-requirements.txt
	23	+ -rrequirements/plugin-requirements.txt
24	24
25	25	[testenv:docs]
26	26	commands =
...	...	@@ -30,8 +30,8 @@ deps =
30	30	sphinx==1.7.9
31	31	sphinx-click
32	32	sphinx_rtd_theme
33		- -rtools/requirements.txt
34		- -rtools/plugin-requirements.txt
	33	+ -rrequirements/requirements.txt
	34	+ -rrequirements/plugin-requirements.txt
35	35	passenv =
36	36	BST_FORCE_SESSION_REBUILD
37	37	BST_SOURCE_CACHE

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.