[Notes] [Git][BuildStream/buildstream][phil/tox-vev-environment] 11 commits: doc/../arch_dependency_model: transient/transitive

[Phil Dawson <gitlab mg gitlab com>]

Title: GitLab

Phil Dawson pushed to branch phil/tox-vev-environment at BuildStream / buildstream

Commits:

• 40de8204
  by Angelos Evripiotis at 2019-02-20T14:07:24Z

  doc/../arch_dependency_model: transient/transitive
  

• 5a9143b2
  by Angelos Evripiotis at 2019-02-20T14:07:24Z

  doc/./arch_scheduler: fixup "it's" typos
  
  "it's" is always short for "it is" or "it has".
  
  The possesive pronoun for "it" is "its", like "his".
  

• c2263902
  by Angelos Evripiotis at 2019-02-20T14:07:24Z

  doc/./arch_scheduler: fix 'imerative' typo
  

• 3cceaada
  by Angelos Evripiotis at 2019-02-20T14:07:24Z

  doc/./arch_cachekeys: JSON now, not dict pickle
  

• 53f94d05
  by Angelos Evripiotis at 2019-02-20T14:07:24Z

  doc/./arch_cachekeys: consistent full-stops
  

• 9382001c
  by Angelos Evripiotis at 2019-02-20T14:07:24Z

  doc/./arch_cachekeys: note no direct runtime deps
  
  I was mistaken on this point in my first reading. The sentence on strong
  and weak key equivalency made me check my assumptions.
  
  Help others with the same misunderstanding by explicitly calling this
  out in a new paragraph.
  

• e5c2e708
  by Angelos Evripiotis at 2019-02-20T14:07:24Z

  doc/./arch_sandboxing: 'read only'->'read-only'
  
  I read this wrong initially, the hyphen would have helped me here.
  

• 10006748
  by Angelos Evripiotis at 2019-02-20T14:07:24Z

  doc/./arch_sandboxing: reword 'user provided user'
  

• 54515138
  by Angelos Evripiotis at 2019-02-20T14:07:24Z

  doc/./arch_sandboxing: no OSTree artifact cache
  
  As of commit 1f8b4aa290a908a697f008a29ea143a9320dd639, we're no longer
  using the OSTree artifact cache. Update accordingly.
  

• 80a0832c
  by Angelos Evripiotis at 2019-02-20T15:13:59Z

  Merge branch 'aevri/doc_arch_deps' into 'master'
  
  Architecture docs: minor corrections, clarifications, and nitpicks
  
  See merge request BuildStream/buildstream!1170

• eee9e283
  by Phil Dawson at 2019-02-20T15:14:42Z

  tox.ini: Add 'venv' environment to run arbitrary commands in a venv
  
  This is a handy way for developers to make use of the venvs we're
  constructing for our test suite to run in.
  

6 changed files:

• CONTRIBUTING.rst
• doc/source/arch_cachekeys.rst
• doc/source/arch_dependency_model.rst
• doc/source/arch_sandboxing.rst
• doc/source/arch_scheduler.rst
• tox.ini

Changes:

CONTRIBUTING.rst

---

...	...	@@ -1589,6 +1589,15 @@ can run ``tox`` with ``-r`` or ``--recreate`` option.
1589	1589
1590	1590	./setup.py test --addopts 'tests/frontend/buildtrack.py::test_build_track'
1591	1591
	1592	+.. tip::
	1593	+
	1594	+ We also have an environment called 'venv' which takes any arguments
	1595	+ you give it and runs them inside the same virtualenv we use for our
	1596	+ tests::
	1597	+
	1598	+ tox -e venv -- <your command(s) here>
	1599	+
	1600	+ Any commands after ``--`` will be run a virtualenv managed by tox.
1592	1601
1593	1602	Observing coverage
1594	1603	~~~~~~~~~~~~~~~~~~

doc/source/arch_cachekeys.rst

---

...	...	@@ -10,14 +10,14 @@ for the purpose of reusing artifacts in a well-defined, predictable way.
10	10
11	11	Structure
12	12	---------
13		-Cache keys are SHA256 hash values generated from a pickled Python dict that
	13	+Cache keys are SHA256 hash values generated from a UTF-8 JSON document that
14	14	includes:
15	15
16		-* Environment (e.g., project configuration and variables)
17		-* Element configuration (details depend on element kind, ``Element.get_unique_key()``)
18		-* Sources (``Source.get_unique_key()``)
19		-* Dependencies (depending on cache key type, see below)
20		-* Public data
	16	+* Environment (e.g., project configuration and variables).
	17	+* Element configuration (details depend on element kind, ``Element.get_unique_key()``).
	18	+* Sources (``Source.get_unique_key()``).
	19	+* Dependencies (depending on cache key type, see below).
	20	+* Public data.
21	21
22	22	Cache key types
23	23	---------------
...	...	@@ -42,6 +42,9 @@ or the environment changes but it will not change when a dependency is updated.
42	42	For elements without build dependencies the ``strong`` cache key is identical
43	43	to the ``weak`` cache key.
44	44
	45	+Note that dependencies which are not required at build time do not affect
	46	+either kind of key.
	47	+
45	48	Strict build plan
46	49	-----------------
47	50	This is the default build plan that exclusively uses ``strong`` cache keys

doc/source/arch_dependency_model.rst

---

...	...	@@ -42,7 +42,8 @@ Scope
42	42	* **Scope.RUN**
43	43
44	44	In the :func:`Scope.RUN <buildstream.types.Scope.RUN>` scope, only elements
45		- which are required to run are considered, including the element itself.
	45	+ which are required to run are considered, including the element itself. Note
	46	+ that these are transitive - the service also requires the base runtime.
46	47
47	48	This is used when for example, launching a ``bst shell`` environment
48	49	for the purpose of running, or in any case we need to consider which
...	...	@@ -60,7 +61,7 @@ Scope
60	61	.. image:: images/arch-dependency-model-build.svg
61	62	:align: center
62	63
63		- Note that build type dependencies are not transient, which is why the
	64	+ Note that build type dependencies are not transitive, which is why the
64	65	*Bootstrap* element is not selected when pulling in the *Compiler* to
65	66	build the *Application*.
66	67

doc/source/arch_sandboxing.rst

---

...	...	@@ -51,7 +51,7 @@ well.
51	51	Filesystem access
52	52	~~~~~~~~~~~~~~~~~
53	53
54		-The filesystem inside sandboxes should be read only during element assembly,
	54	+The filesystem inside sandboxes should be read-only during element assembly,
55	55	except for certain directories which element plugins can mark as being
56	56	read/write. Most elements plugins derive from :mod:`BuildElement
57	57	<buildstream.buildelement>`, which marks ``%{build-root}`` and
...	...	@@ -158,17 +158,17 @@ and will refuse to push any artifacts built on such a system to a remote cache.
158	158	For more information, see `issue #92
159	159	<https://gitlab.com/BuildStream/buildstream/issues/92>`_.
160	160
161		-The Linux platform can operate as a standard user provided user namespace
	161	+The Linux platform can operate as a standard user, if user namespace
162	162	support is available. If user namespace support is not available you have the
163	163	option of installing bubblewrap as a setuid binary to avoid needing to run the
164	164	entire ``bst`` process as the ``root`` user.
165	165
166		-The artifact cache on Linux systems is implemented using `OSTree
167		-<https://github.com/ostreedev/ostree>`_, which can allow us to stage artifacts
168		-using hardlinks instead of copying them. To avoid cache corruption it is
169		-vital that hardlinked files cannot be overwritten. In cases where the root
170		-filesystem inside the sandbox needs to be writable, a custom FUSE filesystem
171		-named SafeHardlinks is used which provides a copy-on-write layer.
	166	+The artifact cache on Linux systems is implemented using a content-addressable
	167	+hardlink farm, which can allow us to stage artifacts using hardlinks instead of
	168	+copying them. To avoid cache corruption it is vital that hardlinked files
	169	+cannot be overwritten. In cases where the root filesystem inside the sandbox
	170	+needs to be writable, a custom FUSE filesystem named SafeHardlinks is used
	171	+which provides a copy-on-write layer.
172	172
173	173	Some of the operations on filesystem metadata listed above are not prohibited
174	174	by the sandbox, but will instead be silently dropped when an artifact is

doc/source/arch_scheduler.rst

---

...	...	@@ -19,9 +19,9 @@ The Job class has the following responsibilities:
19	19	* Offering an abstract method for subclasses to handle the outcome of
20	20	a job when it completes.
21	21
22		-* Forcefully terminating it's subprocess.
	22	+* Forcefully terminating its subprocess.
23	23
24		-* Suspending and resuming it's subprocess.
	24	+* Suspending and resuming its subprocess.
25	25
26	26	* Declaring the types of resources it will require, and which resources
27	27	it will require exclusively.
...	...	@@ -73,7 +73,7 @@ The Queue implementations are:
73	73
74	74	The pull queue tries to obtain a built artifact from a remote artifact server,
75	75	it should be placed in advance of the fetch queue if used, since we can safely
76		- avoid fetching if fetching is not imerative, and we already have a cached
	76	+ avoid fetching if fetching is not imperative, and we already have a cached
77	77	artifact.
78	78
79	79	* **Fetch**
...	...	@@ -84,7 +84,7 @@ The Queue implementations are:
84	84
85	85	* **Build**
86	86
87		- The build queue attempts to build the element if it's artifact is not locally
	87	+ The build queue attempts to build the element if its artifact is not locally
88	88	present.
89	89
90	90	* **Push**

tox.ini

---

...	...	@@ -91,3 +91,14 @@ commands =
91	91	deps =
92	92	click-man >= 0.3.0
93	93	-rrequirements/requirements.txt
	94	+
	95	+#
	96	+# Usefull for running arbitrary scripts in a BuildStream virtual env
	97	+#
	98	+[testenv:venv]
	99	+commands = {posargs}
	100	+deps =
	101	+ -rrequirements/requirements.txt
	102	+ -rrequirements/dev-requirements.txt
	103	+ -rrequirements/plugin-requirements.txt
	104	+whitelist_externals = *

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.