[Notes] [Git][BuildGrid/buildgrid][finn/parser-docs] 6 commits: Locked sphinx to version 1.7.8.

[finn <gitlab mg gitlab com>]

Title: GitLab

finn pushed to branch finn/parser-docs at BuildGrid / buildgrid

Commits:

• 9401c87d
  by Finn at 2018-10-02T16:33:03Z

  Locked sphinx to version 1.7.8.
  
  Breaks rtd-theme search function at higher versions currently.
  

• 51f661fd
  by Jürg Billeter at 2018-10-03T07:50:34Z

  docs: Fix typos
  

• febe2ced
  by Martin Blanchard at 2018-10-03T08:08:56Z

  storage/disk.py: Port to BuildStream compatible path scheme
  
  https://gitlab.com/BuildGrid/buildgrid/issues/93
  

• e80b6503
  by Martin Blanchard at 2018-10-03T08:08:56Z

  client/cas.py: Fix assertion in batch downloading client
  
  https://gitlab.com/BuildGrid/buildgrid/issues/93
  

• a305c9a1
  by Finn at 2018-10-03T09:22:30Z

  An action cache miss should not be an error.
  

• b5196dc4
  by Finn at 2018-10-03T09:30:03Z

  Added doc strings to YAML parser.
  

9 changed files:

• buildgrid/_app/settings/parser.py
• buildgrid/client/cas.py
• buildgrid/server/actioncache/service.py
• buildgrid/server/cas/storage/disk.py
• docs/source/reference.rst
• + docs/source/reference_server_config.rst
• docs/source/using_bazel.rst
• docs/source/using_buildstream.rst
• setup.py

Changes:

buildgrid/_app/settings/parser.py

---

...	...	@@ -35,6 +35,9 @@ from ..cli import Context
35	35
36	36
37	37	class YamlFactory(yaml.YAMLObject):
	38	+ """ Base class for contructing maps or scalars from tags.
	39	+ """
	40	+
38	41	@classmethod
39	42	def from_yaml(cls, loader, node):
40	43	if isinstance(node, yaml.ScalarNode):
...	...	@@ -47,6 +50,21 @@ class YamlFactory(yaml.YAMLObject):
47	50
48	51
49	52	class Channel(YamlFactory):
	53	+ """Creates a GRPC channel.
	54	+
	55	+ The :class:`Channel` class returns a `grpc.Channel` and is generated from the tag ``!channel``.
	56	+ Creates either a secure or insecure channel.
	57	+
	58	+ Args:
	59	+ port (int): A port for the channel.
	60	+ insecure_mode (bool): If ``True``, generates an insecure channel, even if there are
	61	+ credentials. Defaults to ``True``.
	62	+ credentials (dict, optional): A dictionary in the form::
	63	+
	64	+ tls-server-key: /path/to/server-key
	65	+ tls-server-cert: /path/to/server-cert
	66	+ tls-client-certs: /path/to/client-certs
	67	+ """
50	68
51	69	yaml_tag = u'!channel'
52	70
...	...	@@ -69,6 +87,13 @@ class Channel(YamlFactory):
69	87
70	88
71	89	class ExpandPath(YamlFactory):
	90	+ """Returns a string of the user's path after expansion.
	91	+
	92	+ The :class:`ExpandPath` class returns a string and is generated from the tag ``!expand-path``.
	93	+
	94	+ Args:
	95	+ path (str): Can be used with strings such as: ``~/dir/to/something`` or ``$HOME/certs``
	96	+ """
72	97
73	98	yaml_tag = u'!expand-path'
74	99
...	...	@@ -79,14 +104,30 @@ class ExpandPath(YamlFactory):
79	104
80	105
81	106	class Disk(YamlFactory):
	107	+ """Generates :class:`buildgrid.server.cas.storage.disk.DiskStorage` using the tag ``!disk-storage``.
	108	+
	109	+ Args:
	110	+ path (str): Path to directory to storage.
	111	+
	112	+ """
82	113
83	114	yaml_tag = u'!disk-storage'
84	115
85	116	def __new__(cls, path):
	117	+ """Creates a new disk
	118	+
	119	+ Args:
	120	+ path (str): Some path
	121	+ """
86	122	return DiskStorage(path)
87	123
88	124
89	125	class LRU(YamlFactory):
	126	+ """Generates :class:`buildgrid.server.cas.storage.lru_memory_cache.LRUMemoryCache` using the tag ``!lru-storage``.
	127	+
	128	+ Args:
	129	+ size (int): Size e.g ``10kb``. Size parsed with :meth:`buildgrid._app.settings.parser._parse_size`.
	130	+ """
90	131
91	132	yaml_tag = u'!lru-storage'
92	133
...	...	@@ -95,6 +136,12 @@ class LRU(YamlFactory):
95	136
96	137
97	138	class S3(YamlFactory):
	139	+ """Generates :class:`buildgrid.server.cas.storage.s3.S3Storage` using the tag ``!s3-storage``.
	140	+
	141	+ Args:
	142	+ bucket (str): Name of bucket
	143	+ endpoint (str): URL of endpoint.
	144	+ """
98	145
99	146	yaml_tag = u'!s3-storage'
100	147
...	...	@@ -103,6 +150,18 @@ class S3(YamlFactory):
103	150
104	151
105	152	class Remote(YamlFactory):
	153	+ """Generates :class:`buildgrid.server.cas.storage.remote.RemoteStorage`
	154	+ using the tag ``!remote-storage``.
	155	+
	156	+ Args:
	157	+ url (str): URL to remote storage. If used with ``https``, needs credentials.
	158	+ instance_name (str): Instance of the remote to connect to.
	159	+ credentials (dict, optional): A dictionary in the form::
	160	+
	161	+ tls-client-key: /path/to/client-key
	162	+ tls-client-cert: /path/to/client-cert
	163	+ tls-server-cert: /path/to/server-cert
	164	+ """
106	165
107	166	yaml_tag = u'!remote-storage'
108	167
...	...	@@ -144,6 +203,18 @@ class Remote(YamlFactory):
144	203
145	204
146	205	class WithCache(YamlFactory):
	206	+ """Generates :class:`buildgrid.server.cas.storage.with_cache.WithCacheStorage`
	207	+ using the tag ``!with-cache-storage``.
	208	+
	209	+ Args:
	210	+ url (str): URL to remote storage. If used with ``https``, needs credentials.
	211	+ instance_name (str): Instance of the remote to connect to.
	212	+ credentials (dict, optional): A dictionary in the form::
	213	+
	214	+ tls-client-key: /path/to/certs
	215	+ tls-client-cert: /path/to/certs
	216	+ tls-server-cert: /path/to/certs
	217	+ """
147	218
148	219	yaml_tag = u'!with-cache-storage'
149	220
...	...	@@ -152,6 +223,13 @@ class WithCache(YamlFactory):
152	223
153	224
154	225	class Execution(YamlFactory):
	226	+ """Generates :class:`buildgrid.server.execution.service.ExecutionService`
	227	+ using the tag ``!execution``.
	228	+
	229	+ Args:
	230	+ storage(:class:`buildgrid.server.cas.storage.storage_abc.StorageABC`): Instance of storage to use.
	231	+ action_cache(:class:`Action`): Instance of action cache to use.
	232	+ """
155	233
156	234	yaml_tag = u'!execution'
157	235
...	...	@@ -160,6 +238,14 @@ class Execution(YamlFactory):
160	238
161	239
162	240	class Action(YamlFactory):
	241	+ """Generates :class:`buildgrid.server.actioncache.service.ActionCacheService`
	242	+ using the tag ``!action-cache``.
	243	+
	244	+ Args:
	245	+ storage(:class:`buildgrid.server.cas.storage.storage_abc.StorageABC`): Instance of storage to use.
	246	+ max_cached_refs(int): Max number of cached actions.
	247	+ allow_updates(bool): Allow updates pushed to CAS. Defaults to ``True``.
	248	+ """
163	249
164	250	yaml_tag = u'!action-cache'
165	251
...	...	@@ -168,6 +254,14 @@ class Action(YamlFactory):
168	254
169	255
170	256	class Reference(YamlFactory):
	257	+ """Generates :class:`buildgrid.server.referencestorage.service.ReferenceStorageService`
	258	+ using the tag ``!reference-cache``.
	259	+
	260	+ Args:
	261	+ storage(:class:`buildgrid.server.cas.storage.storage_abc.StorageABC`): Instance of storage to use.
	262	+ max_cached_refs(int): Max number of cached actions.
	263	+ allow_updates(bool): Allow updates pushed to CAS. Defauled to ``True``.
	264	+ """
171	265
172	266	yaml_tag = u'!reference-cache'
173	267
...	...	@@ -176,6 +270,12 @@ class Reference(YamlFactory):
176	270
177	271
178	272	class CAS(YamlFactory):
	273	+ """Generates :class:`buildgrid.server.cas.service.ContentAddressableStorageService`
	274	+ using the tag ``!cas``.
	275	+
	276	+ Args:
	277	+ storage(:class:`buildgrid.server.cas.storage.storage_abc.StorageABC`): Instance of storage to use.
	278	+ """
179	279
180	280	yaml_tag = u'!cas'
181	281
...	...	@@ -184,6 +284,12 @@ class CAS(YamlFactory):
184	284
185	285
186	286	class ByteStream(YamlFactory):
	287	+ """Generates :class:`buildgrid.server.cas.service.ByteStreamService`
	288	+ using the tag ``!bytestream``.
	289	+
	290	+ Args:
	291	+ storage(:class:`buildgrid.server.cas.storage.storage_abc.StorageABC`): Instance of storage to use.
	292	+ """
187	293
188	294	yaml_tag = u'!bytestream'
189	295

buildgrid/client/cas.py

---

...	...	@@ -283,7 +283,7 @@ class Downloader:
283	283	try:
284	284	batch_response = self.__cas_stub.BatchReadBlobs(batch_request)
285	285	for response in batch_response.responses:
286		- assert response.digest.hash in digests
	286	+ assert response.digest in digests
287	287
288	288	read_blobs.append(response.data)
289	289

buildgrid/server/actioncache/service.py

---

...	...	@@ -52,7 +52,7 @@ class ActionCacheService(remote_execution_pb2_grpc.ActionCacheServicer):
52	52	context.set_code(grpc.StatusCode.INVALID_ARGUMENT)
53	53
54	54	except NotFoundError as e:
55		- self.logger.error(e)
	55	+ self.logger.info(e)
56	56	context.set_code(grpc.StatusCode.NOT_FOUND)
57	57
58	58	return remote_execution_pb2.ActionResult()

buildgrid/server/cas/storage/disk.py

---

...	...	@@ -21,7 +21,6 @@ A CAS storage provider that stores files as blobs on disk.
21	21	"""
22	22
23	23	import os
24		-import pathlib
25	24	import tempfile
26	25
27	26	from .storage_abc import StorageABC
...	...	@@ -30,28 +29,41 @@ from .storage_abc import StorageABC
30	29	class DiskStorage(StorageABC):
31	30
32	31	def __init__(self, path):
33		- self._path = pathlib.Path(path)
34		- os.makedirs(str(self._path / "temp"), exist_ok=True)
	32	+ if not os.path.isabs(path):
	33	+ self.__root_path = os.path.abspath(path)
	34	+ else:
	35	+ self.__root_path = path
	36	+ self.__cas_path = os.path.join(self.__root_path, 'cas')
	37	+
	38	+ self.objects_path = os.path.join(self.__cas_path, 'objects')
	39	+ self.temp_path = os.path.join(self.__root_path, 'tmp')
	40	+
	41	+ os.makedirs(self.objects_path, exist_ok=True)
	42	+ os.makedirs(self.temp_path, exist_ok=True)
35	43
36	44	def has_blob(self, digest):
37		- return (self._path / (digest.hash + "_" + str(digest.size_bytes))).exists()
	45	+ return os.path.exists(self._get_object_path(digest))
38	46
39	47	def get_blob(self, digest):
40	48	try:
41		- return (self._path / (digest.hash + "_" + str(digest.size_bytes))).open('rb')
	49	+ return open(self._get_object_path(digest), 'rb')
42	50	except FileNotFoundError:
43	51	return None
44	52
45		- def begin_write(self, _digest):
46		- return tempfile.NamedTemporaryFile("wb", dir=str(self._path / "temp"))
	53	+ def begin_write(self, digest):
	54	+ return tempfile.NamedTemporaryFile("wb", dir=self.temp_path)
47	55
48	56	def commit_write(self, digest, write_session):
49		- # Atomically move the temporary file into place.
50		- path = self._path / (digest.hash + "_" + str(digest.size_bytes))
51		- os.replace(write_session.name, str(path))
	57	+ object_path = self._get_object_path(digest)
	58	+
52	59	try:
53		- write_session.close()
54		- except FileNotFoundError:
55		- # We moved the temporary file to a new location, so when Python
56		- # tries to delete its old location, it'll fail.
	60	+ os.makedirs(os.path.dirname(object_path), exist_ok=True)
	61	+ os.link(write_session.name, object_path)
	62	+ except FileExistsError:
	63	+ # Object is already there!
57	64	pass
	65	+
	66	+ write_session.close()
	67	+
	68	+ def _get_object_path(self, digest):
	69	+ return os.path.join(self.objects_path, digest.hash[:2], digest.hash[2:])

docs/source/reference.rst

---

1		-
2	1	.. _reference:
3	2
4	3	Reference
...	...	@@ -11,3 +10,4 @@ This section contains BuildGrid API and CLI reference documentation.
11	10
12	11	reference_api.rst
13	12	reference_cli.rst
	13	+ reference_server_config.rst

docs/source/reference_server_config.rst

---

	1	+.. _parser:
	2	+
	3	+Server configuration reference
	4	+==============================
	5	+
	6	+BuildGrid's server configuration. To be used with::
	7	+
	8	+ bgd server start server.conf
	9	+
	10	+.. automodule:: buildgrid._app.settings.parser
	11	+ :members:
	12	+ :undoc-members:
	13	+ :show-inheritance:

docs/source/using_bazel.rst

---

...	...	@@ -98,7 +98,7 @@ has ``gcc`` installed, run:
98	98
99	99	The ``--remote`` option is used to specify the server location (running on the
100	100	same machine here, and listening to port 50051). The ``--parent`` option is used
101		-to specify the server instance you except the bot to be attached to. Refer to
	101	+to specify the server instance you expect the bot to be attached to. Refer to
102	102	the :ref:`CLI reference section <invoking-bgd-bot-temp-directory>` for command
103	103	line interface details.
104	104
...	...	@@ -120,7 +120,7 @@ You can finally have Bazel to build the example workspace by running:
120	120
121	121	bazel build //main:hello-world
122	122
123		-You can verify that the example has been successfully build by running the
	123	+You can verify that the example has been successfully built by running the
124	124	generated executable. Simply invoke:
125	125
126	126	.. code-block:: sh

docs/source/using_buildstream.rst

---

...	...	@@ -149,7 +149,7 @@ that the ``buildbox`` tool is functional on your machine (refer to
149	149
150	150	The ``--remote`` option is used to specify the server location (running on the
151	151	same machine here, and listening to port 50051). The ``--parent`` option is
152		-used to specify the server instance you except the bot to be attached to (empty
	152	+used to specify the server instance you expect the bot to be attached to (empty
153	153	here). ``--fuse-dir`` and ``--local-cas`` are specific to the ``buildbox`` bot
154	154	and respectively specify the sandbox mount point and local CAS cache locations.
155	155	Refer to the :ref:`CLI reference section <invoking-bgd-bot-buildbox>` for
...	...	@@ -178,7 +178,7 @@ You can finally have BuildStream to build the example project by running:
178	178
179	179	bst build hello.bst
180	180
181		-You can verify that the example has been successfully build by running the
	181	+You can verify that the example has been successfully built by running the
182	182	generated executable. Simply invoke:
183	183
184	184	.. code-block:: sh

setup.py

---

...	...	@@ -97,7 +97,8 @@ tests_require = [
97	97	]
98	98
99	99	docs_require = [
100		- 'sphinx',
	100	+ # rtd-theme broken in Sphinx >= 1.8, this breaks search functionality.
	101	+ 'sphinx == 1.7.8',
101	102	'sphinx-click',
102	103	'sphinx-rtd-theme',
103	104	'sphinxcontrib-apidoc',

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.