[Notes] [Git][BuildGrid/buildgrid][finn/81-precon-fail] 8 commits: Instance and operation name errors now formatted with `[{}]`.

[finn <gitlab mg gitlab com>]

Title: GitLab

finn pushed to branch finn/81-precon-fail at BuildGrid / buildgrid

Commits:

• f7052140
  by finnball at 2018-09-17T13:40:08Z

  Instance and operation name errors now formatted with `[{}]`.
  
  Helps diagnose empty instance and operation name errors.
  

• 53257bac
  by Laurence Urhegyi at 2018-09-17T17:26:05Z

  Update README.rst
  
  Adds a link to Bazel in addition to BuildStream as a client.
  Amends some of the formatting of paragraphs so they are in a more logical order.

• f8e2b7c0
  by Laurence Urhegyi at 2018-09-17T17:34:59Z

  Update README.rst
  
  Adds links as hyperlinks so that the README page looks neater

• 37e735e5
  by Laurence Urhegyi at 2018-09-17T17:39:20Z

  Update index.rst
  
  Amends links to be hyperlinks so that the webpages look neater.

• c714a098
  by finnball at 2018-09-19T09:18:58Z

  Fails precondition if cannot fetch action from CAS.
  
  Also added new exception for precondition fail.
  

• 97a44af0
  by finnball at 2018-09-19T09:18:58Z

  Tests now account for precondition fail if no action in CAS
  

• 069db6ef
  by finnball at 2018-09-19T09:18:58Z

  Added new upload-dummy command.
  
  Uploads dummy action to CAS.
  

• d21db77b
  by finnball at 2018-09-19T09:18:58Z

  Included new upload-dummy command to CI and docs.
  

19 changed files:

• .gitlab-ci.yml
• README.rst
• buildgrid/_app/commands/cmd_cas.py
• buildgrid/_app/commands/cmd_execute.py
• buildgrid/bot/bot_session.py
• buildgrid/server/_exceptions.py
• buildgrid/server/actioncache/service.py
• buildgrid/server/bots/instance.py
• buildgrid/server/bots/service.py
• buildgrid/server/cas/service.py
• buildgrid/server/execution/instance.py
• buildgrid/server/execution/service.py
• buildgrid/server/operations/instance.py
• buildgrid/server/operations/service.py
• buildgrid/server/referencestorage/service.py
• docs/source/index.rst
• docs/source/using_dummy_build.rst
• tests/integration/execution_service.py
• tests/integration/operations_service.py

Changes:

.gitlab-ci.yml

---

...	...	@@ -33,6 +33,7 @@ before_script:
33	33	- ${BGD} server start buildgrid/_app/settings/default.yml &
34	34	- sleep 1 # Allow server to boot
35	35	- ${BGD} bot dummy &
	36	+ - ${BGD} cas upload-dummy
36	37	- ${BGD} execute request-dummy --wait-for-completion
37	38
38	39

README.rst

---

1		-
2	1	.. _about:
3	2
4		-About
5		-=====
	3	+
6	4
7	5	.. image:: https://gitlab.com/Buildgrid/buildgrid/badges/master/pipeline.svg
8	6	:target: https://gitlab.com/BuildStream/buildstream/commits/master
9	7
10	8	.. image:: https://gitlab.com/BuildGrid/buildgrid/badges/master/coverage.svg?job=coverage
11	9	:target: https://buildgrid.gitlab.io/buildgrid/coverage
	10	+
	11	+About BuildGrid
	12	+===============
	13	+
	14	+What is BuildGrid?
	15	+------------------
12	16
13	17	BuildGrid is a Python remote execution service which implements Google's
14	18	`Remote Execution API`_ and the `Remote Workers API`_. The project's goal is to
15	19	be able to execute build jobs remotely on a grid of computers in order to
16	20	massively speed up build times. Workers on the grid should be able to run with
17		-different environments. It is designed to work with but not exclusively
	21	+different environments. It is designed to work with clients such as `Bazel`_ and
18	22	`BuildStream`_.
19	23
20	24	.. _Remote Execution API: https://github.com/bazelbuild/remote-apis
21	25	.. _Remote Workers API: https://docs.google.com/document/d/1s_AzRRD2mdyktKUj2HWBn99rMg_3tcPvdjx3MPbFidU/edit#heading=h.1u2taqr2h940
22	26	.. _BuildStream: https://wiki.gnome.org/Projects/BuildStream
	27	+.. _Bazel: https://bazel.build
23	28
24	29
25	30	.. _getting-started:
...	...	@@ -40,10 +45,15 @@ instructions.
40	45	Resources
41	46	---------
42	47
43		-- Homepage: https://buildgrid.build
44		-- GitLab repository: https://gitlab.com/BuildGrid/buildgrid
45		-- Bug tracking: https://gitlab.com/BuildGrid/buildgrid/issues
46		-- Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
47		-- Slack channel: https://buildteamworld.slack.com/messages/CC9MKC203 [`invite link`_]
48		-
	48	+- `Homepage`_
	49	+- `GitLab repository`_
	50	+- `Bug tracking`_
	51	+- `Mailing list`_
	52	+- `Slack channel`_ [`invite link`_]
	53	+
	54	+.. _Homepage: https://buildgrid.build
	55	+.. _GitLab repository: https://gitlab.com/BuildGrid/buildgrid
	56	+.. _Bug tracking: https://gitlab.com/BuildGrid/buildgrid/issues
	57	+.. _Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
	58	+.. _Slack channel: https://buildteamworld.slack.com/messages/CC9MKC203
49	59	.. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU

buildgrid/_app/commands/cmd_cas.py

---

...	...	@@ -65,6 +65,23 @@ def cli(context, remote, instance_name, client_key, client_cert, server_cert):
65	65	context.logger.debug("Starting for remote {}".format(context.remote))
66	66
67	67
	68	+@cli.command('upload-dummy', short_help="Upload a dummy action. Should be used with `execute dummy-request`")
	69	+@pass_context
	70	+def upload_dummy(context):
	71	+ context.logger.info("Uploading dummy action...")
	72	+ action = remote_execution_pb2.Action(do_not_cache=True)
	73	+ action_digest = create_digest(action.SerializeToString())
	74	+
	75	+ request = remote_execution_pb2.BatchUpdateBlobsRequest(instance_name=context.instance_name)
	76	+ request.requests.add(digest=action_digest,
	77	+ data=action.SerializeToString())
	78	+
	79	+ stub = remote_execution_pb2_grpc.ContentAddressableStorageStub(context.channel)
	80	+ response = stub.BatchUpdateBlobs(request)
	81	+
	82	+ context.logger.info(response)
	83	+
	84	+
68	85	@cli.command('upload-files', short_help="Upload files to the CAS server.")
69	86	@click.argument('files', nargs=-1, type=click.File('rb'), required=True)
70	87	@pass_context

buildgrid/_app/commands/cmd_execute.py

---

...	...	@@ -76,9 +76,11 @@ def cli(context, remote, instance_name, client_key, client_cert, server_cert):
76	76	help="Stream updates until jobs are completed.")
77	77	@pass_context
78	78	def request_dummy(context, number, wait_for_completion):
79		- action_digest = remote_execution_pb2.Digest()
80	79
81	80	context.logger.info("Sending execution request...")
	81	+ action = remote_execution_pb2.Action(do_not_cache=True)
	82	+ action_digest = create_digest(action.SerializeToString())
	83	+
82	84	stub = remote_execution_pb2_grpc.ExecutionStub(context.channel)
83	85
84	86	request = remote_execution_pb2.ExecuteRequest(instance_name=context.instance_name,
...	...	@@ -90,9 +92,18 @@ def request_dummy(context, number, wait_for_completion):
90	92	responses.append(stub.Execute(request))
91	93
92	94	for response in responses:
	95	+
93	96	if wait_for_completion:
	97	+ result = None
94	98	for stream in response:
95		- context.logger.info(stream)
	99	+ result = stream
	100	+ context.logger.info(result)
	101	+
	102	+ if not result.done:
	103	+ click.echo("Result did not return True." +
	104	+ "Was the action uploaded to CAS?", err=True)
	105	+ sys.exit(-1)
	106	+
96	107	else:
97	108	context.logger.info(next(response))
98	109

buildgrid/bot/bot_session.py

---

...	...	@@ -99,13 +99,13 @@ class BotSession:
99	99	session = self._interface.create_bot_session(self._parent, self.get_pb2())
100	100	self._name = session.name
101	101
102		- self.logger.info("Created bot session with name: {}".format(self._name))
	102	+ self.logger.info("Created bot session with name: [{}]".format(self._name))
103	103
104	104	for lease in session.leases:
105	105	self._update_lease_from_server(lease)
106	106
107	107	def update_bot_session(self):
108		- self.logger.debug("Updating bot session: {}".format(self._bot_id))
	108	+ self.logger.debug("Updating bot session: [{}]".format(self._bot_id))
109	109	session = self._interface.update_bot_session(self.get_pb2())
110	110	for k, v in list(self._leases.items()):
111	111	if v.state == LeaseState.COMPLETED.value:
...	...	@@ -141,12 +141,12 @@ class BotSession:
141	141	asyncio.ensure_future(self.create_work(lease))
142	142
143	143	async def create_work(self, lease):
144		- self.logger.debug("Work created: {}".format(lease.id))
	144	+ self.logger.debug("Work created: [{}]".format(lease.id))
145	145
146	146	loop = asyncio.get_event_loop()
147	147	lease = await loop.run_in_executor(None, self._work, self._context, lease)
148	148
149		- self.logger.debug("Work complete: {}".format(lease.id))
	149	+ self.logger.debug("Work complete: [{}]".format(lease.id))
150	150	self.lease_completed(lease)
151	151
152	152
...	...	@@ -161,14 +161,14 @@ class Worker:
161	161	if k == 'pool':
162	162	self.properties[k] = v
163	163	else:
164		- raise KeyError('Key not supported: {}'.format(k))
	164	+ raise KeyError('Key not supported: [{}]'.format(k))
165	165
166	166	if configs:
167	167	for k, v in configs.items():
168	168	if k == 'DockerImage':
169	169	self.configs[k] = v
170	170	else:
171		- raise KeyError('Key not supported: {}'.format(k))
	171	+ raise KeyError('Key not supported: [{}]'.format(k))
172	172
173	173	@property
174	174	def configs(self):
...	...	@@ -214,11 +214,11 @@ class Device:
214	214
215	215	elif k == 'docker':
216	216	if v not in ('True', 'False'):
217		- raise ValueError('Value not supported: {}'.format(v))
	217	+ raise ValueError('Value not supported: [{}]'.format(v))
218	218	self._properties[k] = v
219	219
220	220	else:
221		- raise KeyError('Key not supported: {}'.format(k))
	221	+ raise KeyError('Key not supported: [{}]'.format(k))
222	222
223	223	@property
224	224	def name(self):

buildgrid/server/_exceptions.py

---

...	...	@@ -46,3 +46,12 @@ class OutOfRangeError(BgdError):
46	46
47	47	def __init__(self, message, detail=None, reason=None):
48	48	super().__init__(message, detail=detail, domain=ErrorDomain.SERVER, reason=reason)
	49	+
	50	+
	51	+class FailedPreconditionError(BgdError):
	52	+ """ One or more errors occurred in setting up the action requested, such as a missing input
	53	+ or command or no worker being available. The client may be able to fix the errors and retry.
	54	+ """
	55	+
	56	+ def __init__(self, message, detail=None, reason=None):
	57	+ super().__init__(message, detail=detail, domain=ErrorDomain.SERVER, reason=reason)

buildgrid/server/actioncache/service.py

---

...	...	@@ -77,4 +77,4 @@ class ActionCacheService(remote_execution_pb2_grpc.ActionCacheServicer):
77	77	return self._instances[instance_name]
78	78
79	79	except KeyError:
80		- raise InvalidArgumentError("Invalid instance name: {}".format(instance_name))
	80	+ raise InvalidArgumentError("Invalid instance name: [{}]".format(instance_name))

buildgrid/server/bots/instance.py

---

...	...	@@ -60,7 +60,7 @@ class BotsInterface:
60	60
61	61	self._bot_ids[name] = bot_id
62	62	self._bot_sessions[name] = bot_session
63		- self.logger.info("Created bot session name={} with bot_id={}".format(name, bot_id))
	63	+ self.logger.info("Created bot session name=[{}] with bot_id=[{}]".format(name, bot_id))
64	64
65	65	for lease in self._scheduler.create_leases():
66	66	bot_session.leases.extend([lease])
...	...	@@ -92,7 +92,7 @@ class BotsInterface:
92	92	try:
93	93	server_lease = self._scheduler.get_job_lease(client_lease.id)
94	94	except KeyError:
95		- raise InvalidArgumentError("Lease not found on server: {}".format(client_lease))
	95	+ raise InvalidArgumentError("Lease not found on server: [{}]".format(client_lease))
96	96
97	97	server_state = LeaseState(server_lease.state)
98	98	client_state = LeaseState(client_lease.state)
...	...	@@ -105,7 +105,7 @@ class BotsInterface:
105	105	# TODO: Lease was rejected
106	106	raise NotImplementedError("'Not Accepted' is unsupported")
107	107	else:
108		- raise OutofSyncError("Server lease: {}. Client lease: {}".format(server_lease, client_lease))
	108	+ raise OutofSyncError("Server lease: [{}]. Client lease: [{}]".format(server_lease, client_lease))
109	109
110	110	elif server_state == LeaseState.ACTIVE:
111	111
...	...	@@ -118,10 +118,10 @@ class BotsInterface:
118	118	return None
119	119
120	120	else:
121		- raise OutofSyncError("Server lease: {}. Client lease: {}".format(server_lease, client_lease))
	121	+ raise OutofSyncError("Server lease: [{}]. Client lease: [{}]".format(server_lease, client_lease))
122	122
123	123	elif server_state == LeaseState.COMPLETED:
124		- raise OutofSyncError("Server lease: {}. Client lease: {}".format(server_lease, client_lease))
	124	+ raise OutofSyncError("Server lease: [{}]. Client lease: [{}]".format(server_lease, client_lease))
125	125
126	126	elif server_state == LeaseState.CANCELLED:
127	127	raise NotImplementedError("Cancelled states not supported yet")
...	...	@@ -138,19 +138,19 @@ class BotsInterface:
138	138	if name is not None:
139	139	_bot_id = self._bot_ids.get(name)
140	140	if _bot_id is None:
141		- raise InvalidArgumentError('Name not registered on server: {}'.format(name))
	141	+ raise InvalidArgumentError('Name not registered on server: [{}]'.format(name))
142	142	elif _bot_id != bot_id:
143	143	self._close_bot_session(name)
144	144	raise InvalidArgumentError(
145		- 'Bot id invalid. ID sent: {} with name: {}.'
146		- 'ID registered: {} for that name'.format(bot_id, name, _bot_id))
	145	+ 'Bot id invalid. ID sent: [{}] with name: [{}].'
	146	+ 'ID registered: [{}] for that name'.format(bot_id, name, _bot_id))
147	147	else:
148	148	for _name, _bot_id in self._bot_ids.items():
149	149	if bot_id == _bot_id:
150	150	self._close_bot_session(_name)
151	151	raise InvalidArgumentError(
152		- 'Bot id already registered. ID sent: {}.'
153		- 'Id registered: {} with name: {}'.format(bot_id, _bot_id, _name))
	152	+ 'Bot id already registered. ID sent: [{}].'
	153	+ 'Id registered: [{}] with name: [{}]'.format(bot_id, _bot_id, _name))
154	154
155	155	def _close_bot_session(self, name):
156	156	""" Before removing the session, close any leases and
...	...	@@ -159,14 +159,14 @@ class BotsInterface:
159	159	bot_id = self._bot_ids.get(name)
160	160
161	161	if bot_id is None:
162		- raise InvalidArgumentError("Bot id does not exist: {}".format(name))
	162	+ raise InvalidArgumentError("Bot id does not exist: [{}]".format(name))
163	163
164		- self.logger.debug("Attempting to close {} with name: {}".format(bot_id, name))
	164	+ self.logger.debug("Attempting to close [{}] with name: [{}]".format(bot_id, name))
165	165	for lease in self._bot_sessions[name].leases:
166	166	if lease.state != LeaseState.COMPLETED.value:
167	167	# TODO: Be wary here, may need to handle rejected leases in future
168	168	self._scheduler.retry_job(lease.id)
169	169
170		- self.logger.debug("Closing bot session: {}".format(name))
	170	+ self.logger.debug("Closing bot session: [{}]".format(name))
171	171	self._bot_ids.pop(name)
172		- self.logger.info("Closed bot {} with name: {}".format(bot_id, name))
	172	+ self.logger.info("Closed bot [{}] with name: [{}]".format(bot_id, name))

buildgrid/server/bots/service.py

---

...	...	@@ -90,4 +90,4 @@ class BotsService(bots_pb2_grpc.BotsServicer):
90	90	return self._instances[name]
91	91
92	92	except KeyError:
93		- raise InvalidArgumentError("Instance doesn't exist on server: {}".format(name))
	93	+ raise InvalidArgumentError("Instance doesn't exist on server: [{}]".format(name))

buildgrid/server/cas/service.py

---

...	...	@@ -70,7 +70,7 @@ class ContentAddressableStorageService(remote_execution_pb2_grpc.ContentAddressa
70	70	return self._instances[instance_name]
71	71
72	72	except KeyError:
73		- raise InvalidArgumentError("Invalid instance name: {}".format(instance_name))
	73	+ raise InvalidArgumentError("Invalid instance name: [{}]".format(instance_name))
74	74
75	75
76	76	class ByteStreamService(bytestream_pb2_grpc.ByteStreamServicer):
...	...	@@ -89,15 +89,15 @@ class ByteStreamService(bytestream_pb2_grpc.ByteStreamServicer):
89	89	# TODO: Decide on default instance name
90	90	if path[0] == "blobs":
91	91	if len(path) < 3 or not path[2].isdigit():
92		- raise InvalidArgumentError("Invalid resource name: {}".format(request.resource_name))
	92	+ raise InvalidArgumentError("Invalid resource name: [{}]".format(request.resource_name))
93	93	instance_name = ""
94	94
95	95	elif path[1] == "blobs":
96	96	if len(path) < 4 or not path[3].isdigit():
97		- raise InvalidArgumentError("Invalid resource name: {}".format(request.resource_name))
	97	+ raise InvalidArgumentError("Invalid resource name: [{}]".format(request.resource_name))
98	98
99	99	else:
100		- raise InvalidArgumentError("Invalid resource name: {}".format(request.resource_name))
	100	+ raise InvalidArgumentError("Invalid resource name: [{}]".format(request.resource_name))
101	101
102	102	instance = self._get_instance(instance_name)
103	103	yield from instance.read(path,
...	...	@@ -134,15 +134,15 @@ class ByteStreamService(bytestream_pb2_grpc.ByteStreamServicer):
134	134	# TODO: Sort out no instance name
135	135	if path[0] == "uploads":
136	136	if len(path) < 5 or path[2] != "blobs" or not path[4].isdigit():
137		- raise InvalidArgumentError("Invalid resource name: {}".format(first_request.resource_name))
	137	+ raise InvalidArgumentError("Invalid resource name: [{}]".format(first_request.resource_name))
138	138	instance_name = ""
139	139
140	140	elif path[1] == "uploads":
141	141	if len(path) < 6 or path[3] != "blobs" or not path[5].isdigit():
142		- raise InvalidArgumentError("Invalid resource name: {}".format(first_request.resource_name))
	142	+ raise InvalidArgumentError("Invalid resource name: [{}]".format(first_request.resource_name))
143	143
144	144	else:
145		- raise InvalidArgumentError("Invalid resource name: {}".format(first_request.resource_name))
	145	+ raise InvalidArgumentError("Invalid resource name: [{}]".format(first_request.resource_name))
146	146
147	147	instance = self._get_instance(instance_name)
148	148	return instance.write(requests)
...	...	@@ -169,4 +169,4 @@ class ByteStreamService(bytestream_pb2_grpc.ByteStreamServicer):
169	169	return self._instances[instance_name]
170	170
171	171	except KeyError:
172		- raise InvalidArgumentError("Invalid instance name: {}".format(instance_name))
	172	+ raise InvalidArgumentError("Invalid instance name: [{}]".format(instance_name))

buildgrid/server/execution/instance.py

---

...	...	@@ -24,12 +24,12 @@ import logging
24	24	from buildgrid._protos.build.bazel.remote.execution.v2.remote_execution_pb2 import Action
25	25
26	26	from ..job import Job
27		-from .._exceptions import InvalidArgumentError
	27	+from .._exceptions import InvalidArgumentError, FailedPreconditionError
28	28
29	29
30	30	class ExecutionInstance:
31	31
32		- def __init__(self, scheduler, storage=None):
	32	+ def __init__(self, scheduler, storage):
33	33	self.logger = logging.getLogger(__name__)
34	34	self._storage = storage
35	35	self._scheduler = scheduler
...	...	@@ -40,14 +40,13 @@ class ExecutionInstance:
40	40	this action.
41	41	"""
42	42
43		- do_not_cache = False
44		- if self._storage is not None:
45		- action = self._storage.get_message(action_digest, Action)
46		- if action is not None:
47		- do_not_cache = action.do_not_cache
	43	+ action = self._storage.get_message(action_digest, Action)
48	44
49		- job = Job(action_digest, do_not_cache, message_queue)
50		- self.logger.info("Operation name: {}".format(job.name))
	45	+ if not action:
	46	+ raise FailedPreconditionError("Could not get action from storage.")
	47	+
	48	+ job = Job(action_digest, action.do_not_cache, message_queue)
	49	+ self.logger.info("Operation name: [{}]".format(job.name))
51	50
52	51	self._scheduler.append_job(job, skip_cache_lookup)
53	52
...	...	@@ -58,14 +57,14 @@ class ExecutionInstance:
58	57	self._scheduler.register_client(name, queue)
59	58
60	59	except KeyError:
61		- raise InvalidArgumentError("Operation name does not exist: {}".format(name))
	60	+ raise InvalidArgumentError("Operation name does not exist: [{}]".format(name))
62	61
63	62	def unregister_message_client(self, name, queue):
64	63	try:
65	64	self._scheduler.unregister_client(name, queue)
66	65
67	66	except KeyError:
68		- raise InvalidArgumentError("Operation name does not exist: {}".format(name))
	67	+ raise InvalidArgumentError("Operation name does not exist: [{}]".format(name))
69	68
70	69	def stream_operation_updates(self, message_queue, operation_name):
71	70	operation = message_queue.get()

buildgrid/server/execution/service.py

---

...	...	@@ -30,7 +30,7 @@ from buildgrid._protos.build.bazel.remote.execution.v2 import remote_execution_p
30	30
31	31	from buildgrid._protos.google.longrunning import operations_pb2
32	32
33		-from .._exceptions import InvalidArgumentError
	33	+from .._exceptions import InvalidArgumentError, FailedPreconditionError
34	34
35	35
36	36	class ExecutionService(remote_execution_pb2_grpc.ExecutionServicer):
...	...	@@ -61,6 +61,12 @@ class ExecutionService(remote_execution_pb2_grpc.ExecutionServicer):
61	61	context.set_code(grpc.StatusCode.INVALID_ARGUMENT)
62	62	yield operations_pb2.Operation()
63	63
	64	+ except FailedPreconditionError as e:
	65	+ self.logger.error(e)
	66	+ context.set_details(str(e))
	67	+ context.set_code(grpc.StatusCode.FAILED_PRECONDITION)
	68	+ yield operations_pb2.Operation()
	69	+
64	70	def WaitExecution(self, request, context):
65	71	try:
66	72	names = request.name.split("/")
...	...	@@ -92,4 +98,4 @@ class ExecutionService(remote_execution_pb2_grpc.ExecutionServicer):
92	98	return self._instances[name]
93	99
94	100	except KeyError:
95		- raise InvalidArgumentError("Instance doesn't exist on server: {}".format(name))
	101	+ raise InvalidArgumentError("Instance doesn't exist on server: [{}]".format(name))

buildgrid/server/operations/instance.py

---

...	...	@@ -34,7 +34,7 @@ class OperationsInstance:
34	34	operation = self._scheduler.jobs.get(name)
35	35
36	36	if operation is None:
37		- raise InvalidArgumentError("Operation name does not exist: {}".format(name))
	37	+ raise InvalidArgumentError("Operation name does not exist: [{}]".format(name))
38	38
39	39	else:
40	40	return operation.get_operation()
...	...	@@ -49,21 +49,21 @@ class OperationsInstance:
49	49	self._scheduler.jobs.pop(name)
50	50
51	51	except KeyError:
52		- raise InvalidArgumentError("Operation name does not exist: {}".format(name))
	52	+ raise InvalidArgumentError("Operation name does not exist: [{}]".format(name))
53	53
54	54	def register_message_client(self, name, queue):
55	55	try:
56	56	self._scheduler.register_client(name, queue)
57	57
58	58	except KeyError:
59		- raise InvalidArgumentError("Operation name does not exist: {}".format(name))
	59	+ raise InvalidArgumentError("Operation name does not exist: [{}]".format(name))
60	60
61	61	def unregister_message_client(self, name, queue):
62	62	try:
63	63	self._scheduler.unregister_client(name, queue)
64	64
65	65	except KeyError:
66		- raise InvalidArgumentError("Operation name does not exist: {}".format(name))
	66	+ raise InvalidArgumentError("Operation name does not exist: [{}]".format(name))
67	67
68	68	def stream_operation_updates(self, message_queue, operation_name):
69	69	operation = message_queue.get()

buildgrid/server/operations/service.py

---

...	...	@@ -132,4 +132,4 @@ class OperationsService(operations_pb2_grpc.OperationsServicer):
132	132	return self._instances[instance_name]
133	133
134	134	except KeyError:
135		- raise InvalidArgumentError("Instance doesn't exist on server: {}".format(name))
	135	+ raise InvalidArgumentError("Instance doesn't exist on server: [{}]".format(name))

buildgrid/server/referencestorage/service.py

---

...	...	@@ -86,4 +86,4 @@ class ReferenceStorageService(buildstream_pb2_grpc.ReferenceStorageServicer):
86	86	return self._instances[instance_name]
87	87
88	88	except KeyError:
89		- raise InvalidArgumentError("Invalid instance name: {}".format(instance_name))
	89	+ raise InvalidArgumentError("Invalid instance name: [{}]".format(instance_name))

docs/source/index.rst

---

...	...	@@ -24,10 +24,15 @@ Remote execution service implementing Google's REAPI and RWAPI.
24	24	Resources
25	25	---------
26	26
27		-- Homepage: https://buildgrid.build
28		-- GitLab repository: https://gitlab.com/BuildGrid/buildgrid
29		-- Bug tracking: https://gitlab.com/BuildGrid/buildgrid/issues
30		-- Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
31		-- Slack channel: https://buildteamworld.slack.com/messages/CC9MKC203 [`invite link`_]
32		-
	27	+- `Homepage`_
	28	+- `GitLab repository`_
	29	+- `Bug tracking`_
	30	+- `Mailing list`_
	31	+- `Slack channel`_ [`invite link`_]
	32	+
	33	+.. _Homepage: https://buildgrid.build
	34	+.. _GitLab repository: https://gitlab.com/BuildGrid/buildgrid
	35	+.. _Bug tracking: https://gitlab.com/BuildGrid/buildgrid/issues
	36	+.. _Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
	37	+.. _Slack channel: https://buildteamworld.slack.com/messages/CC9MKC203
33	38	.. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU

docs/source/using_dummy_build.rst

---

...	...	@@ -9,7 +9,13 @@ In one terminal, start a server:
9	9
10	10	bgd server start buildgrid/_app/settings/default.yml
11	11
12		-In another terminal, send a request for work:
	12	+In another terminal, upload an action to CAS:
	13	+
	14	+.. code-block::sh
	15	+
	16	+ bgd cas upload-dummy
	17	+
	18	+Then send a request for work:
13	19
14	20	.. code-block:: sh
15	21

tests/integration/execution_service.py

---

...	...	@@ -28,6 +28,7 @@ import pytest
28	28	from buildgrid._protos.build.bazel.remote.execution.v2 import remote_execution_pb2
29	29	from buildgrid._protos.google.longrunning import operations_pb2
30	30
	31	+from buildgrid.utils import create_digest
31	32	from buildgrid.server import job
32	33	from buildgrid.server.controller import ExecutionController
33	34	from buildgrid.server.cas.storage import lru_memory_cache
...	...	@@ -37,6 +38,8 @@ from buildgrid.server.execution.service import ExecutionService
37	38
38	39
39	40	server = mock.create_autospec(grpc.server)
	41	+action = remote_execution_pb2.Action(do_not_cache=True)
	42	+action_digest = create_digest(action.SerializeToString())
40	43
41	44
42	45	@pytest.fixture
...	...	@@ -47,12 +50,16 @@ def context():
47	50
48	51	@pytest.fixture(params=["action-cache", "no-action-cache"])
49	52	def controller(request):
	53	+ storage = lru_memory_cache.LRUMemoryCache(1024 * 1024)
	54	+ write_session = storage.begin_write(action_digest)
	55	+ storage.commit_write(action_digest, write_session)
	56	+
50	57	if request.param == "action-cache":
51		- storage = lru_memory_cache.LRUMemoryCache(1024 * 1024)
52	58	cache = ActionCache(storage, 50)
53	59	yield ExecutionController(cache, storage)
	60	+
54	61	else:
55		- yield ExecutionController()
	62	+ yield ExecutionController(None, storage)
56	63
57	64
58	65	# Instance to test
...	...	@@ -65,9 +72,6 @@ def instance(controller):
65	72
66	73	@pytest.mark.parametrize("skip_cache_lookup", [True, False])
67	74	def test_execute(skip_cache_lookup, instance, context):
68		- action_digest = remote_execution_pb2.Digest()
69		- action_digest.hash = 'zhora'
70		-
71	75	request = remote_execution_pb2.ExecuteRequest(instance_name='',
72	76	action_digest=action_digest,
73	77	skip_cache_lookup=skip_cache_lookup)
...	...	@@ -90,10 +94,16 @@ def test_wrong_execute_instance(instance, context):
90	94	context.set_code.assert_called_once_with(grpc.StatusCode.INVALID_ARGUMENT)
91	95
92	96
93		-def test_wait_execution(instance, controller, context):
94		- action_digest = remote_execution_pb2.Digest()
95		- action_digest.hash = 'zhora'
	97	+def test_no_action_digest_in_storage(instance, context):
	98	+ request = remote_execution_pb2.ExecuteRequest(instance_name='',
	99	+ skip_cache_lookup=True)
	100	+ response = instance.Execute(request, context)
	101	+
	102	+ next(response)
	103	+ context.set_code.assert_called_once_with(grpc.StatusCode.FAILED_PRECONDITION)
96	104
	105	+
	106	+def test_wait_execution(instance, controller, context):
97	107	j = job.Job(action_digest, None)
98	108	j._operation.done = True
99	109

tests/integration/operations_service.py

---

...	...	@@ -24,18 +24,21 @@ import grpc
24	24	from grpc._server import _Context
25	25	import pytest
26	26
27		-from buildgrid._protos.build.bazel.remote.execution.v2 import remote_execution_pb2
28		-from buildgrid._protos.google.longrunning import operations_pb2
29		-
	27	+from buildgrid.utils import create_digest
30	28	from buildgrid.server.controller import ExecutionController
31		-from buildgrid.server._exceptions import InvalidArgumentError
32		-
	29	+from buildgrid.server.cas.storage import lru_memory_cache
33	30	from buildgrid.server.operations import service
34	31	from buildgrid.server.operations.service import OperationsService
	32	+from buildgrid.server._exceptions import InvalidArgumentError
	33	+
	34	+from buildgrid._protos.build.bazel.remote.execution.v2 import remote_execution_pb2
	35	+from buildgrid._protos.google.longrunning import operations_pb2
35	36
36	37
37	38	server = mock.create_autospec(grpc.server)
38	39	instance_name = "blade"
	40	+action = remote_execution_pb2.Action(do_not_cache=True)
	41	+action_digest = create_digest(action.SerializeToString())
39	42
40	43
41	44	# Can mock this
...	...	@@ -47,9 +50,6 @@ def context():
47	50	# Requests to make
48	51	@pytest.fixture
49	52	def execute_request():
50		- action_digest = remote_execution_pb2.Digest()
51		- action_digest.hash = 'zhora'
52		-
53	53	yield remote_execution_pb2.ExecuteRequest(instance_name='',
54	54	action_digest=action_digest,
55	55	skip_cache_lookup=True)
...	...	@@ -57,7 +57,11 @@ def execute_request():
57	57
58	58	@pytest.fixture
59	59	def controller():
60		- yield ExecutionController()
	60	+ storage = lru_memory_cache.LRUMemoryCache(1024 * 1024)
	61	+ write_session = storage.begin_write(action_digest)
	62	+ storage.commit_write(action_digest, write_session)
	63	+
	64	+ yield ExecutionController(None, storage)
61	65
62	66
63	67	# Instance to test

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.