Martin Blanchard pushed to branch mablanch/connection-drop-simulation at BuildGrid / buildgrid
Commits:
-
6afc4f91
by Raoul Hidalgo Charman at 2018-10-04T17:19:49Z
-
0ea2b629
by Martin Blanchard at 2018-10-05T09:29:53Z
-
e82d70b1
by Martin Blanchard at 2018-10-05T09:29:53Z
-
d7d43431
by Martin Blanchard at 2018-10-05T11:41:46Z
-
2df25ff7
by Martin Blanchard at 2018-10-08T15:49:32Z
-
8f5a93c6
by Marios Hadjimichael at 2018-10-08T18:22:53Z
-
6f82144b
by Laurence Urhegyi at 2018-10-08T20:13:31Z
-
3aec6c0d
by Laurence Urhegyi at 2018-10-08T20:16:48Z
-
025bcdee
by Finn at 2018-10-10T19:23:05Z
-
1631a659
by Laurence Urhegyi at 2018-10-10T19:30:29Z
-
c2554302
by Laurence Urhegyi at 2018-10-10T19:33:44Z
-
a49581a6
by Laurence Urhegyi at 2018-10-10T19:35:05Z
-
81ee2399
by Finn at 2018-10-11T17:16:03Z
-
1417a8bf
by Laurence Urhegyi at 2018-10-11T19:20:02Z
-
5e8eb6a9
by Laurence Urhegyi at 2018-10-15T18:39:59Z
-
cee31b4a
by Laurence Urhegyi at 2018-10-15T18:55:20Z
-
b032b482
by Martin Blanchard at 2018-10-22T11:00:36Z
14 changed files:
- .pylintrc
- CONTRIBUTING.rst
- + Dockerfile
- README.rst
- buildgrid/_app/commands/cmd_execute.py
- buildgrid/_app/commands/cmd_operation.py
- buildgrid/server/execution/instance.py
- buildgrid/server/execution/service.py
- buildgrid/server/operations/service.py
- docs/source/index.rst
- docs/source/installation.rst
- + docs/source/resources.rst
- setup.cfg
- setup.py
Changes:
... | ... | @@ -455,8 +455,12 @@ known-standard-library= |
455 | 455 |
|
456 | 456 |
# Force import order to recognize a module as part of a third party library.
|
457 | 457 |
known-third-party=boto3,
|
458 |
+ click,
|
|
458 | 459 |
enchant,
|
459 |
- grpc
|
|
460 |
+ google,
|
|
461 |
+ grpc,
|
|
462 |
+ moto,
|
|
463 |
+ yaml
|
|
460 | 464 |
|
461 | 465 |
|
462 | 466 |
[DESIGN]
|
1 |
- |
|
2 | 1 |
.. _contributing:
|
3 | 2 |
|
4 | 3 |
Contributing
|
... | ... | @@ -18,9 +17,13 @@ discuss with us before submitting anything, as we may well have some important |
18 | 17 |
context which will could help guide your efforts.
|
19 | 18 |
|
20 | 19 |
Any major feature additions should be raised first as a proposal on the
|
21 |
-`BuildGrid mailing list`_ to be discussed, and then eventually followed up with
|
|
22 |
-an issue on GitLab. We recommend that you propose the feature in advance of
|
|
23 |
-commencing work.
|
|
20 |
+BuildGrid mailing list to be discussed between the core contributors. Once
|
|
21 |
+this discussion has taken place and there is agreement on how to proceed,
|
|
22 |
+it should be followed by with a Gitlab issue being raised which summarizes
|
|
23 |
+the tasks required.
|
|
24 |
+ |
|
25 |
+We strongly recommend that you propose the feature in advance of
|
|
26 |
+commencing any work.
|
|
24 | 27 |
|
25 | 28 |
The author of any patch is expected to take ownership of that code and is to
|
26 | 29 |
support it for a reasonable time-frame. This means addressing any unforeseen
|
... | ... | @@ -204,17 +207,16 @@ Committer access |
204 | 207 |
We'll hand out commit access to anyone who has successfully landed a single
|
205 | 208 |
patch to the code base. Please request this via Slack or the mailing list.
|
206 | 209 |
|
207 |
-This of course relies on contributors being responsive and show willingness to
|
|
208 |
-address problems after landing branches there should not be any problems here.
|
|
210 |
+This of course relies on contributors being responsive and showing willingness
|
|
211 |
+to address any problems that may arise after landing branches.
|
|
209 | 212 |
|
210 |
-What we are expecting of committers here in general is basically to escalate the
|
|
211 |
-review in cases of uncertainty:
|
|
213 |
+When submitting a merge request, please obtain a review from another committer
|
|
214 |
+who is familiar with the area of the code base which the branch effects. An
|
|
215 |
+approval from another committer who is not the patch author will be needed
|
|
216 |
+before any merge (we use gitlab's 'approval' feature for this).
|
|
212 | 217 |
|
213 |
-- If the change is very trivial (obvious few line changes, typos…), and you are
|
|
214 |
- confident of the change, there is no need for review.
|
|
215 |
-- If the change is non trivial, please obtain a review from another committer
|
|
216 |
- who is familiar with the area which the branch effects. An approval from
|
|
217 |
- someone who is not the patch author will be needed before any merge.
|
|
218 |
+What we are expecting of committers here in general is basically to escalate the
|
|
219 |
+review in cases of uncertainty.
|
|
218 | 220 |
|
219 | 221 |
.. note::
|
220 | 222 |
|
... | ... | @@ -239,21 +241,49 @@ following goals: |
239 | 241 |
for the viewer to digest.
|
240 | 242 |
- Ensure that we keep it simple and easy to contribute to the project.
|
241 | 243 |
|
242 |
-We are currenlty using the following GitLab features:
|
|
244 |
+Explanation of how the project is currenlty using some GitLab features:
|
|
243 | 245 |
|
244 | 246 |
- `Milestones`_: we have seen them used in the same way as `Epics`_ in other
|
245 |
- projects. BuildGrid milestones must be time-line based, can overlap and we can
|
|
246 |
- be working towards multiple milestones at any one time. They allow us to group
|
|
247 |
- together all sub tasks into an overall aim. See our `BuildGrid milestones`_.
|
|
248 |
-- `Labels`_: allow us to filter tickets in useful ways. They do complexity and
|
|
249 |
- effort as they grow in number and usage, though, so the general approach is
|
|
250 |
- to have the minimum possible. See our `BuildGrid labels`_.
|
|
247 |
+ projects and are trying not to do that here. Instead we are going to
|
|
248 |
+ use milestones to denote development cycles (ie, two week 'sprints'). See the
|
|
249 |
+ `BuildGrid milestones`_.
|
|
250 |
+- `Labels`_: allow us to filter tickets (ie, 'issues' in gitlab terminology)
|
|
251 |
+ in useful ways. They add complexity and effort as they grow in number, so the
|
|
252 |
+ general approach is to have the minimum possible but
|
|
253 |
+ ensure we use them consistently. See the `BuildGrid labels`_.
|
|
251 | 254 |
- `Boards`_: allow us to visualise and manage issues and labels in a simple way.
|
252 |
- For now, we are only utilising one boards. Issues start life in the
|
|
253 |
- ``Backlog`` column by default, and we move them into ``ToDo`` when they are
|
|
254 |
- coming up in the next few weeks. ``Doing`` is only for when an item is
|
|
255 |
- currently being worked on. Moving an issue from column to column automatically
|
|
256 |
- adjust the tagged labels. See our `BuildGrid boards`_.
|
|
255 |
+ Issues start life in the ``Backlog`` column by default, and we move them into
|
|
256 |
+ ``ToDo`` when we aim to complete them in the current development cycle.
|
|
257 |
+ ``Doing`` is only for when an item is currently being worked on. When on the
|
|
258 |
+ Board view, dragging and dropping an issue from column to column automatically
|
|
259 |
+ adjusts the relevant labels. See the `BuildGrid boards`_.
|
|
260 |
+
|
|
261 |
+
|
|
262 |
+Guidelines for using GitLab features when working on this project:
|
|
263 |
+
|
|
264 |
+- When raising an issue, please:
|
|
265 |
+
|
|
266 |
+ - check to see if there already is an issue to cover this task (if not then
|
|
267 |
+ raise a new one)
|
|
268 |
+ - assign the appropriate label or labels (tip: the vast majority of issues
|
|
269 |
+ raised will be either an enhancement or a bug)
|
|
270 |
+
|
|
271 |
+- If you plan to work on an issue, please:
|
|
272 |
+ |
|
273 |
+ - self-assign the ticket
|
|
274 |
+ - ensure it's captured in the current sprint (ie, Gitlab milestone)
|
|
275 |
+ - ensure the ticket is in the ``ToDo`` column of the board if you aim to
|
|
276 |
+ complete in the current sprint but aren't yet working on it, and
|
|
277 |
+ the ``Doing`` column if you are working on it currently.
|
|
278 |
+ |
|
279 |
+- Please note that Gitlab issues are for either 'tasks' or 'bugs' - ie not for
|
|
280 |
+ long discussions (where the mailing list is a better choice) or for ranting,
|
|
281 |
+ for example.
|
|
282 |
+
|
|
283 |
+The above may seem like a lot to take in, but please don't worry about getting
|
|
284 |
+it right the first few times. The worst that can happen is that you'll get a
|
|
285 |
+friendly message from a current contributor who explains the process. We welcome
|
|
286 |
+and value all contributions to the project!
|
|
257 | 287 |
|
258 | 288 |
.. _Milestones: https://docs.gitlab.com/ee/user/project/milestones
|
259 | 289 |
.. _Epics: https://docs.gitlab.com/ee/user/group/epics
|
1 |
+FROM python:3.5-stretch
|
|
2 |
+ |
|
3 |
+# Point the path to where buildgrid gets installed
|
|
4 |
+ENV PATH=$PATH:/root/.local/bin/
|
|
5 |
+ |
|
6 |
+# Upgrade python modules
|
|
7 |
+RUN python3 -m pip install --upgrade setuptools pip
|
|
8 |
+ |
|
9 |
+# Use /app as the current working directory
|
|
10 |
+WORKDIR /app
|
|
11 |
+ |
|
12 |
+# Copy the repo contents (source, config files, etc) in the WORKDIR
|
|
13 |
+COPY . .
|
|
14 |
+ |
|
15 |
+# Install BuildGrid
|
|
16 |
+RUN pip install --user --editable .
|
|
17 |
+ |
|
18 |
+# Entry Point of the image (should get an additional argument from CMD, the path to the config file)
|
|
19 |
+ENTRYPOINT ["bgd", "-v", "server", "start"]
|
|
20 |
+ |
|
21 |
+# Default config file (used if no CMD specified when running)
|
|
22 |
+CMD ["buildgrid/_app/settings/default.yml"]
|
|
23 |
+ |
1 |
- |
|
2 | 1 |
.. _about:
|
3 | 2 |
|
4 | 3 |
About
|
... | ... | @@ -14,13 +13,15 @@ BuildGrid is a Python remote execution service which implements Google's |
14 | 13 |
`Remote Execution API`_ and the `Remote Workers API`_. The project's goal is to
|
15 | 14 |
be able to execute build jobs remotely on a grid of computers in order to
|
16 | 15 |
massively speed up build times. Workers on the grid should be able to run with
|
17 |
-different environments. It is designed to work with clients such as `Bazel`_ and
|
|
18 |
-`BuildStream`_.
|
|
16 |
+different environments. It works with clients such as `Bazel`_,
|
|
17 |
+`BuildStream`_ and `RECC`_, and is designed to be able to work with any client
|
|
18 |
+that conforms to the above API protocols.
|
|
19 | 19 |
|
20 | 20 |
.. _Remote Execution API: https://github.com/bazelbuild/remote-apis
|
21 | 21 |
.. _Remote Workers API: https://docs.google.com/document/d/1s_AzRRD2mdyktKUj2HWBn99rMg_3tcPvdjx3MPbFidU/edit#heading=h.1u2taqr2h940
|
22 | 22 |
.. _BuildStream: https://wiki.gnome.org/Projects/BuildStream
|
23 | 23 |
.. _Bazel: https://bazel.build
|
24 |
+.. _RECC: https://gitlab.com/bloomberg/recc
|
|
24 | 25 |
|
25 | 26 |
|
26 | 27 |
.. _getting-started:
|
... | ... | @@ -49,7 +50,7 @@ Resources |
49 | 50 |
|
50 | 51 |
.. _Homepage: https://buildgrid.build
|
51 | 52 |
.. _GitLab repository: https://gitlab.com/BuildGrid/buildgrid
|
52 |
-.. _Bug tracking: https://gitlab.com/BuildGrid/buildgrid/issues
|
|
53 |
+.. _Bug tracking: https://gitlab.com/BuildGrid/buildgrid/boards
|
|
53 | 54 |
.. _Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
|
54 | 55 |
.. _Slack channel: https://buildteamworld.slack.com/messages/CC9MKC203
|
55 | 56 |
.. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU
|
... | ... | @@ -169,6 +169,7 @@ def run_command(context, input_root, commands, output_file, output_directory): |
169 | 169 |
|
170 | 170 |
downloader.download_file(output_file_response.digest, path)
|
171 | 171 |
|
172 |
- if output_file_response.path in output_executeables:
|
|
173 |
- st = os.stat(path)
|
|
174 |
- os.chmod(path, st.st_mode | stat.S_IXUSR)
|
|
172 |
+ for output_file_response in execute_response.result.output_files:
|
|
173 |
+ if output_file_response.path in output_executeables:
|
|
174 |
+ st = os.stat(path)
|
|
175 |
+ os.chmod(path, st.st_mode | stat.S_IXUSR)
|
... | ... | @@ -101,8 +101,7 @@ def lists(context): |
101 | 101 |
@pass_context
|
102 | 102 |
def wait(context, operation_name):
|
103 | 103 |
stub = remote_execution_pb2_grpc.ExecutionStub(context.channel)
|
104 |
- request = remote_execution_pb2.WaitExecutionRequest(instance_name=context.instance_name,
|
|
105 |
- name=operation_name)
|
|
104 |
+ request = remote_execution_pb2.WaitExecutionRequest(name=operation_name)
|
|
106 | 105 |
|
107 | 106 |
response = stub.WaitExecution(request)
|
108 | 107 |
|
... | ... | @@ -21,8 +21,11 @@ An instance of the Remote Execution Service. |
21 | 21 |
|
22 | 22 |
import logging
|
23 | 23 |
|
24 |
+import grpc
|
|
25 |
+ |
|
24 | 26 |
from buildgrid._exceptions import FailedPreconditionError, InvalidArgumentError
|
25 | 27 |
from buildgrid._protos.build.bazel.remote.execution.v2.remote_execution_pb2 import Action
|
28 |
+from buildgrid._protos.google.longrunning import operations_pb2
|
|
26 | 29 |
|
27 | 30 |
from ..job import Job
|
28 | 31 |
|
... | ... | @@ -71,7 +74,12 @@ class ExecutionInstance: |
71 | 74 |
|
72 | 75 |
def stream_operation_updates(self, message_queue, operation_name):
|
73 | 76 |
operation = message_queue.get()
|
77 |
+ last_operation = operation
|
|
74 | 78 |
while not operation.done:
|
75 | 79 |
yield operation
|
80 |
+ last_operation = operation
|
|
76 | 81 |
operation = message_queue.get()
|
82 |
+ if not isinstance(operation, operations_pb2.Operation):
|
|
83 |
+ message_queue.context_.cancel()
|
|
84 |
+ operation = last_operation
|
|
77 | 85 |
yield operation
|
... | ... | @@ -52,6 +52,8 @@ class ExecutionService(remote_execution_pb2_grpc.ExecutionServicer): |
52 | 52 |
context.add_callback(partial(instance.unregister_message_client,
|
53 | 53 |
operation.name, message_queue))
|
54 | 54 |
|
55 |
+ message_queue.context_ = context
|
|
56 |
+ |
|
55 | 57 |
yield from instance.stream_operation_updates(message_queue,
|
56 | 58 |
operation.name)
|
57 | 59 |
|
... | ... | @@ -68,9 +70,10 @@ class ExecutionService(remote_execution_pb2_grpc.ExecutionServicer): |
68 | 70 |
yield operations_pb2.Operation()
|
69 | 71 |
|
70 | 72 |
def WaitExecution(self, request, context):
|
73 |
+ print('Client reopened the operation stream!')
|
|
74 |
+ |
|
71 | 75 |
try:
|
72 | 76 |
names = request.name.split("/")
|
73 |
- |
|
74 | 77 |
# Operation name should be in format:
|
75 | 78 |
# {instance/name}/{operation_id}
|
76 | 79 |
instance_name = ''.join(names[0:-1])
|
... | ... | @@ -43,14 +43,16 @@ class OperationsService(operations_pb2_grpc.OperationsServicer): |
43 | 43 |
|
44 | 44 |
def GetOperation(self, request, context):
|
45 | 45 |
try:
|
46 |
- name = request.name
|
|
47 |
- operation_name = self._get_operation_name(name)
|
|
46 |
+ operation_name, job = request.name, None
|
|
47 |
+ for instance in self._instances.values():
|
|
48 |
+ if operation_name in instance._scheduler.jobs:
|
|
49 |
+ job = instance._scheduler.jobs.get(operation_name)
|
|
50 |
+ break
|
|
48 | 51 |
|
49 |
- instance = self._get_instance(name)
|
|
52 |
+ print('Connection drop simulation request received')
|
|
50 | 53 |
|
51 |
- operation = instance.get_operation(operation_name)
|
|
52 |
- operation.name = name
|
|
53 |
- return operation
|
|
54 |
+ for queue in job._operation_update_queues:
|
|
55 |
+ queue.put('drop')
|
|
54 | 56 |
|
55 | 57 |
except InvalidArgumentError as e:
|
56 | 58 |
self.logger.error(e)
|
... | ... | @@ -19,6 +19,7 @@ Remote execution service implementing Google's REAPI and RWAPI. |
19 | 19 |
using.rst
|
20 | 20 |
reference.rst
|
21 | 21 |
contributing.rst
|
22 |
+ resources.rst
|
|
22 | 23 |
|
23 | 24 |
|
24 | 25 |
Resources
|
... | ... | @@ -28,11 +29,11 @@ Resources |
28 | 29 |
- `GitLab repository`_
|
29 | 30 |
- `Bug tracking`_
|
30 | 31 |
- `Mailing list`_
|
31 |
-- `Slack channel`_ [`invite link`_]
|
|
32 |
+- `Slack channel`_ [`invite link`_]
|
|
32 | 33 |
|
33 | 34 |
.. _Homepage: https://buildgrid.build
|
34 | 35 |
.. _GitLab repository: https://gitlab.com/BuildGrid/buildgrid
|
35 |
-.. _Bug tracking: https://gitlab.com/BuildGrid/buildgrid/issues
|
|
36 |
+.. _Bug tracking: https://gitlab.com/BuildGrid/buildgrid/boards
|
|
36 | 37 |
.. _Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
|
37 | 38 |
.. _Slack channel: https://buildteamworld.slack.com/messages/CC9MKC203
|
38 |
-.. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU
|
|
39 |
+.. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU
|
|
\ No newline at end of file |
1 |
- |
|
2 | 1 |
.. _installation:
|
3 | 2 |
|
4 | 3 |
Installation
|
5 | 4 |
============
|
6 | 5 |
|
7 |
-How to install BuildGrid onto your machine.
|
|
6 |
+.. _install-on-host:
|
|
7 |
+ |
|
8 |
+Installation onto host machine
|
|
9 |
+------------------------------
|
|
10 |
+ |
|
11 |
+How to install BuildGrid directly onto your machine.
|
|
8 | 12 |
|
9 | 13 |
.. note::
|
10 | 14 |
|
11 |
- BuildGrid server currently only support *Linux*, *macOS* and *Windows*
|
|
15 |
+ BuildGrid server currently only support *Linux*. *macOS* and *Windows*
|
|
12 | 16 |
platforms are **not** supported.
|
13 | 17 |
|
14 | 18 |
|
15 |
-.. _install-prerequisites:
|
|
19 |
+.. _install-host-prerequisites:
|
|
16 | 20 |
|
17 | 21 |
Prerequisites
|
18 |
--------------
|
|
22 |
+~~~~~~~~~~~~~
|
|
19 | 23 |
|
20 | 24 |
BuildGrid only supports ``python3 >= 3.5`` but has no system requirements. Main
|
21 |
-Python dependencies, automatically handle during installation, includes:
|
|
25 |
+Python dependencies, automatically handled during installation, include:
|
|
22 | 26 |
|
23 | 27 |
- `boto3`_: the Amazon Web Services (AWS) SDK for Python.
|
24 | 28 |
- `click`_: a Python composable command line library.
|
... | ... | @@ -33,10 +37,10 @@ Python dependencies, automatically handle during installation, includes: |
33 | 37 |
.. _protocol-buffers: https://developers.google.com/protocol-buffers
|
34 | 38 |
|
35 | 39 |
|
36 |
-.. _source-install:
|
|
40 |
+.. _install-host-source-install:
|
|
37 | 41 |
|
38 | 42 |
Install from sources
|
39 |
---------------------
|
|
43 |
+~~~~~~~~~~~~~~~~~~~~
|
|
40 | 44 |
|
41 | 45 |
BuildGrid has ``setuptools`` support. In order to install it to your home
|
42 | 46 |
directory, typically under ``~/.local``, simply run:
|
... | ... | @@ -46,7 +50,7 @@ directory, typically under ``~/.local``, simply run: |
46 | 50 |
git clone https://gitlab.com/BuildGrid/buildgrid.git && cd buildgrid
|
47 | 51 |
pip3 install --user --editable .
|
48 | 52 |
|
49 |
-Additionally, and if your distribution does not already includes it, you may
|
|
53 |
+Additionally, and if your distribution does not already include it, you may
|
|
50 | 54 |
have to adjust your ``PATH``, in ``~/.bashrc``, with:
|
51 | 55 |
|
52 | 56 |
.. code-block:: sh
|
... | ... | @@ -63,3 +67,62 @@ have to adjust your ``PATH``, in ``~/.bashrc``, with: |
63 | 67 |
.. code-block:: sh
|
64 | 68 |
|
65 | 69 |
pip3 install --user --editable ".[docs,tests]"
|
70 |
+ |
|
71 |
+ |
|
72 |
+ |
|
73 |
+.. install-docker:
|
|
74 |
+ |
|
75 |
+Installation through docker
|
|
76 |
+---------------------------
|
|
77 |
+ |
|
78 |
+How to build a Docker image that runs BuildGrid.
|
|
79 |
+ |
|
80 |
+.. _install-docker-prerequisites:
|
|
81 |
+ |
|
82 |
+Prerequisites
|
|
83 |
+~~~~~~~~~~~~~
|
|
84 |
+ |
|
85 |
+A working Docker installation. Please consult `Docker's Getting Started Guide`_ if you don't already have it installed.
|
|
86 |
+ |
|
87 |
+.. _`Docker's Getting Started Guide`: https://www.docker.com/get-started
|
|
88 |
+ |
|
89 |
+ |
|
90 |
+.. _install-docker-build:
|
|
91 |
+ |
|
92 |
+Docker Container from Sources
|
|
93 |
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
94 |
+ |
|
95 |
+To clone the source code and build a Docker image, simply run:
|
|
96 |
+ |
|
97 |
+.. code-block:: sh
|
|
98 |
+ |
|
99 |
+ git clone https://gitlab.com/BuildGrid/buildgrid.git && cd buildgrid
|
|
100 |
+ docker build -t buildgrid_server .
|
|
101 |
+ |
|
102 |
+.. note::
|
|
103 |
+ |
|
104 |
+ The image built will contain the contents of the source code directory, including
|
|
105 |
+ configuration files.
|
|
106 |
+
|
|
107 |
+.. hint::
|
|
108 |
+ |
|
109 |
+ Whenever the source code is updated or new configuration files are made, you need to re-build
|
|
110 |
+ the image.
|
|
111 |
+ |
|
112 |
+After building the Docker image, to run BuildGrid using the default configuration file
|
|
113 |
+(found in `buildgrid/_app/settings/default.yml`), simply run:
|
|
114 |
+ |
|
115 |
+.. code-block:: sh
|
|
116 |
+ |
|
117 |
+ docker run -i -p 50051:50051 buildgrid_server
|
|
118 |
+ |
|
119 |
+.. note::
|
|
120 |
+ |
|
121 |
+ To run BuildGrid using a different configuration file, include the relative path to the
|
|
122 |
+ configuration file at the end of the command above. For example, to run the default
|
|
123 |
+ standalone CAS server (without an execution service), simply run:
|
|
124 |
+ |
|
125 |
+ .. code-block:: sh
|
|
126 |
+ |
|
127 |
+ docker run -i -p 50052:50052 buildgrid_server buildgrid/_app/settings/cas.yml
|
|
128 |
+ |
1 |
+.. _external-resources:
|
|
2 |
+ |
|
3 |
+Resources
|
|
4 |
+=========
|
|
5 |
+ |
|
6 |
+Remote execution and worker API useful links:
|
|
7 |
+ |
|
8 |
+- `REAPI design document`_
|
|
9 |
+- `REAPI protobuf specification`_
|
|
10 |
+- `RWAPI design document`_
|
|
11 |
+- `RWAPI protobuf specification`_
|
|
12 |
+- `Bazel`_ `remote caching and execution documentation`_
|
|
13 |
+- `RECC usage instructions`_
|
|
14 |
+- `BuildStream webside`_ and `documentation`_
|
|
15 |
+- `BuildStream-externals repository`_
|
|
16 |
+- `BuildBox repository`_
|
|
17 |
+- `FUSE on wikipedia`_ and `kernel documentation`_
|
|
18 |
+- `bubblewrap repository`_
|
|
19 |
+- `Buildfarm reference REAPI implementation`_
|
|
20 |
+- `Buildbarn Golang REAPI implementation`_
|
|
21 |
+- `Demonstration of RECC with BuildGrid`_
|
|
22 |
+- `Demonstration of Bazel with BuildGrid`_
|
|
23 |
+- `Demonstration of BuildStream with BuildGrid`_
|
|
24 |
+ |
|
25 |
+.. _REAPI design document: https://docs.google.com/document/d/1AaGk7fOPByEvpAbqeXIyE8HX_A3_axxNnvroblTZ_6s
|
|
26 |
+.. _REAPI protobuf specification: https://github.com/bazelbuild/remote-apis/blob/master/build/bazel/remote/execution/v2/remote_execution.proto
|
|
27 |
+.. _RWAPI design document: https://docs.google.com/document/d/1s_AzRRD2mdyktKUj2HWBn99rMg_3tcPvdjx3MPbFidU
|
|
28 |
+.. _RWAPI protobuf specification: https://github.com/googleapis/googleapis/blob/master/google/devtools/remoteworkers/v1test2/bots.proto
|
|
29 |
+.. _Bazel: https://www.bazel.build
|
|
30 |
+.. _remote caching and execution documentation: https://docs.bazel.build/versions/master/remote-caching.html
|
|
31 |
+.. _RECC usage instructions: https://gitlab.com/bloomberg/recc#running-recc
|
|
32 |
+.. _BuildStream webside: https://buildstream.build
|
|
33 |
+.. _documentation: https://docs.buildstream.build
|
|
34 |
+.. _BuildStream-externals repository: https://gitlab.com/BuildStream/bst-external
|
|
35 |
+.. _FUSE on wikipedia: https://en.wikipedia.org/wiki/Filesystem_in_Userspace
|
|
36 |
+.. _kernel documentation: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/fuse.txt
|
|
37 |
+.. _BuildBox repository: https://gitlab.com/BuildStream/buildbox
|
|
38 |
+.. _bubblewrap repository: https://github.com/projectatomic/bubblewrap
|
|
39 |
+.. _Buildfarm reference REAPI implementation: https://github.com/bazelbuild/bazel-buildfarm
|
|
40 |
+.. _Buildbarn Golang REAPI implementation: https://github.com/EdSchouten/bazel-buildbarn
|
|
41 |
+.. _Demonstration of RECC with BuildGrid: https://asciinema.org/a/0FjExIqrTGSlpSUIS8Ehf5gUg
|
|
42 |
+.. _Demonstration of Bazel with BuildGrid: https://asciinema.org/a/uVHFWOxpivwJ4ari23CEerR8N
|
|
43 |
+.. _Demonstration of BuildStream with BuildGrid: https://asciinema.org/a/QfkYGqhfhEQz4o8prlBdEBFP7
|
... | ... | @@ -14,3 +14,6 @@ pep8ignore = |
14 | 14 |
docs/source/conf.py ALL
|
15 | 15 |
*_pb2.py ALL
|
16 | 16 |
*_pb2_grpc.py ALL
|
17 |
+filterwarnings =
|
|
18 |
+ ignore::DeprecationWarning
|
|
19 |
+ ignore::PendingDeprecationWarning
|
|
\ No newline at end of file |
... | ... | @@ -90,7 +90,7 @@ tests_require = [ |
90 | 90 |
'moto',
|
91 | 91 |
'pep8',
|
92 | 92 |
'psutil',
|
93 |
- 'pytest == 3.6.4',
|
|
93 |
+ 'pytest >= 3.8.0',
|
|
94 | 94 |
'pytest-cov >= 2.6.0',
|
95 | 95 |
'pytest-pep8',
|
96 | 96 |
'pytest-pylint',
|