Martin Blanchard pushed to branch mablanch/51-clients-usage at BuildGrid / buildgrid
Commits:
-
2f02bd54
by Martin Blanchard at 2018-09-19T08:35:06Z
-
88f5fc94
by Martin Blanchard at 2018-09-19T08:35:08Z
-
4c2639d4
by Martin Blanchard at 2018-09-19T08:52:13Z
7 changed files:
- CONTRIBUTING.rst
- README.rst
- + docs/source/data/bazel-example-server.conf
- docs/source/using.rst
- + docs/source/using_bazel.rst
- − docs/source/using_dummy_build.rst
- docs/source/using_simple_build.rst → docs/source/using_internal.rst
Changes:
... | ... | @@ -142,13 +142,67 @@ their containing *package*, as such; modules which are entirely private to |
142 | 142 |
BuildGrid are named as such, e.g. ``_roy.py``.
|
143 | 143 |
|
144 | 144 |
|
145 |
+.. _codebase-testing:
|
|
146 |
+ |
|
147 |
+Testing
|
|
148 |
+-------
|
|
149 |
+ |
|
150 |
+BuildGrid is using `pytest`_ for regression and newly added code testing. The
|
|
151 |
+test suite contains a serie of unit-tests and also run linting tools in order to
|
|
152 |
+detect coding-style_ breakage. The full test suite is automatically executed by
|
|
153 |
+GitLab CI system for every push to the server. Passing all the tests is a
|
|
154 |
+mandatory requirement for any merge request to the trunk.
|
|
155 |
+ |
|
156 |
+.. _pytest: https://docs.pytest.org
|
|
157 |
+ |
|
158 |
+ |
|
159 |
+Running tests
|
|
160 |
+~~~~~~~~~~~~~
|
|
161 |
+ |
|
162 |
+In order to run the entire test suite, simply run:
|
|
163 |
+ |
|
164 |
+.. code-block:: sh
|
|
165 |
+ |
|
166 |
+ python3 setup.py test
|
|
167 |
+ |
|
168 |
+You can use the ``--addopt`` function to feed arguments to pytest. For example,
|
|
169 |
+if you want to see the ``stdout`` and ``stderr`` generated y the test, run:
|
|
170 |
+ |
|
171 |
+.. code-block:: sh
|
|
172 |
+ |
|
173 |
+ python3 setup.py test --addopts -s
|
|
174 |
+ |
|
175 |
+If you want run a specific test instead of the entire suite use:
|
|
176 |
+ |
|
177 |
+.. code-block:: sh
|
|
178 |
+ |
|
179 |
+ python3 setup.py test --addopts tests/cas/test_client
|
|
180 |
+ |
|
181 |
+pyest's `usage documentation section`_ details the different command line
|
|
182 |
+options that can be used when invoking the test runner.
|
|
183 |
+ |
|
184 |
+.. _usage documentation section: https://docs.pytest.org/en/latest/usage.html
|
|
185 |
+ |
|
186 |
+ |
|
187 |
+Test coverage
|
|
188 |
+~~~~~~~~~~~~~
|
|
189 |
+ |
|
190 |
+We are doing our best at keeping BuildGrid's test coverage score as high as
|
|
191 |
+possible. Doing so, we ask for any merge request to include necessary test
|
|
192 |
+additions and/or modifications in order to maintain that coverage level. A
|
|
193 |
+detailed `coverage report`_ is produced and publish for any change merged to the
|
|
194 |
+trunk.
|
|
195 |
+ |
|
196 |
+.. _coverage report: https://buildgrid.gitlab.io/buildgrid/coverage/
|
|
197 |
+ |
|
198 |
+ |
|
145 | 199 |
.. _committer-access:
|
146 | 200 |
|
147 | 201 |
Committer access
|
148 | 202 |
----------------
|
149 | 203 |
|
150 | 204 |
We'll hand out commit access to anyone who has successfully landed a single
|
151 |
-patch to the code base. Please request this via irc or the mailing list.
|
|
205 |
+patch to the code base. Please request this via Slack or the mailing list.
|
|
152 | 206 |
|
153 | 207 |
This of course relies on contributors being responsive and show willingness to
|
154 | 208 |
address problems after landing branches there should not be any problems here.
|
1 |
-.. _about:
|
|
2 |
- |
|
3 |
- |
|
4 | 1 |
|
5 | 2 |
.. image:: https://gitlab.com/Buildgrid/buildgrid/badges/master/pipeline.svg
|
6 | 3 |
:target: https://gitlab.com/BuildStream/buildstream/commits/master
|
7 | 4 |
|
8 | 5 |
.. image:: https://gitlab.com/BuildGrid/buildgrid/badges/master/coverage.svg?job=coverage
|
9 | 6 |
:target: https://buildgrid.gitlab.io/buildgrid/coverage
|
10 |
-
|
|
7 |
+ |
|
8 |
+ |
|
9 |
+.. _about:
|
|
10 |
+ |
|
11 | 11 |
About BuildGrid
|
12 | 12 |
===============
|
13 | 13 |
|
14 |
+ |
|
15 |
+.. _what-is-it:
|
|
16 |
+ |
|
14 | 17 |
What is BuildGrid?
|
15 | 18 |
------------------
|
16 | 19 |
|
... | ... | @@ -49,7 +52,7 @@ Resources |
49 | 52 |
- `GitLab repository`_
|
50 | 53 |
- `Bug tracking`_
|
51 | 54 |
- `Mailing list`_
|
52 |
-- `Slack channel`_ [`invite link`_]
|
|
55 |
+- `Slack channel`_ [`invite link`_]
|
|
53 | 56 |
|
54 | 57 |
.. _Homepage: https://buildgrid.build
|
55 | 58 |
.. _GitLab repository: https://gitlab.com/BuildGrid/buildgrid
|
1 |
+server:
|
|
2 |
+ port: 50051
|
|
3 |
+ insecure-mode: true
|
|
4 |
+ |
|
5 |
+instances:
|
|
6 |
+ - name: main
|
|
7 |
+ |
|
8 |
+ storages:
|
|
9 |
+ - !lru-storage &main-storage
|
|
10 |
+ size: 512MB
|
|
11 |
+ |
|
12 |
+ services:
|
|
13 |
+ - !action-cache &main-action
|
|
14 |
+ storage: *main-storage
|
|
15 |
+ max_cached_refs: 256
|
|
16 |
+ allow_updates: true
|
|
17 |
+ - !execution
|
|
18 |
+ storage: *main-storage
|
|
19 |
+ action_cache: *main-action
|
|
20 |
+ - !cas
|
|
21 |
+ storage: *main-storage
|
|
22 |
+ - !bytestream
|
|
23 |
+ storage: *main-storage
|
|
\ No newline at end of file |
... | ... | @@ -9,5 +9,5 @@ This section covers how to run an use the BuildGrid build service. |
9 | 9 |
.. toctree::
|
10 | 10 |
:maxdepth: 2
|
11 | 11 |
|
12 |
- using_dummy_build.rst
|
|
13 |
- using_simple_build.rst
|
|
12 |
+ using_internal.rst
|
|
13 |
+ using_bazel.rst
|
1 |
+ |
|
2 |
+.. _bazel-builds:
|
|
3 |
+ |
|
4 |
+Bazel builds
|
|
5 |
+============
|
|
6 |
+ |
|
7 |
+`Bazel`_ is a *“fast, scalable, multi-language and extensible build system”*
|
|
8 |
+that supports remote build execution using the remote execution API (REAPI) v2
|
|
9 |
+since its `0.17 release`_.
|
|
10 |
+ |
|
11 |
+.. _Bazel: https://bazel.build
|
|
12 |
+.. _0.17 release: https://blog.bazel.build/2018/09/14/bazel-0.17.html
|
|
13 |
+ |
|
14 |
+ |
|
15 |
+.. _bazel-configuration:
|
|
16 |
+ |
|
17 |
+Configuration
|
|
18 |
+-------------
|
|
19 |
+ |
|
20 |
+Bazel accepts many options that can either be specified as command line
|
|
21 |
+arguments when involking the ``bazel`` tool or stored in a `.bazelrc`_
|
|
22 |
+configuration file. In order to activate remote build execution, the
|
|
23 |
+``bazel build`` subcommand needs to be given specific `build options`_,
|
|
24 |
+including:
|
|
25 |
+ |
|
26 |
+- ``--remote_executor``: remote execution endpoint's location, ``{host}:{port}``.
|
|
27 |
+- ``--remote_instance_name``: remote execution instance's name.
|
|
28 |
+- ``--spawn_strategy``: action execution method.
|
|
29 |
+- ``--genrule_strategy``: `genrules`_ execution method.
|
|
30 |
+ |
|
31 |
+Spawn and genrule strategies need to be set to ``remote`` for remote execution.
|
|
32 |
+ |
|
33 |
+As an example, in order to activate remote execution on the ``main`` instance of
|
|
34 |
+the remote execution server available at ``controller.grid.build`` on port
|
|
35 |
+``50051`` you should amend your ``.bazelrc`` with:
|
|
36 |
+ |
|
37 |
+.. code-block:: sh
|
|
38 |
+ |
|
39 |
+ build --spawn_strategy=remote --genrule_strategy=remote --remote_executor=controller.grid.build:50051 --remote_instance_name=main
|
|
40 |
+ |
|
41 |
+.. _.bazelrc: https://docs.bazel.build/versions/master/user-manual.html#bazelrc
|
|
42 |
+.. _build options: https://docs.bazel.build/versions/master/command-line-reference.html#build-options
|
|
43 |
+.. _genrules: https://docs.bazel.build/versions/master/be/general.html#genrule
|
|
44 |
+ |
|
45 |
+ |
|
46 |
+.. _bazel-example:
|
|
47 |
+ |
|
48 |
+Example build
|
|
49 |
+-------------
|
|
50 |
+ |
|
51 |
+The `bazel-examples`_ repository contains example Bazel workspaces that can be
|
|
52 |
+used for testing purpose. We'll focus here on instructions on how to build the
|
|
53 |
+`stage3 CPP example`_ running Bazel and BuildGrid on your local machine,
|
|
54 |
+compiling the C++ source code using host-tools.
|
|
55 |
+ |
|
56 |
+First, you need to checkout the bazel-examples repository sources:
|
|
57 |
+ |
|
58 |
+.. code-block:: sh
|
|
59 |
+ |
|
60 |
+ git clone https://github.com/bazelbuild/examples.git bazel-examples
|
|
61 |
+ |
|
62 |
+Next, change the current directory to the Bazel workspace root:
|
|
63 |
+ |
|
64 |
+.. code-block:: sh
|
|
65 |
+ |
|
66 |
+ cd bazel-examples/cpp-tutorial/stage3
|
|
67 |
+ |
|
68 |
+.. note::
|
|
69 |
+ |
|
70 |
+ All the commands in the instructions below are expected to be executed from
|
|
71 |
+ that root directory (the stage3 example workspace's root directory).
|
|
72 |
+ |
|
73 |
+Before running Bazel and building the example workspace, we'll have to setup and
|
|
74 |
+run a BuildGrid server and bot. A minimal server's configuration is given below,
|
|
75 |
+paste it in a ``server.conf`` file in the root directory:
|
|
76 |
+ |
|
77 |
+.. literalinclude:: ./data/bazel-example-server.conf
|
|
78 |
+ :language: yaml
|
|
79 |
+ |
|
80 |
+This defines a single ``main`` server instance implementing a
|
|
81 |
+``ContentAddressableStorage`` (CAS) + ``ByteStream`` service together with an
|
|
82 |
+``Execution`` + ``ActionCache`` service, both using the same in-memory storage.
|
|
83 |
+You can then start the BuildGrid server deamon using that configuration by
|
|
84 |
+running:
|
|
85 |
+ |
|
86 |
+.. code-block:: sh
|
|
87 |
+ |
|
88 |
+ bgd server start server.conf
|
|
89 |
+ |
|
90 |
+In order to perform the actual build work, we need to attached a worker bot to
|
|
91 |
+that server for that ``main`` instance. A simple host-tools based bot should be
|
|
92 |
+enought to build the stage3 CPP example. Once you've make sure that your machine
|
|
93 |
+has ``gcc`` installed, run:
|
|
94 |
+ |
|
95 |
+.. code-block:: sh
|
|
96 |
+ |
|
97 |
+ bgd bot --remote=http://localhost:50051 --parent=main temp-directory
|
|
98 |
+ |
|
99 |
+The ``--remote`` option is used to specify the server location (running on the
|
|
100 |
+same machine here, and listenning to port 50051). The ``--parent`` option is
|
|
101 |
+used to specifiy the server instance you except the bot to be attached to.
|
|
102 |
+ |
|
103 |
+BuildGrid server is now ready to accept jobs and execute them. Bazel needs some
|
|
104 |
+:ref:`configuration <bazel-configuration>` in order to run remote builds. Below
|
|
105 |
+is the minimal build options you should use, paste them in a ``.bazelrc`` file
|
|
106 |
+in the root directory:
|
|
107 |
+ |
|
108 |
+.. code-block:: sh
|
|
109 |
+ |
|
110 |
+ build --spawn_strategy=remote --genrule_strategy=remote --remote_executor=localhost:50051 --remote_instance_name=main
|
|
111 |
+ |
|
112 |
+This ativates Bazel's remote execution mode and points to the ``main`` remote
|
|
113 |
+execution server instance at ``localhost:50051``.
|
|
114 |
+ |
|
115 |
+You can finally have Bazel to build the example workspace by running:
|
|
116 |
+ |
|
117 |
+.. code-block:: sh
|
|
118 |
+ |
|
119 |
+ bazel build //main:hello-world
|
|
120 |
+ |
|
121 |
+You can verify that the example has been successfully build by runnig the
|
|
122 |
+generated executable. Simply invoke:
|
|
123 |
+ |
|
124 |
+.. code-block:: sh
|
|
125 |
+ |
|
126 |
+ ./bazel-bin/main/hello-world
|
|
127 |
+ |
|
128 |
+.. _bazel-examples: https://github.com/bazelbuild/examples
|
|
129 |
+.. _stage3 CPP example: https://github.com/bazelbuild/examples/tree/master/cpp-tutorial/stage3
|
1 |
-.. _dummy-build:
|
|
2 |
- |
|
3 |
-Dummy build
|
|
4 |
-===========
|
|
5 |
- |
|
6 |
-In one terminal, start a server:
|
|
7 |
- |
|
8 |
-.. code-block:: sh
|
|
9 |
- |
|
10 |
- bgd server start buildgrid/_app/settings/default.yml
|
|
11 |
- |
|
12 |
-In another terminal, send a request for work:
|
|
13 |
- |
|
14 |
-.. code-block:: sh
|
|
15 |
- |
|
16 |
- bgd execute request-dummy
|
|
17 |
- |
|
18 |
-The stage should show as ``QUEUED`` as it awaits a bot to pick up the work:
|
|
19 |
- |
|
20 |
-.. code-block:: sh
|
|
21 |
- |
|
22 |
- bgd execute list
|
|
23 |
- |
|
24 |
-Create a bot session:
|
|
25 |
- |
|
26 |
-.. code-block:: sh
|
|
27 |
- |
|
28 |
- bgd bot dummy
|
|
29 |
- |
|
30 |
-Show the work as completed:
|
|
31 |
- |
|
32 |
-.. code-block:: sh
|
|
33 |
- |
|
34 |
- bgd execute list
|
1 |
+ |
|
2 |
+.. _internal-builds:
|
|
3 |
+ |
|
4 |
+Internal builds
|
|
5 |
+===============
|
|
6 |
+ |
|
7 |
+ |
|
8 |
+.. _dummy-test:
|
|
9 |
+ |
|
10 |
+Dummy test
|
|
11 |
+----------
|
|
12 |
+ |
|
13 |
+In one terminal, start a server:
|
|
14 |
+ |
|
15 |
+.. code-block:: sh
|
|
16 |
+ |
|
17 |
+ bgd server start buildgrid/_app/settings/default.yml
|
|
18 |
+ |
|
19 |
+In another terminal, send a request for work:
|
|
20 |
+ |
|
21 |
+.. code-block:: sh
|
|
22 |
+ |
|
23 |
+ bgd execute request-dummy
|
|
24 |
+ |
|
25 |
+The stage should show as ``QUEUED`` as it awaits a bot to pick up the work:
|
|
26 |
+ |
|
27 |
+.. code-block:: sh
|
|
28 |
+ |
|
29 |
+ bgd execute list
|
|
30 |
+ |
|
31 |
+Create a bot session:
|
|
32 |
+ |
|
33 |
+.. code-block:: sh
|
|
34 |
+ |
|
35 |
+ bgd bot dummy
|
|
36 |
+ |
|
37 |
+Show the work as completed:
|
|
38 |
+ |
|
39 |
+.. code-block:: sh
|
|
40 |
+ |
|
41 |
+ bgd execute list
|
|
42 |
+ |
|
43 |
+ |
|
1 | 44 |
.. _simple-build:
|
2 | 45 |
|
3 | 46 |
Simple build
|
4 |
-============
|
|
47 |
+------------
|
|
5 | 48 |
|
6 | 49 |
This example covers a simple build. The user will upload a directory containing
|
7 | 50 |
a C file and a command to the CAS. The bot will then fetch the uploaded
|