|
|
1
|
+.. _architecture-overview:
|
|
|
2
|
+
|
|
|
3
|
+Remote execution overview
|
|
|
4
|
+=========================
|
|
|
5
|
+
|
|
|
6
|
+Remote execution aims to speed up a build process and to rely on two separate
|
|
|
7
|
+but related concepts that are remote caching and remote execution itself. Remote
|
|
|
8
|
+caching allows users to share build outputs while remote execution allows the running
|
|
|
9
|
+of operations on a remote cluster of machines which may be more powerful than what
|
|
|
10
|
+the user has access to locally.
|
|
|
11
|
+
|
|
|
12
|
+The `Remote Execution API`_ (REAPI) describes a `gRPC`_ + `protocol-buffers`_
|
|
|
13
|
+interface that has three main services for remote caching and execution:
|
|
|
14
|
+
|
|
|
15
|
+- A ``ContentAddressableStorage`` (CAS) service: a remote storage end-point
|
|
|
16
|
+ where content is addressed by digests, a digest being a pair of the hash and
|
|
|
17
|
+ size of the data stored or retrieved.
|
|
|
18
|
+- An ``ActionCache`` (AC) service: a mapping between build actions already
|
|
|
19
|
+ performed and their corresponding resulting artifact.
|
|
|
20
|
+- An ``Execution`` service: the main end-point allowing one to request build
|
|
|
21
|
+ job to be perform against the build farm.
|
|
|
22
|
+
|
|
|
23
|
+The `Remote Worker API`_ (RWAPI) describes another `gRPC`_ + `protocol-buffers`_
|
|
|
24
|
+interface that allows a central ``BotsService`` to manage a farm of pluggable workers.
|
|
|
25
|
+
|
|
|
26
|
+BuildGrid is combining these two interfaces in order to provide a complete
|
|
|
27
|
+remote caching and execution service. The high level architecture can be
|
|
|
28
|
+represented like this:
|
|
|
29
|
+
|
|
|
30
|
+.. graphviz::
|
|
|
31
|
+ :align: center
|
|
|
32
|
+
|
|
|
33
|
+ digraph remote_execution_overview {
|
|
|
34
|
+ node [shape = record,
|
|
|
35
|
+ width=2,
|
|
|
36
|
+ height=1];
|
|
|
37
|
+
|
|
|
38
|
+ ranksep = 2
|
|
|
39
|
+ compound=true
|
|
|
40
|
+ edge[arrowtail="vee"];
|
|
|
41
|
+ edge[arrowhead="vee"];
|
|
|
42
|
+
|
|
|
43
|
+ client [label = "Client",
|
|
|
44
|
+ color="#0342af",
|
|
|
45
|
+ fillcolor="#37c1e8",
|
|
|
46
|
+ style=filled,
|
|
|
47
|
+ shape=box]
|
|
|
48
|
+
|
|
|
49
|
+ subgraph cluster_controller{
|
|
|
50
|
+ label = "Controller";
|
|
|
51
|
+ labeljust = "c";
|
|
|
52
|
+ fillcolor="#42edae";
|
|
|
53
|
+ style=filled;
|
|
|
54
|
+ controller [label = "{ExecutionService|BotsInterface\n}",
|
|
|
55
|
+ fillcolor="#17e86a",
|
|
|
56
|
+ style=filled];
|
|
|
57
|
+
|
|
|
58
|
+ }
|
|
|
59
|
+
|
|
|
60
|
+ subgraph cluster_worker0 {
|
|
|
61
|
+ label = "Worker 1";
|
|
|
62
|
+ labeljust = "c";
|
|
|
63
|
+ color="#8e7747";
|
|
|
64
|
+ fillcolor="#ffda8e";
|
|
|
65
|
+ style=filled;
|
|
|
66
|
+ bot0 [label = "{Bot|Host-tools}"
|
|
|
67
|
+ fillcolor="#ffb214",
|
|
|
68
|
+ style=filled];
|
|
|
69
|
+ }
|
|
|
70
|
+
|
|
|
71
|
+ subgraph cluster_worker1 {
|
|
|
72
|
+ label = "Worker 2";
|
|
|
73
|
+ labeljust = "c";
|
|
|
74
|
+ color="#8e7747";
|
|
|
75
|
+ fillcolor="#ffda8e";
|
|
|
76
|
+ style=filled;
|
|
|
77
|
+ bot1 [label = "{Bot|BuildBox}",
|
|
|
78
|
+ fillcolor="#ffb214",
|
|
|
79
|
+ style=filled];
|
|
|
80
|
+ }
|
|
|
81
|
+
|
|
|
82
|
+ client -> controller [
|
|
|
83
|
+ dir = "both",
|
|
|
84
|
+ headlabel = "REAPI",
|
|
|
85
|
+ labelangle = 20.0,
|
|
|
86
|
+ labeldistance = 9,
|
|
|
87
|
+ labelfontsize = 15.0,
|
|
|
88
|
+ lhead=cluster_controller];
|
|
|
89
|
+
|
|
|
90
|
+ controller -> bot0 [
|
|
|
91
|
+ dir = "both",
|
|
|
92
|
+ labelangle= 340.0,
|
|
|
93
|
+ labeldistance = 7.5,
|
|
|
94
|
+ labelfontsize = 15.0,
|
|
|
95
|
+ taillabel = "RWAPI ",
|
|
|
96
|
+ lhead=cluster_worker0,
|
|
|
97
|
+ ltail=cluster_controller];
|
|
|
98
|
+
|
|
|
99
|
+ controller -> bot1 [
|
|
|
100
|
+ dir = "both",
|
|
|
101
|
+ labelangle= 20.0,
|
|
|
102
|
+ labeldistance = 7.5,
|
|
|
103
|
+ labelfontsize = 15.0,
|
|
|
104
|
+ taillabel = " RWAPI",
|
|
|
105
|
+ lhead=cluster_worker1,
|
|
|
106
|
+ ltail=cluster_controller];
|
|
|
107
|
+
|
|
|
108
|
+ }
|
|
|
109
|
+
|
|
|
110
|
+.. _Remote Execution API: https://github.com/bazelbuild/remote-apis/blob/master/build/bazel/remote/execution/v2
|
|
|
111
|
+.. _gRPC: https://grpc.io
|
|
|
112
|
+.. _protocol-buffers: https://developers.google.com/protocol-buffers
|
|
|
113
|
+.. _Remote Worker API: https://github.com/googleapis/googleapis/tree/master/google/devtools/remoteworkers/v1test2
|
