GNOME.org
 

[Notes] [Git][BuildGrid/buildgrid][mablanch/117-job-scheduler-refactoring] 11 commits: docs: Add a resources page

Title: GitLab

Martin Blanchard pushed to branch mablanch/117-job-scheduler-refactoring at BuildGrid / buildgrid

Commits:

12 changed files:

Changes:

  • CONTRIBUTING.rst
    1 
    -
    2 1 
     .. _contributing:
    3 2
    4 3 
     Contributing
    ... ... @@ -204,17 +203,16 @@ Committer access
    204 203 
     We'll hand out commit access to anyone who has successfully landed a single
    205 204 
     patch to the code base. Please request this via Slack or the mailing list.
    206 205
    207 
    -This of course relies on contributors being responsive and show willingness to
    208 
    -address problems after landing branches there should not be any problems here.
    206 
    +This of course relies on contributors being responsive and showing willingness
    207 
    +to address any problems that may arise after landing branches.
    209 208
    210 
    -What we are expecting of committers here in general is basically to escalate the
    211 
    -review in cases of uncertainty:
    209 
    +When submitting a merge request, please obtain a review from another committer
    210 
    +who is familiar with the area of the code base which the branch effects. An
    211 
    +approval from another committer who is not the patch author will be needed
    212 
    +before any merge (we use gitlab's 'approval' feature for this).
    212 213
    213 
    -- If the change is very trivial (obvious few line changes, typos…), and you are
    214 
    -  confident of the change, there is no need for review.
    215 
    -- If the change is non trivial, please obtain a review from another committer
    216 
    -  who is familiar with the area which the branch effects. An approval from
    217 
    -  someone who is not the patch author will be needed before any merge.
    214 
    +What we are expecting of committers here in general is basically to escalate the
    215 
    +review in cases of uncertainty.
    218 216
    219 217 
     .. note::
    220 218

  • Dockerfile
    1 
    +FROM python:3.5-stretch
    2 
    +
    3 
    +# Point the path to where buildgrid gets installed
    4 
    +ENV PATH=$PATH:/root/.local/bin/
    5 
    +
    6 
    +# Upgrade python modules
    7 
    +RUN python3 -m pip install --upgrade setuptools pip
    8 
    +
    9 
    +# Use /app as the current working directory
    10 
    +WORKDIR /app
    11 
    +
    12 
    +# Copy the repo contents (source, config files, etc) in the WORKDIR
    13 
    +COPY . .
    14 
    +
    15 
    +# Install BuildGrid
    16 
    +RUN pip install --user --editable .
    17 
    +
    18 
    +# Entry Point of the image (should get an additional argument from CMD, the path to the config file)
    19 
    +ENTRYPOINT ["bgd", "-v", "server", "start"]
    20 
    +
    21 
    +# Default config file (used if no CMD specified when running)
    22 
    +CMD ["buildgrid/_app/settings/default.yml"]
    23 
    +

  • README.rst
    1 
    -
    2 1 
     .. _about:
    3 2
    4 3 
     About
    ... ... @@ -14,13 +13,15 @@ BuildGrid is a Python remote execution service which implements Google's
    14 13 
     `Remote Execution API`_ and the `Remote Workers API`_. The project's goal is to
    15 14 
     be able to execute build jobs remotely on a grid of computers in order to
    16 15 
     massively speed up build times. Workers on the grid should be able to run with
    17 
    -different environments. It is designed to work with clients such as `Bazel`_ and
    18 
    -`BuildStream`_.
    16 
    +different environments. It works with clients such as `Bazel`_,
    17 
    +`BuildStream`_ and `RECC`_, and is designed to be able to work with any client
    18 
    +that conforms to the above API protocols.
    19 19
    20 20 
     .. _Remote Execution API: https://github.com/bazelbuild/remote-apis
    21 21 
     .. _Remote Workers API: https://docs.google.com/document/d/1s_AzRRD2mdyktKUj2HWBn99rMg_3tcPvdjx3MPbFidU/edit#heading=h.1u2taqr2h940
    22 22 
     .. _BuildStream: https://wiki.gnome.org/Projects/BuildStream
    23 23 
     .. _Bazel: https://bazel.build
    24 
    +.. _RECC: https://gitlab.com/bloomberg/recc
    24 25
    25 26
    26 27 
     .. _getting-started:
    ... ... @@ -49,7 +50,7 @@ Resources
    49 50
    50 51 
     .. _Homepage: https://buildgrid.build
    51 52 
     .. _GitLab repository: https://gitlab.com/BuildGrid/buildgrid
    52 
    -.. _Bug tracking: https://gitlab.com/BuildGrid/buildgrid/issues
    53 
    +.. _Bug tracking: https://gitlab.com/BuildGrid/buildgrid/boards
    53 54 
     .. _Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
    54 55 
     .. _Slack channel: https://buildteamworld.slack.com/messages/CC9MKC203
    55 56 
     .. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU

  • buildgrid/_app/bots/buildbox.py
    ... ... @@ -17,8 +17,6 @@ import os
    17 17 
     import subprocess
    18 18 
     import tempfile
    19 19
    20 
    -from google.protobuf import any_pb2
    21 
    -
    22 20 
     from buildgrid.client.cas import download, upload
    23 21 
     from buildgrid._exceptions import BotError
    24 22 
     from buildgrid._protos.build.bazel.remote.execution.v2 import remote_execution_pb2
    ... ... @@ -29,13 +27,14 @@ from buildgrid.utils import read_file, write_file
    29 27 
     def work_buildbox(context, lease):
    30 28 
         """Executes a lease for a build action, using buildbox.
    31 29 
         """
    32 
    -
    33 30 
         local_cas_directory = context.local_cas
    34 31 
         # instance_name = context.parent
    35 32 
         logger = context.logger
    36 33
    37 34 
         action_digest = remote_execution_pb2.Digest()
    35 
    +
    38 36 
         lease.payload.Unpack(action_digest)
    37 
    +    lease.result.Clear()
    39 38
    40 39 
         with download(context.cas_channel) as downloader:
    41 40 
             action = downloader.get_message(action_digest,
    ... ... @@ -131,10 +130,7 @@ def work_buildbox(context, lease):
    131 130
    132 131 
                 action_result.output_directories.extend([output_directory])
    133 132
    134 
    -            action_result_any = any_pb2.Any()
    135 
    -            action_result_any.Pack(action_result)
    136 
    -
    137 
    -            lease.result.CopyFrom(action_result_any)
    133 
    +            lease.result.Pack(action_result)
    138 134
    139 135 
         return lease
    140 136

  • buildgrid/_app/bots/host.py
    ... ... @@ -17,8 +17,6 @@ import os
    17 17 
     import subprocess
    18 18 
     import tempfile
    19 19
    20 
    -from google.protobuf import any_pb2
    21 
    -
    22 20 
     from buildgrid.client.cas import download, upload
    23 21 
     from buildgrid._protos.build.bazel.remote.execution.v2 import remote_execution_pb2
    24 22 
     from buildgrid.utils import output_file_maker, output_directory_maker
    ... ... @@ -27,12 +25,13 @@ from buildgrid.utils import output_file_maker, output_directory_maker
    27 25 
     def work_host_tools(context, lease):
    28 26 
         """Executes a lease for a build action, using host tools.
    29 27 
         """
    30 
    -
    31 28 
         instance_name = context.parent
    32 29 
         logger = context.logger
    33 30
    34 31 
         action_digest = remote_execution_pb2.Digest()
    32 
    +
    35 33 
         lease.payload.Unpack(action_digest)
    34 
    +    lease.result.Clear()
    36 35
    37 36 
         with tempfile.TemporaryDirectory() as temp_directory:
    38 37 
             with download(context.cas_channel, instance=instance_name) as downloader:
    ... ... @@ -122,9 +121,6 @@ def work_host_tools(context, lease):
    122 121
    123 122 
                 action_result.output_directories.extend(output_directories)
    124 123
    125 
    -        action_result_any = any_pb2.Any()
    126 
    -        action_result_any.Pack(action_result)
    127 
    -
    128 
    -        lease.result.CopyFrom(action_result_any)
    124 
    +        lease.result.Pack(action_result)
    129 125
    130 126 
         return lease

  • buildgrid/server/execution/instance.py
    ... ... @@ -53,7 +53,7 @@ class ExecutionInstance:
    53 53
    54 54 
             self._scheduler.append_job(job, skip_cache_lookup)
    55 55
    56 
    -        return job.get_operation()
    56 
    +        return job._operation
    57 57
    58 58 
         def register_message_client(self, name, queue):
    59 59 
             try:

  • buildgrid/server/job.py
    ... ... @@ -19,57 +19,33 @@ import logging
    19 19 
     import uuid
    20 20 
     from enum import Enum
    21 21
    22 
    -from google.protobuf import any_pb2
    23 
    -
    24 22 
     from buildgrid._protos.build.bazel.remote.execution.v2 import remote_execution_pb2
    25 23 
     from buildgrid._protos.google.devtools.remoteworkers.v1test2 import bots_pb2
    26 24 
     from buildgrid._protos.google.longrunning import operations_pb2
    27 25
    28 26
    29 
    -class ExecuteStage(Enum):
    27 
    +class OperationStage(Enum):
    28 
    +    # Initially unknown stage.
    30 29 
         UNKNOWN = remote_execution_pb2.ExecuteOperationMetadata.Stage.Value('UNKNOWN')
    31 
    -
    32 30 
         # Checking the result against the cache.
    33 31 
         CACHE_CHECK = remote_execution_pb2.ExecuteOperationMetadata.Stage.Value('CACHE_CHECK')
    34 
    -
    35 32 
         # Currently idle, awaiting a free machine to execute.
    36 33 
         QUEUED = remote_execution_pb2.ExecuteOperationMetadata.Stage.Value('QUEUED')
    37 
    -
    38 34 
         # Currently being executed by a worker.
    39 35 
         EXECUTING = remote_execution_pb2.ExecuteOperationMetadata.Stage.Value('EXECUTING')
    40 
    -
    41 36 
         # Finished execution.
    42 37 
         COMPLETED = remote_execution_pb2.ExecuteOperationMetadata.Stage.Value('COMPLETED')
    43 38
    44 39
    45 
    -class BotStatus(Enum):
    46 
    -    BOT_STATUS_UNSPECIFIED = bots_pb2.BotStatus.Value('BOT_STATUS_UNSPECIFIED')
    47 
    -
    48 
    -    # The bot is healthy, and will accept leases as normal.
    49 
    -    OK = bots_pb2.BotStatus.Value('OK')
    50 
    -
    51 
    -    # The bot is unhealthy and will not accept new leases.
    52 
    -    UNHEALTHY = bots_pb2.BotStatus.Value('UNHEALTHY')
    53 
    -
    54 
    -    # The bot has been asked to reboot the host.
    55 
    -    HOST_REBOOTING = bots_pb2.BotStatus.Value('HOST_REBOOTING')
    56 
    -
    57 
    -    # The bot has been asked to shut down.
    58 
    -    BOT_TERMINATING = bots_pb2.BotStatus.Value('BOT_TERMINATING')
    59 
    -
    60 
    -
    61 40 
     class LeaseState(Enum):
    41 
    +    # Initially unknown state.
    62 42 
         LEASE_STATE_UNSPECIFIED = bots_pb2.LeaseState.Value('LEASE_STATE_UNSPECIFIED')
    63 
    -
    64 43 
         # The server expects the bot to accept this lease.
    65 44 
         PENDING = bots_pb2.LeaseState.Value('PENDING')
    66 
    -
    67 45 
         # The bot has accepted this lease.
    68 46 
         ACTIVE = bots_pb2.LeaseState.Value('ACTIVE')
    69 
    -
    70 47 
         # The bot is no longer leased.
    71 48 
         COMPLETED = bots_pb2.LeaseState.Value('COMPLETED')
    72 
    -
    73 49 
         # The bot should immediately release all resources associated with the lease.
    74 50 
         CANCELLED = bots_pb2.LeaseState.Value('CANCELLED')
    75 51
    ... ... @@ -77,18 +53,24 @@ class LeaseState(Enum):
    77 53 
     class Job:
    78 54
    79 55 
         def __init__(self, action_digest, do_not_cache=False, message_queue=None):
    80 
    -        self.lease = None
    81 56 
             self.logger = logging.getLogger(__name__)
    82 
    -        self.n_tries = 0
    83 57 
             self.result = None
    84 58 
             self.result_cached = False
    85 59
    60 
    +        self._name = str(uuid.uuid4())
    86 61 
             self._action_digest = action_digest
    87 62 
             self._do_not_cache = do_not_cache
    88 
    -        self._execute_stage = ExecuteStage.UNKNOWN
    89 
    -        self._name = str(uuid.uuid4())
    90 
    -        self._operation = operations_pb2.Operation(name=self._name)
    63 
    +        self._operation = operations_pb2.Operation()
    64 
    +        self._lease = None
    65 
    +        self._n_tries = 0
    66 
    +
    67 
    +        self.__operation_metadata = remote_execution_pb2.ExecuteOperationMetadata()
    68 
    +
    69 
    +        self.__operation_metadata.action_digest.CopyFrom(self._action_digest)
    70 
    +        self.__operation_metadata.stage = OperationStage.UNKNOWN.value
    71 
    +
    91 72 
             self._operation_update_queues = []
    73 
    +        self._operation.name = self._name
    92 74
    93 75 
             if message_queue is not None:
    94 76 
                 self.register_client(message_queue)
    ... ... @@ -105,6 +87,26 @@ class Job:
    105 87 
         def do_not_cache(self):
    106 88 
             return self._do_not_cache
    107 89
    90 
    +    @property
    91 
    +    def operation(self):
    92 
    +        return self._operation
    93 
    +
    94 
    +    @property
    95 
    +    def operation_stage(self):
    96 
    +        return OperationStage(self.__operation_metadata.state)
    97 
    +
    98 
    +    @property
    99 
    +    def lease(self):
    100 
    +        return self._lease
    101 
    +
    102 
    +    @property
    103 
    +    def lease_state(self):
    104 
    +        return LeaseState(self._lease.state)
    105 
    +
    106 
    +    @property
    107 
    +    def n_tries(self):
    108 
    +        return self._n_tries
    109 
    +
    108 110 
         def check_job_finished(self):
    109 111 
             if not self._operation_update_queues:
    110 112 
                 return self._operation.done
    ... ... @@ -112,50 +114,70 @@ class Job:
    112 114
    113 115 
         def register_client(self, queue):
    114 116 
             self._operation_update_queues.append(queue)
    115 
    -        queue.put(self.get_operation())
    117 
    +        queue.put(self._operation)
    116 118
    117 119 
         def unregister_client(self, queue):
    118 120 
             self._operation_update_queues.remove(queue)
    119 121
    120 
    -    def get_operation(self):
    121 
    -        self._operation.metadata.CopyFrom(self._pack_any(self.get_operation_meta()))
    122 
    -        if self.result is not None:
    123 
    -            self._operation.done = True
    124 
    -            response = remote_execution_pb2.ExecuteResponse(result=self.result,
    125 
    -                                                            cached_result=self.result_cached)
    122 
    +    # def get_operation(self):
    123 
    +    #     if self.result is not None:
    124 
    +    #         self._operation.done = True
    125 
    +    #         response = remote_execution_pb2.ExecuteResponse(result=self.result,
    126 
    +    #                                                         cached_result=self.result_cached)
    126 127
    127 
    -            if not self.result_cached:
    128 
    -                response.status.CopyFrom(self.lease.status)
    128 
    +    #         if not self.result_cached:
    129 
    +    #             response.status.CopyFrom(self.lease.status)
    129 130
    130 
    -            self._operation.response.CopyFrom(self._pack_any(response))
    131 
    +    #         self._operation.response.Pack(response)
    131 132
    132 
    -        return self._operation
    133 
    +    #     return self._operation
    133 134
    134 
    -    def get_operation_meta(self):
    135 
    -        meta = remote_execution_pb2.ExecuteOperationMetadata()
    136 
    -        meta.stage = self._execute_stage.value
    137 
    -        meta.action_digest.CopyFrom(self._action_digest)
    135 
    +    # def get_operation_meta(self):
    136 
    +    #     meta = remote_execution_pb2.ExecuteOperationMetadata()
    137 
    +    #     meta.stage = self._execute_stage.value
    138 
    +    #     meta.action_digest.CopyFrom(self._action_digest)
    138 139
    139 
    -        return meta
    140 
    +    #     return meta
    140 141
    141 142 
         def create_lease(self):
    142 
    -        action_digest = self._pack_any(self._action_digest)
    143 
    +        self._lease = bots_pb2.Lease()
    144 
    +        self._lease.id = self._name
    145 
    +        self._lease.payload.Pack(self._action_digest)
    146 
    +        self._lease.state = LeaseState.PENDING.value
    147 
    +
    148 
    +        return self._lease
    149 
    +
    150 
    +    def update_lease_state(self, state, status=None, action_result=None):
    151 
    +        if state.value == self._lease.state:
    152 
    +            return
    153 
    +
    154 
    +        self._lease.state = state.value
    155 
    +
    156 
    +        if self._lease.state == LeaseState.COMPLETED:
    157 
    +            response = remote_execution_pb2.ExecuteResponse()
    158 
    +            response.result.CopyFrom(action_result)
    159 
    +            response.cached_result = False
    160 
    +            response.status.CopyFrom(status)
    161 
    +
    162 
    +            self._operation.response.Pack(response)
    163 
    +            self._operation.done = True
    164 
    +
    165 
    +    def cancel_lease(self):
    166 
    +        pass
    167 
    +
    168 
    +    def update_operation_stage(self, stage):
    169 
    +        if stage.value == self.__operation_metadata.stage:
    170 
    +            return
    171 
    +
    172 
    +        self.__operation_metadata.stage = stage.value
    143 173
    144 
    -        lease = bots_pb2.Lease(id=self.name,
    145 
    -                               payload=action_digest,
    146 
    -                               state=LeaseState.PENDING.value)
    147 
    -        self.lease = lease
    148 
    -        return lease
    174 
    +        if self.__operation_metadata.stage == OperationStage.QUEUED.value:
    175 
    +            self._n_tries += 1
    149 176
    150 
    -    def get_operations(self):
    151 
    -        return operations_pb2.ListOperationsResponse(operations=[self.get_operation()])
    177 
    +        self._operation.metadata.Pack(self.__operation_metadata)
    152 178
    153 
    -    def update_execute_stage(self, stage):
    154 
    -        self._execute_stage = stage
    155 179 
             for queue in self._operation_update_queues:
    156 
    -            queue.put(self.get_operation())
    180 
    +            queue.put(self._operation)
    157 181
    158 
    -    def _pack_any(self, pack):
    159 
    -        some_any = any_pb2.Any()
    160 
    -        some_any.Pack(pack)
    161 
    -        return some_any
    182 
    +    def cancel_operation(self):
    183 
    +        pass

  • buildgrid/server/scheduler.py
    ... ... @@ -27,7 +27,7 @@ from buildgrid._exceptions import NotFoundError
    27 27 
     from buildgrid._protos.build.bazel.remote.execution.v2 import remote_execution_pb2
    28 28 
     from buildgrid._protos.google.longrunning import operations_pb2
    29 29
    30 
    -from .job import ExecuteStage, LeaseState
    30 
    +from .job import OperationStage, LeaseState
    31 31
    32 32
    33 33 
     class Scheduler:
    ... ... @@ -55,27 +55,26 @@ class Scheduler:
    55 55 
                     cached_result = self._action_cache.get_action_result(job.action_digest)
    56 56 
                 except NotFoundError:
    57 57 
                     self.queue.append(job)
    58 
    -                job.update_execute_stage(ExecuteStage.QUEUED)
    58 
    +                job.update_operation_stage(OperationStage.QUEUED)
    59 59
    60 60 
                 else:
    61 61 
                     job.result = cached_result
    62 62 
                     job.result_cached = True
    63 
    -                job.update_execute_stage(ExecuteStage.COMPLETED)
    63 
    +                job.update_operation_stage(OperationStage.COMPLETED)
    64 64
    65 65 
             else:
    66 66 
                 self.queue.append(job)
    67 
    -            job.update_execute_stage(ExecuteStage.QUEUED)
    67 
    +            job.update_operation_stage(OperationStage.QUEUED)
    68 68
    69 69 
         def retry_job(self, name):
    70 70 
             if name in self.jobs:
    71 71 
                 job = self.jobs[name]
    72 72 
                 if job.n_tries >= self.MAX_N_TRIES:
    73 73 
                     # TODO: Decide what to do with these jobs
    74 
    -                job.update_execute_stage(ExecuteStage.COMPLETED)
    74 
    +                job.update_operation_stage(OperationStage.COMPLETED)
    75 75 
                     # TODO: Mark these jobs as done
    76 76 
                 else:
    77 
    -                job.update_execute_stage(ExecuteStage.QUEUED)
    78 
    -                job.n_tries += 1
    77 
    +                job.update_operation_stage(OperationStage.QUEUED)
    79 78 
                     self.queue.appendleft(job)
    80 79
    81 80 
         def job_complete(self, name, result, status):
    ... ... @@ -83,16 +82,17 @@ class Scheduler:
    83 82 
             job.lease.status.CopyFrom(status)
    84 83 
             action_result = remote_execution_pb2.ActionResult()
    85 84 
             result.Unpack(action_result)
    86 
    -        job.result = action_result
    85 
    +        # job.result = action_result
    86 
    +        job.update_lease_state(LeaseState.COMPLETED, action_result=action_result)
    87 87 
             if not job.do_not_cache and self._action_cache is not None:
    88 88 
                 if not job.lease.status.code:
    89 89 
                     self._action_cache.update_action_result(job.action_digest, action_result)
    90 
    -        job.update_execute_stage(ExecuteStage.COMPLETED)
    90 
    +        job.update_operation_stage(OperationStage.COMPLETED)
    91 91
    92 92 
         def get_operations(self):
    93 93 
             response = operations_pb2.ListOperationsResponse()
    94 94 
             for v in self.jobs.values():
    95 
    -            response.operations.extend([v.get_operation()])
    95 
    +            response.operations.extend([v.operation])
    96 96 
             return response
    97 97
    98 98 
         def update_job_lease_state(self, name, state):
    ... ... @@ -111,8 +111,7 @@ class Scheduler:
    111 111 
         def create_lease(self):
    112 112 
             if self.queue:
    113 113 
                 job = self.queue.popleft()
    114 
    -            job.update_execute_stage(ExecuteStage.EXECUTING)
    114 
    +            job.update_operation_stage(OperationStage.EXECUTING)
    115 115 
                 job.create_lease()
    116 
    -            job.lease.state = LeaseState.PENDING.value
    117 116 
                 return job.lease
    118 117 
             return None

  • docs/source/index.rst
    ... ... @@ -19,6 +19,7 @@ Remote execution service implementing Google's REAPI and RWAPI.
    19 19 
        using.rst
    20 20 
        reference.rst
    21 21 
        contributing.rst
    22 
    +   resources.rst
    22 23
    23 24
    24 25 
     Resources
    ... ... @@ -35,4 +36,4 @@ Resources
    35 36 
     .. _Bug tracking: https://gitlab.com/BuildGrid/buildgrid/issues
    36 37 
     .. _Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
    37 38 
     .. _Slack channel: https://buildteamworld.slack.com/messages/CC9MKC203
    38 
    -.. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU
    39 
    +.. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU
    \ No newline at end of file

  • docs/source/installation.rst
    1 
    -
    2 1 
     .. _installation:
    3 2
    4 3 
     Installation
    5 4 
     ============
    6 5
    7 
    -How to install BuildGrid onto your machine.
    6 
    +.. _install-on-host:
    7 
    +
    8 
    +Installation onto host machine
    9 
    +------------------------------
    10 
    +
    11 
    +How to install BuildGrid directly onto your machine.
    8 12
    9 13 
     .. note::
    10 14
    11 
    -   BuildGrid server currently only support *Linux*, *macOS* and *Windows*
    15 
    +   BuildGrid server currently only support *Linux*. *macOS* and *Windows*
    12 16 
        platforms are **not** supported.
    13 17
    14 18
    15 
    -.. _install-prerequisites:
    19 
    +.. _install-host-prerequisites:
    16 20
    17 21 
     Prerequisites
    18 
    --------------
    22 
    +~~~~~~~~~~~~~
    19 23
    20 24 
     BuildGrid only supports ``python3 >= 3.5`` but has no system requirements. Main
    21 
    -Python dependencies, automatically handle during installation, includes:
    25 
    +Python dependencies, automatically handled during installation, include:
    22 26
    23 27 
     - `boto3`_: the Amazon Web Services (AWS) SDK for Python.
    24 28 
     - `click`_: a Python composable command line library.
    ... ... @@ -33,10 +37,10 @@ Python dependencies, automatically handle during installation, includes:
    33 37 
     .. _protocol-buffers: https://developers.google.com/protocol-buffers
    34 38
    35 39
    36 
    -.. _source-install:
    40 
    +.. _install-host-source-install:
    37 41
    38 42 
     Install from sources
    39 
    ---------------------
    43 
    +~~~~~~~~~~~~~~~~~~~~
    40 44
    41 45 
     BuildGrid has ``setuptools`` support. In order to install it to your home
    42 46 
     directory, typically under ``~/.local``, simply run:
    ... ... @@ -46,7 +50,7 @@ directory, typically under ``~/.local``, simply run:
    46 50 
        git clone https://gitlab.com/BuildGrid/buildgrid.git && cd buildgrid
    47 51 
        pip3 install --user --editable .
    48 52
    49 
    -Additionally, and if your distribution does not already includes it, you may
    53 
    +Additionally, and if your distribution does not already include it, you may
    50 54 
     have to adjust your ``PATH``, in ``~/.bashrc``, with:
    51 55
    52 56 
     .. code-block:: sh
    ... ... @@ -63,3 +67,62 @@ have to adjust your ``PATH``, in ``~/.bashrc``, with:
    63 67 
        .. code-block:: sh
    64 68
    65 69 
           pip3 install --user --editable ".[docs,tests]"
    70 
    +
    71 
    +
    72 
    +
    73 
    +.. install-docker:
    74 
    +
    75 
    +Installation through docker
    76 
    +---------------------------
    77 
    +
    78 
    +How to build a Docker image that runs BuildGrid.
    79 
    +
    80 
    +.. _install-docker-prerequisites:
    81 
    +
    82 
    +Prerequisites
    83 
    +~~~~~~~~~~~~~
    84 
    +
    85 
    +A working Docker installation. Please consult `Docker's Getting Started Guide`_ if you don't already have it installed.
    86 
    +
    87 
    +.. _`Docker's Getting Started Guide`: https://www.docker.com/get-started
    88 
    +
    89 
    +
    90 
    +.. _install-docker-build:
    91 
    +
    92 
    +Docker Container from Sources
    93 
    +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    94 
    +
    95 
    +To clone the source code and build a Docker image, simply run:
    96 
    +
    97 
    +.. code-block:: sh
    98 
    +
    99 
    +   git clone https://gitlab.com/BuildGrid/buildgrid.git && cd buildgrid
    100 
    +   docker build -t buildgrid_server .
    101 
    +
    102 
    +.. note::
    103 
    +
    104 
    +   The image built will contain the contents of the source code directory, including
    105 
    +   configuration files.
    106 
    +
    107 
    +.. hint::
    108 
    +
    109 
    +    Whenever the source code is updated or new configuration files are made, you need to re-build
    110 
    +    the image.
    111 
    +
    112 
    +After building the Docker image, to run BuildGrid using the default configuration file
    113 
    +(found in `buildgrid/_app/settings/default.yml`), simply run:
    114 
    +
    115 
    +.. code-block:: sh
    116 
    +
    117 
    +   docker run -i -p 50051:50051 buildgrid_server
    118 
    +
    119 
    +.. note::
    120 
    +
    121 
    +    To run BuildGrid using a different configuration file, include the relative path to the
    122 
    +    configuration file at the end of the command above. For example, to run the default
    123 
    +    standalone CAS server (without an execution service), simply run:
    124 
    +
    125 
    +       .. code-block:: sh
    126 
    +
    127 
    +            docker run -i -p 50052:50052 buildgrid_server buildgrid/_app/settings/cas.yml
    128 
    +

  • docs/source/resources.rst
    1 
    +.. _external-resources:
    2 
    +
    3 
    +Resources
    4 
    +=========
    5 
    +
    6 
    +Remote execution and worker API useful links:
    7 
    +
    8 
    +- `REAPI design document`_
    9 
    +- `REAPI protobuf specification`_
    10 
    +- `RWAPI design document`_
    11 
    +- `RWAPI protobuf specification`_
    12 
    +- `Bazel`_ `remote caching and execution documentation`_
    13 
    +- `RECC usage instructions`_
    14 
    +- `BuildStream webside`_ and `documentation`_
    15 
    +- `BuildStream-externals repository`_
    16 
    +- `BuildBox repository`_
    17 
    +- `FUSE on wikipedia`_ and `kernel documentation`_
    18 
    +- `bubblewrap repository`_
    19 
    +- `Buildfarm reference REAPI implementation`_
    20 
    +- `Buildbarn Golang REAPI implementation`_
    21 
    +- `Demonstration of RECC with BuildGrid`_
    22 
    +- `Demonstration of Bazel with BuildGrid`_
    23 
    +- `Demonstration of BuildStream with BuildGrid`_
    24 
    +
    25 
    +.. _REAPI design document: https://docs.google.com/document/d/1AaGk7fOPByEvpAbqeXIyE8HX_A3_axxNnvroblTZ_6s
    26 
    +.. _REAPI protobuf specification: https://github.com/bazelbuild/remote-apis/blob/master/build/bazel/remote/execution/v2/remote_execution.proto
    27 
    +.. _RWAPI design document: https://docs.google.com/document/d/1s_AzRRD2mdyktKUj2HWBn99rMg_3tcPvdjx3MPbFidU
    28 
    +.. _RWAPI protobuf specification: https://github.com/googleapis/googleapis/blob/master/google/devtools/remoteworkers/v1test2/bots.proto
    29 
    +.. _Bazel: https://www.bazel.build
    30 
    +.. _remote caching and execution documentation: https://docs.bazel.build/versions/master/remote-caching.html
    31 
    +.. _RECC usage instructions: https://gitlab.com/bloomberg/recc#running-recc
    32 
    +.. _BuildStream webside: https://buildstream.build
    33 
    +.. _documentation: https://docs.buildstream.build
    34 
    +.. _BuildStream-externals repository: https://gitlab.com/BuildStream/bst-external
    35 
    +.. _FUSE on wikipedia: https://en.wikipedia.org/wiki/Filesystem_in_Userspace
    36 
    +.. _kernel documentation: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/fuse.txt
    37 
    +.. _BuildBox repository: https://gitlab.com/BuildStream/buildbox
    38 
    +.. _bubblewrap repository: https://github.com/projectatomic/bubblewrap
    39 
    +.. _Buildfarm reference REAPI implementation: https://github.com/bazelbuild/bazel-buildfarm
    40 
    +.. _Buildbarn Golang REAPI implementation: https://github.com/EdSchouten/bazel-buildbarn
    41 
    +.. _Demonstration of RECC with BuildGrid: https://asciinema.org/a/0FjExIqrTGSlpSUIS8Ehf5gUg
    42 
    +.. _Demonstration of Bazel with BuildGrid: https://asciinema.org/a/uVHFWOxpivwJ4ari23CEerR8N
    43 
    +.. _Demonstration of BuildStream with BuildGrid: https://asciinema.org/a/QfkYGqhfhEQz4o8prlBdEBFP7

  • tests/integration/execution_service.py
    ... ... @@ -82,7 +82,7 @@ def test_execute(skip_cache_lookup, instance, context):
    82 82 
         assert isinstance(result, operations_pb2.Operation)
    83 83 
         metadata = remote_execution_pb2.ExecuteOperationMetadata()
    84 84 
         result.metadata.Unpack(metadata)
    85 
    -    assert metadata.stage == job.ExecuteStage.QUEUED.value
    85 
    +    assert metadata.stage == job.OperationStage.QUEUED.value
    86 86 
         assert uuid.UUID(result.name, version=4)
    87 87 
         assert result.done is False
    88 88
    ... ... @@ -116,7 +116,7 @@ def test_wait_execution(instance, controller, context):
    116 116 
         action_result = remote_execution_pb2.ActionResult()
    117 117 
         action_result_any.Pack(action_result)
    118 118
    119 
    -    j.update_execute_stage(job.ExecuteStage.COMPLETED)
    119 
    +    j.update_operation_stage(job.OperationStage.COMPLETED)
    120 120
    121 121 
         response = instance.WaitExecution(request, context)
    122 122
    ... ... @@ -125,7 +125,7 @@ def test_wait_execution(instance, controller, context):
    125 125 
         assert isinstance(result, operations_pb2.Operation)
    126 126 
         metadata = remote_execution_pb2.ExecuteOperationMetadata()
    127 127 
         result.metadata.Unpack(metadata)
    128 
    -    assert metadata.stage == job.ExecuteStage.COMPLETED.value
    128 
    +    assert metadata.stage == job.OperationStage.COMPLETED.value
    129 129 
         assert uuid.UUID(result.name, version=4)
    130 130 
         assert result.done is True
    131 131

    • —
    View it on GitLab.
    You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.

    [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]