GNOME.org
 

[Notes] [Git][BuildGrid/buildgrid][raoul/logging] 14 commits: docs: Add a resources page

Title: GitLab

Raoul Hidalgo Charman pushed to branch raoul/logging at BuildGrid / buildgrid

Commits:

10 changed files:

Changes:

  • CONTRIBUTING.rst
    1 
    -
    2 1 
     .. _contributing:
    3 2
    4 3 
     Contributing
    ... ... @@ -18,9 +17,13 @@ discuss with us before submitting anything, as we may well have some important
    18 17 
     context which will could help guide your efforts.
    19 18
    20 19 
     Any major feature additions should be raised first as a proposal on the
    21 
    -`BuildGrid mailing list`_ to be discussed, and then eventually followed up with
    22 
    -an issue on GitLab. We recommend that you propose the feature in advance of
    23 
    -commencing work.
    20 
    +BuildGrid mailing list to be discussed between the core contributors. Once
    21 
    +this discussion has taken place and there is agreement on how to proceed,
    22 
    +it should be followed by with a Gitlab issue being raised which summarizes
    23 
    +the tasks required.
    24 
    +
    25 
    +We strongly recommend that you propose the feature in advance of
    26 
    +commencing any work.
    24 27
    25 28 
     The author of any patch is expected to take ownership of that code and is to
    26 29 
     support it for a reasonable time-frame. This means addressing any unforeseen
    ... ... @@ -204,17 +207,16 @@ Committer access
    204 207 
     We'll hand out commit access to anyone who has successfully landed a single
    205 208 
     patch to the code base. Please request this via Slack or the mailing list.
    206 209
    207 
    -This of course relies on contributors being responsive and show willingness to
    208 
    -address problems after landing branches there should not be any problems here.
    210 
    +This of course relies on contributors being responsive and showing willingness
    211 
    +to address any problems that may arise after landing branches.
    209 212
    210 
    -What we are expecting of committers here in general is basically to escalate the
    211 
    -review in cases of uncertainty:
    213 
    +When submitting a merge request, please obtain a review from another committer
    214 
    +who is familiar with the area of the code base which the branch effects. An
    215 
    +approval from another committer who is not the patch author will be needed
    216 
    +before any merge (we use gitlab's 'approval' feature for this).
    212 217
    213 
    -- If the change is very trivial (obvious few line changes, typos…), and you are
    214 
    -  confident of the change, there is no need for review.
    215 
    -- If the change is non trivial, please obtain a review from another committer
    216 
    -  who is familiar with the area which the branch effects. An approval from
    217 
    -  someone who is not the patch author will be needed before any merge.
    218 
    +What we are expecting of committers here in general is basically to escalate the
    219 
    +review in cases of uncertainty.
    218 220
    219 221 
     .. note::
    220 222
    ... ... @@ -239,21 +241,49 @@ following goals:
    239 241 
       for the viewer to digest.
    240 242 
     - Ensure that we keep it simple and easy to contribute to the project.
    241 243
    242 
    -We are currenlty using the following GitLab features:
    244 
    +Explanation of how the project is currenlty using some GitLab features:
    243 245
    244 246 
     - `Milestones`_: we have seen them used in the same way as `Epics`_ in other
    245 
    -  projects. BuildGrid milestones must be time-line based, can overlap and we can
    246 
    -  be working towards multiple milestones at any one time. They allow us to group
    247 
    -  together all sub tasks into an overall aim. See our `BuildGrid milestones`_.
    248 
    -- `Labels`_: allow us to filter tickets in useful ways. They do complexity and
    249 
    -  effort as they grow in number and usage, though, so the general approach is
    250 
    -  to have the minimum possible. See our `BuildGrid labels`_.
    247 
    +  projects and are trying not to do that here. Instead we are going to
    248 
    +  use milestones to denote development cycles (ie, two week 'sprints'). See the
    249 
    +  `BuildGrid milestones`_.
    250 
    +- `Labels`_: allow us to filter tickets (ie, 'issues' in gitlab terminology)
    251 
    +  in useful ways. They add complexity and effort as they grow in number, so the
    252 
    +  general approach is to have the minimum possible but
    253 
    +  ensure we use them consistently. See the `BuildGrid labels`_.
    251 254 
     - `Boards`_: allow us to visualise and manage issues and labels in a simple way.
    252 
    -  For now, we are only utilising one boards. Issues start life in the
    253 
    -  ``Backlog`` column by default, and we move them into ``ToDo`` when they are
    254 
    -  coming up in the next few weeks. ``Doing`` is only for when an item is
    255 
    -  currently being worked on. Moving an issue from column to column automatically
    256 
    -  adjust the tagged labels. See our `BuildGrid boards`_.
    255 
    +  Issues start life in the ``Backlog`` column by default, and we move them into
    256 
    +  ``ToDo`` when we aim to complete them in the current development cycle.
    257 
    +  ``Doing`` is only for when an item is currently being worked on. When on the
    258 
    +  Board view, dragging and dropping an issue from column to column automatically
    259 
    +  adjusts the relevant labels. See the `BuildGrid boards`_.
    260 
    +
    261 
    +
    262 
    +Guidelines for using GitLab features when working on this project:
    263 
    +
    264 
    +- When raising an issue, please:
    265 
    +
    266 
    +  - check to see if there already is an issue to cover this task (if not then
    267 
    +    raise a new one)
    268 
    +  - assign the appropriate label or labels (tip: the vast majority of issues
    269 
    +    raised will be either an enhancement or a bug)
    270 
    +
    271 
    +- If you plan to work on an issue, please:
    272 
    +
    273 
    +  - self-assign the ticket
    274 
    +  - ensure it's captured in the current sprint (ie, Gitlab milestone)
    275 
    +  - ensure the ticket is in the ``ToDo`` column of the board if you aim to
    276 
    +    complete in the current sprint but aren't yet working on it, and
    277 
    +    the ``Doing`` column if you are working on it currently.
    278 
    +
    279 
    +- Please note that Gitlab issues are for either 'tasks' or 'bugs' - ie not for
    280 
    +  long discussions (where the mailing list is a better choice) or for ranting,
    281 
    +  for example.
    282 
    +
    283 
    +The above may seem like a lot to take in, but please don't worry about getting
    284 
    +it right the first few times. The worst that can happen is that you'll get a
    285 
    +friendly message from a current contributor who explains the process. We welcome
    286 
    +and value all contributions to the project!
    257 287
    258 288 
     .. _Milestones: https://docs.gitlab.com/ee/user/project/milestones
    259 289 
     .. _Epics: https://docs.gitlab.com/ee/user/group/epics

  • Dockerfile
    1 
    +FROM python:3.5-stretch
    2 
    +
    3 
    +# Point the path to where buildgrid gets installed
    4 
    +ENV PATH=$PATH:/root/.local/bin/
    5 
    +
    6 
    +# Upgrade python modules
    7 
    +RUN python3 -m pip install --upgrade setuptools pip
    8 
    +
    9 
    +# Use /app as the current working directory
    10 
    +WORKDIR /app
    11 
    +
    12 
    +# Copy the repo contents (source, config files, etc) in the WORKDIR
    13 
    +COPY . .
    14 
    +
    15 
    +# Install BuildGrid
    16 
    +RUN pip install --user --editable .
    17 
    +
    18 
    +# Entry Point of the image (should get an additional argument from CMD, the path to the config file)
    19 
    +ENTRYPOINT ["bgd", "-v", "server", "start"]
    20 
    +
    21 
    +# Default config file (used if no CMD specified when running)
    22 
    +CMD ["buildgrid/_app/settings/default.yml"]
    23 
    +

  • README.rst
    1 
    -
    2 1 
     .. _about:
    3 2
    4 3 
     About
    ... ... @@ -14,13 +13,15 @@ BuildGrid is a Python remote execution service which implements Google's
    14 13 
     `Remote Execution API`_ and the `Remote Workers API`_. The project's goal is to
    15 14 
     be able to execute build jobs remotely on a grid of computers in order to
    16 15 
     massively speed up build times. Workers on the grid should be able to run with
    17 
    -different environments. It is designed to work with clients such as `Bazel`_ and
    18 
    -`BuildStream`_.
    16 
    +different environments. It works with clients such as `Bazel`_,
    17 
    +`BuildStream`_ and `RECC`_, and is designed to be able to work with any client
    18 
    +that conforms to the above API protocols.
    19 19
    20 20 
     .. _Remote Execution API: https://github.com/bazelbuild/remote-apis
    21 21 
     .. _Remote Workers API: https://docs.google.com/document/d/1s_AzRRD2mdyktKUj2HWBn99rMg_3tcPvdjx3MPbFidU/edit#heading=h.1u2taqr2h940
    22 22 
     .. _BuildStream: https://wiki.gnome.org/Projects/BuildStream
    23 23 
     .. _Bazel: https://bazel.build
    24 
    +.. _RECC: https://gitlab.com/bloomberg/recc
    24 25
    25 26
    26 27 
     .. _getting-started:
    ... ... @@ -49,7 +50,7 @@ Resources
    49 50
    50 51 
     .. _Homepage: https://buildgrid.build
    51 52 
     .. _GitLab repository: https://gitlab.com/BuildGrid/buildgrid
    52 
    -.. _Bug tracking: https://gitlab.com/BuildGrid/buildgrid/issues
    53 
    +.. _Bug tracking: https://gitlab.com/BuildGrid/buildgrid/boards
    53 54 
     .. _Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
    54 55 
     .. _Slack channel: https://buildteamworld.slack.com/messages/CC9MKC203
    55 56 
     .. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU

  • buildgrid/_app/cli.py
    ... ... @@ -187,4 +187,3 @@ def cli(context, verbose):
    187 187 
             logger.setLevel(logging.DEBUG)
    188 188 
         elif verbose == 2:
    189 189 
             logger.setLevel(logging.VERBOSE)
    190 
    -

  • buildgrid/_app/commands/cmd_execute.py
    ... ... @@ -169,6 +169,7 @@ def run_command(context, input_root, commands, output_file, output_directory):
    169 169
    170 170 
                 downloader.download_file(output_file_response.digest, path)
    171 171
    172 
    -            if output_file_response.path in output_executeables:
    173 
    -                st = os.stat(path)
    174 
    -                os.chmod(path, st.st_mode | stat.S_IXUSR)
    172 
    +    for output_file_response in execute_response.result.output_files:
    173 
    +        if output_file_response.path in output_executeables:
    174 
    +            st = os.stat(path)
    175 
    +            os.chmod(path, st.st_mode | stat.S_IXUSR)

  • buildgrid/server/job.py
    ... ... @@ -76,9 +76,11 @@ class LeaseState(Enum):
    76 76
    77 77 
     class JobLogger(logging.LoggerAdapter):
    78 78 
         """ Add job id to job logging """
    79 
    +
    79 80 
         def process(self, msg, kwargs):
    80 81 
             return "[Job id: %s] %s" % (self.extra['job_id'], msg), kwargs
    81 82
    83 
    +
    82 84 
     class Job:
    83 85
    84 86 
         def __init__(self, action_digest, do_not_cache=False, message_queue=None):

  • buildgrid/server/operations/service.py
    ... ... @@ -44,7 +44,7 @@ class OperationsService(operations_pb2_grpc.OperationsServicer):
    44 44 
         def GetOperation(self, request, context):
    45 45 
             try:
    46 46 
                 self.logger.debug("GetOperation request from [%s]"
    47 
    -                              % (context.peer()))
    47 
    +                              .format(context.peer()))
    48 48 
                 name = request.name
    49 49 
                 operation_name = self._get_operation_name(name)
    50 50

  • docs/source/index.rst
    ... ... @@ -19,6 +19,7 @@ Remote execution service implementing Google's REAPI and RWAPI.
    19 19 
        using.rst
    20 20 
        reference.rst
    21 21 
        contributing.rst
    22 
    +   resources.rst
    22 23
    23 24
    24 25 
     Resources
    ... ... @@ -28,11 +29,11 @@ Resources
    28 29 
     - `GitLab repository`_
    29 30 
     - `Bug tracking`_
    30 31 
     - `Mailing list`_
    31 
    -- `Slack channel`_  [`invite link`_]
    32 
    +- `Slack channel`_ [`invite link`_]
    32 33
    33 34 
     .. _Homepage: https://buildgrid.build
    34 35 
     .. _GitLab repository: https://gitlab.com/BuildGrid/buildgrid
    35 
    -.. _Bug tracking: https://gitlab.com/BuildGrid/buildgrid/issues
    36 
    +.. _Bug tracking: https://gitlab.com/BuildGrid/buildgrid/boards
    36 37 
     .. _Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
    37 38 
     .. _Slack channel: https://buildteamworld.slack.com/messages/CC9MKC203
    38 
    -.. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU
    39 
    +.. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU
    \ No newline at end of file

  • docs/source/installation.rst
    1 
    -
    2 1 
     .. _installation:
    3 2
    4 3 
     Installation
    5 4 
     ============
    6 5
    7 
    -How to install BuildGrid onto your machine.
    6 
    +.. _install-on-host:
    7 
    +
    8 
    +Installation onto host machine
    9 
    +------------------------------
    10 
    +
    11 
    +How to install BuildGrid directly onto your machine.
    8 12
    9 13 
     .. note::
    10 14
    11 
    -   BuildGrid server currently only support *Linux*, *macOS* and *Windows*
    15 
    +   BuildGrid server currently only support *Linux*. *macOS* and *Windows*
    12 16 
        platforms are **not** supported.
    13 17
    14 18
    15 
    -.. _install-prerequisites:
    19 
    +.. _install-host-prerequisites:
    16 20
    17 21 
     Prerequisites
    18 
    --------------
    22 
    +~~~~~~~~~~~~~
    19 23
    20 24 
     BuildGrid only supports ``python3 >= 3.5`` but has no system requirements. Main
    21 
    -Python dependencies, automatically handle during installation, includes:
    25 
    +Python dependencies, automatically handled during installation, include:
    22 26
    23 27 
     - `boto3`_: the Amazon Web Services (AWS) SDK for Python.
    24 28 
     - `click`_: a Python composable command line library.
    ... ... @@ -33,10 +37,10 @@ Python dependencies, automatically handle during installation, includes:
    33 37 
     .. _protocol-buffers: https://developers.google.com/protocol-buffers
    34 38
    35 39
    36 
    -.. _source-install:
    40 
    +.. _install-host-source-install:
    37 41
    38 42 
     Install from sources
    39 
    ---------------------
    43 
    +~~~~~~~~~~~~~~~~~~~~
    40 44
    41 45 
     BuildGrid has ``setuptools`` support. In order to install it to your home
    42 46 
     directory, typically under ``~/.local``, simply run:
    ... ... @@ -46,7 +50,7 @@ directory, typically under ``~/.local``, simply run:
    46 50 
        git clone https://gitlab.com/BuildGrid/buildgrid.git && cd buildgrid
    47 51 
        pip3 install --user --editable .
    48 52
    49 
    -Additionally, and if your distribution does not already includes it, you may
    53 
    +Additionally, and if your distribution does not already include it, you may
    50 54 
     have to adjust your ``PATH``, in ``~/.bashrc``, with:
    51 55
    52 56 
     .. code-block:: sh
    ... ... @@ -63,3 +67,62 @@ have to adjust your ``PATH``, in ``~/.bashrc``, with:
    63 67 
        .. code-block:: sh
    64 68
    65 69 
           pip3 install --user --editable ".[docs,tests]"
    70 
    +
    71 
    +
    72 
    +
    73 
    +.. install-docker:
    74 
    +
    75 
    +Installation through docker
    76 
    +---------------------------
    77 
    +
    78 
    +How to build a Docker image that runs BuildGrid.
    79 
    +
    80 
    +.. _install-docker-prerequisites:
    81 
    +
    82 
    +Prerequisites
    83 
    +~~~~~~~~~~~~~
    84 
    +
    85 
    +A working Docker installation. Please consult `Docker's Getting Started Guide`_ if you don't already have it installed.
    86 
    +
    87 
    +.. _`Docker's Getting Started Guide`: https://www.docker.com/get-started
    88 
    +
    89 
    +
    90 
    +.. _install-docker-build:
    91 
    +
    92 
    +Docker Container from Sources
    93 
    +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    94 
    +
    95 
    +To clone the source code and build a Docker image, simply run:
    96 
    +
    97 
    +.. code-block:: sh
    98 
    +
    99 
    +   git clone https://gitlab.com/BuildGrid/buildgrid.git && cd buildgrid
    100 
    +   docker build -t buildgrid_server .
    101 
    +
    102 
    +.. note::
    103 
    +
    104 
    +   The image built will contain the contents of the source code directory, including
    105 
    +   configuration files.
    106 
    +
    107 
    +.. hint::
    108 
    +
    109 
    +    Whenever the source code is updated or new configuration files are made, you need to re-build
    110 
    +    the image.
    111 
    +
    112 
    +After building the Docker image, to run BuildGrid using the default configuration file
    113 
    +(found in `buildgrid/_app/settings/default.yml`), simply run:
    114 
    +
    115 
    +.. code-block:: sh
    116 
    +
    117 
    +   docker run -i -p 50051:50051 buildgrid_server
    118 
    +
    119 
    +.. note::
    120 
    +
    121 
    +    To run BuildGrid using a different configuration file, include the relative path to the
    122 
    +    configuration file at the end of the command above. For example, to run the default
    123 
    +    standalone CAS server (without an execution service), simply run:
    124 
    +
    125 
    +       .. code-block:: sh
    126 
    +
    127 
    +            docker run -i -p 50052:50052 buildgrid_server buildgrid/_app/settings/cas.yml
    128 
    +

  • docs/source/resources.rst
    1 
    +.. _external-resources:
    2 
    +
    3 
    +Resources
    4 
    +=========
    5 
    +
    6 
    +Remote execution and worker API useful links:
    7 
    +
    8 
    +- `REAPI design document`_
    9 
    +- `REAPI protobuf specification`_
    10 
    +- `RWAPI design document`_
    11 
    +- `RWAPI protobuf specification`_
    12 
    +- `Bazel`_ `remote caching and execution documentation`_
    13 
    +- `RECC usage instructions`_
    14 
    +- `BuildStream webside`_ and `documentation`_
    15 
    +- `BuildStream-externals repository`_
    16 
    +- `BuildBox repository`_
    17 
    +- `FUSE on wikipedia`_ and `kernel documentation`_
    18 
    +- `bubblewrap repository`_
    19 
    +- `Buildfarm reference REAPI implementation`_
    20 
    +- `Buildbarn Golang REAPI implementation`_
    21 
    +- `Demonstration of RECC with BuildGrid`_
    22 
    +- `Demonstration of Bazel with BuildGrid`_
    23 
    +- `Demonstration of BuildStream with BuildGrid`_
    24 
    +
    25 
    +.. _REAPI design document: https://docs.google.com/document/d/1AaGk7fOPByEvpAbqeXIyE8HX_A3_axxNnvroblTZ_6s
    26 
    +.. _REAPI protobuf specification: https://github.com/bazelbuild/remote-apis/blob/master/build/bazel/remote/execution/v2/remote_execution.proto
    27 
    +.. _RWAPI design document: https://docs.google.com/document/d/1s_AzRRD2mdyktKUj2HWBn99rMg_3tcPvdjx3MPbFidU
    28 
    +.. _RWAPI protobuf specification: https://github.com/googleapis/googleapis/blob/master/google/devtools/remoteworkers/v1test2/bots.proto
    29 
    +.. _Bazel: https://www.bazel.build
    30 
    +.. _remote caching and execution documentation: https://docs.bazel.build/versions/master/remote-caching.html
    31 
    +.. _RECC usage instructions: https://gitlab.com/bloomberg/recc#running-recc
    32 
    +.. _BuildStream webside: https://buildstream.build
    33 
    +.. _documentation: https://docs.buildstream.build
    34 
    +.. _BuildStream-externals repository: https://gitlab.com/BuildStream/bst-external
    35 
    +.. _FUSE on wikipedia: https://en.wikipedia.org/wiki/Filesystem_in_Userspace
    36 
    +.. _kernel documentation: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/filesystems/fuse.txt
    37 
    +.. _BuildBox repository: https://gitlab.com/BuildStream/buildbox
    38 
    +.. _bubblewrap repository: https://github.com/projectatomic/bubblewrap
    39 
    +.. _Buildfarm reference REAPI implementation: https://github.com/bazelbuild/bazel-buildfarm
    40 
    +.. _Buildbarn Golang REAPI implementation: https://github.com/EdSchouten/bazel-buildbarn
    41 
    +.. _Demonstration of RECC with BuildGrid: https://asciinema.org/a/0FjExIqrTGSlpSUIS8Ehf5gUg
    42 
    +.. _Demonstration of Bazel with BuildGrid: https://asciinema.org/a/uVHFWOxpivwJ4ari23CEerR8N
    43 
    +.. _Demonstration of BuildStream with BuildGrid: https://asciinema.org/a/QfkYGqhfhEQz4o8prlBdEBFP7

    • —
    View it on GitLab.
    You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.

    [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]