GNOME.org
 

[Notes] [Git][BuildGrid/buildgrid][master] 8 commits: Add a Description Of A Project (DOAP) file

Title: GitLab

Martin Blanchard pushed to branch master at BuildGrid / buildgrid

Commits:

20 changed files:

Changes:

  • .gitignore
    ... ... @@ -16,3 +16,7 @@ dist/
    16 16 
     .coverage.*
    17 17 
     .pylint.d/
    18 18 
     .pytest_cache/
    19 
    +
    20 
    +# Documentation related things:
    21 
    +docs/build/
    22 
    +docs/source/api/

  • BuildGrid.doap
    1 
    +<?xml version="1.0" encoding="UTF-8"?>
    2 
    +<Project xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    3 
    +         xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
    4 
    +         xmlns:foaf="http://xmlns.com/foaf/0.1/"
    5 
    +         xmlns="http://usefulinc.com/ns/doap#">
    6 
    +
    7 
    +  <name xml:lang="en">BuildGrid</name>
    8 
    +  <shortdesc xml:lang="en">Remote execution service implementing Google's REAPI &amp; RWAPI</shortdesc>
    9 
    +  <homepage rdf:resource="https://buildgrid.build" />
    10 
    +  <mailing-list rdf:resource="https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid" />
    11 
    +  <download-page rdf:resource="https://gitlab.com/BuildGrid/buildgrid" />
    12 
    +  <bug-database rdf:resource="https://gitlab.com/BuildGrid/buildgrid/issues" />
    13 
    +
    14 
    +  <programming-language>python3</programming-language>
    15 
    +
    16 
    +  <maintainer>
    17 
    +    <foaf:Person>
    18 
    +      <foaf:name>Finn Ball</foaf:name>
    19 
    +      <foaf:mbox rdf:resource="mailto:finn ball codethink co uk" />
    20 
    +    </foaf:Person>
    21 
    +  </maintainer>
    22 
    +</Project>
    \ No newline at end of file

  • CONTRIBUTING.rst
    1 
    -Contributing to BuildGrid
    2 
    -=========================
    3 
    -Some guidelines for people wanting to contribute. Also please always feel free to speak to us, we're very friendly :-)
    4 1
    5 
    -Feature Additions
    6 
    ------------------
    2 
    +.. _contributing:
    3 
    +
    4 
    +Contributing
    5 
    +============
    7 6
    8 
    -We welcome contributions in the form of bug fixes or feature additions / enhancements. Please discuss with us before submitting anything, as we may well have some important context which will could help guide your efforts.
    7 
    +Some guidelines for people wanting to contribute. Also please always feel free
    8 
    +to speak to us, we're very friendly :-)
    9 9
    10 
    -Any major feature additions should be raised as a proposal on the `Mailing List <https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid/>`_ to be discussed, and then eventually followed up with an issue here on gitlab. We recommend that you propose the feature in advance of commencing work. We are also on irc - you can find us on #buildgrid on freenode.
    11 10
    12 
    -The author of any patch is expected to take ownership of that code and is to support it for a reasonable time-frame. This means addressing any unforeseen side effects and quirks the feature may have introduced. More on this below in 'Granting Committer Access'.
    11 
    +.. _feature-additions:
    13 12
    14 
    -Patch Submissions
    13 
    +Feature additions
    15 14 
     -----------------
    16 15
    17 
    -We will be running `trunk based development <https://trunkbaseddevelopment.com>`_. The idea behind this is that merge requests to the trunk will be small and made often, thus making the review and merge process as fast as possible. We do not want to end up with a huge backlog of outstanding merge requests. If possible,
    18 
    -it is preferred that merge requests address specific points and clearly outline what problem they are solving.
    16 
    +We welcome contributions in the form of bug fixes or feature additions. Please
    17 
    +discuss with us before submitting anything, as we may well have some important
    18 
    +context which will could help guide your efforts.
    19 
    +
    20 
    +Any major feature additions should be raised first as a proposal on the
    21 
    +`BuildGrid mailing list`_ to be discussed, and then eventually followed up with
    22 
    +an issue on GitLab. We recommend that you propose the feature in advance of
    23 
    +commencing work.
    24 
    +
    25 
    +The author of any patch is expected to take ownership of that code and is to
    26 
    +support it for a reasonable time-frame. This means addressing any unforeseen
    27 
    +side effects and quirks the feature may have introduced. More on this below in
    28 
    +:ref:`Committer access <committer-access>`.
    19 29
    20 
    -Branches must be submitted as merge requests on gitlab and should be associated with an issue report on gitlab, whenever possible. If it's a small change, we'll accept an MR without it being associated to a gitlab issue, but generally we prefer an issue to be raised in advance. This is so that we can track the work that is currently in progress on the project - please see our Gitlab policy below.
    30 
    +.. _BuildGrid mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
    31 
    +
    32 
    +
    33 
    +.. _patch-submissions:
    34 
    +
    35 
    +Patch submissions
    36 
    +-----------------
    21 37
    22 
    -Each commit should address a specific gitlab issue number in the commit message. This is really important for provenance reasons.
    38 
    +We are running `trunk based development`_. The idea behind this is that merge
    39 
    +requests to the trunk will be small and made often, thus making the review and
    40 
    +merge process as fast as possible. We do not want to end up with a huge backlog
    41 
    +of outstanding merge requests. If possible, it is preferred that merge requests
    42 
    +address specific points and clearly outline what problem they are solving.
    23 43
    24 
    -Merge requests that are not yet ready for review must be prefixed with the `WIP:` identifier, but if we stick to trunk based development then the 'WIP:' identifier will not stay around for very long on a merge request.
    44 
    +Branches must be submitted as merge requests (MR) on GitLab and should be
    45 
    +associated with an issue, whenever possible. If it's a small change, we'll
    46 
    +accept an MR without it being associated to an issue, but generally we prefer an
    47 
    +issue to be raised in advance. This is so that we can track the work that is
    48 
    +currently in progress on the project.
    25 49
    26 
    -When a merge request is ready for review, please find someone willing to do the review (ideally a maintainer) and assign them the MR on gitlab, leaving a comment asking for their review.
    50 
    +Below is a list of good patch submission good practice:
    27 51
    28 
    -Submitted branches should not contain a history of work done.
    52 
    +- Each commit should address a specific issue number in the commit message. This
    53 
    +  is really important for provenance reasons.
    54 
    +- Merge requests that are not yet ready for review must be prefixed with the
    55 
    +  ``WIP:`` identifier, but if we stick to trunk based development then the
    56 
    +  ``WIP:`` identifier will not stay around for very long on a merge request.
    57 
    +- When a merge request is ready for review, please find someone willing to do
    58 
    +  the review (ideally a maintainer) and assign them the MR, leaving a comment
    59 
    +  asking for their review.
    60 
    +- Submitted branches should not contain a history of work done.
    61 
    +- Unit tests should be a separate commit.
    62 
    +
    63 
    +.. _trunk based development: https://trunkbaseddevelopment.com
    29 64
    30 
    -Unit tests should be a separate commit.
    31 65
    32 66 
     Commit messages
    33 67 
     ~~~~~~~~~~~~~~~
    34 
    -Commit messages must be formatted with a brief summary line, optionally followed by an empty line and then a
    35 
    -free form detailed description of the change.
    36 
    -
    37 
    -The summary line must start with what changed, followed by a colon and a very brief description of the
    38 
    -change.
    39 68
    40 
    -If there is an associated issue, it **must** be mentioned somewhere in the commit message.
    69 
    +Commit messages must be formatted with a brief summary line, optionally followed
    70 
    +by an empty line and then a free form detailed description of the change. The
    71 
    +summary line must start with what changed, followed by a colon and a very brief
    72 
    +description of the change. If there is an associated issue, it **must** be
    73 
    +mentioned somewhere in the commit message.
    41 74
    42 75 
     **Example**::
    43 76
    44 
    -  worker.py: Fixed to be more human than human
    77 
    +   worker.py: Fixed to be more human than human
    78 
    +
    79 
    +   Gifted the worker with a past so we can create
    80 
    +   a cushion or a pillow for their emotions and
    81 
    +   consequently, we can control them better.
    82 
    +
    83 
    +   This fixes issue #8.
    45 84
    46 
    -  Gifted the worker with a past so we can create
    47 
    -  a cushion or a pillow for their emotions and
    48 
    -  consequently, we can control them better.
    49 
    -
    50 
    -  This fixes issue #8
    85 
    +For more tips, please read `The seven rules of a great Git commit message`_.
    51 86
    52 
    -
    53 
    -For more tips, please see `this <https://chris.beams.io/posts/git-commit/#seven-rules/>`_ article.
    87 
    +.. _The seven rules of a great Git commit message: https://chris.beams.io/posts/git-commit/#seven-rules
    88 
    +
    89 
    +
    90 
    +.. _coding-style:
    54 91
    55 92 
     Coding style
    56 93 
     ------------
    57 
    -Coding style details for BuildGrid.
    58 94
    95 
    +Python coding style for BuildGrid is `PEP 8`_. We do have a couple of minor
    96 
    +exceptions to this standard, we dont want to compromise code readability by
    97 
    +being overly restrictive on line length for instance.
    98 
    +
    99 
    +BuildGrid's test suite includes a PEP8 style compliance check phase (using
    100 
    +`pep8`_) and a code linting phase (using `pylint`_). That test suite is
    101 
    +automatically run for every change submitted to the GitLab server and the merge
    102 
    +request sytem requires the test suite execution to succed before changes can
    103 
    +be pulled upstream. This means you have to respect the BuildGrid coding style.
    59 104
    60 
    -Style guide
    61 
    -~~~~~~~~~~~
    62 
    -Python coding style for BuildGrid is pep8, which is documented here: https://www.python.org/dev/peps/pep-0008/
    105 
    +Configuration and exceptions for ``pep8`` and ``pylint`` can be found in:
    63 106
    64 
    -We have a couple of minor exceptions to this standard, we dont want to compromise
    65 
    -code readability by being overly restrictive on line length for instance.
    107 
    +- The `setup.cfg`_ file for ``pep8``.
    108 
    +- The `.pylintrc`_ file for ``pylint``.
    109 
    +
    110 
    +.. _PEP 8: https://www.python.org/dev/peps/pep-0008
    111 
    +.. _pep8: https://pep8.readthedocs.io
    112 
    +.. _pylint: https://pylint.readthedocs.io
    113 
    +.. _setup.cfg: https://gitlab.com/BuildGrid/buildgrid/blob/master/setup.cfg
    114 
    +.. _.pylintrc: https://gitlab.com/BuildGrid/buildgrid/blob/master/.pylintrc
    66 115
    67 116
    68 117 
     Imports
    69 118 
     ~~~~~~~
    119 
    +
    70 120 
     Module imports inside BuildGrid are done with relative ``.`` notation
    71 121
    72 122 
     Good::
    ... ... @@ -77,99 +127,85 @@ Bad::
    77 127
    78 128 
       from buildgrid.worker import Worker
    79 129
    80 
    -Ordering
    81 
    -''''''''
    82 
    -For better readability and consistency, we try to keep private symbols below
    83 
    -public symbols. In the case of public modules where we may have a mix of
    84 
    -*API private* and *local private* symbols, *API private* symbols should come
    85 
    -before *local private* symbols.
    86 
    -
    87 130
    88 131 
     Symbol naming
    89 132 
     '''''''''''''
    90 
    -Any private symbol must start with a single leading underscore for two reasons:
    91 
    -
    92 
    -* So that it does not bleed into documentation and *public API*.
    93 
    -
    94 
    -* So that it is clear to developers which symbols are not used outside of the declaring *scope*
    95 
    -
    96 
    -Remember that with python, the modules (python files) are also symbols
    97 
    -within their containing *package*, as such; modules which are entirely
    98 
    -private to BuildGrid are named as such, e.g. ``_roy.py``.
    99 133
    100 
    -Granting Committer Access
    101 
    --------------------------
    102 
    -
    103 
    -We'll hand out commit access to anyone who has successfully landed a single patch to the code base. Please request this via irc or the mailing list.
    104 
    -
    105 
    -This of course relies on contributors being responsive and show willingness to address problems after landing branches there should not be any problems here.
    106 
    -
    107 
    -What we are expecting of committers here in general is basically to
    108 
    -escalate the review in cases of uncertainty:
    109 
    -
    110 
    -* If the patch/branch is very trivial (obvious few line changes or typos etc), and you are confident of the change, there is no need for review.
    111 
    -
    112 
    -* If the patch/branch is non trivial, please obtain a review from another committer who is familiar with the area which the branch effects. An approval from someone who is not the patch author will be needed before any merge.
    113 
    -
    114 
    -We don't have any detailed policy for "bad actors", but will of course handle things on a case by case basis - commit access should not result in commit wars or be used as a tool to subvert the project when disagreements arise, such incidents (if any) would surely lead to temporary suspension of commit rights.
    134 
    +Any private symbol must start with a single leading underscore for two reasons:
    115 135
    116 
    -BuildGrid policy for use of Gitlab features
    117 
    --------------------------------------------
    136 
    +- So that it does not bleed into documentation and *public API*.
    137 
    +- So that it is clear to developers which symbols are not used outside of the
    138 
    +  declaring *scope*.
    118 139
    119 
    -We intend to make use of some of gitlab's features in order to structure the activity of the BuildGrid project. In doing so we are trying to achieve the following goals:
    140 
    +Remember that with python, the modules (python files) are also symbols within
    141 
    +their containing *package*, as such; modules which are entirely private to
    142 
    +BuildGrid are named as such, e.g. ``_roy.py``.
    120 143
    121 
    -* Full transparency of the current WIP items
    122 
    -* Provide a view of all current and planned activity which is relatively easy for the viewer to digest
    123 
    -* Ensure that we keep it simple and easy to contribute to the project
    124 144
    125 
    -We propose to make use of the following Gitlab features:
    145 
    +.. _committer-access:
    126 146
    127 
    -* Milestones
    128 
    -* Labels
    129 
    -* Boards
    130 
    -* Templates
    147 
    +Committer access
    148 
    +----------------
    131 149
    132 
    -Milestones
    133 
    -~~~~~~~~~~
    134 
    -`Milestones <https://docs.gitlab.com/ee/user/project/milestones/>`_ are based on periods of time and what we want to achieve within those periods of time.
    150 
    +We'll hand out commit access to anyone who has successfully landed a single
    151 
    +patch to the code base. Please request this via irc or the mailing list.
    135 152
    136 
    -We have seen them used in the same way as `Epics <https://docs.gitlab.com/ee/user/group/epics/index.html#doc-nav/>`_ in other projects (since the Epic feature is only available with GitLab Ultimate) and this does not work. Milestones must be time-line based.
    153 
    +This of course relies on contributors being responsive and show willingness to
    154 
    +address problems after landing branches there should not be any problems here.
    137 155
    138 
    -Milestones can overlap, and we can be working towards multiple milestones at any one time. They allow us to group together all sub tasks into an overall aim.
    156 
    +What we are expecting of committers here in general is basically to escalate the
    157 
    +review in cases of uncertainty:
    139 158
    140 
    -Labels
    141 
    -~~~~~~
    142 
    -`Labels <https://docs.gitlab.com/ee/user/project/labels.html/>`_ allow us to filter tickets on gitlab in useful ways. They do complexity and effort as they grow in number and usage, though, so the general approach is to have the minimum possible.
    159 
    +- If the change is very trivial (obvious few line changes, typos…), and you are
    160 
    +  confident of the change, there is no need for review.
    161 
    +- If the change is non trivial, please obtain a review from another committer
    162 
    +  who is familiar with the area which the branch effects. An approval from
    163 
    +  someone who is not the patch author will be needed before any merge.
    143 164
    144 
    -Type Labels
    145 
    -'''''''''''
    146 
    -We have:
    165 
    +.. note::
    147 166
    148 
    -* Bug
    149 
    -* Documentation
    150 
    -* Enhancement
    151 
    -* Tests
    167 
    +   We don't have any detailed policy for "bad actors", but will of course handle
    168 
    +   things on a case by case basis - commit access should not result in commit
    169 
    +   wars or be used as a tool to subvert the project when disagreements arise.
    170 
    +   Such incidents (if any) would surely lead to temporary suspension of commit
    171 
    +   rights.
    152 172
    153 
    -This is useful for filtering different types of issues. We may expand this at some point.
    154 173
    155 
    -Priority Labels
    156 
    -'''''''''''''''
    157 
    -For now, we only have 'High Priority', which indicates an urgent task. We may add more granularity if we get more contributors.
    174 
    +.. _gitlab-features:
    158 175
    159 
    -Status
    160 
    -'''''
    161 
    -We have:
    176 
    +GitLab features
    177 
    +---------------
    162 178
    163 
    -* ToDo
    164 
    -* Doing
    179 
    +We intend to make use of some of GitLab's features in order to structure the
    180 
    +activity of the BuildGrid project. In doing so we are trying to achieve the
    181 
    +following goals:
    165 182
    166 
    -These labels are used when structuring tickets on a Board. GitLab issues start life in the 'Backlog' column by default, and we move them into 'ToDo' when they are coming up in the next few weeks. 'Doing' is only for when an item is currently being worked on. These labels don't have to be manually applied, they are applied by GitLab when moving the issue from column to column when using a Board - see below.
    183 
    +- Full transparency of the current work in progress items.
    184 
    +- Provide a view of all current and planned activity which is relatively easy
    185 
    +  for the viewer to digest.
    186 
    +- Ensure that we keep it simple and easy to contribute to the project.
    167 187
    168 
    -Issue Boards
    169 
    -~~~~~~~~~~~~
    170 
    -`Boards <https://docs.gitlab.com/ee/user/project/issue_board.html#doc-nav/>`_ allow you to visualise and manage issues in a simple way, and we can create different types of board by filtering labels. For now, we are just utilising Boards in order to be able to see all of the currently in flight items at a glance.
    188 
    +We are currenlty using the following GitLab features:
    171 189
    172 
    -Templates
    173 
    -~~~~~~~~~
    174 
    -`Issue templates <https://docs.gitlab.com/ee/user/project/description_templates.html#doc-nav/>`_ help us to receive good quality information in issues.
    190 
    +- `Milestones`_: we have seen them used in the same way as `Epics`_ in other
    191 
    +  projects. BuildGrid milestones must be time-line based, can overlap and we can
    192 
    +  be working towards multiple milestones at any one time. They allow us to group
    193 
    +  together all sub tasks into an overall aim. See our `BuildGrid milestones`_.
    194 
    +- `Labels`_: allow us to filter tickets in useful ways. They do complexity and
    195 
    +  effort as they grow in number and usage, though, so the general approach is
    196 
    +  to have the minimum possible. See our `BuildGrid labels`_.
    197 
    +- `Boards`_: allow us to visualise and manage issues and labels in a simple way.
    198 
    +  For now, we are only utilising one boards. Issues start life in the
    199 
    +  ``Backlog`` column by default, and we move them into ``ToDo`` when they are
    200 
    +  coming up in the next few weeks. ``Doing`` is only for when an item is
    201 
    +  currently being worked on. Moving an issue from column to column automatically
    202 
    +  adjust the tagged labels. See our `BuildGrid boards`_.
    175 203
    204 
    +.. _Milestones: https://docs.gitlab.com/ee/user/project/milestones
    205 
    +.. _Epics: https://docs.gitlab.com/ee/user/group/epics
    206 
    +.. _BuildGrid milestones: https://gitlab.com/BuildGrid/buildgrid/milestones
    207 
    +.. _Labels: https://docs.gitlab.com/ee/user/project/labels.html
    208 
    +.. _BuildGrid labels: https://gitlab.com/BuildGrid/buildgrid/labels
    209 
    +.. _Boards: https://docs.gitlab.com/ee/user/project/issue_board.html
    210 
    +.. _BuildGrid boards: https://gitlab.com/BuildGrid/buildgrid/boards
    211 
    +.. _Templates: https://docs.gitlab.com/ee/user/project/description_templates.html

  • README.rst
    1 
    -BuildGrid
    2 
    -=========
    1 
    +
    2 
    +.. _about:
    3 
    +
    4 
    +About
    5 
    +=====
    3 6
    4 7 
     .. image:: https://gitlab.com/Buildgrid/buildgrid/badges/master/pipeline.svg
    5 8 
        :target: https://gitlab.com/BuildStream/buildstream/commits/master
    ... ... @@ -7,79 +10,40 @@ BuildGrid
    7 10 
     .. image:: https://gitlab.com/BuildGrid/buildgrid/badges/master/coverage.svg?job=coverage
    8 11 
        :target: https://gitlab.com/BuildGrid/buildgrid/commits/master
    9 12
    13 
    +BuildGrid is a Python remote execution service which implements Google's
    14 
    +`Remote Execution API`_ and the `Remote Workers API`_. The project's goal is to
    15 
    +be able to execute build jobs remotely on a grid of computers in order to
    16 
    +massively speed up build times. Workers on the grid should be able to run with
    17 
    +different environments. It is designed to work with but not exclusively
    18 
    +`BuildStream`_.
    10 19
    11 
    -BuildGrid is a python remote execution service which implements the `Remote Execution API <https://github.com/bazelbuild/remote-apis//>`_ and the `Remote Workers API <https://docs.google.com/document/d/1s_AzRRD2mdyktKUj2HWBn99rMg_3tcPvdjx3MPbFidU/edit#heading=h.1u2taqr2h940/>`_.
    12 
    -
    13 
    -The goal is to be able to execute build jobs remotely on a grid of computers to massively speed up build times. Workers on the system will also be able to run with different environments. It is designed to work with but not exclusively with `BuildStream <https://wiki.gnome.org/Projects/BuildStream/>`_.
    14 
    -
    15 
    -Install
    16 
    --------
    17 
    -
    18 
    -To install::
    19 
    -
    20 
    -   git clone https://gitlab.com/BuildGrid/buildgrid.git
    21 
    -   cd buildgrid
    22 
    -   pip3 install --user -e .
    23 
    -
    24 
    -This will install BuildGrid's python dependencies into your user’s homedir in ~/.local
    25 
    -and will run BuildGrid directly from the git checkout. It is recommended you adjust
    26 
    -your path with::
    27 
    -
    28 
    -  export PATH="${PATH}:${HOME}/.local/bin"
    29 
    -
    30 
    -Which you can add to the end of your `~/.bashrc`.
    31 
    -
    32 
    -Instructions for a Dummy Work
    33 
    -----------------------
    34 
    -
    35 
    -In one terminal, start a server::
    36 
    -
    37 
    -  bgd server start
    38 
    -
    39 
    -In another terminal, send a request for work::
    40 
    -
    41 
    -  bgd execute request-dummy
    42 
    -
    43 
    -The stage should show as `QUEUED` as it awaits a bot to pick up the work::
    44 
    -
    45 
    -  bgd execute list
    46 
    -
    47 
    -Create a bot session::
    48 
    -
    49 
    -  bgd bot dummy
    50 
    -
    51 
    -Show the work as completed::
    52 
    -
    53 
    -  bgd execute list
    54 
    -
    55 
    -Instructions for a Simple Build
    56 
    --------------------------------
    57 
    -
    58 
    -This example covers a simple build. The user will upload a directory containing a C file and a command to the CAS. The bot will then fetch the uploaded directory and command which will then be run inside a temporary directory. The result will then be uploaded to the CAS and downloaded by the user. This is an early demo and still lacks a few features such as symlink support and checking to see if files exist in the CAS before executing a command.
    59 
    -
    60 
    -Create a new directory called `test-buildgrid/` and place the following C file in it called `hello.c`::
    20 
    +.. _Remote Execution API: https://github.com/bazelbuild/remote-apis
    21 
    +.. _Remote Workers API: https://docs.google.com/document/d/1s_AzRRD2mdyktKUj2HWBn99rMg_3tcPvdjx3MPbFidU/edit#heading=h.1u2taqr2h940
    22 
    +.. _BuildStream: https://wiki.gnome.org/Projects/BuildStream
    61 23
    62 
    -  #include <stdio.h>
    63 
    -  int main()
    64 
    -  {
    65 
    -   printf("Hello, World!\n");
    66 
    -   return 0;
    67 
    -  }
    68 24
    69 
    -Now start a BuildGrid server, passing it a directory it can write a CAS to::
    25 
    +.. _getting-started:
    70 26
    71 
    -  bgd server start --cas disk --cas-cache disk --cas-disk-directory /path/to/empty/directory
    27 
    +Getting started
    28 
    +---------------
    72 29
    73 
    -Start the following bot session::
    30 
    +Please refer to the `documentation`_ for `installation`_ and `usage`_
    31 
    +instructions.
    74 32
    75 
    -  bgd bot temp-directory
    33 
    +.. _documentation: https://buildgrid.gitlab.io/buildgrid
    34 
    +.. _installation: https://buildgrid.gitlab.io/buildgrid/installation.html
    35 
    +.. _usage: https://buildgrid.gitlab.io/buildgrid/using.html
    76 36
    77 
    -Upload the directory containing the C file::
    78 37
    79 
    -  bgd cas upload-dir /path/to/test-buildgrid
    38 
    +.. _about-resources:
    80 39
    81 
    -Now we send an execution request to the bot with the name of the epxected `output-file`, a boolean describing if it is executeable, the path to the directory we uploaded in order to calculate the digest and finally the command to run on the bot::
    40 
    +Resources
    41 
    +---------
    82 42
    83 
    -  bgd execute command --output-file hello True /path/to/test-buildgrid -- gcc -Wall hello.c -o hello
    43 
    +- Homepage: https://buildgrid.build
    44 
    +- GitLab repository: https://gitlab.com/BuildGrid/buildgrid
    45 
    +- Bug tracking: https://gitlab.com/BuildGrid/buildgrid/issues
    46 
    +- Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
    47 
    +- Slack channel: https://buildteamworld.slack.com/messages/CC9MKC203 [`invite link`_]
    84 48
    85 
    -The resulting executeable should have returned to a new directory called `testing/`
    49 
    +.. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU

  • app/commands/cmd_bot.py
    ... ... @@ -36,10 +36,13 @@ from ..bots import buildbox, dummy, temp_directory
    36 36 
     from ..cli import pass_context
    37 37
    38 38
    39 
    -@click.group(short_help="Create a bot client")
    40 
    -@click.option('--parent', default='bgd_test')
    41 
    -@click.option('--port', default='50051')
    42 
    -@click.option('--host', default='localhost')
    39 
    +@click.group(name='bot', short_help="Create and register bot clients.")
    40 
    +@click.option('--parent', type=click.STRING, default='bgd_test', show_default=True,
    41 
    +              help="Targeted farm resource.")
    42 
    +@click.option('--port', type=click.INT, default='50051', show_default=True,
    43 
    +              help="Remote server's port number.")
    44 
    +@click.option('--host', type=click.STRING, default='localhost', show_default=True,
    45 
    +              help="Renote server's hostname.")
    43 46 
     @pass_context
    44 47 
     def cli(context, host, port, parent):
    45 48 
         channel = grpc.insecure_channel('{}:{}'.format(host, port))
    ... ... @@ -58,7 +61,7 @@ def cli(context, host, port, parent):
    58 61 
         context.bot_session = bot_session
    59 62
    60 63
    61 
    -@cli.command('dummy', short_help='Create a dummy bot session which just returns lease')
    64 
    +@cli.command('dummy', short_help="Run a dummy session simply returning leases.")
    62 65 
     @pass_context
    63 66 
     def run_dummy(context):
    64 67 
         """
    ... ... @@ -69,13 +72,13 @@ def run_dummy(context):
    69 72 
             b = bot.Bot(context.bot_session)
    70 73 
             b.session(dummy.work_dummy,
    71 74 
                       context)
    72 
    -
    73 75 
         except KeyboardInterrupt:
    74 76 
             pass
    75 77
    76 78
    77 
    -@cli.command('temp-directory', short_help='Runs commands in temp directory and uploads results')
    78 
    -@click.option('--instance-name', default='testing')
    79 
    +@cli.command('temp-directory', short_help="Runs commands in temp directory and uploads results.")
    80 
    +@click.option('--instance-name', type=click.STRING, default='testing', show_default=True,
    81 
    +              help="Targeted farm instance name.")
    79 82 
     @pass_context
    80 83 
     def run_temp_directory(context, instance_name):
    81 84 
         """ Downloads files and command from CAS and runs
    ... ... @@ -86,19 +89,25 @@ def run_temp_directory(context, instance_name):
    86 89 
             b = bot.Bot(context.bot_session)
    87 90 
             b.session(temp_directory.work_temp_directory,
    88 91 
                       context)
    89 
    -
    90 92 
         except KeyboardInterrupt:
    91 93 
             pass
    92 94
    93 95
    94 
    -@cli.command('buildbox', short_help="Create a bot session with busybox")
    95 
    -@click.option('--fuse-dir', show_default=True, default=str(PurePath(Path.home(), 'fuse')))
    96 
    -@click.option('--local-cas', show_default=True, default=str(PurePath(Path.home(), 'cas')))
    97 
    -@click.option('--client-cert', show_default=True, default=str(PurePath(Path.home(), 'client.crt')))
    98 
    -@click.option('--client-key', show_default=True, default=str(PurePath(Path.home(), 'client.key')))
    99 
    -@click.option('--server-cert', show_default=True, default=str(PurePath(Path.home(), 'server.crt')))
    100 
    -@click.option('--port', show_default=True, default=11001)
    101 
    -@click.option('--remote', show_default=True, default='localhost')
    96 
    +@cli.command('buildbox', short_help="Run commands using the BuildBox tool.")
    97 
    +@click.option('--fuse-dir', type=click.Path(readable=False), default=str(PurePath(Path.home(), 'fuse')),
    98 
    +              help="Main mount-point location.")
    99 
    +@click.option('--local-cas', type=click.Path(readable=False), default=str(PurePath(Path.home(), 'cas')),
    100 
    +              help="Local CAS cache directory.")
    101 
    +@click.option('--client-cert', type=click.Path(readable=False), default=str(PurePath(Path.home(), 'client.crt')),
    102 
    +              help="Public client certificate for TLS (PEM-encoded).")
    103 
    +@click.option('--client-key', type=click.Path(readable=False), default=str(PurePath(Path.home(), 'client.key')),
    104 
    +              help="Private client key for TLS (PEM-encoded).")
    105 
    +@click.option('--server-cert', type=click.Path(readable=False), default=str(PurePath(Path.home(), 'server.crt')),
    106 
    +              help="Public server certificate for TLS (PEM-encoded).")
    107 
    +@click.option('--port', type=click.INT, default=11001, show_default=True,
    108 
    +              help="Remote CAS server port.")
    109 
    +@click.option('--remote', type=click.STRING, default='localhost', show_default=True,
    110 
    +              help="Remote CAS server hostname.")
    102 111 
     @pass_context
    103 112 
     def run_buildbox(context, remote, port, server_cert, client_key, client_cert, local_cas, fuse_dir):
    104 113 
         """
    ... ... @@ -116,7 +125,6 @@ def run_buildbox(context, remote, port, server_cert, client_key, client_cert, lo
    116 125 
         context.fuse_dir = fuse_dir
    117 126
    118 127 
         try:
    119 
    -
    120 128 
             b = bot.Bot(context.bot_session)
    121 129 
             b.session(work=buildbox.work_buildbox,
    122 130 
                       context=context)

  • app/commands/cmd_cas.py
    ... ... @@ -30,9 +30,11 @@ from buildgrid._protos.build.bazel.remote.execution.v2 import remote_execution_p
    30 30 
     from ..cli import pass_context
    31 31
    32 32
    33 
    -@click.group(short_help='Interact with the CAS')
    34 
    -@click.option('--port', default='50051')
    35 
    -@click.option('--host', default='localhost')
    33 
    +@click.group(name='cas', short_help="Interact with the CAS server.")
    34 
    +@click.option('--port', type=click.INT, default='50051', show_default=True,
    35 
    +              help="Remote server's port number.")
    36 
    +@click.option('--host', type=click.STRING, default='localhost', show_default=True,
    37 
    +              help="Remote server's hostname.")
    36 38 
     @pass_context
    37 39 
     def cli(context, host, port):
    38 40 
         context.logger = logging.getLogger(__name__)
    ... ... @@ -42,9 +44,10 @@ def cli(context, host, port):
    42 44 
         context.port = port
    43 45
    44 46
    45 
    -@cli.command('upload-files', short_help='Upload files')
    46 
    -@click.argument('files', nargs=-1, type=click.File('rb'))
    47 
    -@click.option('--instance-name', default='testing')
    47 
    +@cli.command('upload-files', short_help="Upload files to the CAS server.")
    48 
    +@click.option('--instance-name', type=click.STRING, default='testing', show_default=True,
    49 
    +              help="Targeted farm instance name.")
    50 
    +@click.argument('files', nargs=-1, type=click.File('rb'), required=True)
    48 51 
     @pass_context
    49 52 
     def upload_files(context, files, instance_name):
    50 53 
         stub = remote_execution_pb2_grpc.ContentAddressableStorageStub(context.channel)
    ... ... @@ -63,9 +66,10 @@ def upload_files(context, files, instance_name):
    63 66 
         context.logger.info("Response: {}".format(response))
    64 67
    65 68
    66 
    -@cli.command('upload-dir', short_help='Upload files')
    67 
    -@click.argument('directory')
    68 
    -@click.option('--instance-name', default='testing')
    69 
    +@cli.command('upload-dir', short_help="Upload a directory to the CAS server.")
    70 
    +@click.option('--instance-name', type=click.STRING, default='testing', show_default=True,
    71 
    +              help="Targeted farm instance name.")
    72 
    +@click.argument('directory', nargs=1, type=click.Path(), required=True)
    69 73 
     @pass_context
    70 74 
     def upload_dir(context, directory, instance_name):
    71 75 
         context.logger.info("Uploading directory to cas")

  • app/commands/cmd_execute.py
    ... ... @@ -37,9 +37,11 @@ from buildgrid._protos.google.longrunning import operations_pb2, operations_pb2_
    37 37 
     from ..cli import pass_context
    38 38
    39 39
    40 
    -@click.group(short_help='Simple execute client')
    41 
    -@click.option('--port', default='50051')
    42 
    -@click.option('--host', default='localhost')
    40 
    +@click.group(name='execute', short_help="Execute simple operations.")
    41 
    +@click.option('--port', type=click.INT, default='50051', show_default=True,
    42 
    +              help="Remote server's port number.")
    43 
    +@click.option('--host', type=click.STRING, default='localhost', show_default=True,
    44 
    +              help="Remote server's hostname.")
    43 45 
     @pass_context
    44 46 
     def cli(context, host, port):
    45 47 
         context.logger = logging.getLogger(__name__)
    ... ... @@ -49,10 +51,13 @@ def cli(context, host, port):
    49 51 
         context.port = port
    50 52
    51 53
    52 
    -@cli.command('request-dummy', short_help='Send a dummy action')
    53 
    -@click.option('--number', default=1)
    54 
    -@click.option('--instance-name', default='testing')
    55 
    -@click.option('--wait-for-completion', is_flag=True, help='Stream updates until jobs are completed')
    54 
    +@cli.command('request-dummy', short_help="Send a dummy action.")
    55 
    +@click.option('--instance-name', type=click.STRING, default='testing', show_default=True,
    56 
    +              help="Targeted farm instance name.")
    57 
    +@click.option('--number', type=click.INT, default=1, show_default=True,
    58 
    +              help="Number of request to send.")
    59 
    +@click.option('--wait-for-completion', is_flag=True,
    60 
    +              help="Stream updates until jobs are completed.")
    56 61 
     @pass_context
    57 62 
     def request_dummy(context, number, instance_name, wait_for_completion):
    58 63 
         action_digest = remote_execution_pb2.Digest()
    ... ... @@ -76,8 +81,8 @@ def request_dummy(context, number, instance_name, wait_for_completion):
    76 81 
                 context.logger.info(next(response))
    77 82
    78 83
    79 
    -@cli.command('status', short_help='Get the status of an operation')
    80 
    -@click.argument('operation-name')
    84 
    +@cli.command('status', short_help="Get the status of an operation.")
    85 
    +@click.argument('operation-name', nargs=1, type=click.STRING, required=True)
    81 86 
     @pass_context
    82 87 
     def operation_status(context, operation_name):
    83 88 
         context.logger.info("Getting operation status...")
    ... ... @@ -89,7 +94,7 @@ def operation_status(context, operation_name):
    89 94 
         context.logger.info(response)
    90 95
    91 96
    92 
    -@cli.command('list', short_help='List operations')
    97 
    +@cli.command('list', short_help="List operations.")
    93 98 
     @pass_context
    94 99 
     def list_operations(context):
    95 100 
         context.logger.info("Getting list of operations")
    ... ... @@ -107,8 +112,8 @@ def list_operations(context):
    107 112 
             context.logger.info(op)
    108 113
    109 114
    110 
    -@cli.command('wait', short_help='Streams an operation until it is complete')
    111 
    -@click.argument('operation-name')
    115 
    +@cli.command('wait', short_help="Streams an operation until it is complete.")
    116 
    +@click.argument('operation-name', nargs=1, type=click.STRING, required=True)
    112 117 
     @pass_context
    113 118 
     def wait_execution(context, operation_name):
    114 119 
         stub = remote_execution_pb2_grpc.ExecutionStub(context.channel)
    ... ... @@ -120,13 +125,15 @@ def wait_execution(context, operation_name):
    120 125 
             context.logger.info(stream)
    121 126
    122 127
    123 
    -@cli.command('command', short_help='Send a command to be executed')
    124 
    -@click.argument('input-root')
    125 
    -@click.argument('commands', nargs=-1)
    126 
    -@click.option('--output-file', nargs=2, type=(str, bool), multiple=True,
    127 
    -              help='{Expected output file, is_executeable flag}')
    128 
    -@click.option('--output-directory', default='testing', help='Output directory for output files')
    129 
    -@click.option('--instance-name', default='testing')
    128 
    +@cli.command('command', short_help="Send a command to be executed.")
    129 
    +@click.option('--instance-name', type=click.STRING, default='testing', show_default=True,
    130 
    +              help="Targeted farm instance name.")
    131 
    +@click.option('--output-file', nargs=2, type=(click.STRING, click.BOOL), multiple=True,
    132 
    +              help="Tuple of expected output file and is-executeable flag.")
    133 
    +@click.option('--output-directory', default='testing', show_default=True,
    134 
    +              help="Output directory for the output files.")
    135 
    +@click.argument('input-root', nargs=1, type=click.Path(), required=True)
    136 
    +@click.argument('commands', nargs=-1, type=click.STRING, required=True)
    130 137 
     @pass_context
    131 138 
     def command(context, input_root, commands, output_file, output_directory, instance_name):
    132 139 
         stub = remote_execution_pb2_grpc.ExecutionStub(context.channel)

  • app/commands/cmd_server.py
    ... ... @@ -39,34 +39,34 @@ from ..cli import pass_context
    39 39 
     _SIZE_PREFIXES = {'k': 2 ** 10, 'm': 2 ** 20, 'g': 2 ** 30, 't': 2 ** 40}
    40 40
    41 41
    42 
    -@click.group(short_help="Start local server")
    42 
    +@click.group(name='server', short_help="Start a local server instance.")
    43 43 
     @pass_context
    44 44 
     def cli(context):
    45 45 
         context.logger = logging.getLogger(__name__)
    46 46 
         context.logger.info("BuildGrid server booting up")
    47 47
    48 48
    49 
    -@cli.command('start', short_help="Starts server")
    50 
    -@click.option('--port', default='50051')
    51 
    -@click.option('--max-cached-actions', type=int, default=50,
    49 
    +@cli.command('start', short_help="Setup a new server instance.")
    50 
    +@click.option('--port', type=click.INT, default='50051', show_default=True,
    51 
    +              help="The port number to be listened.")
    52 
    +@click.option('--max-cached-actions', type=click.INT, default=50, show_default=True,
    52 53 
                   help="Maximum number of actions to keep in the ActionCache.")
    53 54 
     @click.option('--allow-update-action-result/--forbid-update-action-result',
    54 
    -              'allow_uar', default=True,
    55 
    +              'allow_uar', default=True, show_default=True,
    55 56 
                   help="Whether or not to allow clients to manually edit the action cache.")
    56 
    -@click.option('--cas',
    57 
    -              type=click.Choice(('lru', 's3', 'disk', 'with-cache')),
    58 
    -              help="CAS storage type to use.")
    59 
    -@click.option('--cas-cache',
    60 
    -              type=click.Choice(('lru', 's3', 'disk')),
    57 
    +@click.option('--cas', type=click.Choice(('lru', 's3', 'disk', 'with-cache')),
    58 
    +              help="The CAS storage type to use.")
    59 
    +@click.option('--cas-cache', type=click.Choice(('lru', 's3', 'disk')),
    61 60 
                   help="For --cas=with-cache, the CAS storage to use as the cache.")
    62 
    -@click.option('--cas-fallback',
    63 
    -              type=click.Choice(('lru', 's3', 'disk')),
    61 
    +@click.option('--cas-fallback', type=click.Choice(('lru', 's3', 'disk')),
    64 62 
                   help="For --cas=with-cache, the CAS storage to use as the fallback.")
    65 
    -@click.option('--cas-lru-size', help="For --cas=lru, the LRU cache's memory limit.")
    66 
    -@click.option('--cas-s3-bucket', help="For --cas=s3, the bucket name.")
    67 
    -@click.option('--cas-s3-endpoint', help="For --cas=s3, the endpoint URI.")
    68 
    -@click.option('--cas-disk-directory',
    69 
    -              type=click.Path(file_okay=False, dir_okay=True, writable=True),
    63 
    +@click.option('--cas-lru-size', type=click.STRING,
    64 
    +              help="For --cas=lru, the LRU cache's memory limit.")
    65 
    +@click.option('--cas-s3-bucket', type=click.STRING,
    66 
    +              help="For --cas=s3, the bucket name.")
    67 
    +@click.option('--cas-s3-endpoint', type=click.STRING,
    68 
    +              help="For --cas=s3, the endpoint URI.")
    69 
    +@click.option('--cas-disk-directory', type=click.Path(file_okay=False, dir_okay=True, writable=True),
    70 70 
                   help="For --cas=disk, the folder to store CAS blobs in.")
    71 71 
     @pass_context
    72 72 
     def start(context, port, max_cached_actions, allow_uar, cas, **cas_args):
    ... ... @@ -96,7 +96,7 @@ def start(context, port, max_cached_actions, allow_uar, cas, **cas_args):
    96 96 
             loop.close()
    97 97
    98 98
    99 
    -@cli.command('stop', short_help="Stops server")
    99 
    +@cli.command('stop', short_help="Request a server to teardown.")
    100 100 
     @pass_context
    101 101 
     def stop(context):
    102 102 
         context.logger.error("Not implemented yet")

  • buildgrid/server/action_cache.py
    ... ... @@ -17,9 +17,17 @@
    17 17
    18 18 
     """
    19 19 
     ActionCache
    20 
    -==================
    20 
    +===========
    21 21
    22 22 
     Implements a simple in-memory action cache.
    23 
    +
    24 
    +The action cache maps Action to their corresponding ActionResult. An
    25 
    +ActionResult may be found in cache, for any given Action, if that action has
    26 
    +already been executed.
    27 
    +
    28 
    +Note:
    29 
    +    Action and ActionResult are referenced by their Digest and mapping is stored
    30 
    +    in-memory.
    23 31 
     """
    24 32
    25 33 
     import collections
    ... ... @@ -28,15 +36,29 @@ from buildgrid._protos.build.bazel.remote.execution.v2 import remote_execution_p
    28 36
    29 37
    30 38 
     class ActionCache:
    39 
    +    """In-memory Action to ActionResult associative array.
    40 
    +    """
    31 41
    32 42 
         def __init__(self, storage, max_cached_actions):
    43 
    +        """Initialises a new ActionCache instance.
    44 
    +
    45 
    +        Args:
    46 
    +            storage (StorageABC): storage backend instance to be used.
    47 
    +            max_cached_actions (int): maximun number of entries to cache.
    48 
    +        """
    33 49 
             self._storage = storage
    34 50 
             self._max_cached_actions = max_cached_actions
    35 51 
             self._digest_map = collections.OrderedDict()
    36 52
    37 53 
         def get_action_result(self, action_digest):
    38 
    -        """Return the cached ActionResult for the given Action digest, or None
    39 
    -        if there isn't one.
    54 
    +        """Retrieves the cached ActionResult for the given Action digest.
    55 
    +
    56 
    +        Args:
    57 
    +            action_digest (Digest): digest of the Action to query.
    58 
    +
    59 
    +        Returns:
    60 
    +            The cached ActionResult matching the given Action digest or None if
    61 
    +            the nothing hass been cached yet for that Action.
    40 62 
             """
    41 63 
             key = (action_digest.hash, action_digest.size_bytes)
    42 64 
             if key in self._digest_map:
    ... ... @@ -50,8 +72,15 @@ class ActionCache:
    50 72 
             return None
    51 73
    52 74 
         def put_action_result(self, action_digest, action_result):
    53 
    -        """Add the given ActionResult to the cache for the given Action
    54 
    -        digest.
    75 
    +        """Stores an ActionResult in cache for the given Action digest.
    76 
    +
    77 
    +        If the cache size limit has been reached, the oldest cache entries will
    78 
    +        be dropped before insertion so that the cache size never exceeds the
    79 
    +        maximum numbers of entries allowed.
    80 
    +
    81 
    +        Args:
    82 
    +            action_digest (Digest): digest of the Action to select.
    83 
    +            action_result (ActionResult): result object to store.
    55 84 
             """
    56 85 
             if self._max_cached_actions == 0:
    57 86 
                 return
    ... ... @@ -64,8 +93,14 @@ class ActionCache:
    64 93 
             self._digest_map[key] = action_result_digest
    65 94
    66 95 
         def _blobs_still_exist(self, action_result):
    67 
    -        """Return True if all the CAS blobs referenced by the given
    68 
    -        ActionResult are present in CAS.
    96 
    +        """Checks CAS for ActionResult output blobs existance.
    97 
    +
    98 
    +        Args:
    99 
    +            action_result (ActionResult): ActionResult to search referenced
    100 
    +                output blobs for.
    101 
    +
    102 
    +        Returns:
    103 
    +            True if all referenced blobs are present in CAS, False otherwise.
    69 104 
             """
    70 105 
             blobs_needed = []
    71 106

  • docs/source/about.rst
    1 
    +.. include:: ../../README.rst
    \ No newline at end of file

  • docs/source/conf.py
    ... ... @@ -24,8 +24,8 @@ from _version import __version__
    24 24 
     # -- Project information -----------------------------------------------------
    25 25
    26 26 
     project = 'BuildGrid'
    27 
    -copyright = '2018, The BuildGrid Contributors'
    28 
    -author = 'The BuildGrid Contributors'
    27 
    +copyright = '2018, Bloomberg LP'
    28 
    +author = None
    29 29
    30 30 
     # The short X.Y version
    31 31 
     version = __version__
    ... ... @@ -45,7 +45,8 @@ release = __version__
    45 45 
     extensions = [
    46 46 
         'sphinx.ext.autodoc',
    47 47 
         'sphinx.ext.napoleon',
    48 
    -    'sphinx_click.ext'
    48 
    +    'sphinx_click.ext',
    49 
    +    'sphinxcontrib.apidoc'
    49 50 
     ]
    50 51
    51 52 
     # Add any paths that contain templates here, relative to this directory.
    ... ... @@ -76,6 +77,26 @@ exclude_patterns = []
    76 77 
     pygments_style = 'sphinx'
    77 78
    78 79
    80 
    +# -- Options for apidoc ------------------------------------------------------
    81 
    +
    82 
    +# The path to the module to document. This must be a path to a Python package.
    83 
    +# This path can be a path relative to the documentation source directory or an
    84 
    +# absolute path.
    85 
    +apidoc_module_dir = '../../buildgrid'
    86 
    +
    87 
    +# The output directory. If it does not exist, it is created. This path is
    88 
    +# relative to the documentation source directory.
    89 
    +apidoc_output_dir = 'api'
    90 
    +
    91 
    +# An optional list of modules to exclude. These should be paths relative to
    92 
    +# apidoc_module_dir. fnmatch-style wildcarding is supported.
    93 
    +apidoc_excluded_paths = ['_protos']
    94 
    +
    95 
    +# Put documentation for each module on its own page. Otherwise there will be
    96 
    +# one page per (sub)package.
    97 
    +apidoc_separate_modules = True
    98 
    +
    99 
    +
    79 100 
     # -- Options for HTML output -------------------------------------------------
    80 101
    81 102 
     # The theme to use for HTML and HTML Help pages.  See the documentation for

  • docs/source/contributing.rst
    1 
    +.. include:: ../../CONTRIBUTING.rst
    \ No newline at end of file

  • docs/source/index.rst
    ... ... @@ -3,19 +3,30 @@
    3 3 
        You can adapt this file completely to your liking, but it should at least
    4 4 
        contain the root `toctree` directive.
    5 5
    6 
    +
    6 7 
     BuildGrid's documentation
    7 8 
     =========================
    8 9
    10 
    +Remote execution service implementing Google's REAPI and RWAPI.
    11 
    +
    9 12 
     .. toctree::
    10 13 
        :maxdepth: 1
    11 14 
        :caption: Contents:
    12 15
    16 
    +   about.rst
    17 
    +   installation.rst
    18 
    +   using.rst
    19 
    +   reference.rst
    20 
    +   contributing.rst
    21 
    +
    13 22
    14 23 
     Resources
    15 24 
     ---------
    16 25
    17 
    -* Homepage: https://buildgrid.build
    18 
    -* GitLab repository: https://gitlab.com/BuildGrid/buildgrid
    19 
    -* Bug Tracking: https://gitlab.com/BuildGrid/buildgrid/issues
    20 
    -* Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
    21 
    -* IRC Channel: irc://chat.freenode.net/#buildgrid
    26 
    +- Homepage: https://buildgrid.build
    27 
    +- GitLab repository: https://gitlab.com/BuildGrid/buildgrid
    28 
    +- Bug tracking: https://gitlab.com/BuildGrid/buildgrid/issues
    29 
    +- Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
    30 
    +- Slack channel: https://buildteamworld.slack.com/messages/CC9MKC203 [`invite link`_]
    31 
    +
    32 
    +.. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU

  • docs/source/installation.rst
    1 
    +
    2 
    +.. _installation:
    3 
    +
    4 
    +Installation
    5 
    +============
    6 
    +
    7 
    +How to install BuildGrid onto your machine.
    8 
    +
    9 
    +.. note::
    10 
    +
    11 
    +   BuildGrid server currently only support *Linux*, *macOS* and *Windows*
    12 
    +   platforms are **not** supported.
    13 
    +
    14 
    +
    15 
    +.. _install-prerequisites:
    16 
    +
    17 
    +Prerequisites
    18 
    +-------------
    19 
    +
    20 
    +BuildGrid only supports ``python3 >= 3.5`` but has no system requirements. Main
    21 
    +Python dependencies, automatically handle during installation, includes:
    22 
    +
    23 
    +- `boto3`_: the Amazon Web Services (AWS) SDK for Python.
    24 
    +- `click`_: a Python composable command line library.
    25 
    +- `grpcio`_: Google's `gRPC`_ Python interface.
    26 
    +- `protobuf`_: Google's `protocol-buffers`_ Python interface.
    27 
    +
    28 
    +.. _boto3: https://pypi.org/project/boto3
    29 
    +.. _click: https://pypi.org/project/click
    30 
    +.. _grpcio: https://pypi.org/project/grpcio
    31 
    +.. _gRPC: https://grpc.io
    32 
    +.. _protobuf: https://pypi.org/project/protobuf
    33 
    +.. _protocol-buffers: https://developers.google.com/protocol-buffers
    34 
    +
    35 
    +
    36 
    +.. _source-install:
    37 
    +
    38 
    +Install from sources
    39 
    +--------------------
    40 
    +
    41 
    +BuildGrid has ``setuptools`` support. In order to install it to your home
    42 
    +directory, typically under ``~/.local``, simply run:
    43 
    +
    44 
    +.. code-block:: sh
    45 
    +
    46 
    +   git clone https://gitlab.com/BuildGrid/buildgrid.git && cd buildgrid
    47 
    +   pip3 install --user --editable .
    48 
    +
    49 
    +Additionally, and if your distribution does not already includes it, you may
    50 
    +have to adjust your ``PATH``, in ``~/.bashrc``, with:
    51 
    +
    52 
    +.. code-block:: sh
    53 
    +
    54 
    +   export PATH="${PATH}:${HOME}/.local/bin"
    55 
    +
    56 
    +.. note::
    57 
    +
    58 
    +   The ``setup.py`` script defines two extra targets, ``docs`` and ``tests``,
    59 
    +   that declare required dependency for, respectively, generating documentation
    60 
    +   and running unit-tests. They can be use as helpers for setting up a
    61 
    +   development environment. To use them simply run:
    62 
    +
    63 
    +   .. code-block:: sh
    64 
    +
    65 
    +      pip3 install --user --editable ".[docs,tests]"

  • docs/source/reference.rst
    1 
    +
    2 
    +.. _reference:
    3 
    +
    4 
    +Reference
    5 
    +=========
    6 
    +
    7 
    +This section contains BuildGrid API and CLI reference documentation.
    8 
    +
    9 
    +.. toctree::
    10 
    +   :maxdepth: 2
    11 
    +
    12 
    +   reference_api.rst
    13 
    +   reference_cli.rst

  • docs/source/reference_api.rst
    1 
    +
    2 
    +.. _api_reference:
    3 
    +
    4 
    +API reference
    5 
    +=============
    6 
    +
    7 
    +BuildGrid's Application Programming Interface (API) reference documentation.
    8 
    +
    9 
    +.. toctree::
    10 
    +   :maxdepth: 1
    11 
    +
    12 
    +   api/modules.rst

  • docs/source/reference_cli.rst
    1 
    +
    2 
    +.. _cli_reference:
    3 
    +
    4 
    +CLI reference
    5 
    +=============
    6 
    +
    7 
    +BuildGrid's Command Line Interface (CLI) reference documentation.
    8 
    +
    9 
    +----
    10 
    +
    11 
    +.. _invoking_bgd:
    12 
    +
    13 
    +.. click:: app:cli
    14 
    +   :prog: bgd
    15 
    +
    16 
    +----
    17 
    +
    18 
    +.. _invoking_bgd_bot:
    19 
    +
    20 
    +.. click:: app.commands.cmd_bot:cli
    21 
    +   :prog: bgd bot
    22 
    +
    23 
    +----
    24 
    +
    25 
    +.. _invoking_bgd_bot_buildbox:
    26 
    +
    27 
    +.. click:: app.commands.cmd_bot:run_buildbox
    28 
    +   :prog: bgd bot buildbot
    29 
    +
    30 
    +----
    31 
    +
    32 
    +.. _invoking_bgd_bot_dummy:
    33 
    +
    34 
    +.. click:: app.commands.cmd_bot:run_dummy
    35 
    +   :prog: bgd bot dummy
    36 
    +
    37 
    +----
    38 
    +
    39 
    +.. _invoking_bgd_bot_temp_directory:
    40 
    +
    41 
    +.. click:: app.commands.cmd_bot:run_temp_directory
    42 
    +   :prog: bgd bot temp-directory
    43 
    +
    44 
    +----
    45 
    +
    46 
    +.. _invoking_bgd_cas:
    47 
    +
    48 
    +.. click:: app.commands.cmd_cas:cli
    49 
    +   :prog: bgd cas
    50 
    +
    51 
    +----
    52 
    +
    53 
    +.. _invoking_bgd_cas_upload_dir:
    54 
    +
    55 
    +.. click:: app.commands.cmd_cas:upload_dir
    56 
    +   :prog: bgd cas upload-dir
    57 
    +
    58 
    +----
    59 
    +
    60 
    +.. _invoking_bgd_cas_upload_files:
    61 
    +
    62 
    +.. click:: app.commands.cmd_cas:upload_files
    63 
    +   :prog: bgd cas upload-files
    64 
    +
    65 
    +----
    66 
    +
    67 
    +.. _invoking_bgd_execute:
    68 
    +
    69 
    +.. click:: app.commands.cmd_execute:cli
    70 
    +   :prog: bgd execute
    71 
    +
    72 
    +----
    73 
    +
    74 
    +.. _invoking_bgd_execute_command:
    75 
    +
    76 
    +.. click:: app.commands.cmd_execute:command
    77 
    +   :prog: bgd execute command
    78 
    +
    79 
    +----
    80 
    +
    81 
    +.. _invoking_bgd_execute_list:
    82 
    +
    83 
    +.. click:: app.commands.cmd_execute:list_operations
    84 
    +   :prog: bgd execute list
    85 
    +
    86 
    +----
    87 
    +
    88 
    +.. _invoking_bgd_execute_ request_dummy:
    89 
    +
    90 
    +.. click:: app.commands.cmd_execute:request_dummy
    91 
    +   :prog: bgd execute request-dummy
    92 
    +
    93 
    +----
    94 
    +
    95 
    +.. _invoking_bgd_execute_status:
    96 
    +
    97 
    +.. click:: app.commands.cmd_execute:operation_status
    98 
    +   :prog: bgd execute status
    99 
    +
    100 
    +----
    101 
    +
    102 
    +.. _invoking_bgd_execute_wait:
    103 
    +
    104 
    +.. click:: app.commands.cmd_execute:wait_execution
    105 
    +   :prog: bgd execute wait
    106 
    +
    107 
    +----
    108 
    +
    109 
    +.. _invoking_bgd_server:
    110 
    +
    111 
    +.. click:: app.commands.cmd_server:cli
    112 
    +   :prog: bgd server
    113 
    +
    114 
    +----
    115 
    +
    116 
    +.. _invoking_bgd_server_start:
    117 
    +
    118 
    +.. click:: app.commands.cmd_server:start
    119 
    +   :prog: bgd server start
    120 
    +
    121 
    +----
    122 
    +
    123 
    +.. _invoking_bgd_server_stop:
    124 
    +
    125 
    +.. click:: app.commands.cmd_server:stop
    126 
    +   :prog: bgd server stop

  • docs/source/using.rst
    1 
    +
    2 
    +.. _using:
    3 
    +
    4 
    +Using
    5 
    +=====
    6 
    +
    7 
    +This section covers how to run an use the BuildGrid build service.
    8 
    +
    9 
    +.. toctree::
    10 
    +   :maxdepth: 2
    11 
    +
    12 
    +   using_dummy_build.rst
    13 
    +   using_simple_build.rst

  • docs/source/using_dummy_build.rst
    1 
    +
    2 
    +.. _dummy-build:
    3 
    +
    4 
    +Dummy build
    5 
    +===========
    6 
    +
    7 
    +In one terminal, start a server:
    8 
    +
    9 
    +.. code-block:: sh
    10 
    +
    11 
    +   bgd server start
    12 
    +
    13 
    +In another terminal, send a request for work:
    14 
    +
    15 
    +.. code-block:: sh
    16 
    +
    17 
    +   bgd execute request-dummy
    18 
    +
    19 
    +The stage should show as ``QUEUED`` as it awaits a bot to pick up the work:
    20 
    +
    21 
    +.. code-block:: sh
    22 
    +
    23 
    +   bgd execute list
    24 
    +
    25 
    +Create a bot session:
    26 
    +
    27 
    +.. code-block:: sh
    28 
    +
    29 
    +   bgd bot dummy
    30 
    +
    31 
    +Show the work as completed:
    32 
    +
    33 
    +.. code-block:: sh
    34 
    +
    35 
    +   bgd execute list

  • docs/source/using_simple_build.rst
    1 
    +
    2 
    +.. _simple-build:
    3 
    +
    4 
    +Simple build
    5 
    +============
    6 
    +
    7 
    +This example covers a simple build. The user will upload a directory containing
    8 
    +a C file and a command to the CAS. The bot will then fetch the uploaded
    9 
    +directory and command which will then be run inside a temporary directory. The
    10 
    +result will then be uploaded to the CAS and downloaded by the user. This is an
    11 
    +early demo and still lacks a few features such as symlink support and checking
    12 
    +to see if files exist in the CAS before executing a command.
    13 
    +
    14 
    +Create a new directory called `test-buildgrid/` and place the following C file
    15 
    +in it called `hello.c`:
    16 
    +
    17 
    +.. code-block:: C
    18 
    +
    19 
    +   #include <stdio.h>
    20 
    +   int main()
    21 
    +   {
    22 
    +     printf("Hello, World!\n");
    23 
    +     return 0;
    24 
    +   }
    25 
    +
    26 
    +Now start a BuildGrid server, passing it a directory it can write a CAS to:
    27 
    +
    28 
    +.. code-block:: sh
    29 
    +
    30 
    +   bgd server start --cas disk --cas-cache disk --cas-disk-directory /path/to/empty/directory
    31 
    +
    32 
    +Start the following bot session:
    33 
    +
    34 
    +.. code-block:: sh
    35 
    +
    36 
    +   bgd bot temp-directory
    37 
    +
    38 
    +Upload the directory containing the C file:
    39 
    +
    40 
    +.. code-block:: sh
    41 
    +
    42 
    +   bgd cas upload-dir /path/to/test-buildgrid
    43 
    +
    44 
    +Now we send an execution request to the bot with the name of the epxected
    45 
    +``output-file``, a boolean describing if it is executeable, the path to the
    46 
    +directory we uploaded in order to calculate the digest and finally the command
    47 
    +to run on the bot:
    48 
    +
    49 
    +.. code-block:: sh
    50 
    +
    51 
    +   bgd execute command --output-file hello True /path/to/test-buildgrid -- gcc -Wall hello.c -o hello
    52 
    +
    53 
    +The resulting executeable should have returned to a new directory called
    54 
    +``testing``.

    • —
    View it on GitLab.
    You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.

    [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]