[Notes] [Git][BuildGrid/buildgrid][mablanch/51-user-facing-docs] 6 commits: Turn existing instruction material into documentation



Title: GitLab

Martin Blanchard pushed to branch mablanch/51-user-facing-docs at BuildGrid / buildgrid

Commits:

19 changed files:

Changes:

  • .gitignore
    ... ... @@ -16,3 +16,7 @@ dist/
    16 16
     .coverage.*
    
    17 17
     .pylint.d/
    
    18 18
     .pytest_cache/
    
    19
    +
    
    20
    +# Documentation related things:
    
    21
    +docs/build/
    
    22
    +docs/source/api/

  • CONTRIBUTING.rst
    1
    -Contributing to BuildGrid
    
    2
    -=========================
    
    3
    -Some guidelines for people wanting to contribute. Also please always feel free to speak to us, we're very friendly :-)
    
    4 1
     
    
    5
    -Feature Additions
    
    6
    ------------------
    
    2
    +.. _contributing:
    
    3
    +
    
    4
    +Contributing
    
    5
    +============
    
    7 6
     
    
    8
    -We welcome contributions in the form of bug fixes or feature additions / enhancements. Please discuss with us before submitting anything, as we may well have some important context which will could help guide your efforts.  
    
    7
    +Some guidelines for people wanting to contribute. Also please always feel free
    
    8
    +to speak to us, we're very friendly :-)
    
    9 9
     
    
    10
    -Any major feature additions should be raised as a proposal on the `Mailing List <https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid/>`_ to be discussed, and then eventually followed up with an issue here on gitlab. We recommend that you propose the feature in advance of commencing work. We are also on irc - you can find us on #buildgrid on freenode.
    
    11 10
     
    
    12
    -The author of any patch is expected to take ownership of that code and is to support it for a reasonable time-frame. This means addressing any unforeseen side effects and quirks the feature may have introduced. More on this below in 'Granting Committer Access'.
    
    11
    +.. _feature-additions:
    
    13 12
     
    
    14
    -Patch Submissions
    
    13
    +Feature additions
    
    15 14
     -----------------
    
    16 15
     
    
    17
    -We will be running `trunk based development <https://trunkbaseddevelopment.com>`_. The idea behind this is that merge requests to the trunk will be small and made often, thus making the review and merge process as fast as possible. We do not want to end up with a huge backlog of outstanding merge requests. If possible,
    
    18
    -it is preferred that merge requests address specific points and clearly outline what problem they are solving.
    
    16
    +We welcome contributions in the form of bug fixes or feature additions. Please
    
    17
    +discuss with us before submitting anything, as we may well have some important
    
    18
    +context which will could help guide your efforts.
    
    19
    +
    
    20
    +Any major feature additions should be raised first as a proposal on the
    
    21
    +`BuildGrid mailing list`_ to be discussed, and then eventually followed up with
    
    22
    +an issue on GitLab. We recommend that you propose the feature in advance of
    
    23
    +commencing work.
    
    24
    +
    
    25
    +The author of any patch is expected to take ownership of that code and is to
    
    26
    +support it for a reasonable time-frame. This means addressing any unforeseen
    
    27
    +side effects and quirks the feature may have introduced. More on this below in
    
    28
    +:ref:`Committer access <committer-access>`.
    
    19 29
     
    
    20
    -Branches must be submitted as merge requests on gitlab and should be associated with an issue report on gitlab, whenever possible. If it's a small change, we'll accept an MR without it being associated to a gitlab issue, but generally we prefer an issue to be raised in advance. This is so that we can track the work that is currently in progress on the project - please see our Gitlab policy below.
    
    30
    +.. _BuildGrid mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
    
    31
    +
    
    32
    +
    
    33
    +.. _patch-submissions:
    
    34
    +
    
    35
    +Patch submissions
    
    36
    +-----------------
    
    21 37
     
    
    22
    -Each commit should address a specific gitlab issue number in the commit message. This is really important for provenance reasons.
    
    38
    +We are running `trunk based development`_. The idea behind this is that merge
    
    39
    +requests to the trunk will be small and made often, thus making the review and
    
    40
    +merge process as fast as possible. We do not want to end up with a huge backlog
    
    41
    +of outstanding merge requests. If possible, it is preferred that merge requests
    
    42
    +address specific points and clearly outline what problem they are solving.
    
    23 43
     
    
    24
    -Merge requests that are not yet ready for review must be prefixed with the `WIP:` identifier, but if we stick to trunk based development then the 'WIP:' identifier will not stay around for very long on a merge request.
    
    44
    +Branches must be submitted as merge requests (MR) on GitLab and should be
    
    45
    +associated with an issue, whenever possible. If it's a small change, we'll
    
    46
    +accept an MR without it being associated to an issue, but generally we prefer an
    
    47
    +issue to be raised in advance. This is so that we can track the work that is
    
    48
    +currently in progress on the project.
    
    25 49
     
    
    26
    -When a merge request is ready for review, please find someone willing to do the review (ideally a maintainer) and assign them the MR on gitlab, leaving a comment asking for their review. 
    
    50
    +Below is a list of good patch submission good practice:
    
    27 51
     
    
    28
    -Submitted branches should not contain a history of work done.
    
    52
    +- Each commit should address a specific issue number in the commit message. This
    
    53
    +  is really important for provenance reasons.
    
    54
    +- Merge requests that are not yet ready for review must be prefixed with the
    
    55
    +  ``WIP:`` identifier, but if we stick to trunk based development then the
    
    56
    +  ``WIP:`` identifier will not stay around for very long on a merge request.
    
    57
    +- When a merge request is ready for review, please find someone willing to do
    
    58
    +  the review (ideally a maintainer) and assign them the MR, leaving a comment
    
    59
    +  asking for their review.
    
    60
    +- Submitted branches should not contain a history of work done.
    
    61
    +- Unit tests should be a separate commit.
    
    62
    +
    
    63
    +.. _trunk based development: https://trunkbaseddevelopment.com
    
    29 64
     
    
    30
    -Unit tests should be a separate commit.
    
    31 65
     
    
    32 66
     Commit messages
    
    33 67
     ~~~~~~~~~~~~~~~
    
    34
    -Commit messages must be formatted with a brief summary line, optionally followed by an empty line and then a
    
    35
    -free form detailed description of the change.
    
    36
    -
    
    37
    -The summary line must start with what changed, followed by a colon and a very brief description of the
    
    38
    -change.
    
    39 68
     
    
    40
    -If there is an associated issue, it **must** be mentioned somewhere in the commit message.
    
    69
    +Commit messages must be formatted with a brief summary line, optionally followed
    
    70
    +by an empty line and then a free form detailed description of the change. The
    
    71
    +summary line must start with what changed, followed by a colon and a very brief
    
    72
    +description of the change. If there is an associated issue, it **must** be
    
    73
    +mentioned somewhere in the commit message.
    
    41 74
     
    
    42 75
     **Example**::
    
    43 76
     
    
    44
    -  worker.py: Fixed to be more human than human
    
    77
    +   worker.py: Fixed to be more human than human
    
    78
    +
    
    79
    +   Gifted the worker with a past so we can create
    
    80
    +   a cushion or a pillow for their emotions and
    
    81
    +   consequently, we can control them better.
    
    82
    +
    
    83
    +   This fixes issue #8.
    
    45 84
     
    
    46
    -  Gifted the worker with a past so we can create
    
    47
    -  a cushion or a pillow for their emotions and
    
    48
    -  consequently, we can control them better.
    
    49
    -  
    
    50
    -  This fixes issue #8
    
    85
    +For more tips, please read `The seven rules of a great Git commit message`_.
    
    51 86
     
    
    52
    -  
    
    53
    -For more tips, please see `this <https://chris.beams.io/posts/git-commit/#seven-rules/>`_ article.
    
    87
    +.. _The seven rules of a great Git commit message: https://chris.beams.io/posts/git-commit/#seven-rules
    
    88
    +
    
    89
    +
    
    90
    +.. _coding-style:
    
    54 91
     
    
    55 92
     Coding style
    
    56 93
     ------------
    
    57
    -Coding style details for BuildGrid.
    
    58 94
     
    
    95
    +Python coding style for BuildGrid is `PEP 8`_. We do have a couple of minor
    
    96
    +exceptions to this standard, we dont want to compromise code readability by
    
    97
    +being overly restrictive on line length for instance.
    
    98
    +
    
    99
    +BuildGrid's test suite includes a PEP8 style compliance check phase (using
    
    100
    +`pep8`_) and a code linting phase (using `pylint`_). That test suite is
    
    101
    +automatically run for every change submitted to the GitLab server and the merge
    
    102
    +request sytem requires the test suite execution to succed before changes can
    
    103
    +be pulled upstream. This means you have to respect the BuildGrid coding style.
    
    59 104
     
    
    60
    -Style guide
    
    61
    -~~~~~~~~~~~
    
    62
    -Python coding style for BuildGrid is pep8, which is documented here: https://www.python.org/dev/peps/pep-0008/
    
    105
    +Configuration and exceptions for ``pep8`` and ``pylint`` can be found in:
    
    63 106
     
    
    64
    -We have a couple of minor exceptions to this standard, we dont want to compromise
    
    65
    -code readability by being overly restrictive on line length for instance.
    
    107
    +- The `setup.cfg`_ file for ``pep8``.
    
    108
    +- The `.pylintrc`_ file for ``pylint``.
    
    109
    +
    
    110
    +.. _PEP 8: https://www.python.org/dev/peps/pep-0008
    
    111
    +.. _pep8: https://pep8.readthedocs.io
    
    112
    +.. _pylint: https://pylint.readthedocs.io
    
    113
    +.. _setup.cfg: https://gitlab.com/BuildGrid/buildgrid/blob/master/setup.cfg
    
    114
    +.. _.pylintrc: https://gitlab.com/BuildGrid/buildgrid/blob/master/.pylintrc
    
    66 115
     
    
    67 116
     
    
    68 117
     Imports
    
    69 118
     ~~~~~~~
    
    119
    +
    
    70 120
     Module imports inside BuildGrid are done with relative ``.`` notation
    
    71 121
     
    
    72 122
     Good::
    
    ... ... @@ -77,99 +127,85 @@ Bad::
    77 127
     
    
    78 128
       from buildgrid.worker import Worker
    
    79 129
     
    
    80
    -Ordering
    
    81
    -''''''''
    
    82
    -For better readability and consistency, we try to keep private symbols below
    
    83
    -public symbols. In the case of public modules where we may have a mix of
    
    84
    -*API private* and *local private* symbols, *API private* symbols should come
    
    85
    -before *local private* symbols.
    
    86
    -
    
    87 130
     
    
    88 131
     Symbol naming
    
    89 132
     '''''''''''''
    
    90
    -Any private symbol must start with a single leading underscore for two reasons:
    
    91
    -
    
    92
    -* So that it does not bleed into documentation and *public API*.
    
    93
    -
    
    94
    -* So that it is clear to developers which symbols are not used outside of the declaring *scope*
    
    95
    -
    
    96
    -Remember that with python, the modules (python files) are also symbols
    
    97
    -within their containing *package*, as such; modules which are entirely
    
    98
    -private to BuildGrid are named as such, e.g. ``_roy.py``.
    
    99 133
     
    
    100
    -Granting Committer Access
    
    101
    --------------------------
    
    102
    -
    
    103
    -We'll hand out commit access to anyone who has successfully landed a single patch to the code base. Please request this via irc or the mailing list.
    
    104
    -
    
    105
    -This of course relies on contributors being responsive and show willingness to address problems after landing branches there should not be any problems here.
    
    106
    -
    
    107
    -What we are expecting of committers here in general is basically to
    
    108
    -escalate the review in cases of uncertainty:
    
    109
    -
    
    110
    -* If the patch/branch is very trivial (obvious few line changes or typos etc), and you are confident of the change, there is no need for review.
    
    111
    -
    
    112
    -* If the patch/branch is non trivial, please obtain a review from another committer who is familiar with the area which the branch effects. An approval from someone who is not the patch author will be needed before any merge. 
    
    113
    -
    
    114
    -We don't have any detailed policy for "bad actors", but will of course handle things on a case by case basis - commit access should not result in commit wars or be used as a tool to subvert the project when disagreements arise, such incidents (if any) would surely lead to temporary suspension of commit rights.
    
    134
    +Any private symbol must start with a single leading underscore for two reasons:
    
    115 135
     
    
    116
    -BuildGrid policy for use of Gitlab features
    
    117
    --------------------------------------------
    
    136
    +- So that it does not bleed into documentation and *public API*.
    
    137
    +- So that it is clear to developers which symbols are not used outside of the
    
    138
    +  declaring *scope*.
    
    118 139
     
    
    119
    -We intend to make use of some of gitlab's features in order to structure the activity of the BuildGrid project. In doing so we are trying to achieve the following goals:
    
    140
    +Remember that with python, the modules (python files) are also symbols within
    
    141
    +their containing *package*, as such; modules which are entirely private to
    
    142
    +BuildGrid are named as such, e.g. ``_roy.py``.
    
    120 143
     
    
    121
    -* Full transparency of the current WIP items 
    
    122
    -* Provide a view of all current and planned activity which is relatively easy for the viewer to digest
    
    123
    -* Ensure that we keep it simple and easy to contribute to the project
    
    124 144
     
    
    125
    -We propose to make use of the following Gitlab features:
    
    145
    +.. _committer-access:
    
    126 146
     
    
    127
    -* Milestones
    
    128
    -* Labels
    
    129
    -* Boards
    
    130
    -* Templates
    
    147
    +Committer access
    
    148
    +----------------
    
    131 149
     
    
    132
    -Milestones
    
    133
    -~~~~~~~~~~
    
    134
    -`Milestones <https://docs.gitlab.com/ee/user/project/milestones/>`_ are based on periods of time and what we want to achieve within those periods of time.
    
    150
    +We'll hand out commit access to anyone who has successfully landed a single
    
    151
    +patch to the code base. Please request this via irc or the mailing list.
    
    135 152
     
    
    136
    -We have seen them used in the same way as `Epics <https://docs.gitlab.com/ee/user/group/epics/index.html#doc-nav/>`_ in other projects (since the Epic feature is only available with GitLab Ultimate) and this does not work. Milestones must be time-line based.
    
    153
    +This of course relies on contributors being responsive and show willingness to
    
    154
    +address problems after landing branches there should not be any problems here.
    
    137 155
     
    
    138
    -Milestones can overlap, and we can be working towards multiple milestones at any one time. They allow us to group together all sub tasks into an overall aim.
    
    156
    +What we are expecting of committers here in general is basically to escalate the
    
    157
    +review in cases of uncertainty:
    
    139 158
     
    
    140
    -Labels
    
    141
    -~~~~~~
    
    142
    -`Labels <https://docs.gitlab.com/ee/user/project/labels.html/>`_ allow us to filter tickets on gitlab in useful ways. They do complexity and effort as they grow in number and usage, though, so the general approach is to have the minimum possible.
    
    159
    +- If the change is very trivial (obvious few line changes, typos…), and you are
    
    160
    +  confident of the change, there is no need for review.
    
    161
    +- If the change is non trivial, please obtain a review from another committer
    
    162
    +  who is familiar with the area which the branch effects. An approval from
    
    163
    +  someone who is not the patch author will be needed before any merge.
    
    143 164
     
    
    144
    -Type Labels
    
    145
    -'''''''''''
    
    146
    -We have:
    
    165
    +.. note::
    
    147 166
     
    
    148
    -* Bug
    
    149
    -* Documentation
    
    150
    -* Enhancement
    
    151
    -* Tests
    
    167
    +   We don't have any detailed policy for "bad actors", but will of course handle
    
    168
    +   things on a case by case basis - commit access should not result in commit
    
    169
    +   wars or be used as a tool to subvert the project when disagreements arise.
    
    170
    +   Such incidents (if any) would surely lead to temporary suspension of commit
    
    171
    +   rights.
    
    152 172
     
    
    153
    -This is useful for filtering different types of issues. We may expand this at some point.
    
    154 173
     
    
    155
    -Priority Labels
    
    156
    -'''''''''''''''
    
    157
    -For now, we only have 'High Priority', which indicates an urgent task. We may add more granularity if we get more contributors. 
    
    174
    +.. _gitlab-features:
    
    158 175
     
    
    159
    -Status
    
    160
    -'''''
    
    161
    -We have:
    
    176
    +GitLab features
    
    177
    +---------------
    
    162 178
     
    
    163
    -* ToDo
    
    164
    -* Doing
    
    179
    +We intend to make use of some of GitLab's features in order to structure the
    
    180
    +activity of the BuildGrid project. In doing so we are trying to achieve the
    
    181
    +following goals:
    
    165 182
     
    
    166
    -These labels are used when structuring tickets on a Board. GitLab issues start life in the 'Backlog' column by default, and we move them into 'ToDo' when they are coming up in the next few weeks. 'Doing' is only for when an item is currently being worked on. These labels don't have to be manually applied, they are applied by GitLab when moving the issue from column to column when using a Board - see below.
    
    183
    +- Full transparency of the current work in progress items.
    
    184
    +- Provide a view of all current and planned activity which is relatively easy
    
    185
    +  for the viewer to digest.
    
    186
    +- Ensure that we keep it simple and easy to contribute to the project.
    
    167 187
     
    
    168
    -Issue Boards
    
    169
    -~~~~~~~~~~~~
    
    170
    -`Boards <https://docs.gitlab.com/ee/user/project/issue_board.html#doc-nav/>`_ allow you to visualise and manage issues in a simple way, and we can create different types of board by filtering labels. For now, we are just utilising Boards in order to be able to see all of the currently in flight items at a glance.
    
    188
    +We are currenlty using the following GitLab features:
    
    171 189
     
    
    172
    -Templates
    
    173
    -~~~~~~~~~
    
    174
    -`Issue templates <https://docs.gitlab.com/ee/user/project/description_templates.html#doc-nav/>`_ help us to receive good quality information in issues.
    
    190
    +- `Milestones`_: we have seen them used in the same way as `Epics`_ in other
    
    191
    +  projects. BuildGrid milestones must be time-line based, can overlap and we can
    
    192
    +  be working towards multiple milestones at any one time. They allow us to group
    
    193
    +  together all sub tasks into an overall aim. See our `BuildGrid milestones`_.
    
    194
    +- `Labels`_: allow us to filter tickets in useful ways. They do complexity and
    
    195
    +  effort as they grow in number and usage, though, so the general approach is
    
    196
    +  to have the minimum possible. See our `BuildGrid labels`_.
    
    197
    +- `Boards`_: allow us to visualise and manage issues and labels in a simple way.
    
    198
    +  For now, we are only utilising one boards. Issues start life in the
    
    199
    +  ``Backlog`` column by default, and we move them into ``ToDo`` when they are
    
    200
    +  coming up in the next few weeks. ``Doing`` is only for when an item is
    
    201
    +  currently being worked on. Moving an issue from column to column automatically
    
    202
    +  adjust the tagged labels. See our `BuildGrid boards`_.
    
    175 203
     
    
    204
    +.. _Milestones: https://docs.gitlab.com/ee/user/project/milestones
    
    205
    +.. _Epics: https://docs.gitlab.com/ee/user/group/epics
    
    206
    +.. _BuildGrid milestones: https://gitlab.com/BuildGrid/buildgrid/milestones
    
    207
    +.. _Labels: https://docs.gitlab.com/ee/user/project/labels.html
    
    208
    +.. _BuildGrid labels: https://gitlab.com/BuildGrid/buildgrid/labels
    
    209
    +.. _Boards: https://docs.gitlab.com/ee/user/project/issue_board.html
    
    210
    +.. _BuildGrid boards: https://gitlab.com/BuildGrid/buildgrid/boards
    
    211
    +.. _Templates: https://docs.gitlab.com/ee/user/project/description_templates.html

  • README.rst
    1
    -BuildGrid
    
    2
    -=========
    
    1
    +
    
    2
    +.. _about:
    
    3
    +
    
    4
    +About
    
    5
    +=====
    
    3 6
     
    
    4 7
     .. image:: https://gitlab.com/Buildgrid/buildgrid/badges/master/pipeline.svg
    
    5 8
        :target: https://gitlab.com/BuildStream/buildstream/commits/master
    
    ... ... @@ -7,79 +10,40 @@ BuildGrid
    7 10
     .. image:: https://gitlab.com/BuildGrid/buildgrid/badges/master/coverage.svg?job=coverage
    
    8 11
        :target: https://gitlab.com/BuildGrid/buildgrid/commits/master
    
    9 12
     
    
    13
    +BuildGrid is a Python remote execution service which implements Google's
    
    14
    +`Remote Execution API`_ and the `Remote Workers API`_. The project's goal is to
    
    15
    +be able to execute build jobs remotely on a grid of computers in order to
    
    16
    +massively speed up build times. Workers on the grid should be able to run with
    
    17
    +different environments. It is designed to work with but not exclusively
    
    18
    +`BuildStream`_.
    
    10 19
     
    
    11
    -BuildGrid is a python remote execution service which implements the `Remote Execution API <https://github.com/bazelbuild/remote-apis//>`_ and the `Remote Workers API <https://docs.google.com/document/d/1s_AzRRD2mdyktKUj2HWBn99rMg_3tcPvdjx3MPbFidU/edit#heading=h.1u2taqr2h940/>`_.
    
    12
    -
    
    13
    -The goal is to be able to execute build jobs remotely on a grid of computers to massively speed up build times. Workers on the system will also be able to run with different environments. It is designed to work with but not exclusively with `BuildStream <https://wiki.gnome.org/Projects/BuildStream/>`_.
    
    14
    -
    
    15
    -Install
    
    16
    --------
    
    17
    -
    
    18
    -To install::
    
    19
    -
    
    20
    -   git clone https://gitlab.com/BuildGrid/buildgrid.git
    
    21
    -   cd buildgrid
    
    22
    -   pip3 install --user -e .
    
    23
    -
    
    24
    -This will install BuildGrid's python dependencies into your user’s homedir in ~/.local
    
    25
    -and will run BuildGrid directly from the git checkout. It is recommended you adjust
    
    26
    -your path with::
    
    27
    -
    
    28
    -  export PATH="${PATH}:${HOME}/.local/bin"
    
    29
    -
    
    30
    -Which you can add to the end of your `~/.bashrc`.
    
    31
    -
    
    32
    -Instructions for a Dummy Work
    
    33
    -----------------------
    
    34
    -
    
    35
    -In one terminal, start a server::
    
    36
    -
    
    37
    -  bgd server start
    
    38
    -
    
    39
    -In another terminal, send a request for work::
    
    40
    -
    
    41
    -  bgd execute request-dummy
    
    42
    -
    
    43
    -The stage should show as `QUEUED` as it awaits a bot to pick up the work::
    
    44
    -
    
    45
    -  bgd execute list
    
    46
    -
    
    47
    -Create a bot session::
    
    48
    -
    
    49
    -  bgd bot dummy
    
    50
    -
    
    51
    -Show the work as completed::
    
    52
    -
    
    53
    -  bgd execute list
    
    54
    -
    
    55
    -Instructions for a Simple Build
    
    56
    --------------------------------
    
    57
    -
    
    58
    -This example covers a simple build. The user will upload a directory containing a C file and a command to the CAS. The bot will then fetch the uploaded directory and command which will then be run inside a temporary directory. The result will then be uploaded to the CAS and downloaded by the user. This is an early demo and still lacks a few features such as symlink support and checking to see if files exist in the CAS before executing a command.
    
    59
    -
    
    60
    -Create a new directory called `test-buildgrid/` and place the following C file in it called `hello.c`::
    
    20
    +.. _Remote Execution API: https://github.com/bazelbuild/remote-apis
    
    21
    +.. _Remote Workers API: https://docs.google.com/document/d/1s_AzRRD2mdyktKUj2HWBn99rMg_3tcPvdjx3MPbFidU/edit#heading=h.1u2taqr2h940
    
    22
    +.. _BuildStream: https://wiki.gnome.org/Projects/BuildStream
    
    61 23
     
    
    62
    -  #include <stdio.h>
    
    63
    -  int main()
    
    64
    -  {
    
    65
    -   printf("Hello, World!\n");
    
    66
    -   return 0;
    
    67
    -  }
    
    68 24
     
    
    69
    -Now start a BuildGrid server, passing it a directory it can write a CAS to::
    
    25
    +.. _getting-started:
    
    70 26
     
    
    71
    -  bgd server start --cas disk --cas-cache disk --cas-disk-directory /path/to/empty/directory
    
    27
    +Getting started
    
    28
    +---------------
    
    72 29
     
    
    73
    -Start the following bot session::
    
    30
    +Please refer to the `documentation`_ for `installation`_ and `usage`_
    
    31
    +instructions.
    
    74 32
     
    
    75
    -  bgd bot temp-directory
    
    33
    +.. _documentation: https://buildgrid.gitlab.io/buildgrid
    
    34
    +.. _installation: https://buildgrid.gitlab.io/buildgrid/installation.html
    
    35
    +.. _usage: https://buildgrid.gitlab.io/buildgrid/using.html
    
    76 36
     
    
    77
    -Upload the directory containing the C file::
    
    78 37
     
    
    79
    -  bgd cas upload-dir /path/to/test-buildgrid
    
    38
    +.. _about-resources:
    
    80 39
     
    
    81
    -Now we send an execution request to the bot with the name of the epxected `output-file`, a boolean describing if it is executeable, the path to the directory we uploaded in order to calculate the digest and finally the command to run on the bot::
    
    40
    +Resources
    
    41
    +---------
    
    82 42
     
    
    83
    -  bgd execute command --output-file hello True /path/to/test-buildgrid -- gcc -Wall hello.c -o hello
    
    43
    +- Homepage: https://buildgrid.build
    
    44
    +- GitLab repository: https://gitlab.com/BuildGrid/buildgrid
    
    45
    +- Bug tracking: https://gitlab.com/BuildGrid/buildgrid/issues
    
    46
    +- Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
    
    47
    +- Slack channel: https://buildteamworld.slack.com/messages/CC9MKC203 [`invite link`_]
    
    84 48
     
    
    85
    -The resulting executeable should have returned to a new directory called `testing/`
    49
    +.. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU

  • app/commands/cmd_bot.py
    ... ... @@ -36,10 +36,13 @@ from ..bots import buildbox, dummy, temp_directory
    36 36
     from ..cli import pass_context
    
    37 37
     
    
    38 38
     
    
    39
    -@click.group(short_help="Create a bot client")
    
    40
    -@click.option('--parent', default='bgd_test')
    
    41
    -@click.option('--port', default='50051')
    
    42
    -@click.option('--host', default='localhost')
    
    39
    +@click.group(name='bot', short_help="Create and register bot clients.")
    
    40
    +@click.option('--parent', type=click.STRING, default='bgd_test', show_default=True,
    
    41
    +              help="Targeted farm resource.")
    
    42
    +@click.option('--port', type=click.INT, default='50051', show_default=True,
    
    43
    +              help="Remote server's port number.")
    
    44
    +@click.option('--host', type=click.STRING, default='localhost', show_default=True,
    
    45
    +              help="Renote server's hostname.")
    
    43 46
     @pass_context
    
    44 47
     def cli(context, host, port, parent):
    
    45 48
         channel = grpc.insecure_channel('{}:{}'.format(host, port))
    
    ... ... @@ -58,7 +61,7 @@ def cli(context, host, port, parent):
    58 61
         context.bot_session = bot_session
    
    59 62
     
    
    60 63
     
    
    61
    -@cli.command('dummy', short_help='Create a dummy bot session which just returns lease')
    
    64
    +@cli.command('dummy', short_help="Run a dummy session simply returning leases.")
    
    62 65
     @pass_context
    
    63 66
     def run_dummy(context):
    
    64 67
         """
    
    ... ... @@ -69,13 +72,13 @@ def run_dummy(context):
    69 72
             b = bot.Bot(context.bot_session)
    
    70 73
             b.session(dummy.work_dummy,
    
    71 74
                       context)
    
    72
    -
    
    73 75
         except KeyboardInterrupt:
    
    74 76
             pass
    
    75 77
     
    
    76 78
     
    
    77
    -@cli.command('temp-directory', short_help='Runs commands in temp directory and uploads results')
    
    78
    -@click.option('--instance-name', default='testing')
    
    79
    +@cli.command('temp-directory', short_help="Runs commands in temp directory and uploads results.")
    
    80
    +@click.option('--instance-name', type=click.STRING, default='testing', show_default=True,
    
    81
    +              help="Targeted farm instance name.")
    
    79 82
     @pass_context
    
    80 83
     def run_temp_directory(context, instance_name):
    
    81 84
         """ Downloads files and command from CAS and runs
    
    ... ... @@ -86,19 +89,25 @@ def run_temp_directory(context, instance_name):
    86 89
             b = bot.Bot(context.bot_session)
    
    87 90
             b.session(temp_directory.work_temp_directory,
    
    88 91
                       context)
    
    89
    -
    
    90 92
         except KeyboardInterrupt:
    
    91 93
             pass
    
    92 94
     
    
    93 95
     
    
    94
    -@cli.command('buildbox', short_help="Create a bot session with busybox")
    
    95
    -@click.option('--fuse-dir', show_default=True, default=str(PurePath(Path.home(), 'fuse')))
    
    96
    -@click.option('--local-cas', show_default=True, default=str(PurePath(Path.home(), 'cas')))
    
    97
    -@click.option('--client-cert', show_default=True, default=str(PurePath(Path.home(), 'client.crt')))
    
    98
    -@click.option('--client-key', show_default=True, default=str(PurePath(Path.home(), 'client.key')))
    
    99
    -@click.option('--server-cert', show_default=True, default=str(PurePath(Path.home(), 'server.crt')))
    
    100
    -@click.option('--port', show_default=True, default=11001)
    
    101
    -@click.option('--remote', show_default=True, default='localhost')
    
    96
    +@cli.command('buildbox', short_help="Run commands using the BuildBox tool.")
    
    97
    +@click.option('--fuse-dir', type=click.Path(readable=False), default=str(PurePath(Path.home(), 'fuse')),
    
    98
    +              help="Main mount-point location.")
    
    99
    +@click.option('--local-cas', type=click.Path(readable=False), default=str(PurePath(Path.home(), 'cas')),
    
    100
    +              help="Local CAS cache directory.")
    
    101
    +@click.option('--client-cert', type=click.Path(readable=False), default=str(PurePath(Path.home(), 'client.crt')),
    
    102
    +              help="Public client certificate for TLS (PEM-encoded).")
    
    103
    +@click.option('--client-key', type=click.Path(readable=False), default=str(PurePath(Path.home(), 'client.key')),
    
    104
    +              help="Private client key for TLS (PEM-encoded).")
    
    105
    +@click.option('--server-cert', type=click.Path(readable=False), default=str(PurePath(Path.home(), 'server.crt')),
    
    106
    +              help="Public server certificate for TLS (PEM-encoded).")
    
    107
    +@click.option('--port', type=click.INT, default=11001, show_default=True,
    
    108
    +              help="Remote CAS server port.")
    
    109
    +@click.option('--remote', type=click.STRING, default='localhost', show_default=True,
    
    110
    +              help="Remote CAS server hostname.")
    
    102 111
     @pass_context
    
    103 112
     def run_buildbox(context, remote, port, server_cert, client_key, client_cert, local_cas, fuse_dir):
    
    104 113
         """
    
    ... ... @@ -116,7 +125,6 @@ def run_buildbox(context, remote, port, server_cert, client_key, client_cert, lo
    116 125
         context.fuse_dir = fuse_dir
    
    117 126
     
    
    118 127
         try:
    
    119
    -
    
    120 128
             b = bot.Bot(context.bot_session)
    
    121 129
             b.session(work=buildbox.work_buildbox,
    
    122 130
                       context=context)
    

  • app/commands/cmd_cas.py
    ... ... @@ -30,9 +30,11 @@ from buildgrid._protos.build.bazel.remote.execution.v2 import remote_execution_p
    30 30
     from ..cli import pass_context
    
    31 31
     
    
    32 32
     
    
    33
    -@click.group(short_help='Interact with the CAS')
    
    34
    -@click.option('--port', default='50051')
    
    35
    -@click.option('--host', default='localhost')
    
    33
    +@click.group(name='cas', short_help="Interact with the CAS server.")
    
    34
    +@click.option('--port', type=click.INT, default='50051', show_default=True,
    
    35
    +              help="Remote server's port number.")
    
    36
    +@click.option('--host', type=click.STRING, default='localhost', show_default=True,
    
    37
    +              help="Remote server's hostname.")
    
    36 38
     @pass_context
    
    37 39
     def cli(context, host, port):
    
    38 40
         context.logger = logging.getLogger(__name__)
    
    ... ... @@ -42,9 +44,10 @@ def cli(context, host, port):
    42 44
         context.port = port
    
    43 45
     
    
    44 46
     
    
    45
    -@cli.command('upload-files', short_help='Upload files')
    
    46
    -@click.argument('files', nargs=-1, type=click.File('rb'))
    
    47
    -@click.option('--instance-name', default='testing')
    
    47
    +@cli.command('upload-files', short_help="Upload files to the CAS server.")
    
    48
    +@click.option('--instance-name', type=click.STRING, default='testing', show_default=True,
    
    49
    +              help="Targeted farm instance name.")
    
    50
    +@click.argument('files', nargs=-1, type=click.File('rb'), required=True)
    
    48 51
     @pass_context
    
    49 52
     def upload_files(context, files, instance_name):
    
    50 53
         stub = remote_execution_pb2_grpc.ContentAddressableStorageStub(context.channel)
    
    ... ... @@ -63,9 +66,10 @@ def upload_files(context, files, instance_name):
    63 66
         context.logger.info("Response: {}".format(response))
    
    64 67
     
    
    65 68
     
    
    66
    -@cli.command('upload-dir', short_help='Upload files')
    
    67
    -@click.argument('directory')
    
    68
    -@click.option('--instance-name', default='testing')
    
    69
    +@cli.command('upload-dir', short_help="Upload a directory to the CAS server.")
    
    70
    +@click.option('--instance-name', type=click.STRING, default='testing', show_default=True,
    
    71
    +              help="Targeted farm instance name.")
    
    72
    +@click.argument('directory', nargs=1, type=click.Path(), required=True)
    
    69 73
     @pass_context
    
    70 74
     def upload_dir(context, directory, instance_name):
    
    71 75
         context.logger.info("Uploading directory to cas")
    

  • app/commands/cmd_execute.py
    ... ... @@ -37,9 +37,11 @@ from buildgrid._protos.google.longrunning import operations_pb2, operations_pb2_
    37 37
     from ..cli import pass_context
    
    38 38
     
    
    39 39
     
    
    40
    -@click.group(short_help='Simple execute client')
    
    41
    -@click.option('--port', default='50051')
    
    42
    -@click.option('--host', default='localhost')
    
    40
    +@click.group(name='execute', short_help="Execute simple operations.")
    
    41
    +@click.option('--port', type=click.INT, default='50051', show_default=True,
    
    42
    +              help="Remote server's port number.")
    
    43
    +@click.option('--host', type=click.STRING, default='localhost', show_default=True,
    
    44
    +              help="Remote server's hostname.")
    
    43 45
     @pass_context
    
    44 46
     def cli(context, host, port):
    
    45 47
         context.logger = logging.getLogger(__name__)
    
    ... ... @@ -49,10 +51,13 @@ def cli(context, host, port):
    49 51
         context.port = port
    
    50 52
     
    
    51 53
     
    
    52
    -@cli.command('request-dummy', short_help='Send a dummy action')
    
    53
    -@click.option('--number', default=1)
    
    54
    -@click.option('--instance-name', default='testing')
    
    55
    -@click.option('--wait-for-completion', is_flag=True, help='Stream updates until jobs are completed')
    
    54
    +@cli.command('request-dummy', short_help="Send a dummy action.")
    
    55
    +@click.option('--instance-name', type=click.STRING, default='testing', show_default=True,
    
    56
    +              help="Targeted farm instance name.")
    
    57
    +@click.option('--number', type=click.INT, default=1, show_default=True,
    
    58
    +              help="Number of request to send.")
    
    59
    +@click.option('--wait-for-completion', is_flag=True,
    
    60
    +              help="Stream updates until jobs are completed.")
    
    56 61
     @pass_context
    
    57 62
     def request_dummy(context, number, instance_name, wait_for_completion):
    
    58 63
         action_digest = remote_execution_pb2.Digest()
    
    ... ... @@ -76,8 +81,8 @@ def request_dummy(context, number, instance_name, wait_for_completion):
    76 81
                 context.logger.info(next(response))
    
    77 82
     
    
    78 83
     
    
    79
    -@cli.command('status', short_help='Get the status of an operation')
    
    80
    -@click.argument('operation-name')
    
    84
    +@cli.command('status', short_help="Get the status of an operation.")
    
    85
    +@click.argument('operation-name', nargs=1, type=click.STRING, required=True)
    
    81 86
     @pass_context
    
    82 87
     def operation_status(context, operation_name):
    
    83 88
         context.logger.info("Getting operation status...")
    
    ... ... @@ -89,7 +94,7 @@ def operation_status(context, operation_name):
    89 94
         context.logger.info(response)
    
    90 95
     
    
    91 96
     
    
    92
    -@cli.command('list', short_help='List operations')
    
    97
    +@cli.command('list', short_help="List operations.")
    
    93 98
     @pass_context
    
    94 99
     def list_operations(context):
    
    95 100
         context.logger.info("Getting list of operations")
    
    ... ... @@ -107,8 +112,8 @@ def list_operations(context):
    107 112
             context.logger.info(op)
    
    108 113
     
    
    109 114
     
    
    110
    -@cli.command('wait', short_help='Streams an operation until it is complete')
    
    111
    -@click.argument('operation-name')
    
    115
    +@cli.command('wait', short_help="Streams an operation until it is complete.")
    
    116
    +@click.argument('operation-name', nargs=1, type=click.STRING, required=True)
    
    112 117
     @pass_context
    
    113 118
     def wait_execution(context, operation_name):
    
    114 119
         stub = remote_execution_pb2_grpc.ExecutionStub(context.channel)
    
    ... ... @@ -120,13 +125,15 @@ def wait_execution(context, operation_name):
    120 125
             context.logger.info(stream)
    
    121 126
     
    
    122 127
     
    
    123
    -@cli.command('command', short_help='Send a command to be executed')
    
    124
    -@click.argument('input-root')
    
    125
    -@click.argument('commands', nargs=-1)
    
    126
    -@click.option('--output-file', nargs=2, type=(str, bool), multiple=True,
    
    127
    -              help='{Expected output file, is_executeable flag}')
    
    128
    -@click.option('--output-directory', default='testing', help='Output directory for output files')
    
    129
    -@click.option('--instance-name', default='testing')
    
    128
    +@cli.command('command', short_help="Send a command to be executed.")
    
    129
    +@click.option('--instance-name', type=click.STRING, default='testing', show_default=True,
    
    130
    +              help="Targeted farm instance name.")
    
    131
    +@click.option('--output-file', nargs=2, type=(click.STRING, click.BOOL), multiple=True,
    
    132
    +              help="Tuple of expected output file and is-executeable flag.")
    
    133
    +@click.option('--output-directory', default='testing', show_default=True,
    
    134
    +              help="Output directory for the output files.")
    
    135
    +@click.argument('input-root', nargs=1, type=click.Path(), required=True)
    
    136
    +@click.argument('commands', nargs=-1, type=click.STRING, required=True)
    
    130 137
     @pass_context
    
    131 138
     def command(context, input_root, commands, output_file, output_directory, instance_name):
    
    132 139
         stub = remote_execution_pb2_grpc.ExecutionStub(context.channel)
    

  • app/commands/cmd_server.py
    ... ... @@ -39,34 +39,34 @@ from ..cli import pass_context
    39 39
     _SIZE_PREFIXES = {'k': 2 ** 10, 'm': 2 ** 20, 'g': 2 ** 30, 't': 2 ** 40}
    
    40 40
     
    
    41 41
     
    
    42
    -@click.group(short_help="Start local server")
    
    42
    +@click.group(name='server', short_help="Start a local server instance.")
    
    43 43
     @pass_context
    
    44 44
     def cli(context):
    
    45 45
         context.logger = logging.getLogger(__name__)
    
    46 46
         context.logger.info("BuildGrid server booting up")
    
    47 47
     
    
    48 48
     
    
    49
    -@cli.command('start', short_help="Starts server")
    
    50
    -@click.option('--port', default='50051')
    
    51
    -@click.option('--max-cached-actions', type=int, default=50,
    
    49
    +@cli.command('start', short_help="Setup a new server instance.")
    
    50
    +@click.option('--port', type=click.INT, default='50051', show_default=True,
    
    51
    +              help="The port number to be listened.")
    
    52
    +@click.option('--max-cached-actions', type=click.INT, default=50, show_default=True,
    
    52 53
                   help="Maximum number of actions to keep in the ActionCache.")
    
    53 54
     @click.option('--allow-update-action-result/--forbid-update-action-result',
    
    54
    -              'allow_uar', default=True,
    
    55
    +              'allow_uar', default=True, show_default=True,
    
    55 56
                   help="Whether or not to allow clients to manually edit the action cache.")
    
    56
    -@click.option('--cas',
    
    57
    -              type=click.Choice(('lru', 's3', 'disk', 'with-cache')),
    
    58
    -              help="CAS storage type to use.")
    
    59
    -@click.option('--cas-cache',
    
    60
    -              type=click.Choice(('lru', 's3', 'disk')),
    
    57
    +@click.option('--cas', type=click.Choice(('lru', 's3', 'disk', 'with-cache')),
    
    58
    +              help="The CAS storage type to use.")
    
    59
    +@click.option('--cas-cache', type=click.Choice(('lru', 's3', 'disk')),
    
    61 60
                   help="For --cas=with-cache, the CAS storage to use as the cache.")
    
    62
    -@click.option('--cas-fallback',
    
    63
    -              type=click.Choice(('lru', 's3', 'disk')),
    
    61
    +@click.option('--cas-fallback', type=click.Choice(('lru', 's3', 'disk')),
    
    64 62
                   help="For --cas=with-cache, the CAS storage to use as the fallback.")
    
    65
    -@click.option('--cas-lru-size', help="For --cas=lru, the LRU cache's memory limit.")
    
    66
    -@click.option('--cas-s3-bucket', help="For --cas=s3, the bucket name.")
    
    67
    -@click.option('--cas-s3-endpoint', help="For --cas=s3, the endpoint URI.")
    
    68
    -@click.option('--cas-disk-directory',
    
    69
    -              type=click.Path(file_okay=False, dir_okay=True, writable=True),
    
    63
    +@click.option('--cas-lru-size', type=click.STRING,
    
    64
    +              help="For --cas=lru, the LRU cache's memory limit.")
    
    65
    +@click.option('--cas-s3-bucket', type=click.STRING,
    
    66
    +              help="For --cas=s3, the bucket name.")
    
    67
    +@click.option('--cas-s3-endpoint', type=click.STRING,
    
    68
    +              help="For --cas=s3, the endpoint URI.")
    
    69
    +@click.option('--cas-disk-directory', type=click.Path(file_okay=False, dir_okay=True, writable=True),
    
    70 70
                   help="For --cas=disk, the folder to store CAS blobs in.")
    
    71 71
     @pass_context
    
    72 72
     def start(context, port, max_cached_actions, allow_uar, cas, **cas_args):
    
    ... ... @@ -96,7 +96,7 @@ def start(context, port, max_cached_actions, allow_uar, cas, **cas_args):
    96 96
             loop.close()
    
    97 97
     
    
    98 98
     
    
    99
    -@cli.command('stop', short_help="Stops server")
    
    99
    +@cli.command('stop', short_help="Request a server to teardown.")
    
    100 100
     @pass_context
    
    101 101
     def stop(context):
    
    102 102
         context.logger.error("Not implemented yet")
    

  • buildgrid/server/action_cache.py
    ... ... @@ -17,9 +17,17 @@
    17 17
     
    
    18 18
     """
    
    19 19
     ActionCache
    
    20
    -==================
    
    20
    +===========
    
    21 21
     
    
    22 22
     Implements a simple in-memory action cache.
    
    23
    +
    
    24
    +The action cache maps Action to their corresponding ActionResult. An
    
    25
    +ActionResult may be found in cache, for any given Action, if that action has
    
    26
    +already been executed.
    
    27
    +
    
    28
    +Note:
    
    29
    +    Action and ActionResult are referenced by their Digest and mapping is stored
    
    30
    +    in-memory.
    
    23 31
     """
    
    24 32
     
    
    25 33
     import collections
    
    ... ... @@ -28,15 +36,29 @@ from buildgrid._protos.build.bazel.remote.execution.v2 import remote_execution_p
    28 36
     
    
    29 37
     
    
    30 38
     class ActionCache:
    
    39
    +    """In-memory Action to ActionResult associative array.
    
    40
    +    """
    
    31 41
     
    
    32 42
         def __init__(self, storage, max_cached_actions):
    
    43
    +        """Initialises a new ActionCache instance.
    
    44
    +
    
    45
    +        Args:
    
    46
    +            storage (StorageABC): storage backend instance to be used.
    
    47
    +            max_cached_actions (int): maximun number of entries to cache.
    
    48
    +        """
    
    33 49
             self._storage = storage
    
    34 50
             self._max_cached_actions = max_cached_actions
    
    35 51
             self._digest_map = collections.OrderedDict()
    
    36 52
     
    
    37 53
         def get_action_result(self, action_digest):
    
    38
    -        """Return the cached ActionResult for the given Action digest, or None
    
    39
    -        if there isn't one.
    
    54
    +        """Retrieves the cached ActionResult for the given Action digest.
    
    55
    +
    
    56
    +        Args:
    
    57
    +            action_digest (Digest): digest of the Action to query.
    
    58
    +
    
    59
    +        Returns:
    
    60
    +            The cached ActionResult matching the given Action digest or None if
    
    61
    +            the nothing hass been cached yet for that Action.
    
    40 62
             """
    
    41 63
             key = (action_digest.hash, action_digest.size_bytes)
    
    42 64
             if key in self._digest_map:
    
    ... ... @@ -50,8 +72,15 @@ class ActionCache:
    50 72
             return None
    
    51 73
     
    
    52 74
         def put_action_result(self, action_digest, action_result):
    
    53
    -        """Add the given ActionResult to the cache for the given Action
    
    54
    -        digest.
    
    75
    +        """Stores an ActionResult in cache for the given Action digest.
    
    76
    +
    
    77
    +        If the cache size limit has been reached, the oldest cache entries will
    
    78
    +        be dropped before insertion so that the cache size never exceeds the
    
    79
    +        maximum numbers of entries allowed.
    
    80
    +
    
    81
    +        Args:
    
    82
    +            action_digest (Digest): digest of the Action to select.
    
    83
    +            action_result (ActionResult): result object to store.
    
    55 84
             """
    
    56 85
             if self._max_cached_actions == 0:
    
    57 86
                 return
    
    ... ... @@ -64,8 +93,14 @@ class ActionCache:
    64 93
             self._digest_map[key] = action_result_digest
    
    65 94
     
    
    66 95
         def _blobs_still_exist(self, action_result):
    
    67
    -        """Return True if all the CAS blobs referenced by the given
    
    68
    -        ActionResult are present in CAS.
    
    96
    +        """Checks CAS for ActionResult output blobs existance.
    
    97
    +
    
    98
    +        Args:
    
    99
    +            action_result (ActionResult): ActionResult to search referenced
    
    100
    +                output blobs for.
    
    101
    +
    
    102
    +        Returns:
    
    103
    +            True if all referenced blobs are present in CAS, False otherwise.
    
    69 104
             """
    
    70 105
             blobs_needed = []
    
    71 106
     
    

  • docs/source/about.rst
    1
    +.. include:: ../../README.rst
    \ No newline at end of file

  • docs/source/conf.py
    ... ... @@ -45,7 +45,8 @@ release = __version__
    45 45
     extensions = [
    
    46 46
         'sphinx.ext.autodoc',
    
    47 47
         'sphinx.ext.napoleon',
    
    48
    -    'sphinx_click.ext'
    
    48
    +    'sphinx_click.ext',
    
    49
    +    'sphinxcontrib.apidoc'
    
    49 50
     ]
    
    50 51
     
    
    51 52
     # Add any paths that contain templates here, relative to this directory.
    
    ... ... @@ -76,6 +77,26 @@ exclude_patterns = []
    76 77
     pygments_style = 'sphinx'
    
    77 78
     
    
    78 79
     
    
    80
    +# -- Options for apidoc ------------------------------------------------------
    
    81
    +
    
    82
    +# The path to the module to document. This must be a path to a Python package.
    
    83
    +# This path can be a path relative to the documentation source directory or an
    
    84
    +# absolute path.
    
    85
    +apidoc_module_dir = '../../buildgrid'
    
    86
    +
    
    87
    +# The output directory. If it does not exist, it is created. This path is
    
    88
    +# relative to the documentation source directory.
    
    89
    +apidoc_output_dir = 'api'
    
    90
    +
    
    91
    +# An optional list of modules to exclude. These should be paths relative to
    
    92
    +# apidoc_module_dir. fnmatch-style wildcarding is supported.
    
    93
    +apidoc_excluded_paths = ['_protos']
    
    94
    +
    
    95
    +# Put documentation for each module on its own page. Otherwise there will be
    
    96
    +# one page per (sub)package.
    
    97
    +apidoc_separate_modules = True
    
    98
    +
    
    99
    +
    
    79 100
     # -- Options for HTML output -------------------------------------------------
    
    80 101
     
    
    81 102
     # The theme to use for HTML and HTML Help pages.  See the documentation for
    

  • docs/source/contributing.rst
    1
    +.. include:: ../../CONTRIBUTING.rst
    \ No newline at end of file

  • docs/source/index.rst
    ... ... @@ -3,19 +3,30 @@
    3 3
        You can adapt this file completely to your liking, but it should at least
    
    4 4
        contain the root `toctree` directive.
    
    5 5
     
    
    6
    +
    
    6 7
     BuildGrid's documentation
    
    7 8
     =========================
    
    8 9
     
    
    10
    +Remote execution service implementing Google's REAPI and RWAPI.
    
    11
    +
    
    9 12
     .. toctree::
    
    10 13
        :maxdepth: 1
    
    11 14
        :caption: Contents:
    
    12 15
     
    
    16
    +   about.rst
    
    17
    +   installation.rst
    
    18
    +   using.rst
    
    19
    +   reference.rst
    
    20
    +   contributing.rst
    
    21
    +
    
    13 22
     
    
    14 23
     Resources
    
    15 24
     ---------
    
    16 25
     
    
    17
    -* Homepage: https://buildgrid.build
    
    18
    -* GitLab repository: https://gitlab.com/BuildGrid/buildgrid
    
    19
    -* Bug Tracking: https://gitlab.com/BuildGrid/buildgrid/issues
    
    20
    -* Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
    
    21
    -* IRC Channel: irc://chat.freenode.net/#buildgrid
    26
    +- Homepage: https://buildgrid.build
    
    27
    +- GitLab repository: https://gitlab.com/BuildGrid/buildgrid
    
    28
    +- Bug tracking: https://gitlab.com/BuildGrid/buildgrid/issues
    
    29
    +- Mailing list: https://lists.buildgrid.build/cgi-bin/mailman/listinfo/buildgrid
    
    30
    +- Slack channel: https://buildteamworld.slack.com/messages/CC9MKC203 [`invite link`_]
    
    31
    +
    
    32
    +.. _invite link: https://join.slack.com/t/buildteamworld/shared_invite/enQtMzkxNzE0MDMyMDY1LTRmZmM1OWE0OTFkMGE1YjU5Njc4ODEzYjc0MGMyOTM5ZTQ5MmE2YTQ1MzQwZDc5MWNhODY1ZmRkZTE4YjFhNjU

  • docs/source/installation.rst
    1
    +
    
    2
    +.. _installation:
    
    3
    +
    
    4
    +Installation
    
    5
    +============
    
    6
    +
    
    7
    +How to install BuildGrid onto your machine.
    
    8
    +
    
    9
    +.. note::
    
    10
    +
    
    11
    +   BuildGrid server currently only support *Linux*, *macOS* and *Windows*
    
    12
    +   platforms are **not** supported.
    
    13
    +
    
    14
    +
    
    15
    +.. _install-prerequisites:
    
    16
    +
    
    17
    +Prerequisites
    
    18
    +-------------
    
    19
    +
    
    20
    +BuildGrid only supports ``python3 >= 3.5`` but has no system requirements. Main
    
    21
    +Python dependencies, automatically handle during installation, includes:
    
    22
    +
    
    23
    +- `boto3`_: the Amazon Web Services (AWS) SDK for Python.
    
    24
    +- `click`_: a Python composable command line library.
    
    25
    +- `grpcio`_: Google's `gRPC`_ Python interface.
    
    26
    +- `protobuf`_: Google's `protocol-buffers`_ Python interface.
    
    27
    +
    
    28
    +.. _boto3: https://pypi.org/project/boto3
    
    29
    +.. _click: https://pypi.org/project/click
    
    30
    +.. _grpcio: https://pypi.org/project/grpcio
    
    31
    +.. _gRPC: https://grpc.io
    
    32
    +.. _protobuf: https://pypi.org/project/protobuf
    
    33
    +.. _protocol-buffers: https://developers.google.com/protocol-buffers
    
    34
    +
    
    35
    +
    
    36
    +.. _source-install:
    
    37
    +
    
    38
    +Install from sources
    
    39
    +--------------------
    
    40
    +
    
    41
    +BuildGrid has ``setuptools`` support. In order to install it to your home
    
    42
    +directory, typically under ``~/.local``, simply run:
    
    43
    +
    
    44
    +.. code-block:: sh
    
    45
    +
    
    46
    +   git clone https://gitlab.com/BuildGrid/buildgrid.git && cd buildgrid
    
    47
    +   pip3 install --user --editable .
    
    48
    +
    
    49
    +Additionally, and if your distribution does not already includes it, you may
    
    50
    +have to adjust your ``PATH``, in ``~/.bashrc``, with:
    
    51
    +
    
    52
    +.. code-block:: sh
    
    53
    +
    
    54
    +   export PATH="${PATH}:${HOME}/.local/bin"
    
    55
    +
    
    56
    +.. note::
    
    57
    +
    
    58
    +   The ``setup.py`` script defines two extra targets, ``docs`` and ``tests``,
    
    59
    +   that declare required dependency for, respectively, generating documentation
    
    60
    +   and running unit-tests. They can be use as helpers for setting up a
    
    61
    +   development environment. To use them simply run:
    
    62
    +
    
    63
    +   .. code-block:: sh
    
    64
    +
    
    65
    +      pip3 install --user --editable ".[docs,tests]"

  • docs/source/reference.rst
    1
    +
    
    2
    +.. _reference:
    
    3
    +
    
    4
    +Reference
    
    5
    +=========
    
    6
    +
    
    7
    +This section contains BuildGrid API and CLI reference documentation.
    
    8
    +
    
    9
    +.. toctree::
    
    10
    +   :maxdepth: 2
    
    11
    +
    
    12
    +   reference_api.rst
    
    13
    +   reference_cli.rst

  • docs/source/reference_api.rst
    1
    +
    
    2
    +.. _api_reference:
    
    3
    +
    
    4
    +API reference
    
    5
    +=============
    
    6
    +
    
    7
    +BuildGrid's Application Programming Interface (API) reference documentation.
    
    8
    +
    
    9
    +.. toctree::
    
    10
    +   :maxdepth: 1
    
    11
    +
    
    12
    +   api/modules.rst

  • docs/source/reference_cli.rst
    1
    +
    
    2
    +.. _cli_reference:
    
    3
    +
    
    4
    +CLI reference
    
    5
    +=============
    
    6
    +
    
    7
    +BuildGrid's Command Line Interface (CLI) reference documentation.
    
    8
    +
    
    9
    +----
    
    10
    +
    
    11
    +.. _invoking_bgd:
    
    12
    +
    
    13
    +.. click:: app:cli
    
    14
    +   :prog: bgd
    
    15
    +
    
    16
    +----
    
    17
    +
    
    18
    +.. _invoking_bgd_bot:
    
    19
    +
    
    20
    +.. click:: app.commands.cmd_bot:cli
    
    21
    +   :prog: bgd bot
    
    22
    +
    
    23
    +----
    
    24
    +
    
    25
    +.. _invoking_bgd_bot_buildbox:
    
    26
    +
    
    27
    +.. click:: app.commands.cmd_bot:run_buildbox
    
    28
    +   :prog: bgd bot buildbot
    
    29
    +
    
    30
    +----
    
    31
    +
    
    32
    +.. _invoking_bgd_bot_dummy:
    
    33
    +
    
    34
    +.. click:: app.commands.cmd_bot:run_dummy
    
    35
    +   :prog: bgd bot dummy
    
    36
    +
    
    37
    +----
    
    38
    +
    
    39
    +.. _invoking_bgd_bot_temp_directory:
    
    40
    +
    
    41
    +.. click:: app.commands.cmd_bot:run_temp_directory
    
    42
    +   :prog: bgd bot temp-directory
    
    43
    +
    
    44
    +----
    
    45
    +
    
    46
    +.. _invoking_bgd_cas:
    
    47
    +
    
    48
    +.. click:: app.commands.cmd_cas:cli
    
    49
    +   :prog: bgd cas
    
    50
    +
    
    51
    +----
    
    52
    +
    
    53
    +.. _invoking_bgd_cas_upload_dir:
    
    54
    +
    
    55
    +.. click:: app.commands.cmd_cas:upload_dir
    
    56
    +   :prog: bgd cas upload-dir
    
    57
    +
    
    58
    +----
    
    59
    +
    
    60
    +.. _invoking_bgd_cas_upload_files:
    
    61
    +
    
    62
    +.. click:: app.commands.cmd_cas:upload_files
    
    63
    +   :prog: bgd cas upload-files
    
    64
    +
    
    65
    +----
    
    66
    +
    
    67
    +.. _invoking_bgd_execute:
    
    68
    +
    
    69
    +.. click:: app.commands.cmd_execute:cli
    
    70
    +   :prog: bgd execute
    
    71
    +
    
    72
    +----
    
    73
    +
    
    74
    +.. _invoking_bgd_execute_command:
    
    75
    +
    
    76
    +.. click:: app.commands.cmd_execute:command
    
    77
    +   :prog: bgd execute command
    
    78
    +
    
    79
    +----
    
    80
    +
    
    81
    +.. _invoking_bgd_execute_list:
    
    82
    +
    
    83
    +.. click:: app.commands.cmd_execute:list_operations
    
    84
    +   :prog: bgd execute list
    
    85
    +
    
    86
    +----
    
    87
    +
    
    88
    +.. _invoking_bgd_execute_ request_dummy:
    
    89
    +
    
    90
    +.. click:: app.commands.cmd_execute:request_dummy
    
    91
    +   :prog: bgd execute request-dummy
    
    92
    +
    
    93
    +----
    
    94
    +
    
    95
    +.. _invoking_bgd_execute_status:
    
    96
    +
    
    97
    +.. click:: app.commands.cmd_execute:operation_status
    
    98
    +   :prog: bgd execute status
    
    99
    +
    
    100
    +----
    
    101
    +
    
    102
    +.. _invoking_bgd_execute_wait:
    
    103
    +
    
    104
    +.. click:: app.commands.cmd_execute:wait_execution
    
    105
    +   :prog: bgd execute wait
    
    106
    +
    
    107
    +----
    
    108
    +
    
    109
    +.. _invoking_bgd_server:
    
    110
    +
    
    111
    +.. click:: app.commands.cmd_server:cli
    
    112
    +   :prog: bgd server
    
    113
    +
    
    114
    +----
    
    115
    +
    
    116
    +.. _invoking_bgd_server_start:
    
    117
    +
    
    118
    +.. click:: app.commands.cmd_server:start
    
    119
    +   :prog: bgd server start
    
    120
    +
    
    121
    +----
    
    122
    +
    
    123
    +.. _invoking_bgd_server_stop:
    
    124
    +
    
    125
    +.. click:: app.commands.cmd_server:stop
    
    126
    +   :prog: bgd server stop

  • docs/source/using.rst
    1
    +
    
    2
    +.. _using:
    
    3
    +
    
    4
    +Using
    
    5
    +=====
    
    6
    +
    
    7
    +This section covers how to run an use the BuildGrid build service.
    
    8
    +
    
    9
    +.. toctree::
    
    10
    +   :maxdepth: 2
    
    11
    +
    
    12
    +   using_dummy_build.rst
    
    13
    +   using_simple_build.rst

  • docs/source/using_dummy_build.rst
    1
    +
    
    2
    +.. _dummy-build:
    
    3
    +
    
    4
    +Dummy build
    
    5
    +===========
    
    6
    +
    
    7
    +In one terminal, start a server:
    
    8
    +
    
    9
    +.. code-block:: sh
    
    10
    +
    
    11
    +   bgd server start
    
    12
    +
    
    13
    +In another terminal, send a request for work:
    
    14
    +
    
    15
    +.. code-block:: sh
    
    16
    +
    
    17
    +   bgd execute request-dummy
    
    18
    +
    
    19
    +The stage should show as ``QUEUED`` as it awaits a bot to pick up the work:
    
    20
    +
    
    21
    +.. code-block:: sh
    
    22
    +
    
    23
    +   bgd execute list
    
    24
    +
    
    25
    +Create a bot session:
    
    26
    +
    
    27
    +.. code-block:: sh
    
    28
    +
    
    29
    +   bgd bot dummy
    
    30
    +
    
    31
    +Show the work as completed:
    
    32
    +
    
    33
    +.. code-block:: sh
    
    34
    +
    
    35
    +   bgd execute list

  • docs/source/using_simple_build.rst
    1
    +
    
    2
    +.. _simple-build:
    
    3
    +
    
    4
    +Simple build
    
    5
    +============
    
    6
    +
    
    7
    +This example covers a simple build. The user will upload a directory containing
    
    8
    +a C file and a command to the CAS. The bot will then fetch the uploaded
    
    9
    +directory and command which will then be run inside a temporary directory. The
    
    10
    +result will then be uploaded to the CAS and downloaded by the user. This is an
    
    11
    +early demo and still lacks a few features such as symlink support and checking
    
    12
    +to see if files exist in the CAS before executing a command.
    
    13
    +
    
    14
    +Create a new directory called `test-buildgrid/` and place the following C file
    
    15
    +in it called `hello.c`:
    
    16
    +
    
    17
    +.. code-block:: C
    
    18
    +
    
    19
    +   #include <stdio.h>
    
    20
    +   int main()
    
    21
    +   {
    
    22
    +     printf("Hello, World!\n");
    
    23
    +     return 0;
    
    24
    +   }
    
    25
    +
    
    26
    +Now start a BuildGrid server, passing it a directory it can write a CAS to:
    
    27
    +
    
    28
    +.. code-block:: sh
    
    29
    +
    
    30
    +   bgd server start --cas disk --cas-cache disk --cas-disk-directory /path/to/empty/directory
    
    31
    +
    
    32
    +Start the following bot session:
    
    33
    +
    
    34
    +.. code-block:: sh
    
    35
    +
    
    36
    +   bgd bot temp-directory
    
    37
    +
    
    38
    +Upload the directory containing the C file:
    
    39
    +
    
    40
    +.. code-block:: sh
    
    41
    +
    
    42
    +   bgd cas upload-dir /path/to/test-buildgrid
    
    43
    +
    
    44
    +Now we send an execution request to the bot with the name of the epxected
    
    45
    +``output-file``, a boolean describing if it is executeable, the path to the
    
    46
    +directory we uploaded in order to calculate the digest and finally the command
    
    47
    +to run on the bot:
    
    48
    +
    
    49
    +.. code-block:: sh
    
    50
    +
    
    51
    +   bgd execute command --output-file hello True /path/to/test-buildgrid -- gcc -Wall hello.c -o hello
    
    52
    +
    
    53
    +The resulting executeable should have returned to a new directory called
    
    54
    +``testing``.



  • [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]