[Notes] [Git][BuildGrid/buildgrid][jmac/expand-user-for-tls] 4 commits: CONTRIBUTING.rst: Add a sub-section about testing



Title: GitLab

finn pushed to branch jmac/expand-user-for-tls at BuildGrid / buildgrid

Commits:

8 changed files:

Changes:

  • CONTRIBUTING.rst
    ... ... @@ -142,13 +142,67 @@ their containing *package*, as such; modules which are entirely private to
    142 142
     BuildGrid are named as such, e.g. ``_roy.py``.
    
    143 143
     
    
    144 144
     
    
    145
    +.. _codebase-testing:
    
    146
    +
    
    147
    +Testing
    
    148
    +-------
    
    149
    +
    
    150
    +BuildGrid is using `pytest`_ for regression and newly added code testing. The
    
    151
    +test suite contains a serie of unit-tests and also run linting tools in order to
    
    152
    +detect coding-style_ breakage. The full test suite is automatically executed by
    
    153
    +GitLab CI system for every push to the server. Passing all the tests is a
    
    154
    +mandatory requirement for any merge request to the trunk.
    
    155
    +
    
    156
    +.. _pytest: https://docs.pytest.org
    
    157
    +
    
    158
    +
    
    159
    +Running tests
    
    160
    +~~~~~~~~~~~~~
    
    161
    +
    
    162
    +In order to run the entire test suite, simply run:
    
    163
    +
    
    164
    +.. code-block:: sh
    
    165
    +
    
    166
    +   python3 setup.py test
    
    167
    +
    
    168
    +You can use the ``--addopt`` function to feed arguments to pytest. For example,
    
    169
    +if you want to see the ``stdout`` and ``stderr`` generated y the test, run:
    
    170
    +
    
    171
    +.. code-block:: sh
    
    172
    +
    
    173
    +   python3 setup.py test  --addopts -s
    
    174
    +
    
    175
    +If you want run a  specific test instead of the entire suite use:
    
    176
    +
    
    177
    +.. code-block:: sh
    
    178
    +
    
    179
    +   python3 setup.py test  --addopts tests/cas/test_client
    
    180
    +
    
    181
    +pyest's `usage documentation section`_ details the different command line
    
    182
    +options that can be used when invoking the test runner.
    
    183
    +
    
    184
    +.. _usage documentation section: https://docs.pytest.org/en/latest/usage.html
    
    185
    +
    
    186
    +
    
    187
    +Test coverage
    
    188
    +~~~~~~~~~~~~~
    
    189
    +
    
    190
    +We are doing our best at keeping BuildGrid's test coverage score as high as
    
    191
    +possible. Doing so, we ask for any merge request to include necessary test
    
    192
    +additions and/or modifications in order to maintain that coverage level. A
    
    193
    +detailed `coverage report`_ is produced and publish for any change merged to the
    
    194
    +trunk.
    
    195
    +
    
    196
    +.. _coverage report: https://buildgrid.gitlab.io/buildgrid/coverage/
    
    197
    +
    
    198
    +
    
    145 199
     .. _committer-access:
    
    146 200
     
    
    147 201
     Committer access
    
    148 202
     ----------------
    
    149 203
     
    
    150 204
     We'll hand out commit access to anyone who has successfully landed a single
    
    151
    -patch to the code base. Please request this via irc or the mailing list.
    
    205
    +patch to the code base. Please request this via Slack or the mailing list.
    
    152 206
     
    
    153 207
     This of course relies on contributors being responsive and show willingness to
    
    154 208
     address problems after landing branches there should not be any problems here.
    

  • README.rst
    1
    -.. _about:
    
    2
    -
    
    3
    -
    
    4 1
     
    
    5 2
     .. image:: https://gitlab.com/Buildgrid/buildgrid/badges/master/pipeline.svg
    
    6 3
        :target: https://gitlab.com/BuildStream/buildstream/commits/master
    
    7 4
     
    
    8 5
     .. image:: https://gitlab.com/BuildGrid/buildgrid/badges/master/coverage.svg?job=coverage
    
    9 6
        :target: https://buildgrid.gitlab.io/buildgrid/coverage
    
    10
    -   
    
    7
    +
    
    8
    +
    
    9
    +.. _about:
    
    10
    +
    
    11 11
     About BuildGrid
    
    12 12
     ===============
    
    13 13
     
    
    14
    +
    
    15
    +.. _what-is-it:
    
    16
    +
    
    14 17
     What is BuildGrid?
    
    15 18
     ------------------
    
    16 19
     
    
    ... ... @@ -49,7 +52,7 @@ Resources
    49 52
     - `GitLab repository`_
    
    50 53
     - `Bug tracking`_
    
    51 54
     - `Mailing list`_
    
    52
    -- `Slack channel`_  [`invite link`_]
    
    55
    +- `Slack channel`_ [`invite link`_]
    
    53 56
     
    
    54 57
     .. _Homepage: https://buildgrid.build
    
    55 58
     .. _GitLab repository: https://gitlab.com/BuildGrid/buildgrid
    

  • buildgrid/_app/settings/parser.py
    ... ... @@ -41,6 +41,15 @@ class YamlFactory(yaml.YAMLObject):
    41 41
             return cls(**values)
    
    42 42
     
    
    43 43
     
    
    44
    +class Path(YamlFactory):
    
    45
    +
    
    46
    +    yaml_tag = u'!path'
    
    47
    +
    
    48
    +    def __new__(cls, path):
    
    49
    +        path = os.path.expanduser(path)
    
    50
    +        return path
    
    51
    +
    
    52
    +
    
    44 53
     class Disk(YamlFactory):
    
    45 54
     
    
    46 55
         yaml_tag = u'!disk-storage'
    
    ... ... @@ -169,6 +178,7 @@ def _parse_size(size):
    169 178
     
    
    170 179
     def get_parser():
    
    171 180
     
    
    181
    +    yaml.SafeLoader.add_constructor(Path.yaml_tag, Path.from_yaml)
    
    172 182
         yaml.SafeLoader.add_constructor(Execution.yaml_tag, Execution.from_yaml)
    
    173 183
         yaml.SafeLoader.add_constructor(Action.yaml_tag, Action.from_yaml)
    
    174 184
         yaml.SafeLoader.add_constructor(Reference.yaml_tag, Reference.from_yaml)
    

  • docs/source/data/bazel-example-server.conf
    1
    +server:
    
    2
    +  port: 50051
    
    3
    +  insecure-mode: true
    
    4
    +
    
    5
    +instances:
    
    6
    +  - name: main
    
    7
    +
    
    8
    +    storages:
    
    9
    +      - !lru-storage &main-storage
    
    10
    +        size: 512MB
    
    11
    +
    
    12
    +    services:
    
    13
    +      - !action-cache &main-action
    
    14
    +        storage: *main-storage
    
    15
    +        max_cached_refs: 256
    
    16
    +        allow_updates: true
    
    17
    +      - !execution
    
    18
    +        storage: *main-storage
    
    19
    +        action_cache: *main-action
    
    20
    +      - !cas
    
    21
    +        storage: *main-storage
    
    22
    +      - !bytestream
    
    23
    +        storage: *main-storage
    \ No newline at end of file

  • docs/source/using.rst
    ... ... @@ -9,5 +9,5 @@ This section covers how to run an use the BuildGrid build service.
    9 9
     .. toctree::
    
    10 10
        :maxdepth: 2
    
    11 11
     
    
    12
    -   using_dummy_build.rst
    
    13
    -   using_simple_build.rst
    12
    +   using_internal.rst
    
    13
    +   using_bazel.rst

  • docs/source/using_bazel.rst
    1
    +
    
    2
    +.. _bazel-builds:
    
    3
    +
    
    4
    +Bazel builds
    
    5
    +============
    
    6
    +
    
    7
    +`Bazel`_ is a *“fast, scalable, multi-language and extensible build system”*
    
    8
    +that supports remote build execution using the remote execution API (REAPI) v2
    
    9
    +since its `0.17 release`_.
    
    10
    +
    
    11
    +.. _Bazel: https://bazel.build
    
    12
    +.. _0.17 release: https://blog.bazel.build/2018/09/14/bazel-0.17.html
    
    13
    +
    
    14
    +
    
    15
    +.. _bazel-configuration:
    
    16
    +
    
    17
    +Configuration
    
    18
    +-------------
    
    19
    +
    
    20
    +Bazel accepts many options that can either be specified as command line
    
    21
    +arguments when involking the ``bazel`` tool or stored in a `.bazelrc`_
    
    22
    +configuration file. In order to activate remote build execution, the
    
    23
    +``bazel build`` subcommand needs to be given specific `build options`_,
    
    24
    +including:
    
    25
    +
    
    26
    +- ``--remote_executor``: remote execution endpoint's location, ``{host}:{port}``.
    
    27
    +- ``--remote_instance_name``: remote execution instance's name.
    
    28
    +- ``--spawn_strategy``: action execution method.
    
    29
    +- ``--genrule_strategy``: `genrules`_ execution method.
    
    30
    +
    
    31
    +Spawn and genrule strategies need to be set to ``remote`` for remote execution.
    
    32
    +
    
    33
    +As an example, in order to activate remote execution on the ``main`` instance of
    
    34
    +the remote execution server available at ``controller.grid.build`` on port
    
    35
    +``50051`` you should amend your ``.bazelrc`` with:
    
    36
    +
    
    37
    +.. code-block:: sh
    
    38
    +
    
    39
    +   build --spawn_strategy=remote --genrule_strategy=remote --remote_executor=controller.grid.build:50051 --remote_instance_name=main
    
    40
    +
    
    41
    +.. _.bazelrc: https://docs.bazel.build/versions/master/user-manual.html#bazelrc
    
    42
    +.. _build options: https://docs.bazel.build/versions/master/command-line-reference.html#build-options
    
    43
    +.. _genrules: https://docs.bazel.build/versions/master/be/general.html#genrule
    
    44
    +
    
    45
    +
    
    46
    +.. _bazel-example:
    
    47
    +
    
    48
    +Example build
    
    49
    +-------------
    
    50
    +
    
    51
    +The `bazel-examples`_ repository contains example Bazel workspaces that can be
    
    52
    +used for testing purpose. We'll focus here on instructions on how to build the
    
    53
    +`stage3 CPP example`_ running Bazel and BuildGrid on your local machine,
    
    54
    +compiling the C++ source code using host-tools.
    
    55
    +
    
    56
    +First, you need to checkout the bazel-examples repository sources:
    
    57
    +
    
    58
    +.. code-block:: sh
    
    59
    +
    
    60
    +   git clone https://github.com/bazelbuild/examples.git bazel-examples
    
    61
    +
    
    62
    +Next, change the current directory to the Bazel workspace root:
    
    63
    +
    
    64
    +.. code-block:: sh
    
    65
    +
    
    66
    +   cd bazel-examples/cpp-tutorial/stage3
    
    67
    +
    
    68
    +.. note::
    
    69
    +
    
    70
    +   All the commands in the instructions below are expected to be executed from
    
    71
    +   that root directory (the stage3 example workspace's root directory).
    
    72
    +
    
    73
    +Before running Bazel and building the example workspace, you'll have to setup
    
    74
    +and run a BuildGrid server and bot. A minimal server's configuration is given
    
    75
    +below, paste it in a ``server.conf`` file in the root directory:
    
    76
    +
    
    77
    +.. literalinclude:: ./data/bazel-example-server.conf
    
    78
    +   :language: yaml
    
    79
    +
    
    80
    +This defines a single ``main`` server instance implementing a
    
    81
    +``ContentAddressableStorage`` (CAS) + ``ByteStream`` service together with an
    
    82
    +``Execution`` + ``ActionCache`` service, both using the same in-memory storage.
    
    83
    +You can then start the BuildGrid server deamon using that configuration by
    
    84
    +running:
    
    85
    +
    
    86
    +.. code-block:: sh
    
    87
    +
    
    88
    +   bgd server start server.conf
    
    89
    +
    
    90
    +In order to perform the actual build work, you need to attach a worker bot to
    
    91
    +that server for that ``main`` instance. A simple host-tools based bot should be
    
    92
    +enough to build the stage3 CPP example. Once you've make sure that your machine
    
    93
    +has ``gcc`` installed, run:
    
    94
    +
    
    95
    +.. code-block:: sh
    
    96
    +
    
    97
    +   bgd bot --remote=http://localhost:50051 --parent=main temp-directory
    
    98
    +
    
    99
    +The ``--remote`` option is used to specify the server location (running on the
    
    100
    +same machine here, and listenning to port 50051). The ``--parent`` option is
    
    101
    +used to specifiy the server instance you except the bot to be attached to.
    
    102
    +
    
    103
    +The BuildGrid server is now ready to accept jobs and execute them. Bazel needs
    
    104
    +some :ref:`configuration <bazel-configuration>` in order to run remote builds.
    
    105
    +Below are the minimal build options you should use, paste them in a ``.bazelrc``
    
    106
    +file in the root directory:
    
    107
    +
    
    108
    +.. code-block:: sh
    
    109
    +
    
    110
    +   build --spawn_strategy=remote --genrule_strategy=remote --remote_executor=localhost:50051 --remote_instance_name=main
    
    111
    +
    
    112
    +This ativates Bazel's remote execution mode and points to the ``main`` remote
    
    113
    +execution server instance at ``localhost:50051``.
    
    114
    +
    
    115
    +You can finally have Bazel to build the example workspace by running:
    
    116
    +
    
    117
    +.. code-block:: sh
    
    118
    +
    
    119
    +   bazel build //main:hello-world
    
    120
    +
    
    121
    +You can verify that the example has been successfully build by runnig the
    
    122
    +generated executable. Simply invoke:
    
    123
    +
    
    124
    +.. code-block:: sh
    
    125
    +
    
    126
    +   ./bazel-bin/main/hello-world
    
    127
    +
    
    128
    +.. _bazel-examples: https://github.com/bazelbuild/examples
    
    129
    +.. _stage3 CPP example: https://github.com/bazelbuild/examples/tree/master/cpp-tutorial/stage3

  • docs/source/using_dummy_build.rst deleted
    1
    -.. _dummy-build:
    
    2
    -
    
    3
    -Dummy build
    
    4
    -===========
    
    5
    -
    
    6
    -In one terminal, start a server:
    
    7
    -
    
    8
    -.. code-block:: sh
    
    9
    -
    
    10
    -   bgd server start buildgrid/_app/settings/default.yml
    
    11
    -
    
    12
    -In another terminal, send a request for work:
    
    13
    -
    
    14
    -.. code-block:: sh
    
    15
    -
    
    16
    -   bgd execute request-dummy
    
    17
    -
    
    18
    -The stage should show as ``QUEUED`` as it awaits a bot to pick up the work:
    
    19
    -
    
    20
    -.. code-block:: sh
    
    21
    -
    
    22
    -   bgd execute list
    
    23
    -
    
    24
    -Create a bot session:
    
    25
    -
    
    26
    -.. code-block:: sh
    
    27
    -
    
    28
    -   bgd bot dummy
    
    29
    -
    
    30
    -Show the work as completed:
    
    31
    -
    
    32
    -.. code-block:: sh
    
    33
    -
    
    34
    -   bgd execute list

  • docs/source/using_simple_build.rstdocs/source/using_internal.rst
    1
    +
    
    2
    +.. _internal-builds:
    
    3
    +
    
    4
    +Internal builds
    
    5
    +===============
    
    6
    +
    
    7
    +
    
    8
    +.. _dummy-test:
    
    9
    +
    
    10
    +Dummy test
    
    11
    +----------
    
    12
    +
    
    13
    +In one terminal, start a server:
    
    14
    +
    
    15
    +.. code-block:: sh
    
    16
    +
    
    17
    +   bgd server start buildgrid/_app/settings/default.yml
    
    18
    +
    
    19
    +In another terminal, send a request for work:
    
    20
    +
    
    21
    +.. code-block:: sh
    
    22
    +
    
    23
    +   bgd execute request-dummy
    
    24
    +
    
    25
    +The stage should show as ``QUEUED`` as it awaits a bot to pick up the work:
    
    26
    +
    
    27
    +.. code-block:: sh
    
    28
    +
    
    29
    +   bgd execute list
    
    30
    +
    
    31
    +Create a bot session:
    
    32
    +
    
    33
    +.. code-block:: sh
    
    34
    +
    
    35
    +   bgd bot dummy
    
    36
    +
    
    37
    +Show the work as completed:
    
    38
    +
    
    39
    +.. code-block:: sh
    
    40
    +
    
    41
    +   bgd execute list
    
    42
    +
    
    43
    +
    
    1 44
     .. _simple-build:
    
    2 45
     
    
    3 46
     Simple build
    
    4
    -============
    
    47
    +------------
    
    5 48
     
    
    6 49
     This example covers a simple build. The user will upload a directory containing
    
    7 50
     a C file and a command to the CAS. The bot will then fetch the uploaded
    



  • [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]