GNOME.org
 

[Notes] [Git][BuildGrid/buildgrid][jmac/expand-user-for-tls] 4 commits: CONTRIBUTING.rst: Add a sub-section about testing

Title: GitLab

finn pushed to branch jmac/expand-user-for-tls at BuildGrid / buildgrid

Commits:

8 changed files:

Changes:

  • CONTRIBUTING.rst
    ... ... @@ -142,13 +142,67 @@ their containing *package*, as such; modules which are entirely private to
    142 142 
     BuildGrid are named as such, e.g. ``_roy.py``.
    143 143
    144 144
    145 
    +.. _codebase-testing:
    146 
    +
    147 
    +Testing
    148 
    +-------
    149 
    +
    150 
    +BuildGrid is using `pytest`_ for regression and newly added code testing. The
    151 
    +test suite contains a serie of unit-tests and also run linting tools in order to
    152 
    +detect coding-style_ breakage. The full test suite is automatically executed by
    153 
    +GitLab CI system for every push to the server. Passing all the tests is a
    154 
    +mandatory requirement for any merge request to the trunk.
    155 
    +
    156 
    +.. _pytest: https://docs.pytest.org
    157 
    +
    158 
    +
    159 
    +Running tests
    160 
    +~~~~~~~~~~~~~
    161 
    +
    162 
    +In order to run the entire test suite, simply run:
    163 
    +
    164 
    +.. code-block:: sh
    165 
    +
    166 
    +   python3 setup.py test
    167 
    +
    168 
    +You can use the ``--addopt`` function to feed arguments to pytest. For example,
    169 
    +if you want to see the ``stdout`` and ``stderr`` generated y the test, run:
    170 
    +
    171 
    +.. code-block:: sh
    172 
    +
    173 
    +   python3 setup.py test  --addopts -s
    174 
    +
    175 
    +If you want run a  specific test instead of the entire suite use:
    176 
    +
    177 
    +.. code-block:: sh
    178 
    +
    179 
    +   python3 setup.py test  --addopts tests/cas/test_client
    180 
    +
    181 
    +pyest's `usage documentation section`_ details the different command line
    182 
    +options that can be used when invoking the test runner.
    183 
    +
    184 
    +.. _usage documentation section: https://docs.pytest.org/en/latest/usage.html
    185 
    +
    186 
    +
    187 
    +Test coverage
    188 
    +~~~~~~~~~~~~~
    189 
    +
    190 
    +We are doing our best at keeping BuildGrid's test coverage score as high as
    191 
    +possible. Doing so, we ask for any merge request to include necessary test
    192 
    +additions and/or modifications in order to maintain that coverage level. A
    193 
    +detailed `coverage report`_ is produced and publish for any change merged to the
    194 
    +trunk.
    195 
    +
    196 
    +.. _coverage report: https://buildgrid.gitlab.io/buildgrid/coverage/
    197 
    +
    198 
    +
    145 199 
     .. _committer-access:
    146 200
    147 201 
     Committer access
    148 202 
     ----------------
    149 203
    150 204 
     We'll hand out commit access to anyone who has successfully landed a single
    151 
    -patch to the code base. Please request this via irc or the mailing list.
    205 
    +patch to the code base. Please request this via Slack or the mailing list.
    152 206
    153 207 
     This of course relies on contributors being responsive and show willingness to
    154 208 
     address problems after landing branches there should not be any problems here.

  • README.rst
    1 
    -.. _about:
    2 
    -
    3 
    -
    4 1
    5 2 
     .. image:: https://gitlab.com/Buildgrid/buildgrid/badges/master/pipeline.svg
    6 3 
        :target: https://gitlab.com/BuildStream/buildstream/commits/master
    7 4
    8 5 
     .. image:: https://gitlab.com/BuildGrid/buildgrid/badges/master/coverage.svg?job=coverage
    9 6 
        :target: https://buildgrid.gitlab.io/buildgrid/coverage
    10 
    -
    7 
    +
    8 
    +
    9 
    +.. _about:
    10 
    +
    11 11 
     About BuildGrid
    12 12 
     ===============
    13 13
    14 
    +
    15 
    +.. _what-is-it:
    16 
    +
    14 17 
     What is BuildGrid?
    15 18 
     ------------------
    16 19
    ... ... @@ -49,7 +52,7 @@ Resources
    49 52 
     - `GitLab repository`_
    50 53 
     - `Bug tracking`_
    51 54 
     - `Mailing list`_
    52 
    -- `Slack channel`_  [`invite link`_]
    55 
    +- `Slack channel`_ [`invite link`_]
    53 56
    54 57 
     .. _Homepage: https://buildgrid.build
    55 58 
     .. _GitLab repository: https://gitlab.com/BuildGrid/buildgrid

  • buildgrid/_app/settings/parser.py
    ... ... @@ -41,6 +41,15 @@ class YamlFactory(yaml.YAMLObject):
    41 41 
             return cls(**values)
    42 42
    43 43
    44 
    +class Path(YamlFactory):
    45 
    +
    46 
    +    yaml_tag = u'!path'
    47 
    +
    48 
    +    def __new__(cls, path):
    49 
    +        path = os.path.expanduser(path)
    50 
    +        return path
    51 
    +
    52 
    +
    44 53 
     class Disk(YamlFactory):
    45 54
    46 55 
         yaml_tag = u'!disk-storage'
    ... ... @@ -169,6 +178,7 @@ def _parse_size(size):
    169 178
    170 179 
     def get_parser():
    171 180
    181 
    +    yaml.SafeLoader.add_constructor(Path.yaml_tag, Path.from_yaml)
    172 182 
         yaml.SafeLoader.add_constructor(Execution.yaml_tag, Execution.from_yaml)
    173 183 
         yaml.SafeLoader.add_constructor(Action.yaml_tag, Action.from_yaml)
    174 184 
         yaml.SafeLoader.add_constructor(Reference.yaml_tag, Reference.from_yaml)

  • docs/source/data/bazel-example-server.conf
    1 
    +server:
    2 
    +  port: 50051
    3 
    +  insecure-mode: true
    4 
    +
    5 
    +instances:
    6 
    +  - name: main
    7 
    +
    8 
    +    storages:
    9 
    +      - !lru-storage &main-storage
    10 
    +        size: 512MB
    11 
    +
    12 
    +    services:
    13 
    +      - !action-cache &main-action
    14 
    +        storage: *main-storage
    15 
    +        max_cached_refs: 256
    16 
    +        allow_updates: true
    17 
    +      - !execution
    18 
    +        storage: *main-storage
    19 
    +        action_cache: *main-action
    20 
    +      - !cas
    21 
    +        storage: *main-storage
    22 
    +      - !bytestream
    23 
    +        storage: *main-storage
    \ No newline at end of file

  • docs/source/using.rst
    ... ... @@ -9,5 +9,5 @@ This section covers how to run an use the BuildGrid build service.
    9 9 
     .. toctree::
    10 10 
        :maxdepth: 2
    11 11
    12 
    -   using_dummy_build.rst
    13 
    -   using_simple_build.rst
    12 
    +   using_internal.rst
    13 
    +   using_bazel.rst

  • docs/source/using_bazel.rst
    1 
    +
    2 
    +.. _bazel-builds:
    3 
    +
    4 
    +Bazel builds
    5 
    +============
    6 
    +
    7 
    +`Bazel`_ is a *“fast, scalable, multi-language and extensible build system”*
    8 
    +that supports remote build execution using the remote execution API (REAPI) v2
    9 
    +since its `0.17 release`_.
    10 
    +
    11 
    +.. _Bazel: https://bazel.build
    12 
    +.. _0.17 release: https://blog.bazel.build/2018/09/14/bazel-0.17.html
    13 
    +
    14 
    +
    15 
    +.. _bazel-configuration:
    16 
    +
    17 
    +Configuration
    18 
    +-------------
    19 
    +
    20 
    +Bazel accepts many options that can either be specified as command line
    21 
    +arguments when involking the ``bazel`` tool or stored in a `.bazelrc`_
    22 
    +configuration file. In order to activate remote build execution, the
    23 
    +``bazel build`` subcommand needs to be given specific `build options`_,
    24 
    +including:
    25 
    +
    26 
    +- ``--remote_executor``: remote execution endpoint's location, ``{host}:{port}``.
    27 
    +- ``--remote_instance_name``: remote execution instance's name.
    28 
    +- ``--spawn_strategy``: action execution method.
    29 
    +- ``--genrule_strategy``: `genrules`_ execution method.
    30 
    +
    31 
    +Spawn and genrule strategies need to be set to ``remote`` for remote execution.
    32 
    +
    33 
    +As an example, in order to activate remote execution on the ``main`` instance of
    34 
    +the remote execution server available at ``controller.grid.build`` on port
    35 
    +``50051`` you should amend your ``.bazelrc`` with:
    36 
    +
    37 
    +.. code-block:: sh
    38 
    +
    39 
    +   build --spawn_strategy=remote --genrule_strategy=remote --remote_executor=controller.grid.build:50051 --remote_instance_name=main
    40 
    +
    41 
    +.. _.bazelrc: https://docs.bazel.build/versions/master/user-manual.html#bazelrc
    42 
    +.. _build options: https://docs.bazel.build/versions/master/command-line-reference.html#build-options
    43 
    +.. _genrules: https://docs.bazel.build/versions/master/be/general.html#genrule
    44 
    +
    45 
    +
    46 
    +.. _bazel-example:
    47 
    +
    48 
    +Example build
    49 
    +-------------
    50 
    +
    51 
    +The `bazel-examples`_ repository contains example Bazel workspaces that can be
    52 
    +used for testing purpose. We'll focus here on instructions on how to build the
    53 
    +`stage3 CPP example`_ running Bazel and BuildGrid on your local machine,
    54 
    +compiling the C++ source code using host-tools.
    55 
    +
    56 
    +First, you need to checkout the bazel-examples repository sources:
    57 
    +
    58 
    +.. code-block:: sh
    59 
    +
    60 
    +   git clone https://github.com/bazelbuild/examples.git bazel-examples
    61 
    +
    62 
    +Next, change the current directory to the Bazel workspace root:
    63 
    +
    64 
    +.. code-block:: sh
    65 
    +
    66 
    +   cd bazel-examples/cpp-tutorial/stage3
    67 
    +
    68 
    +.. note::
    69 
    +
    70 
    +   All the commands in the instructions below are expected to be executed from
    71 
    +   that root directory (the stage3 example workspace's root directory).
    72 
    +
    73 
    +Before running Bazel and building the example workspace, you'll have to setup
    74 
    +and run a BuildGrid server and bot. A minimal server's configuration is given
    75 
    +below, paste it in a ``server.conf`` file in the root directory:
    76 
    +
    77 
    +.. literalinclude:: ./data/bazel-example-server.conf
    78 
    +   :language: yaml
    79 
    +
    80 
    +This defines a single ``main`` server instance implementing a
    81 
    +``ContentAddressableStorage`` (CAS) + ``ByteStream`` service together with an
    82 
    +``Execution`` + ``ActionCache`` service, both using the same in-memory storage.
    83 
    +You can then start the BuildGrid server deamon using that configuration by
    84 
    +running:
    85 
    +
    86 
    +.. code-block:: sh
    87 
    +
    88 
    +   bgd server start server.conf
    89 
    +
    90 
    +In order to perform the actual build work, you need to attach a worker bot to
    91 
    +that server for that ``main`` instance. A simple host-tools based bot should be
    92 
    +enough to build the stage3 CPP example. Once you've make sure that your machine
    93 
    +has ``gcc`` installed, run:
    94 
    +
    95 
    +.. code-block:: sh
    96 
    +
    97 
    +   bgd bot --remote=http://localhost:50051 --parent=main temp-directory
    98 
    +
    99 
    +The ``--remote`` option is used to specify the server location (running on the
    100 
    +same machine here, and listenning to port 50051). The ``--parent`` option is
    101 
    +used to specifiy the server instance you except the bot to be attached to.
    102 
    +
    103 
    +The BuildGrid server is now ready to accept jobs and execute them. Bazel needs
    104 
    +some :ref:`configuration <bazel-configuration>` in order to run remote builds.
    105 
    +Below are the minimal build options you should use, paste them in a ``.bazelrc``
    106 
    +file in the root directory:
    107 
    +
    108 
    +.. code-block:: sh
    109 
    +
    110 
    +   build --spawn_strategy=remote --genrule_strategy=remote --remote_executor=localhost:50051 --remote_instance_name=main
    111 
    +
    112 
    +This ativates Bazel's remote execution mode and points to the ``main`` remote
    113 
    +execution server instance at ``localhost:50051``.
    114 
    +
    115 
    +You can finally have Bazel to build the example workspace by running:
    116 
    +
    117 
    +.. code-block:: sh
    118 
    +
    119 
    +   bazel build //main:hello-world
    120 
    +
    121 
    +You can verify that the example has been successfully build by runnig the
    122 
    +generated executable. Simply invoke:
    123 
    +
    124 
    +.. code-block:: sh
    125 
    +
    126 
    +   ./bazel-bin/main/hello-world
    127 
    +
    128 
    +.. _bazel-examples: https://github.com/bazelbuild/examples
    129 
    +.. _stage3 CPP example: https://github.com/bazelbuild/examples/tree/master/cpp-tutorial/stage3

  • docs/source/using_dummy_build.rst deleted
    1 
    -.. _dummy-build:
    2 
    -
    3 
    -Dummy build
    4 
    -===========
    5 
    -
    6 
    -In one terminal, start a server:
    7 
    -
    8 
    -.. code-block:: sh
    9 
    -
    10 
    -   bgd server start buildgrid/_app/settings/default.yml
    11 
    -
    12 
    -In another terminal, send a request for work:
    13 
    -
    14 
    -.. code-block:: sh
    15 
    -
    16 
    -   bgd execute request-dummy
    17 
    -
    18 
    -The stage should show as ``QUEUED`` as it awaits a bot to pick up the work:
    19 
    -
    20 
    -.. code-block:: sh
    21 
    -
    22 
    -   bgd execute list
    23 
    -
    24 
    -Create a bot session:
    25 
    -
    26 
    -.. code-block:: sh
    27 
    -
    28 
    -   bgd bot dummy
    29 
    -
    30 
    -Show the work as completed:
    31 
    -
    32 
    -.. code-block:: sh
    33 
    -
    34 
    -   bgd execute list

  • docs/source/using_simple_build.rst → docs/source/using_internal.rst
    1 
    +
    2 
    +.. _internal-builds:
    3 
    +
    4 
    +Internal builds
    5 
    +===============
    6 
    +
    7 
    +
    8 
    +.. _dummy-test:
    9 
    +
    10 
    +Dummy test
    11 
    +----------
    12 
    +
    13 
    +In one terminal, start a server:
    14 
    +
    15 
    +.. code-block:: sh
    16 
    +
    17 
    +   bgd server start buildgrid/_app/settings/default.yml
    18 
    +
    19 
    +In another terminal, send a request for work:
    20 
    +
    21 
    +.. code-block:: sh
    22 
    +
    23 
    +   bgd execute request-dummy
    24 
    +
    25 
    +The stage should show as ``QUEUED`` as it awaits a bot to pick up the work:
    26 
    +
    27 
    +.. code-block:: sh
    28 
    +
    29 
    +   bgd execute list
    30 
    +
    31 
    +Create a bot session:
    32 
    +
    33 
    +.. code-block:: sh
    34 
    +
    35 
    +   bgd bot dummy
    36 
    +
    37 
    +Show the work as completed:
    38 
    +
    39 
    +.. code-block:: sh
    40 
    +
    41 
    +   bgd execute list
    42 
    +
    43 
    +
    1 44 
     .. _simple-build:
    2 45
    3 46 
     Simple build
    4 
    -============
    47 
    +------------
    5 48
    6 49 
     This example covers a simple build. The user will upload a directory containing
    7 50 
     a C file and a command to the CAS. The bot will then fetch the uploaded

    • —
    View it on GitLab.
    You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.

    [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]