GNOME.org
 

[Notes] [Git][BuildGrid/buildgrid][mablanch/160-docker-compose] 5 commits: docker-compose.yml: Add manifest for simple grid

Title: GitLab

Martin Blanchard pushed to branch mablanch/160-docker-compose at BuildGrid / buildgrid

Commits:

10 changed files:

Changes:

  • BUILDSTREAM_README.rst deleted
    1 
    -Temp Demo Instructions
    2 
    -======================
    3 
    -
    4 
    -A quick guide to getting remote execution working with BuildStream. Please change URL and certifcates / keys to your own.
    5 
    -
    6 
    -Downloaded and build::
    7 
    -
    8 
    -  https://gitlab.com/BuildStream/buildbox
    9 
    -
    10 
    -Copy build to bin/.
    11 
    -
    12 
    -Checkout branch::
    13 
    -
    14 
    -  https://gitlab.com/BuildStream/buildstream/tree/jmac/source_pushing_experiments
    15 
    -
    16 
    -Update to your URL::
    17 
    -
    18 
    -  https://gitlab.com/BuildStream/buildstream/blob/jmac/source_pushing_experiments/buildstream/sandbox/_sandboxremote.py#L73
    19 
    -
    20 
    -Start artifact server::
    21 
    -
    22 
    -  bst-artifact-server --port 11001 --server-key server.key --server-cert server.crt --client-certs client.crt --enable-push /home/user/
    23 
    -
    24 
    -Start bgd server::
    25 
    -
    26 
    -  bgd server start
    27 
    -
    28 
    -Run::
    29 
    -
    30 
    -  bgd bot buildbox
    31 
    -
    32 
    -Update project.conf in build area with::
    33 
    -
    34 
    -  artifacts:
    35 
    -      url: https://localhost:11001
    36 
    -      server-cert: server.crt
    37 
    -
    38 
    -      # Optional client key pair for authentication
    39 
    -      client-key: client.key
    40 
    -      client-cert: client.crt
    41 
    -
    42 
    -      push: true
    43 
    -
    44 
    -Run build with::
    45 
    -
    46 
    -  bst build --track something.bst

  • buildgrid/_app/commands/cmd_server.py
    ... ... @@ -119,6 +119,14 @@ def _create_server_from_config(configuration):
    119 119 
                 click.echo("Error: Configuration, {}.".format(e), err=True)
    120 120 
                 sys.exit(-1)
    121 121
    122 
    +    if 'thread-pool-size' in configuration:
    123 
    +        try:
    124 
    +            kargs['max_workers'] = int(configuration['thread-pool-size'])
    125 
    +
    126 
    +        except ValueError as e:
    127 
    +            click.echo("Error: Configuration, {}.".format(e), err=True)
    128 
    +            sys.exit(-1)
    129 
    +
    122 130 
         server = BuildGridServer(**kargs)
    123 131
    124 132 
         try:

  • buildgrid/_app/settings/reference.yml
    ... ... @@ -136,3 +136,7 @@ monitoring:
    136 136 
       #  binary  - Protobuf binary format.
    137 137 
       #  json    - JSON format.
    138 138 
       serialization-format: binary
    139 
    +
    140 
    +##
    141 
    +# Maximum number of gRPC threads.
    142 
    +thread-pool-size: 20

  • buildgrid/server/bots/instance.py
    ... ... @@ -32,6 +32,8 @@ class BotsInterface:
    32 32
    33 33 
         def __init__(self, scheduler):
    34 34 
             self.__logger = logging.getLogger(__name__)
    35 
    +        # Turn on debug mode based on log verbosity level:
    36 
    +        self.__debug = self.__logger.getEffectiveLevel() <= logging.DEBUG
    35 37
    36 38 
             self._scheduler = scheduler
    37 39 
             self._instance_name = None
    ... ... @@ -65,13 +67,11 @@ class BotsInterface:
    65 67 
             register with the service, the old one should be closed along
    66 68 
             with all its jobs.
    67 69 
             """
    68 
    -        bot_id = bot_session.bot_id
    69 
    -
    70 
    -        if bot_id == "":
    71 
    -            raise InvalidArgumentError("bot_id needs to be set by client")
    70 
    +        if not bot_session.bot_id:
    71 
    +            raise InvalidArgumentError("Bot's id must be set by client.")
    72 72
    73 73 
             try:
    74 
    -            self._check_bot_ids(bot_id)
    74 
    +            self._check_bot_ids(bot_session.bot_id)
    75 75 
             except InvalidArgumentError:
    76 76 
                 pass
    77 77
    ... ... @@ -79,21 +79,27 @@ class BotsInterface:
    79 79 
             name = "{}/{}".format(parent, str(uuid.uuid4()))
    80 80 
             bot_session.name = name
    81 81
    82 
    -        self._bot_ids[name] = bot_id
    83 
    -
    84 
    -        self.__logger.info("Created bot session name=[%s] with bot_id=[%s]", name, bot_id)
    82 
    +        self._bot_ids[name] = bot_session.bot_id
    85 83
    86 84 
             # We want to keep a copy of lease ids we have assigned
    87 85 
             self._assigned_leases[name] = set()
    88 86
    89 87 
             self._request_leases(bot_session)
    88 
    +
    89 
    +        if self.__debug:
    90 
    +            self.__logger.info("Opened session name=[%s] for bot=[%s], leases=[%s]",
    91 
    +                               bot_session.name, bot_session.bot_id,
    92 
    +                               ",".join([lease.id[:8] for lease in bot_session.leases]))
    93 
    +        else:
    94 
    +            self.__logger.info("Opened session, name=[%s] for bot=[%s]",
    95 
    +                               bot_session.name, bot_session.bot_id)
    96 
    +
    90 97 
             return bot_session
    91 98
    92 99 
         def update_bot_session(self, name, bot_session):
    93 100 
             """ Client updates the server. Any changes in state to the Lease should be
    94 101 
             registered server side. Assigns available leases with work.
    95 102 
             """
    96 
    -        self.__logger.debug("Updating bot session name=[%s]", name)
    97 103 
             self._check_bot_ids(bot_session.bot_id, name)
    98 104 
             self._check_assigned_leases(bot_session)
    99 105
    ... ... @@ -111,6 +117,15 @@ class BotsInterface:
    111 117 
                     bot_session.leases.remove(lease)
    112 118
    113 119 
             self._request_leases(bot_session)
    120 
    +
    121 
    +        if self.__debug:
    122 
    +            self.__logger.info("Sending session update, name=[%s], for bot=[%s], leases=[%s]",
    123 
    +                               bot_session.name, bot_session.bot_id,
    124 
    +                               ",".join([lease.id[:8] for lease in bot_session.leases]))
    125 
    +        else:
    126 
    +            self.__logger.info("Sending session update, name=[%s], for bot=[%s]",
    127 
    +                               bot_session.name, bot_session.bot_id)
    128 
    +
    114 129 
             return bot_session
    115 130
    116 131 
         # --- Private API ---

  • buildgrid/server/instance.py
    ... ... @@ -91,6 +91,8 @@ class BuildGridServer:
    91 91 
             self.__grpc_server = grpc.server(self.__grpc_executor,
    92 92 
                                              options=(('grpc.so_reuseport', 0),))
    93 93
    94 
    +        self.__logger.debug("Setting up gRPC server with thread-limit=[%s]", max_workers)
    95 
    +
    94 96 
             self.__main_loop = asyncio.get_event_loop()
    95 97
    96 98 
             self.__monitoring_bus = None

  • buildgrid/server/scheduler.py
    ... ... @@ -151,14 +151,14 @@ class Scheduler:
    151 151 
                 raise NotFoundError("Operation name does not exist: [{}]"
    152 152 
                                     .format(operation_name))
    153 153
    154 
    -        job.unregister_operation_peer(operation_name, peer)
    155 
    -
    156 154 
             if not job.n_peers_for_operation(operation_name):
    157 155 
                 del self.__jobs_by_operation[operation_name]
    158 156
    159 157 
             if not job.n_peers and job.done and not job.lease:
    160 158 
                 self._delete_job(job.name)
    161 159
    160 
    +        job.unregister_operation_peer(operation_name, peer)
    161 
    +
    162 162 
         def queue_job_action(self, action, action_digest, platform_requirements=None,
    163 163 
                              priority=0, skip_cache_lookup=False):
    164 164 
             """Inserts a newly created job into the execution queue.

  • data/config/controller.conf
    1 
    +server:
    2 
    +  - !channel
    3 
    +    port: 50051
    4 
    +    insecure-mode: true
    5 
    +
    6 
    +description: >
    7 
    +  Docker Compose controller configuration:
    8 
    +    - Unauthenticated plain HTTP at :50051
    9 
    +    - Single instance: local
    10 
    +    - Expects a remote CAS at :50052
    11 
    +    - Hosted services:
    12 
    +       - ActionCache
    13 
    +       - Execute
    14 
    +
    15 
    +authorization:
    16 
    +  method: none
    17 
    +
    18 
    +monitoring:
    19 
    +  enabled: false
    20 
    +
    21 
    +instances:
    22 
    +  - name: local
    23 
    +    description: |
    24 
    +      The unique 'local' instance.
    25 
    +
    26 
    +    storages:
    27 
    +      - !remote-storage &data-store
    28 
    +        url: http://storage:50052
    29 
    +        instance-name: local
    30 
    +
    31 
    +    services:
    32 
    +      - !action-cache &build-cache
    33 
    +        storage: *data-store
    34 
    +        max-cached-refs: 256
    35 
    +        cache-failed-actions: true
    36 
    +        allow-updates: true
    37 
    +
    38 
    +      - !execution
    39 
    +        storage: *data-store
    40 
    +        action-cache: *build-cache
    41 
    +        action-browser-url: http://localhost:8080
    42 
    +
    43 
    +thread-pool-size: 100

  • data/config/storage.conf
    1 
    +server:
    2 
    +  - !channel
    3 
    +    port: 50052
    4 
    +    insecure-mode: true
    5 
    +
    6 
    +description: >
    7 
    +  Docker Compose storage configuration:
    8 
    +    - Unauthenticated plain HTTP at :50052
    9 
    +    - Single instance: local
    10 
    +    - On-disk data stored in /var
    11 
    +    - Hosted services:
    12 
    +       - ContentAddressableStorage
    13 
    +       - ByteStream
    14 
    +
    15 
    +authorization:
    16 
    +  method: none
    17 
    +
    18 
    +monitoring:
    19 
    +  enabled: false
    20 
    +
    21 
    +instances:
    22 
    +  - name: local
    23 
    +    description: |
    24 
    +      The unique 'local' instance.
    25 
    +
    26 
    +    storages:
    27 
    +      - !disk-storage &data-store
    28 
    +        path: /var/lib/buildgrid/store
    29 
    +
    30 
    +    services:
    31 
    +      - !cas
    32 
    +        storage: *data-store
    33 
    +
    34 
    +      - !bytestream
    35 
    +        storage: *data-store
    36 
    +
    37 
    +thread-pool-size: 100

  • docker-compose.yml
    1 
    +##
    2 
    +# BuildGrid's Docker Compose manifest.
    3 
    +#
    4 
    +#   ¡FOR LOCAL DEVELOPMENT ONLY!
    5 
    +#
    6 
    +# Spins-up a 'local' grid instance:
    7 
    +#   - Controller at http://localhost:50051
    8 
    +#   - CAS + AC at: http://localhost:50052
    9 
    +#
    10 
    +# Basic usage:
    11 
    +#  - docker-compose build
    12 
    +#  - docker-compose up --scale bots=10
    13 
    +#  - docker-compose down
    14 
    +#  - docker volume inspect buildgrid_data
    15 
    +#  - docker volume rm buildgrid_data
    16 
    +#  - docker image rm buildgrid:local
    17 
    +#
    18 
    +version: "3.2"
    19 
    +
    20 
    +services:
    21 
    +  storage:
    22 
    +    build:
    23 
    +      context: .
    24 
    +    image: buildgrid:local
    25 
    +    command: [
    26 
    +      "server", "start", "-vvv",
    27 
    +      "/app/config/storage.conf"]
    28 
    +    volumes:
    29 
    +      - type: volume
    30 
    +        source: data
    31 
    +        target: /var/lib/buildgrid/store
    32 
    +        volume:
    33 
    +          nocopy: true
    34 
    +      - type: bind
    35 
    +        source: ./data/config/storage.conf
    36 
    +        target: /app/config/storage.conf
    37 
    +    ports:
    38 
    +      - "50052:50052"
    39 
    +    networks:
    40 
    +      - backend
    41 
    +      - host
    42 
    +
    43 
    +  controller:
    44 
    +    image: buildgrid:local
    45 
    +    command: [
    46 
    +      "server", "start", "-vvv",
    47 
    +      "/app/config/controller.conf"]
    48 
    +    volumes:
    49 
    +      - type: bind
    50 
    +        source: ./data/config/controller.conf
    51 
    +        target: /app/config/controller.conf
    52 
    +    ports:
    53 
    +      - "50051:50051"
    54 
    +    networks:
    55 
    +      - backend
    56 
    +      - host
    57 
    +
    58 
    +  bots:  # To be scaled horizontaly
    59 
    +    image: buildgrid:local
    60 
    +    command: [
    61 
    +      "bot", "--parent=local",
    62 
    +      "--remote=http://controller:50051",
    63 
    +      "--remote-cas=http://storage:50052",
    64 
    +      "host-tools"]
    65 
    +    depends_on:
    66 
    +      - controller
    67 
    +    networks:
    68 
    +      - backend
    69 
    +
    70 
    +networks:
    71 
    +  backend:
    72 
    +  host:
    73 
    +
    74 
    +volumes:
    75 
    +  data:

  • docs/source/installation.rst
    ... ... @@ -21,20 +21,24 @@ How to install BuildGrid directly onto your machine.
    21 21 
     Prerequisites
    22 22 
     ~~~~~~~~~~~~~
    23 23
    24 
    -BuildGrid only supports ``python3 >= 3.5`` but has no system requirements. Main
    25 
    -Python dependencies, automatically handled during installation, include:
    24 
    +BuildGrid only supports ``python3 >= 3.5.3`` but has no system requirements.
    25 
    +Main Python dependencies, automatically handled during installation, include:
    26 26
    27 27 
     - `boto3`_: the Amazon Web Services (AWS) SDK for Python.
    28 28 
     - `click`_: a Python composable command line library.
    29 29 
     - `grpcio`_: Google's `gRPC`_ Python interface.
    30 
    +- `janus`_: a mixed sync-async Python queue.
    30 31 
     - `protobuf`_: Google's `protocol-buffers`_ Python interface.
    32 
    +- `PyYAML`_: a YAML parser and emitter for Python.
    31 33
    32 34 
     .. _boto3: https://pypi.org/project/boto3
    33 35 
     .. _click: https://pypi.org/project/click
    34 36 
     .. _grpcio: https://pypi.org/project/grpcio
    35 37 
     .. _gRPC: https://grpc.io
    38 
    +.. _janus: https://pypi.org/project/janus
    36 39 
     .. _protobuf: https://pypi.org/project/protobuf
    37 40 
     .. _protocol-buffers: https://developers.google.com/protocol-buffers
    41 
    +.. _PyYAML: https://pypi.org/project/PyYAML
    38 42
    39 43
    40 44 
     .. _install-host-source-install:
    ... ... @@ -42,20 +46,30 @@ Python dependencies, automatically handled during installation, include:
    42 46 
     Install from sources
    43 47 
     ~~~~~~~~~~~~~~~~~~~~
    44 48
    45 
    -BuildGrid has ``setuptools`` support. In order to install it to your home
    46 
    -directory, typically under ``~/.local``, simply run:
    49 
    +BuildGrid has ``setuptools`` support. We recommend installing it in a dedicated
    50 
    +`virtual environment`_. In order to do so in an environment named ``env``
    51 
    +placed in the source tree, run:
    47 52
    48 53 
     .. code-block:: sh
    49 54
    50 
    -   git clone https://gitlab.com/BuildGrid/buildgrid.git && cd buildgrid
    51 
    -   pip3 install --user --editable .
    55 
    +   git clone https://gitlab.com/BuildGrid/buildgrid.git
    56 
    +   cd buildgrid
    57 
    +   python3 -m venv env
    58 
    +   env/bin/python -m pip install --upgrade setuptools pip wheel
    59 
    +   env/bin/python -m pip install --editable .
    52 60
    53 
    -Additionally, and if your distribution does not already include it, you may
    54 
    -have to adjust your ``PATH``, in ``~/.bashrc``, with:
    61 
    +.. hint::
    62 
    +
    63 
    +   Once created, the virtual environment can be *activated* by sourcing the
    64 
    +   ``env/bin/activate`` script. In an activated terminal session, simply run
    65 
    +   ``deactivate`` to later *deactivate* it.
    66 
    +
    67 
    +Once completed, you can check that installation succeed by locally starting the
    68 
    +BuildGrid server with default configuration. Simply run:
    55 69
    56 70 
     .. code-block:: sh
    57 71
    58 
    -   export PATH="${PATH}:${HOME}/.local/bin"
    72 
    +   env/bin/bgd server start data/config/default.conf -vvv
    59 73
    60 74 
     .. note::
    61 75
    ... ... @@ -63,66 +77,112 @@ have to adjust your ``PATH``, in ``~/.bashrc``, with:
    63 77 
        ``tests``. They declare required dependency for, respectively, authentication
    64 78 
        and authorization management, generating documentation and running
    65 79 
        unit-tests. They can be use as helpers for setting up a development
    66 
    -   environment. To use them simply run:
    80 
    +   environment. To use them run:
    67 81
    68 82 
        .. code-block:: sh
    69 83
    70 
    -      pip3 install --user --editable ".[auth,docs,tests]"
    84 
    +      env/bin/python -m pip install --editable ".[auth,docs,tests]"
    85 
    +
    86 
    +.. _virtual environment: https://docs.python.org/3/library/venv.html
    71 87
    72 88
    73 89 
     .. install-docker:
    74 90
    75 
    -Installation through Docker
    76 
    ----------------------------
    91 
    +Install through Docker
    92 
    +----------------------
    77 93
    78 
    -How to build a Docker image that runs BuildGrid.
    94 
    +BuildGrid comes with Docker support for local development use-cases.
    79 95
    80 
    -.. _install-docker-prerequisites:
    96 
    +.. caution::
    81 97
    82 
    -Prerequisites
    83 
    -~~~~~~~~~~~~~
    98 
    +   The Docker manifests are intended to be use for **local development only**.
    99 
    +   Do **not** use them in production.
    84 100
    85 
    -A working Docker installation. Please consult `Docker's Getting Started Guide`_ if you don't already have it installed.
    101 
    +Please consult the `Get Started with Docker`_ guide if you are looking for
    102 
    +instructions on how to setup Docker on your machine.
    86 103
    87 
    -.. _`Docker's Getting Started Guide`: https://www.docker.com/get-started
    104 
    +.. _`Get Started with Docker`: https://www.docker.com/get-started
    88 105
    89 106
    90 107 
     .. _install-docker-build:
    91 108
    92 
    -Docker Container from Sources
    93 
    -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    109 
    +Docker build
    110 
    +~~~~~~~~~~~~
    94 111
    95 
    -To clone the source code and build a Docker image, simply run:
    112 
    +BuildGrid ships a ``Dockerfile`` manifest for building images from source using
    113 
    +``docker build``. In order to produce a ``buildgrid:local`` base image, run:
    96 114
    97 115 
     .. code-block:: sh
    98 116
    99 
    -   git clone https://gitlab.com/BuildGrid/buildgrid.git && cd buildgrid
    100 
    -   docker build -t buildgrid_server .
    117 
    +   git clone https://gitlab.com/BuildGrid/buildgrid.git
    118 
    +   cd buildgrid
    119 
    +   docker build --tag buildgrid:local .
    101 120
    102 121 
     .. note::
    103 122
    104 
    -   The image built will contain the contents of the source code directory, including
    105 
    -   configuration files.
    106 
    -
    123 
    +   The image built will contain the Python sources, including example
    124 
    +   configuration files. The main endpoint is the ``bgd`` CLI tools and the
    125 
    +   default command shall run the BuildGrid server loading default configuration.
    126 
    +
    127 
    +Once completed, you can check that build succeed by locally starting in a
    128 
    +container the BuildGrid server with default configuration. Simply run:
    129 
    +
    130 
    +.. code-block:: sh
    131 
    +
    132 
    +   docker run --interactive --publish 50051:50051 buildgrid:local
    133 
    +
    107 134 
     .. hint::
    108 135
    109 
    -    Whenever the source code is updated or new configuration files are made, you need to re-build
    110 
    -    the image.
    136 
    +   You can run any of the BuildGrid CLI tool using that image, simply pass extra
    137 
    +   arguments to ``docker run`` the same way you would pass them to ``bgd``.
    138 
    +
    139 
    +    Bear in mind that whenever the source code or the configuration files are
    140 
    +    updated, you **must** re-build the image.
    141 
    +
    142 
    +
    143 
    +.. _install-docker-compose:
    111 144
    112 
    -After building the Docker image, to run BuildGrid using the default configuration file
    113 
    -(found in `data/config/default.conf`), simply run:
    145 
    +Docker Compose
    146 
    +~~~~~~~~~~~~~~
    147 
    +
    148 
    +BuildGrid ships a ``docker-compose.yml`` manifest for building and running a
    149 
    +grid locally using ``docker-compose``. In order to produce a
    150 
    +``buildgrid:local`` base image, run:
    114 151
    115 152 
     .. code-block:: sh
    116 153
    117 
    -   docker run -i -p 50051:50051 buildgrid_server
    154 
    +   git clone https://gitlab.com/BuildGrid/buildgrid.git
    155 
    +   cd buildgrid
    156 
    +   docker-compose build
    157 
    +
    158 
    +Once completed, you can start a minimal grid by running:
    159 
    +
    160 
    +.. code-block:: sh
    161 
    +
    162 
    +   docker-compose up
    118 163
    119 164 
     .. note::
    120 165
    121 
    -    To run BuildGrid using a different configuration file, include the relative path to the
    122 
    -    configuration file at the end of the command above. For example, to run the default
    123 
    -    standalone CAS server (without an execution service), simply run:
    166 
    +   The grid is composed of three containers:
    167 
    +
    168 
    +   - An execution and action-cache service available at
    169 
    +     ``http://localhost:50051``.
    170 
    +   - An CAS service available at ``http://localhost:50052``.
    171 
    +   - A single ``local`` instance with one host-tools based worker bot attached.
    124 172
    125 
    -       .. code-block:: sh
    173 
    +.. hint::
    174 
    +
    175 
    +   You can spin up more bots by using ``docker-compose`` scaling capabilities:
    176 
    +
    177 
    +   .. code-block:: sh
    178 
    +
    179 
    +      docker-compose up --scale bots=12
    180 
    +
    181 
    +.. hint::
    126 182
    127 
    -            docker run -i -p 50052:50052 buildgrid_server buildgrid/_app/settings/cas.yml
    183 
    +   The contained services configuration files are bind mounted into the
    184 
    +   container, no need to rebuild the base image on configuration update.
    185 
    +   Configuration files are read from:
    128 186
    187 
    +   - ``data/config/controller.conf`` for the execution service.
    188 
    +   - ``data/config/storage.conf`` for the CAS and action-cache service.

    • —
    View it on GitLab.
    You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.

    [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]