GNOME.org
 

[Notes] [Git][BuildGrid/buildgrid][mablanch/160-docker-compose] 5 commits: Move example configuration out of source tree

Title: GitLab

Martin Blanchard pushed to branch mablanch/160-docker-compose at BuildGrid / buildgrid

Commits:

17 changed files:

Changes:

  • .dockerignore
    1 
    +# Ignore everything:
    2 
    +*
    3 
    +
    4 
    +# Whitelist:
    5 
    +!buildgrid
    6 
    +!tests
    7 
    +!LICENSE
    8 
    +!README.rst
    9 
    +!setup.cfg
    10 
    +!setup.py
    11 
    +!_version.py
    12 
    +!.coveragerc
    13 
    +!.pylintrc

  • .gitlab-ci.yml
    ... ... @@ -31,7 +31,7 @@ before_script:
    31 31 
     .run-dummy-job-template: &dummy-job
    32 32 
       stage: test
    33 33 
       script:
    34 
    -    - ${BGD} server start buildgrid/_app/settings/default.yml &
    34 
    +    - ${BGD} server start data/config/default.conf &
    35 35 
         - sleep 1 # Allow server to boot
    36 36 
         - ${BGD} bot dummy &
    37 37 
         - ${BGD} cas upload-dummy

  • Dockerfile
    1 1 
     FROM python:3.5-stretch
    2 2
    3 
    -# Point the path to where buildgrid gets installed
    4 
    -ENV PATH=$PATH:/root/.local/bin/
    3 
    +# Use /app as working directory:
    4 
    +WORKDIR /app
    5 5
    6 
    -# Upgrade python modules
    7 
    -RUN python3 -m pip install --upgrade setuptools pip
    6 
    +# Copy the repo. contents:
    7 
    +COPY . /app
    8 8
    9 
    -# Use /app as the current working directory
    10 
    -WORKDIR /app
    9 
    +# Create a virtual environment:
    10 
    +RUN [ \
    11 
    +"python3", "-m", "venv", "/app/env" \
    12 
    +]
    11 13
    12 
    -# Copy the repo contents (source, config files, etc) in the WORKDIR
    13 
    -COPY . .
    14 
    +# Upgrade Python core modules:
    15 
    +RUN [ \
    16 
    +"/app/env/bin/python", "-m", "pip", \
    17 
    +"install", "--upgrade", \
    18 
    +"setuptools", "pip", "wheel" \
    19 
    +]
    14 20
    15 
    -# Install BuildGrid
    16 
    -RUN pip install --user --editable .
    21 
    +# Install BuildGrid:
    22 
    +RUN [ \
    23 
    +"/app/env/bin/python", "-m", "pip", \
    24 
    +"install", "--editable", \
    25 
    +".[auth,tests]" \
    26 
    +]
    17 27
    18 
    -# Entry Point of the image (should get an additional argument from CMD, the path to the config file)
    19 
    -ENTRYPOINT ["bgd", "server", "start", "-vv"]
    28 
    +# Entry-point for the image:
    29 
    +ENTRYPOINT [ \
    30 
    +"/app/env/bin/bgd" \
    31 
    +]
    20 32
    21 
    -# Default config file (used if no CMD specified when running)
    22 
    -CMD ["buildgrid/_app/settings/default.yml"]
    33 
    +# Default command (default config.):
    34 
    +CMD [ \
    35 
    +"server", "start", \
    36 
    +"data/config/default.conf", \
    37 
    +"-vvv" \
    38 
    +]

  • buildgrid/_app/commands/cmd_server.py
    ... ... @@ -119,6 +119,14 @@ def _create_server_from_config(configuration):
    119 119 
                 click.echo("Error: Configuration, {}.".format(e), err=True)
    120 120 
                 sys.exit(-1)
    121 121
    122 
    +    if 'thread-pool-size' in configuration:
    123 
    +        try:
    124 
    +            kargs['max_workers'] = int(configuration['thread-pool-size'])
    125 
    +
    126 
    +        except ValueError as e:
    127 
    +            click.echo("Error: Configuration, {}.".format(e), err=True)
    128 
    +            sys.exit(-1)
    129 
    +
    122 130 
         server = BuildGridServer(**kargs)
    123 131
    124 132 
         try:

  • buildgrid/_app/settings/default.yml deleted
    1 
    -server:
    2 
    -  - !channel
    3 
    -    port: 50051
    4 
    -    insecure-mode: true
    5 
    -
    6 
    -description: |
    7 
    -  A single default instance.
    8 
    -
    9 
    -instances:
    10 
    -  - name: ''
    11 
    -    description: |
    12 
    -      The main server
    13 
    -
    14 
    -    storages:
    15 
    -      - !disk-storage &main-storage
    16 
    -        path: !expand-path $HOME/cas
    17 
    -
    18 
    -    services:
    19 
    -      - !action-cache &main-action
    20 
    -        storage: *main-storage
    21 
    -        max-cached-refs: 256
    22 
    -        allow-updates: true
    23 
    -
    24 
    -      - !execution
    25 
    -        storage: *main-storage
    26 
    -        action-cache: *main-action
    27 
    -
    28 
    -      - !cas
    29 
    -        storage: *main-storage
    30 
    -
    31 
    -      - !bytestream
    32 
    -        storage: *main-storage

  • buildgrid/_app/settings/reference.yml
    ... ... @@ -136,3 +136,7 @@ monitoring:
    136 136 
       #  binary  - Protobuf binary format.
    137 137 
       #  json    - JSON format.
    138 138 
       serialization-format: binary
    139 
    +
    140 
    +##
    141 
    +# Maximum number of gRPC threads.
    142 
    +thread-pool-size: 20

  • buildgrid/_app/settings/remote-storage.yml deleted
    1 
    -server:
    2 
    -  - !channel
    3 
    -    port: 50051
    4 
    -    insecure-mode: true
    5 
    -
    6 
    -description: |
    7 
    -  A single default instance with remote storage.
    8 
    -
    9 
    -instances:
    10 
    -  - name: ''
    11 
    -    description: |
    12 
    -      The main server
    13 
    -
    14 
    -    storages:
    15 
    -      - !remote-storage &main-storage
    16 
    -        url: http://localhost:50052
    17 
    -        instance-name: main
    18 
    -        # credentials:
    19 
    -        #   tls-client-key: null
    20 
    -        #   tls-client-cert: null
    21 
    -        #   tls-server-cert: null
    22 
    -
    23 
    -    services:
    24 
    -      - !action-cache &main-action
    25 
    -        storage: *main-storage
    26 
    -        max-cached-refs: 256
    27 
    -        allow-updates: true
    28 
    -
    29 
    -      - !execution
    30 
    -        storage: *main-storage
    31 
    -        action-cache: *main-action
    32 
    -
    33 
    -      - !cas
    34 
    -        storage: *main-storage
    35 
    -
    36 
    -      - !bytestream
    37 
    -        storage: *main-storage

  • buildgrid/server/bots/instance.py
    ... ... @@ -93,7 +93,6 @@ class BotsInterface:
    93 93 
             """ Client updates the server. Any changes in state to the Lease should be
    94 94 
             registered server side. Assigns available leases with work.
    95 95 
             """
    96 
    -        self.__logger.debug("Updating bot session name=[%s]", name)
    97 96 
             self._check_bot_ids(bot_session.bot_id, name)
    98 97 
             self._check_assigned_leases(bot_session)
    99 98

  • buildgrid/server/instance.py
    ... ... @@ -91,6 +91,8 @@ class BuildGridServer:
    91 91 
             self.__grpc_server = grpc.server(self.__grpc_executor,
    92 92 
                                              options=(('grpc.so_reuseport', 0),))
    93 93
    94 
    +        self.__logger.debug("Setting up gRPC server with thread-limit=[%s]", max_workers)
    95 
    +
    94 96 
             self.__main_loop = asyncio.get_event_loop()
    95 97
    96 98 
             self.__monitoring_bus = None

  • buildgrid/_app/settings/cas.yml → data/config/artifacts.conf
    ... ... @@ -2,31 +2,34 @@ server:
    2 2 
       - !channel
    3 3 
         port: 50052
    4 4 
         insecure-mode: true
    5 
    -    # credentials:
    6 
    -    #   tls-server-key: null
    7 
    -    #   tls-server-cert: null
    8 
    -    #   tls-client-certs: null
    9 5
    10 
    -description: |
    11 
    -  Just a CAS with some reference storage.
    6 
    +description: >
    7 
    +  Artifact server configuration:
    8 
    +    - Unauthenticated plain HTTP at :50052
    9 
    +    - Single instance: (empty-name)
    10 
    +    - On-disk data stored in $HOME
    11 
    +    - Hosted services:
    12 
    +       - ReferenceStorage
    13 
    +       - ContentAddressableStorage
    14 
    +       - ByteStream
    12 15
    13 16 
     instances:
    14 17 
       - name: ''
    15 18 
         description: |
    16 
    -      The main server
    19 
    +      The unique '' instance.
    17 20
    18 21 
         storages:
    19 
    -      - !disk-storage &main-storage
    20 
    -        path: !expand-path $HOME/cas
    22 
    +      - !disk-storage &data-store
    23 
    +        path: !expand-path $HOME/.cache/buildgrid/store
    21 24
    22 25 
         services:
    23 26 
           - !cas
    24 
    -        storage: *main-storage
    27 
    +        storage: *data-store
    25 28
    26 29 
           - !bytestream
    27 
    -        storage: *main-storage
    30 
    +        storage: *data-store
    28 31
    29 32 
           - !reference-cache
    30 
    -        storage: *main-storage
    31 
    -        max-cached-refs: 256
    33 
    +        storage: *data-store
    34 
    +        max-cached-refs: 512
    32 35 
             allow-updates: true

  • data/config/controller.conf
    1 
    +server:
    2 
    +  - !channel
    3 
    +    port: 50051
    4 
    +    insecure-mode: true
    5 
    +
    6 
    +description: >
    7 
    +  Docker Compose controller configuration:
    8 
    +    - Unauthenticated plain HTTP at :50051
    9 
    +    - Single instance: local
    10 
    +    - Expects a remote CAS at :50052
    11 
    +    - Hosted services:
    12 
    +       - ActionCache
    13 
    +       - Execute
    14 
    +
    15 
    +authorization:
    16 
    +  method: none
    17 
    +
    18 
    +monitoring:
    19 
    +  enabled: false
    20 
    +
    21 
    +instances:
    22 
    +  - name: local
    23 
    +    description: |
    24 
    +      The unique 'local' instance.
    25 
    +
    26 
    +    storages:
    27 
    +      - !remote-storage &data-store
    28 
    +        url: http://storage:50052
    29 
    +        instance-name: local
    30 
    +
    31 
    +    services:
    32 
    +      - !action-cache &build-cache
    33 
    +        storage: *data-store
    34 
    +        max-cached-refs: 256
    35 
    +        cache-failed-actions: true
    36 
    +        allow-updates: true
    37 
    +
    38 
    +      - !execution
    39 
    +        storage: *data-store
    40 
    +        action-cache: *build-cache

  • data/config/default.conf
    1 
    +server:
    2 
    +  - !channel
    3 
    +    port: 50051
    4 
    +    insecure-mode: true
    5 
    +
    6 
    +description: >
    7 
    +  BuildGrid's default configuration:
    8 
    +    - Unauthenticated plain HTTP at :50052
    9 
    +    - Single instance: main
    10 
    +    - In-memory data, max. 2Gio
    11 
    +    - Hosted services:
    12 
    +       - ActionCache
    13 
    +       - Execute
    14 
    +       - ContentAddressableStorage
    15 
    +       - ByteStream
    16 
    +
    17 
    +authorization:
    18 
    +  method: none
    19 
    +
    20 
    +monitoring:
    21 
    +  enabled: false
    22 
    +
    23 
    +instances:
    24 
    +  - name: ''
    25 
    +    description: |
    26 
    +      The unique '' instance.
    27 
    +
    28 
    +    storages:
    29 
    +      - !lru-storage &data-store
    30 
    +        size: 2048M
    31 
    +
    32 
    +    services:
    33 
    +      - !action-cache &build-cache
    34 
    +        storage: *data-store
    35 
    +        max-cached-refs: 256
    36 
    +        cache-failed-actions: true
    37 
    +        allow-updates: true
    38 
    +
    39 
    +      - !execution
    40 
    +        storage: *data-store
    41 
    +        action-cache: *build-cache
    42 
    +
    43 
    +      - !cas
    44 
    +        storage: *data-store
    45 
    +
    46 
    +      - !bytestream
    47 
    +        storage: *data-store

  • data/config/storage.conf
    1 
    +server:
    2 
    +  - !channel
    3 
    +    port: 50052
    4 
    +    insecure-mode: true
    5 
    +
    6 
    +description: >
    7 
    +  Docker Compose storage configuration:
    8 
    +    - Unauthenticated plain HTTP at :50052
    9 
    +    - Single instance: local
    10 
    +    - On-disk data stored in /var
    11 
    +    - Hosted services:
    12 
    +       - ReferenceStorage
    13 
    +       - ContentAddressableStorage
    14 
    +       - ByteStream
    15 
    +
    16 
    +authorization:
    17 
    +  method: none
    18 
    +
    19 
    +monitoring:
    20 
    +  enabled: false
    21 
    +
    22 
    +instances:
    23 
    +  - name: local
    24 
    +    description: |
    25 
    +      The unique 'local' instance.
    26 
    +
    27 
    +    storages:
    28 
    +      - !disk-storage &data-store
    29 
    +        path: /var/lib/buildgrid/store
    30 
    +
    31 
    +    services:
    32 
    +      - !cas
    33 
    +        storage: *data-store
    34 
    +
    35 
    +      - !bytestream
    36 
    +        storage: *data-store
    37 
    +
    38 
    +      - !reference-cache
    39 
    +        storage: *data-store
    40 
    +        max-cached-refs: 1024
    41 
    +        allow-updates: true

  • docker-compose.yml
    1 
    +##
    2 
    +# BuildGrid's Docker Compose manifest.
    3 
    +#
    4 
    +#   ¡FOR LOCAL TESTING ONLY!
    5 
    +#
    6 
    +# Spins-up a 'local' grid instance:
    7 
    +#   - Controller at http://localhost:50051
    8 
    +#   - CAS + AC at: http://localhost:50052
    9 
    +#
    10 
    +# Basic usage:
    11 
    +#  - docker-compose build
    12 
    +#  - docker-compose up --scale bots=10
    13 
    +#  - docker-compose down
    14 
    +#  - docker volume inspect buildgrid_data
    15 
    +#  - docker volume rm buildgrid_data
    16 
    +#
    17 
    +version: "3.2"
    18 
    +
    19 
    +services:
    20 
    +  storage:
    21 
    +    build:
    22 
    +      context: .
    23 
    +    image: buildgrid:local
    24 
    +    command: [
    25 
    +      "server", "start", "-vvv",
    26 
    +      "/app/config/storage.conf"]
    27 
    +    volumes:
    28 
    +      - type: volume
    29 
    +        source: data
    30 
    +        target: /var/lib/buildgrid/store
    31 
    +        volume:
    32 
    +          nocopy: true
    33 
    +      - type: bind
    34 
    +        source: ./data/config/storage.conf
    35 
    +        target: /app/config/storage.conf
    36 
    +    ports:
    37 
    +      - "50052:50052"
    38 
    +    networks:
    39 
    +      - backend
    40 
    +      - host
    41 
    +
    42 
    +  controller:
    43 
    +    image: buildgrid:local
    44 
    +    command: [
    45 
    +      "server", "start", "-vvv",
    46 
    +      "/app/config/controller.conf"]
    47 
    +    volumes:
    48 
    +      - type: bind
    49 
    +        source: ./data/config/controller.conf
    50 
    +        target: /app/config/controller.conf
    51 
    +    ports:
    52 
    +      - "50051:50051"
    53 
    +    networks:
    54 
    +      - backend
    55 
    +      - host
    56 
    +
    57 
    +  bots:  # To be scaled horizontaly
    58 
    +    image: buildgrid:local
    59 
    +    command: [
    60 
    +      "bot", "--parent=local",
    61 
    +      "--remote=http://controller:50051",
    62 
    +      "--remote-cas=http://storage:50052",
    63 
    +      "host-tools"]
    64 
    +    depends_on:
    65 
    +      - controller
    66 
    +    networks:
    67 
    +      - backend
    68 
    +
    69 
    +networks:
    70 
    +  backend:
    71 
    +  host:
    72 
    +
    73 
    +volumes:
    74 
    +  data:

  • docs/source/installation.rst
    ... ... @@ -110,7 +110,7 @@ To clone the source code and build a Docker image, simply run:
    110 110 
         the image.
    111 111
    112 112 
     After building the Docker image, to run BuildGrid using the default configuration file
    113 
    -(found in `buildgrid/_app/settings/default.yml`), simply run:
    113 
    +(found in `data/config/default.conf`), simply run:
    114 114
    115 115 
     .. code-block:: sh
    116 116

  • docs/source/using_cas_server.rst
    ... ... @@ -27,7 +27,7 @@ This defines a single ``main`` instance of the ``CAS``, ``Bytestream`` and ``Ref
    27 27
    28 28 
     .. code-block:: sh
    29 29
    30 
    -   bgd server start example.conf
    30 
    +   bgd server start data/config/artifacts.conf
    31 31
    32 32 
     The server should now be available to use.
    33 33

  • docs/source/using_internal.rst
    ... ... @@ -17,7 +17,7 @@ In one terminal, start a server:
    17 17
    18 18 
     .. code-block:: sh
    19 19
    20 
    -   bgd server start buildgrid/_app/settings/default.yml
    20 
    +   bgd server start data/config/default.conf
    21 21
    22 22 
     In another terminal, upload an action to CAS:
    23 23
    ... ... @@ -78,7 +78,7 @@ Now start a BuildGrid server, passing it a directory it can write a CAS to:
    78 78
    79 79 
     .. code-block:: sh
    80 80
    81 
    -   bgd server start buildgrid/_app/settings/default.yml
    81 
    +   bgd server start data/config/default.conf
    82 82
    83 83 
     Start the following bot session:
    84 84

    • —
    View it on GitLab.
    You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.

    [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]