GNOME.org
 

[Notes] [Git][BuildGrid/buildgrid][mablanch/160-docker-compose] 8 commits: Scheduler: add mutex to job queue

Title: GitLab

Martin Blanchard pushed to branch mablanch/160-docker-compose at BuildGrid / buildgrid

Commits:

24 changed files:

Changes:

  • .dockerignore
    1 
    +# Ignore everything:
    2 
    +*
    3 
    +
    4 
    +# Whitelist:
    5 
    +!buildgrid
    6 
    +!data/config
    7 
    +!tests
    8 
    +!LICENSE
    9 
    +!README.rst
    10 
    +!requirements.txt
    11 
    +!requirements.auth.txt
    12 
    +!setup.cfg
    13 
    +!setup.py
    14 
    +!_version.py
    15 
    +!.coveragerc
    16 
    +!.pylintrc

  • .gitlab-ci.yml
    ... ... @@ -31,7 +31,7 @@ before_script:
    31 31 
     .run-dummy-job-template: &dummy-job
    32 32 
       stage: test
    33 33 
       script:
    34 
    -    - ${BGD} server start buildgrid/_app/settings/default.yml &
    34 
    +    - ${BGD} server start data/config/default.conf &
    35 35 
         - sleep 1 # Allow server to boot
    36 36 
         - ${BGD} bot dummy &
    37 37 
         - ${BGD} cas upload-dummy

  • BUILDSTREAM_README.rst deleted
    1 
    -Temp Demo Instructions
    2 
    -======================
    3 
    -
    4 
    -A quick guide to getting remote execution working with BuildStream. Please change URL and certifcates / keys to your own.
    5 
    -
    6 
    -Downloaded and build::
    7 
    -
    8 
    -  https://gitlab.com/BuildStream/buildbox
    9 
    -
    10 
    -Copy build to bin/.
    11 
    -
    12 
    -Checkout branch::
    13 
    -
    14 
    -  https://gitlab.com/BuildStream/buildstream/tree/jmac/source_pushing_experiments
    15 
    -
    16 
    -Update to your URL::
    17 
    -
    18 
    -  https://gitlab.com/BuildStream/buildstream/blob/jmac/source_pushing_experiments/buildstream/sandbox/_sandboxremote.py#L73
    19 
    -
    20 
    -Start artifact server::
    21 
    -
    22 
    -  bst-artifact-server --port 11001 --server-key server.key --server-cert server.crt --client-certs client.crt --enable-push /home/user/
    23 
    -
    24 
    -Start bgd server::
    25 
    -
    26 
    -  bgd server start
    27 
    -
    28 
    -Run::
    29 
    -
    30 
    -  bgd bot buildbox
    31 
    -
    32 
    -Update project.conf in build area with::
    33 
    -
    34 
    -  artifacts:
    35 
    -      url: https://localhost:11001
    36 
    -      server-cert: server.crt
    37 
    -
    38 
    -      # Optional client key pair for authentication
    39 
    -      client-key: client.key
    40 
    -      client-cert: client.crt
    41 
    -
    42 
    -      push: true
    43 
    -
    44 
    -Run build with::
    45 
    -
    46 
    -  bst build --track something.bst

  • Dockerfile
    1 
    -FROM python:3.5-stretch
    1 
    +##
    2 
    +# BuildGrid's Docker build manifest.
    3 
    +#
    4 
    +#  ¡FOR LOCAL DEVELOPMENT ONLY!
    5 
    +#
    6 
    +# Builds an image from local sources.
    7 
    +#
    2 8
    3 
    -# Point the path to where buildgrid gets installed
    4 
    -ENV PATH=$PATH:/root/.local/bin/
    9 
    +FROM python:3.5-slim-stretch
    5 10
    6 
    -# Upgrade python modules
    7 
    -RUN python3 -m pip install --upgrade setuptools pip
    8 
    -
    9 
    -# Use /app as the current working directory
    11 
    +# Use /app as working directory:
    10 12 
     WORKDIR /app
    11 13
    12 
    -# Copy the repo contents (source, config files, etc) in the WORKDIR
    13 
    -COPY . .
    14 
    +# Create a virtual environment:
    15 
    +RUN [ \
    16 
    +"python3", "-m", "venv", "/app/env" \
    17 
    +]
    18 
    +
    19 
    +# Upgrade Python core modules:
    20 
    +RUN [ \
    21 
    +"/app/env/bin/python", "-m", "pip", \
    22 
    +"install", "--upgrade", \
    23 
    +"setuptools", "pip", "wheel" \
    24 
    +]
    25 
    +
    26 
    +# Install the main requirements:
    27 
    +ADD requirements.txt /app
    28 
    +RUN [ \
    29 
    +"/app/env/bin/python", "-m", "pip", \
    30 
    +"install", "--requirement", \
    31 
    +"requirements.txt" \
    32 
    +]
    33 
    +
    34 
    +# Install the auth. requirements:
    35 
    +ADD requirements.auth.txt /app
    36 
    +RUN [ \
    37 
    +"/app/env/bin/python", "-m", "pip", \
    38 
    +"install", "--requirement", \
    39 
    +"requirements.auth.txt" \
    40 
    +]
    41 
    +
    42 
    +# Copy the repo. contents:
    43 
    +COPY . /app
    14 44
    15 
    -# Install BuildGrid
    16 
    -RUN pip install --user --editable .
    45 
    +# Install BuildGrid:
    46 
    +RUN [ \
    47 
    +"/app/env/bin/python", "-m", "pip", \
    48 
    +"install", "--editable", \
    49 
    +".[auth,tests]" \
    50 
    +]
    17 51
    18 
    -# Entry Point of the image (should get an additional argument from CMD, the path to the config file)
    19 
    -ENTRYPOINT ["bgd", "server", "start", "-vv"]
    52 
    +# Entry-point for the image:
    53 
    +ENTRYPOINT [ \
    54 
    +"/app/env/bin/bgd" \
    55 
    +]
    20 56
    21 
    -# Default config file (used if no CMD specified when running)
    22 
    -CMD ["buildgrid/_app/settings/default.yml"]
    57 
    +# Default command (default config.):
    58 
    +CMD [ \
    59 
    +"server", "start", \
    60 
    +"data/config/default.conf", \
    61 
    +"-vvv" \
    62 
    +]

  • buildgrid/_app/commands/cmd_server.py
    ... ... @@ -119,6 +119,14 @@ def _create_server_from_config(configuration):
    119 119 
                 click.echo("Error: Configuration, {}.".format(e), err=True)
    120 120 
                 sys.exit(-1)
    121 121
    122 
    +    if 'thread-pool-size' in configuration:
    123 
    +        try:
    124 
    +            kargs['max_workers'] = int(configuration['thread-pool-size'])
    125 
    +
    126 
    +        except ValueError as e:
    127 
    +            click.echo("Error: Configuration, {}.".format(e), err=True)
    128 
    +            sys.exit(-1)
    129 
    +
    122 130 
         server = BuildGridServer(**kargs)
    123 131
    124 132 
         try:

  • buildgrid/_app/settings/default.yml deleted
    1 
    -server:
    2 
    -  - !channel
    3 
    -    port: 50051
    4 
    -    insecure-mode: true
    5 
    -
    6 
    -description: |
    7 
    -  A single default instance.
    8 
    -
    9 
    -instances:
    10 
    -  - name: ''
    11 
    -    description: |
    12 
    -      The main server
    13 
    -
    14 
    -    storages:
    15 
    -      - !disk-storage &main-storage
    16 
    -        path: !expand-path $HOME/cas
    17 
    -
    18 
    -    services:
    19 
    -      - !action-cache &main-action
    20 
    -        storage: *main-storage
    21 
    -        max-cached-refs: 256
    22 
    -        allow-updates: true
    23 
    -
    24 
    -      - !execution
    25 
    -        storage: *main-storage
    26 
    -        action-cache: *main-action
    27 
    -
    28 
    -      - !cas
    29 
    -        storage: *main-storage
    30 
    -
    31 
    -      - !bytestream
    32 
    -        storage: *main-storage

  • buildgrid/_app/settings/reference.yml
    ... ... @@ -136,3 +136,7 @@ monitoring:
    136 136 
       #  binary  - Protobuf binary format.
    137 137 
       #  json    - JSON format.
    138 138 
       serialization-format: binary
    139 
    +
    140 
    +##
    141 
    +# Maximum number of gRPC threads.
    142 
    +thread-pool-size: 20

  • buildgrid/_app/settings/remote-storage.yml deleted
    1 
    -server:
    2 
    -  - !channel
    3 
    -    port: 50051
    4 
    -    insecure-mode: true
    5 
    -
    6 
    -description: |
    7 
    -  A single default instance with remote storage.
    8 
    -
    9 
    -instances:
    10 
    -  - name: ''
    11 
    -    description: |
    12 
    -      The main server
    13 
    -
    14 
    -    storages:
    15 
    -      - !remote-storage &main-storage
    16 
    -        url: http://localhost:50052
    17 
    -        instance-name: main
    18 
    -        # credentials:
    19 
    -        #   tls-client-key: null
    20 
    -        #   tls-client-cert: null
    21 
    -        #   tls-server-cert: null
    22 
    -
    23 
    -    services:
    24 
    -      - !action-cache &main-action
    25 
    -        storage: *main-storage
    26 
    -        max-cached-refs: 256
    27 
    -        allow-updates: true
    28 
    -
    29 
    -      - !execution
    30 
    -        storage: *main-storage
    31 
    -        action-cache: *main-action
    32 
    -
    33 
    -      - !cas
    34 
    -        storage: *main-storage
    35 
    -
    36 
    -      - !bytestream
    37 
    -        storage: *main-storage

  • buildgrid/server/bots/instance.py
    ... ... @@ -32,6 +32,8 @@ class BotsInterface:
    32 32
    33 33 
         def __init__(self, scheduler):
    34 34 
             self.__logger = logging.getLogger(__name__)
    35 
    +        # Turn on debug mode based on log verbosity level:
    36 
    +        self.__debug = self.__logger.getEffectiveLevel() <= logging.DEBUG
    35 37
    36 38 
             self._scheduler = scheduler
    37 39 
             self._instance_name = None
    ... ... @@ -65,13 +67,11 @@ class BotsInterface:
    65 67 
             register with the service, the old one should be closed along
    66 68 
             with all its jobs.
    67 69 
             """
    68 
    -        bot_id = bot_session.bot_id
    69 
    -
    70 
    -        if bot_id == "":
    71 
    -            raise InvalidArgumentError("bot_id needs to be set by client")
    70 
    +        if not bot_session.bot_id:
    71 
    +            raise InvalidArgumentError("Bot's id must be set by client.")
    72 72
    73 73 
             try:
    74 
    -            self._check_bot_ids(bot_id)
    74 
    +            self._check_bot_ids(bot_session.bot_id)
    75 75 
             except InvalidArgumentError:
    76 76 
                 pass
    77 77
    ... ... @@ -79,21 +79,27 @@ class BotsInterface:
    79 79 
             name = "{}/{}".format(parent, str(uuid.uuid4()))
    80 80 
             bot_session.name = name
    81 81
    82 
    -        self._bot_ids[name] = bot_id
    83 
    -
    84 
    -        self.__logger.info("Created bot session name=[%s] with bot_id=[%s]", name, bot_id)
    82 
    +        self._bot_ids[name] = bot_session.bot_id
    85 83
    86 84 
             # We want to keep a copy of lease ids we have assigned
    87 85 
             self._assigned_leases[name] = set()
    88 86
    89 87 
             self._request_leases(bot_session)
    88 
    +
    89 
    +        if self.__debug:
    90 
    +            self.__logger.info("Opened session name=[%s] for bot=[%s], leases=[%s]",
    91 
    +                               bot_session.name, bot_session.bot_id,
    92 
    +                               ",".join([lease.id[:8] for lease in bot_session.leases]))
    93 
    +        else:
    94 
    +            self.__logger.info("Opened session, name=[%s] for bot=[%s]",
    95 
    +                               bot_session.name, bot_session.bot_id)
    96 
    +
    90 97 
             return bot_session
    91 98
    92 99 
         def update_bot_session(self, name, bot_session):
    93 100 
             """ Client updates the server. Any changes in state to the Lease should be
    94 101 
             registered server side. Assigns available leases with work.
    95 102 
             """
    96 
    -        self.__logger.debug("Updating bot session name=[%s]", name)
    97 103 
             self._check_bot_ids(bot_session.bot_id, name)
    98 104 
             self._check_assigned_leases(bot_session)
    99 105
    ... ... @@ -111,6 +117,15 @@ class BotsInterface:
    111 117 
                     bot_session.leases.remove(lease)
    112 118
    113 119 
             self._request_leases(bot_session)
    120 
    +
    121 
    +        if self.__debug:
    122 
    +            self.__logger.info("Sending session update, name=[%s], for bot=[%s], leases=[%s]",
    123 
    +                               bot_session.name, bot_session.bot_id,
    124 
    +                               ",".join([lease.id[:8] for lease in bot_session.leases]))
    125 
    +        else:
    126 
    +            self.__logger.info("Sending session update, name=[%s], for bot=[%s]",
    127 
    +                               bot_session.name, bot_session.bot_id)
    128 
    +
    114 129 
             return bot_session
    115 130
    116 131 
         # --- Private API ---

  • buildgrid/server/instance.py
    ... ... @@ -91,6 +91,8 @@ class BuildGridServer:
    91 91 
             self.__grpc_server = grpc.server(self.__grpc_executor,
    92 92 
                                              options=(('grpc.so_reuseport', 0),))
    93 93
    94 
    +        self.__logger.debug("Setting up gRPC server with thread-limit=[%s]", max_workers)
    95 
    +
    94 96 
             self.__main_loop = asyncio.get_event_loop()
    95 97
    96 98 
             self.__monitoring_bus = None

  • buildgrid/server/scheduler.py
    ... ... @@ -22,6 +22,7 @@ Schedules jobs.
    22 22 
     import bisect
    23 23 
     from datetime import timedelta
    24 24 
     import logging
    25 
    +from threading import Lock
    25 26
    26 27 
     from buildgrid._enums import LeaseState, OperationStage
    27 28 
     from buildgrid._exceptions import NotFoundError
    ... ... @@ -53,6 +54,7 @@ class Scheduler:
    53 54 
             self.__jobs_by_name = {}  # Name to Job 1:1 mapping
    54 55
    55 56 
             self.__queue = []
    57 
    +        self.__queue_lock = Lock()
    56 58
    57 59 
             self._is_instrumented = monitor
    58 60
    ... ... @@ -297,27 +299,26 @@ class Scheduler:
    297 299 
                     worker properties, configuration and state at the time of the
    298 300 
                     request.
    299 301 
             """
    300 
    -        if not self.__queue:
    301 
    -            return []
    302 
    -
    303 
    -        # Looking for the first job that could be assigned to the worker...
    304 
    -        for job_index, job in enumerate(self.__queue):
    305 
    -            if self._worker_is_capable(worker_capabilities, job):
    306 
    -                self.__logger.info("Job scheduled to run: [%s]", job.name)
    302 
    +        # TODO: Replace with a more efficient way of doing this.
    303 
    +        with self.__queue_lock:
    304 
    +            # Looking for the first job that could be assigned to the worker...
    305 
    +            for job_index, job in enumerate(self.__queue):
    306 
    +                if self._worker_is_capable(worker_capabilities, job):
    307 
    +                    self.__logger.info("Job scheduled to run: [%s]", job.name)
    307 308
    308 
    -                lease = job.lease
    309 
    +                    lease = job.lease
    309 310
    310 
    -                if not lease:
    311 
    -                    # For now, one lease at a time:
    312 
    -                    lease = job.create_lease()
    311 
    +                    if not lease:
    312 
    +                        # For now, one lease at a time:
    313 
    +                        lease = job.create_lease()
    313 314
    314 
    -                if lease:
    315 
    -                    del self.__queue[job_index]
    316 
    -                    return [lease]
    315 
    +                    if lease:
    316 
    +                        del self.__queue[job_index]
    317 
    +                        return [lease]
    317 318
    318 
    -                return None
    319 
    +                    return []
    319 320
    320 
    -        return None
    321 
    +            return []
    321 322
    322 323 
         def update_job_lease_state(self, job_name, lease):
    323 324 
             """Requests a state transition for a job's current :class:Lease.
    ... ... @@ -551,11 +552,12 @@ class Scheduler:
    551 552 
             """Schedules or reschedules a job."""
    552 553 
             job = self.__jobs_by_name[job_name]
    553 554
    554 
    -        if job.operation_stage == OperationStage.QUEUED:
    555 
    -            self.__queue.sort()
    555 
    +        with self.__queue_lock:
    556 
    +            if job.operation_stage == OperationStage.QUEUED:
    557 
    +                self.__queue.sort()
    556 558
    557 
    -        else:
    558 
    -            bisect.insort(self.__queue, job)
    559 
    +            else:
    560 
    +                bisect.insort(self.__queue, job)
    559 561
    560 562 
             self.__logger.info("Job queued: [%s]", job.name)
    561 563
    ... ... @@ -564,7 +566,8 @@ class Scheduler:
    564 566 
             job = self.__jobs_by_name[job_name]
    565 567
    566 568 
             if job.operation_stage == OperationStage.QUEUED:
    567 
    -            self.__queue.remove(job)
    569 
    +            with self.__queue_lock:
    570 
    +                self.__queue.remove(job)
    568 571
    569 572 
             del self.__jobs_by_action[job.action_digest.hash]
    570 573 
             del self.__jobs_by_name[job.name]

  • buildgrid/_app/settings/cas.yml → data/config/artifacts.conf
    ... ... @@ -2,31 +2,34 @@ server:
    2 2 
       - !channel
    3 3 
         port: 50052
    4 4 
         insecure-mode: true
    5 
    -    # credentials:
    6 
    -    #   tls-server-key: null
    7 
    -    #   tls-server-cert: null
    8 
    -    #   tls-client-certs: null
    9 5
    10 
    -description: |
    11 
    -  Just a CAS with some reference storage.
    6 
    +description: >
    7 
    +  Artifact server configuration:
    8 
    +    - Unauthenticated plain HTTP at :50052
    9 
    +    - Single instance: (empty-name)
    10 
    +    - On-disk data stored in $HOME
    11 
    +    - Hosted services:
    12 
    +       - ReferenceStorage
    13 
    +       - ContentAddressableStorage
    14 
    +       - ByteStream
    12 15
    13 16 
     instances:
    14 17 
       - name: ''
    15 18 
         description: |
    16 
    -      The main server
    19 
    +      The unique '' instance.
    17 20
    18 21 
         storages:
    19 
    -      - !disk-storage &main-storage
    20 
    -        path: !expand-path $HOME/cas
    22 
    +      - !disk-storage &data-store
    23 
    +        path: !expand-path $HOME/.cache/buildgrid/store
    21 24
    22 25 
         services:
    23 26 
           - !cas
    24 
    -        storage: *main-storage
    27 
    +        storage: *data-store
    25 28
    26 29 
           - !bytestream
    27 
    -        storage: *main-storage
    30 
    +        storage: *data-store
    28 31
    29 32 
           - !reference-cache
    30 
    -        storage: *main-storage
    31 
    -        max-cached-refs: 256
    33 
    +        storage: *data-store
    34 
    +        max-cached-refs: 512
    32 35 
             allow-updates: true

  • data/config/controller.conf
    1 
    +server:
    2 
    +  - !channel
    3 
    +    port: 50051
    4 
    +    insecure-mode: true
    5 
    +
    6 
    +description: >
    7 
    +  Docker Compose controller configuration:
    8 
    +    - Unauthenticated plain HTTP at :50051
    9 
    +    - Single instance: local
    10 
    +    - Expects a remote CAS at :50052
    11 
    +    - Hosted services:
    12 
    +       - ActionCache
    13 
    +       - Execute
    14 
    +
    15 
    +authorization:
    16 
    +  method: none
    17 
    +
    18 
    +monitoring:
    19 
    +  enabled: false
    20 
    +
    21 
    +instances:
    22 
    +  - name: local
    23 
    +    description: |
    24 
    +      The unique 'local' instance.
    25 
    +
    26 
    +    storages:
    27 
    +      - !remote-storage &data-store
    28 
    +        url: http://storage:50052
    29 
    +        instance-name: local
    30 
    +
    31 
    +    services:
    32 
    +      - !action-cache &build-cache
    33 
    +        storage: *data-store
    34 
    +        max-cached-refs: 256
    35 
    +        cache-failed-actions: true
    36 
    +        allow-updates: true
    37 
    +
    38 
    +      - !execution
    39 
    +        storage: *data-store
    40 
    +        action-cache: *build-cache
    41 
    +
    42 
    +thread-pool-size: 200

  • data/config/default.conf
    1 
    +server:
    2 
    +  - !channel
    3 
    +    port: 50051
    4 
    +    insecure-mode: true
    5 
    +
    6 
    +description: >
    7 
    +  BuildGrid's default configuration:
    8 
    +    - Unauthenticated plain HTTP at :50052
    9 
    +    - Single instance: main
    10 
    +    - In-memory data, max. 2Gio
    11 
    +    - Hosted services:
    12 
    +       - ActionCache
    13 
    +       - Execute
    14 
    +       - ContentAddressableStorage
    15 
    +       - ByteStream
    16 
    +
    17 
    +authorization:
    18 
    +  method: none
    19 
    +
    20 
    +monitoring:
    21 
    +  enabled: false
    22 
    +
    23 
    +instances:
    24 
    +  - name: ''
    25 
    +    description: |
    26 
    +      The unique '' instance.
    27 
    +
    28 
    +    storages:
    29 
    +      - !lru-storage &data-store
    30 
    +        size: 2048M
    31 
    +
    32 
    +    services:
    33 
    +      - !action-cache &build-cache
    34 
    +        storage: *data-store
    35 
    +        max-cached-refs: 256
    36 
    +        cache-failed-actions: true
    37 
    +        allow-updates: true
    38 
    +
    39 
    +      - !execution
    40 
    +        storage: *data-store
    41 
    +        action-cache: *build-cache
    42 
    +
    43 
    +      - !cas
    44 
    +        storage: *data-store
    45 
    +
    46 
    +      - !bytestream
    47 
    +        storage: *data-store

  • data/config/storage.conf
    1 
    +server:
    2 
    +  - !channel
    3 
    +    port: 50052
    4 
    +    insecure-mode: true
    5 
    +
    6 
    +description: >
    7 
    +  Docker Compose storage configuration:
    8 
    +    - Unauthenticated plain HTTP at :50052
    9 
    +    - Single instance: local
    10 
    +    - On-disk data stored in /var
    11 
    +    - Hosted services:
    12 
    +       - ReferenceStorage
    13 
    +       - ContentAddressableStorage
    14 
    +       - ByteStream
    15 
    +
    16 
    +authorization:
    17 
    +  method: none
    18 
    +
    19 
    +monitoring:
    20 
    +  enabled: false
    21 
    +
    22 
    +instances:
    23 
    +  - name: local
    24 
    +    description: |
    25 
    +      The unique 'local' instance.
    26 
    +
    27 
    +    storages:
    28 
    +      - !disk-storage &data-store
    29 
    +        path: /var/lib/buildgrid/store
    30 
    +
    31 
    +    services:
    32 
    +      - !cas
    33 
    +        storage: *data-store
    34 
    +
    35 
    +      - !bytestream
    36 
    +        storage: *data-store
    37 
    +
    38 
    +      - !reference-cache
    39 
    +        storage: *data-store
    40 
    +        max-cached-refs: 1024
    41 
    +        allow-updates: true

  • docker-compose.yml
    1 
    +##
    2 
    +# BuildGrid's Docker Compose manifest.
    3 
    +#
    4 
    +#   ¡FOR LOCAL DEVELOPMENT ONLY!
    5 
    +#
    6 
    +# Spins-up a 'local' grid instance:
    7 
    +#   - Controller at http://localhost:50051
    8 
    +#   - CAS + AC at: http://localhost:50052
    9 
    +#
    10 
    +# Basic usage:
    11 
    +#  - docker-compose build
    12 
    +#  - docker-compose up --scale bots=10
    13 
    +#  - docker-compose down
    14 
    +#  - docker volume inspect buildgrid_data
    15 
    +#  - docker volume rm buildgrid_data
    16 
    +#  - docker image rm buildgrid:local
    17 
    +#
    18 
    +version: "3.2"
    19 
    +
    20 
    +services:
    21 
    +  storage:
    22 
    +    build:
    23 
    +      context: .
    24 
    +    image: buildgrid:local
    25 
    +    command: [
    26 
    +      "server", "start", "-vvv",
    27 
    +      "/app/config/storage.conf"]
    28 
    +    volumes:
    29 
    +      - type: volume
    30 
    +        source: data
    31 
    +        target: /var/lib/buildgrid/store
    32 
    +        volume:
    33 
    +          nocopy: true
    34 
    +      - type: bind
    35 
    +        source: ./data/config/storage.conf
    36 
    +        target: /app/config/storage.conf
    37 
    +    ports:
    38 
    +      - "50052:50052"
    39 
    +    networks:
    40 
    +      - backend
    41 
    +      - host
    42 
    +
    43 
    +  controller:
    44 
    +    image: buildgrid:local
    45 
    +    command: [
    46 
    +      "server", "start", "-vvv",
    47 
    +      "/app/config/controller.conf"]
    48 
    +    volumes:
    49 
    +      - type: bind
    50 
    +        source: ./data/config/controller.conf
    51 
    +        target: /app/config/controller.conf
    52 
    +    ports:
    53 
    +      - "50051:50051"
    54 
    +    networks:
    55 
    +      - backend
    56 
    +      - host
    57 
    +
    58 
    +  bots:  # To be scaled horizontaly
    59 
    +    image: buildgrid:local
    60 
    +    command: [
    61 
    +      "bot", "--parent=local",
    62 
    +      "--remote=http://controller:50051",
    63 
    +      "--remote-cas=http://storage:50052",
    64 
    +      "host-tools"]
    65 
    +    depends_on:
    66 
    +      - controller
    67 
    +    networks:
    68 
    +      - backend
    69 
    +
    70 
    +networks:
    71 
    +  backend:
    72 
    +  host:
    73 
    +
    74 
    +volumes:
    75 
    +  data:

  • docs/source/installation.rst
    ... ... @@ -21,20 +21,24 @@ How to install BuildGrid directly onto your machine.
    21 21 
     Prerequisites
    22 22 
     ~~~~~~~~~~~~~
    23 23
    24 
    -BuildGrid only supports ``python3 >= 3.5`` but has no system requirements. Main
    25 
    -Python dependencies, automatically handled during installation, include:
    24 
    +BuildGrid only supports ``python3 >= 3.5.3`` but has no system requirements.
    25 
    +Main Python dependencies, automatically handled during installation, include:
    26 26
    27 27 
     - `boto3`_: the Amazon Web Services (AWS) SDK for Python.
    28 28 
     - `click`_: a Python composable command line library.
    29 29 
     - `grpcio`_: Google's `gRPC`_ Python interface.
    30 
    +- `janus`_: a mixed sync-async Python queue.
    30 31 
     - `protobuf`_: Google's `protocol-buffers`_ Python interface.
    32 
    +- `PyYAML`_: a YAML parser and emitter for Python.
    31 33
    32 34 
     .. _boto3: https://pypi.org/project/boto3
    33 35 
     .. _click: https://pypi.org/project/click
    34 36 
     .. _grpcio: https://pypi.org/project/grpcio
    35 37 
     .. _gRPC: https://grpc.io
    38 
    +.. _janus: https://pypi.org/project/janus
    36 39 
     .. _protobuf: https://pypi.org/project/protobuf
    37 40 
     .. _protocol-buffers: https://developers.google.com/protocol-buffers
    41 
    +.. _PyYAML: https://pypi.org/project/PyYAML
    38 42
    39 43
    40 44 
     .. _install-host-source-install:
    ... ... @@ -42,20 +46,30 @@ Python dependencies, automatically handled during installation, include:
    42 46 
     Install from sources
    43 47 
     ~~~~~~~~~~~~~~~~~~~~
    44 48
    45 
    -BuildGrid has ``setuptools`` support. In order to install it to your home
    46 
    -directory, typically under ``~/.local``, simply run:
    49 
    +BuildGrid has ``setuptools`` support. We recommend installing it in a dedicated
    50 
    +`virtual environment`_. In order to do so in an environment named ``env``
    51 
    +placed in the source tree, run:
    47 52
    48 53 
     .. code-block:: sh
    49 54
    50 
    -   git clone https://gitlab.com/BuildGrid/buildgrid.git && cd buildgrid
    51 
    -   pip3 install --user --editable .
    55 
    +   git clone https://gitlab.com/BuildGrid/buildgrid.git
    56 
    +   cd buildgrid
    57 
    +   python3 -m venv env
    58 
    +   env/bin/python -m pip install --upgrade setuptools pip wheel
    59 
    +   env/bin/python -m pip install --editable .
    52 60
    53 
    -Additionally, and if your distribution does not already include it, you may
    54 
    -have to adjust your ``PATH``, in ``~/.bashrc``, with:
    61 
    +.. hint::
    62 
    +
    63 
    +   Once created, the virtual environment can be *activated* by sourcing the
    64 
    +   ``env/bin/activate`` script. In an activated terminal session, simply run
    65 
    +   ``deactivate`` to later *deactivate* it.
    66 
    +
    67 
    +Once completed, you can check that installation succeed by locally starting the
    68 
    +BuildGrid server with default configuration. Simply run:
    55 69
    56 70 
     .. code-block:: sh
    57 71
    58 
    -   export PATH="${PATH}:${HOME}/.local/bin"
    72 
    +   env/bin/bgd server start data/config/default.conf -vvv
    59 73
    60 74 
     .. note::
    61 75
    ... ... @@ -63,66 +77,111 @@ have to adjust your ``PATH``, in ``~/.bashrc``, with:
    63 77 
        ``tests``. They declare required dependency for, respectively, authentication
    64 78 
        and authorization management, generating documentation and running
    65 79 
        unit-tests. They can be use as helpers for setting up a development
    66 
    -   environment. To use them simply run:
    80 
    +   environment. To use them run:
    67 81
    68 82 
        .. code-block:: sh
    69 83
    70 
    -      pip3 install --user --editable ".[auth,docs,tests]"
    84 
    +      env/bin/python -m pip install --editable ".[auth,docs,tests]"
    85 
    +
    86 
    +.. _virtual environment: https://docs.python.org/3/library/venv.html
    71 87
    72 88
    73 89 
     .. install-docker:
    74 90
    75 
    -Installation through Docker
    76 
    ----------------------------
    91 
    +Install through Docker
    92 
    +----------------------
    77 93
    78 
    -How to build a Docker image that runs BuildGrid.
    94 
    +BuildGrid comes with Docker support for local development use-cases.
    79 95
    80 
    -.. _install-docker-prerequisites:
    96 
    +.. caution::
    81 97
    82 
    -Prerequisites
    83 
    -~~~~~~~~~~~~~
    98 
    +   The Docker manifests are intended to be use for **local development only**.
    99 
    +   Do **not** use them in production.
    84 100
    85 
    -A working Docker installation. Please consult `Docker's Getting Started Guide`_ if you don't already have it installed.
    101 
    +Please consult the `Get Started with Docker`_ guide if you are looking for
    102 
    +instructions on how to setup Docker on your machine.
    86 103
    87 
    -.. _`Docker's Getting Started Guide`: https://www.docker.com/get-started
    104 
    +.. _`Get Started with Docker`: https://www.docker.com/get-started
    88 105
    89 106
    90 107 
     .. _install-docker-build:
    91 108
    92 
    -Docker Container from Sources
    93 
    -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    109 
    +Docker build
    110 
    +~~~~~~~~~~~~
    94 111
    95 
    -To clone the source code and build a Docker image, simply run:
    112 
    +BuildGrid ships a ``Dockerfile`` manifest for building images from source using
    113 
    +``docker build``. In order to produce a ``buildgrid:local`` base image, run:
    96 114
    97 115 
     .. code-block:: sh
    98 116
    99 
    -   git clone https://gitlab.com/BuildGrid/buildgrid.git && cd buildgrid
    100 
    -   docker build -t buildgrid_server .
    117 
    +   git clone https://gitlab.com/BuildGrid/buildgrid.git
    118 
    +   cd buildgrid
    119 
    +   docker build --tag buildgrid:local .
    101 120
    102 121 
     .. note::
    103 122
    104 
    -   The image built will contain the contents of the source code directory, including
    105 
    -   configuration files.
    106 
    -
    123 
    +   The image built will contain the Python sources, including example
    124 
    +   configuration files. The main endpoint is the ``bgd`` CLI tools and the
    125 
    +   default command shall run the BuildGrid server loading default configuration.
    126 
    +
    127 
    +Once completed, you can check that build succeed by locally starting in a
    128 
    +container the BuildGrid server with default configuration. Simply run:
    129 
    +
    130 
    +.. code-block:: sh
    131 
    +
    132 
    +   docker run --interactive --publish 50051:50051 buildgrid:local
    133 
    +
    107 134 
     .. hint::
    108 135
    109 
    -    Whenever the source code is updated or new configuration files are made, you need to re-build
    110 
    -    the image.
    136 
    +   You can run any of the BuildGrid CLI tool using that image, simply pass extra
    137 
    +   arguments to ``docker run`` the same way you would pass them to ``bgd``.
    138 
    +
    139 
    +    Bear in mind that whenever the source code or the configuration files are
    140 
    +    updated, you **must** re-build the image.
    141 
    +
    142 
    +
    143 
    +.. _install-docker-compose:
    111 144
    112 
    -After building the Docker image, to run BuildGrid using the default configuration file
    113 
    -(found in `buildgrid/_app/settings/default.yml`), simply run:
    145 
    +Docker Compose
    146 
    +~~~~~~~~~~~~~~
    147 
    +
    148 
    +BuildGrid ships a ``docker-compose.yml`` manifest for building and running a
    149 
    +grid locally using ``docker-compose``. In order to produce a
    150 
    +``buildgrid:local`` base image, run:
    114 151
    115 152 
     .. code-block:: sh
    116 153
    117 
    -   docker run -i -p 50051:50051 buildgrid_server
    154 
    +   git clone https://gitlab.com/BuildGrid/buildgrid.git
    155 
    +   cd buildgrid
    156 
    +   docker-compose build
    157 
    +
    158 
    +Once completed, you can start a minimal grid by running:
    159 
    +
    160 
    +.. code-block:: sh
    161 
    +
    162 
    +   docker-compose up
    118 163
    119 164 
     .. note::
    120 165
    121 
    -    To run BuildGrid using a different configuration file, include the relative path to the
    122 
    -    configuration file at the end of the command above. For example, to run the default
    123 
    -    standalone CAS server (without an execution service), simply run:
    166 
    +   The grid is composed of three containers:
    167 
    +
    168 
    +   - An execution service available at ``http://localhost:50051``.
    169 
    +   - An CAS and action-cache service available at ``http://localhost:50052``.
    170 
    +   - A single ``local`` instance with one host-tools based worker bot attached.
    124 171
    125 
    -       .. code-block:: sh
    172 
    +.. hint::
    173 
    +
    174 
    +   You can spin up more bots by using ``docker-compose`` scaling capabilities:
    175 
    +
    176 
    +   .. code-block:: sh
    177 
    +
    178 
    +      docker-compose up --scale bots=12
    179 
    +
    180 
    +.. hint::
    126 181
    127 
    -            docker run -i -p 50052:50052 buildgrid_server buildgrid/_app/settings/cas.yml
    182 
    +   The contained services configuration files are bind mounted into the
    183 
    +   container, no need to rebuild the base image on configuration update.
    184 
    +   Configuration files are read from:
    128 185
    186 
    +   - ``data/config/controller.conf`` for the execution service.
    187 
    +   - ``data/config/storage.conf`` for the CAS and action-cache service.

  • docs/source/using_cas_server.rst
    ... ... @@ -27,7 +27,7 @@ This defines a single ``main`` instance of the ``CAS``, ``Bytestream`` and ``Ref
    27 27
    28 28 
     .. code-block:: sh
    29 29
    30 
    -   bgd server start example.conf
    30 
    +   bgd server start data/config/artifacts.conf
    31 31
    32 32 
     The server should now be available to use.
    33 33

  • docs/source/using_internal.rst
    ... ... @@ -17,7 +17,7 @@ In one terminal, start a server:
    17 17
    18 18 
     .. code-block:: sh
    19 19
    20 
    -   bgd server start buildgrid/_app/settings/default.yml
    20 
    +   bgd server start data/config/default.conf
    21 21
    22 22 
     In another terminal, upload an action to CAS:
    23 23
    ... ... @@ -78,7 +78,7 @@ Now start a BuildGrid server, passing it a directory it can write a CAS to:
    78 78
    79 79 
     .. code-block:: sh
    80 80
    81 
    -   bgd server start buildgrid/_app/settings/default.yml
    81 
    +   bgd server start data/config/default.conf
    82 82
    83 83 
     Start the following bot session:
    84 84

  • requirements.auth.txt
    1 
    +cryptography >= 1.8.0  # Required by pyjwt for RSA
    2 
    +pyjwt >= 1.5.0

  • requirements.docs.txt
    1 
    +sphinx
    2 
    +sphinx-click
    3 
    +sphinx-rtd-theme >= 0.4.2  # For HTML search fix (upstream #672)
    4 
    +sphinxcontrib-apidoc
    5 
    +sphinxcontrib-napoleon

  • requirements.tests.txt
    1 
    +coverage >= 4.5.0
    2 
    +moto < 1.3.7
    3 
    +pep8
    4 
    +psutil
    5 
    +pytest >= 3.8.0
    6 
    +pytest-cov >= 2.6.0
    7 
    +pytest-pep8
    8 
    +pytest-pylint

  • requirements.txt
    1 
    +boto3 < 1.8.0  # For moto compatibility (BuildGrid/buildgrid#65)
    2 
    +botocore < 1.11.0  # For moto compatibility (BuildGrid/buildgrid#65)
    3 
    +click
    4 
    +grpcio
    5 
    +janus
    6 
    +protobuf >= 3.6.1
    7 
    +pyyaml

  • setup.py
    ... ... @@ -77,63 +77,47 @@ class BuildGRPC(Command):
    77 77 
                             f.write(code)
    78 78
    79 79
    80 
    -def get_cmdclass():
    81 
    -    cmdclass = {
    82 
    -        'build_grpc': BuildGRPC,
    83 
    -    }
    84 
    -    return cmdclass
    85 
    -
    86 
    -auth_require = [
    87 
    -    'cryptography >= 1.8.0',  # Required by pyjwt for RSA
    88 
    -    'pyjwt >= 1.5.0',
    89 
    -]
    90 
    -
    91 
    -tests_require = [
    92 
    -    'coverage >= 4.5.0',
    93 
    -    'moto < 1.3.7',
    94 
    -    'pep8',
    95 
    -    'psutil',
    96 
    -    'pytest >= 3.8.0',
    97 
    -    'pytest-cov >= 2.6.0',
    98 
    -    'pytest-pep8',
    99 
    -    'pytest-pylint',
    100 
    -]
    101 
    -
    102 
    -docs_require = [
    103 
    -    'sphinx',
    104 
    -    'sphinx-click',
    105 
    -    'sphinx-rtd-theme >= 0.4.2',  # For HTML search fix (upstream #672)
    106 
    -    'sphinxcontrib-apidoc',
    107 
    -    'sphinxcontrib-napoleon',
    108 
    -]
    80 
    +# Load main requirements from file:
    81 
    +with open('requirements.txt') as requirements_file:
    82 
    +    install_requirements = requirements_file.read().splitlines()
    83 
    +
    84 
    +auth_requirements = []
    85 
    +# Load 'auth' requirements from dedicated file:
    86 
    +if os.path.isfile('requirements.auth.txt'):
    87 
    +    with open('requirements.auth.txt') as requirements_file:
    88 
    +        auth_requirements = requirements_file.read().splitlines()
    89 
    +
    90 
    +docs_requirements = []
    91 
    +# Load 'docs' requirements from dedicated file:
    92 
    +if os.path.isfile('requirements.docs.txt'):
    93 
    +    with open('requirements.docs.txt') as requirements_file:
    94 
    +        docs_requirements = requirements_file.read().splitlines()
    95 
    +
    96 
    +tests_requirements = []
    97 
    +# Load 'tests' requirements from dedicated file:
    98 
    +if os.path.isfile('requirements.tests.txt'):
    99 
    +    with open('requirements.tests.txt') as requirements_file:
    100 
    +        tests_requirements = requirements_file.read().splitlines()
    109 101
    110 102 
     setup(
    111 103 
         name="BuildGrid",
    112 104 
         version=__version__,
    113 
    -    cmdclass=get_cmdclass(),
    114 105 
         license="Apache License, Version 2.0",
    115 106 
         description="A remote execution service",
    107 
    +    cmdclass={
    108 
    +        'build_grpc': BuildGRPC, },
    116 109 
         packages=find_packages(),
    117 110 
         python_requires='>= 3.5.3',  # janus requirement
    118 
    -    install_requires=[
    119 
    -        'boto3 < 1.8.0',
    120 
    -        'botocore < 1.11.0',
    121 
    -        'click',
    122 
    -        'grpcio',
    123 
    -        'janus',
    124 
    -        'protobuf >= 3.6.1',
    125 
    -        'pyyaml',
    126 
    -    ],
    111 
    +    install_requires=install_requirements,
    112 
    +    setup_requires=['pytest-runner'],
    113 
    +    tests_require=tests_requirements,
    114 
    +    extras_require={
    115 
    +        'auth': auth_requirements,
    116 
    +        'docs': docs_requirements,
    117 
    +        'tests': tests_requirements, },
    127 118 
         entry_points={
    128 119 
             'console_scripts': [
    129 120 
                 'bgd = buildgrid._app:cli',
    130 121 
             ]
    131 
    -    },
    132 
    -    setup_requires=['pytest-runner'],
    133 
    -    tests_require=tests_require,
    134 
    -    extras_require={
    135 
    -        'auth': auth_require,
    136 
    -        'docs': docs_require,
    137 
    -        'tests': tests_require,
    138 
    -    },
    122 
    +    }
    139 123 
     )

    • —
    View it on GitLab.
    You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.

    [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]