GNOME.org
 

[Notes] [Git][BuildStream/buildstream][725-job-cancellation-on-remote-builds] 6 commits: sources: Don't redundantly document directory configuration

Title: GitLab

Raoul Hidalgo Charman pushed to branch 725-job-cancellation-on-remote-builds at BuildStream / buildstream

Commits:

25 changed files:

Changes:

  • buildstream/_yaml.py
    ... ... @@ -204,9 +204,8 @@ def load(filename, shortname=None, copy_tree=False, *, project=None, yaml_cache=
    204 204
    205 205 
             if not data:
    206 206 
                 data = load_data(contents, file, copy_tree=copy_tree)
    207 
    -
    208 
    -        if yaml_cache:
    209 
    -            yaml_cache.put_from_key(project, filename, key, data)
    207 
    +            if yaml_cache:
    208 
    +                yaml_cache.put_from_key(project, filename, key, data)
    210 209
    211 210 
             return data
    212 211 
         except FileNotFoundError as e:

  • buildstream/buildelement.py
    ... ... @@ -23,6 +23,8 @@ BuildElement - Abstract class for build elements
    23 23 
     The BuildElement class is a convenience element one can derive from for
    24 24 
     implementing the most common case of element.
    25 25
    26 
    +.. _core_buildelement_builtins:
    27 
    +
    26 28 
     Built-in functionality
    27 29 
     ----------------------
    28 30

  • buildstream/plugins/elements/autotools.py
    ... ... @@ -50,6 +50,9 @@ Here is the default configuration for the ``autotools`` element in full:
    50 50
    51 51 
       .. literalinclude:: ../../../buildstream/plugins/elements/autotools.yaml
    52 52 
          :language: yaml
    53 
    +
    54 
    +See :ref:`built-in functionality documentation <core_buildelement_builtins>` for
    55 
    +details on common configuration options for build elements.
    53 56 
     """
    54 57
    55 58 
     from buildstream import BuildElement

  • buildstream/plugins/elements/cmake.py
    ... ... @@ -49,6 +49,9 @@ Here is the default configuration for the ``cmake`` element in full:
    49 49
    50 50 
       .. literalinclude:: ../../../buildstream/plugins/elements/cmake.yaml
    51 51 
          :language: yaml
    52 
    +
    53 
    +See :ref:`built-in functionality documentation <core_buildelement_builtins>` for
    54 
    +details on common configuration options for build elements.
    52 55 
     """
    53 56
    54 57 
     from buildstream import BuildElement

  • buildstream/plugins/elements/distutils.py
    ... ... @@ -26,6 +26,9 @@ python distutils
    26 26 
     The distutils default configuration:
    27 27 
       .. literalinclude:: ../../../buildstream/plugins/elements/distutils.yaml
    28 28 
          :language: yaml
    29 
    +
    30 
    +See :ref:`built-in functionality documentation <core_buildelement_builtins>` for
    31 
    +details on common configuration options for build elements.
    29 32 
     """
    30 33
    31 34 
     from buildstream import BuildElement

  • buildstream/plugins/elements/import.py
    ... ... @@ -28,6 +28,9 @@ some configuration data.
    28 28 
     The empty configuration is as such:
    29 29 
       .. literalinclude:: ../../../buildstream/plugins/elements/import.yaml
    30 30 
          :language: yaml
    31 
    +
    32 
    +See :ref:`built-in functionality documentation <core_buildelement_builtins>` for
    33 
    +details on common configuration options for build elements.
    31 34 
     """
    32 35
    33 36 
     import os

  • buildstream/plugins/elements/make.py
    ... ... @@ -31,6 +31,9 @@ Here is the default configuration for the ``make`` element in full:
    31 31
    32 32 
       .. literalinclude:: ../../../buildstream/plugins/elements/make.yaml
    33 33 
          :language: yaml
    34 
    +
    35 
    +See :ref:`built-in functionality documentation <core_buildelement_builtins>` for
    36 
    +details on common configuration options for build elements.
    34 37 
     """
    35 38
    36 39 
     from buildstream import BuildElement

  • buildstream/plugins/elements/makemaker.py
    ... ... @@ -26,6 +26,9 @@ the Perl ExtUtil::MakeMaker build system
    26 26 
     The MakeMaker default configuration:
    27 27 
       .. literalinclude:: ../../../buildstream/plugins/elements/makemaker.yaml
    28 28 
          :language: yaml
    29 
    +
    30 
    +See :ref:`built-in functionality documentation <core_buildelement_builtins>` for
    31 
    +details on common configuration options for build elements.
    29 32 
     """
    30 33
    31 34 
     from buildstream import BuildElement

  • buildstream/plugins/elements/manual.py
    ... ... @@ -26,6 +26,9 @@ add custom build commands to the array understood by the :mod:`BuildElement <bui
    26 26 
     The empty configuration is as such:
    27 27 
       .. literalinclude:: ../../../buildstream/plugins/elements/manual.yaml
    28 28 
          :language: yaml
    29 
    +
    30 
    +See :ref:`built-in functionality documentation <core_buildelement_builtins>` for
    31 
    +details on common configuration options for build elements.
    29 32 
     """
    30 33
    31 34 
     from buildstream import BuildElement

  • buildstream/plugins/elements/meson.py
    ... ... @@ -46,6 +46,9 @@ Here is the default configuration for the ``meson`` element in full:
    46 46
    47 47 
       .. literalinclude:: ../../../buildstream/plugins/elements/meson.yaml
    48 48 
          :language: yaml
    49 
    +
    50 
    +See :ref:`built-in functionality documentation <core_buildelement_builtins>` for
    51 
    +details on common configuration options for build elements.
    49 52 
     """
    50 53
    51 54 
     from buildstream import BuildElement

  • buildstream/plugins/elements/modulebuild.py
    ... ... @@ -26,6 +26,9 @@ the Perl Module::Build build system
    26 26 
     The modulebuild default configuration:
    27 27 
       .. literalinclude:: ../../../buildstream/plugins/elements/modulebuild.yaml
    28 28 
          :language: yaml
    29 
    +
    30 
    +See :ref:`built-in functionality documentation <core_buildelement_builtins>` for
    31 
    +details on common configuration options for build elements.
    29 32 
     """
    30 33
    31 34 
     from buildstream import BuildElement

  • buildstream/plugins/elements/pip.py
    ... ... @@ -26,6 +26,9 @@ Python modules with pip
    26 26 
     The pip default configuration:
    27 27 
       .. literalinclude:: ../../../buildstream/plugins/elements/pip.yaml
    28 28 
          :language: yaml
    29 
    +
    30 
    +See :ref:`built-in functionality documentation <core_buildelement_builtins>` for
    31 
    +details on common configuration options for build elements.
    29 32 
     """
    30 33
    31 34 
     from buildstream import BuildElement

  • buildstream/plugins/elements/qmake.py
    ... ... @@ -26,6 +26,9 @@ the qmake build system
    26 26 
     The qmake default configuration:
    27 27 
       .. literalinclude:: ../../../buildstream/plugins/elements/qmake.yaml
    28 28 
          :language: yaml
    29 
    +
    30 
    +See :ref:`built-in functionality documentation <core_buildelement_builtins>` for
    31 
    +details on common configuration options for build elements.
    29 32 
     """
    30 33
    31 34 
     from buildstream import BuildElement

  • buildstream/plugins/sources/bzr.py
    ... ... @@ -31,9 +31,6 @@ bzr - stage files from a bazaar repository
    31 31 
        # Specify the bzr source kind
    32 32 
        kind: bzr
    33 33
    34 
    -   # Optionally specify a relative staging directory
    35 
    -   # directory: path/to/stage
    36 
    -
    37 34 
        # Specify the bzr url. Bazaar URLs come in many forms, see
    38 35 
        # `bzr help urlspec` for more information. Using an alias defined
    39 36 
        # in your project configuration is encouraged.
    ... ... @@ -53,6 +50,8 @@ bzr - stage files from a bazaar repository
    53 50 
        # revision number to the one on the tip of the branch specified in 'track'.
    54 51 
        ref: 6622
    55 52
    53 
    +See :ref:`built-in functionality doumentation <core_source_builtins>` for
    54 
    +details on common configuration options for sources.
    56 55 
     """
    57 56
    58 57 
     import os

  • buildstream/plugins/sources/deb.py
    ... ... @@ -33,9 +33,6 @@ deb - stage files from .deb packages
    33 33 
        # Specify the deb source kind
    34 34 
        kind: deb
    35 35
    36 
    -   # Optionally specify a relative staging directory
    37 
    -   # directory: path/to/stage
    38 
    -
    39 36 
        # Specify the deb url. Using an alias defined in your project
    40 37 
        # configuration is encouraged. 'bst track' will update the
    41 38 
        # sha256sum in 'ref' to the downloaded file's sha256sum.
    ... ... @@ -47,6 +44,8 @@ deb - stage files from .deb packages
    47 44 
        # Specify the basedir to return only the specified dir and its children
    48 45 
        base-dir: ''
    49 46
    47 
    +See :ref:`built-in functionality doumentation <core_source_builtins>` for
    48 
    +details on common configuration options for sources.
    50 49 
     """
    51 50
    52 51 
     import tarfile

  • buildstream/plugins/sources/git.py
    ... ... @@ -32,9 +32,6 @@ git - stage files from a git repository
    32 32 
        # Specify the git source kind
    33 33 
        kind: git
    34 34
    35 
    -   # Optionally specify a relative staging directory
    36 
    -   # directory: path/to/stage
    37 
    -
    38 35 
        # Specify the repository url, using an alias defined
    39 36 
        # in your project configuration is recommended.
    40 37 
        url: upstream:foo.git
    ... ... @@ -74,6 +71,9 @@ git - stage files from a git repository
    74 71 
            url: upstream:baz.git
    75 72 
            checkout: False
    76 73
    74 
    +See :ref:`built-in functionality doumentation <core_source_builtins>` for
    75 
    +details on common configuration options for sources.
    76 
    +
    77 77 
     **Configurable Warnings:**
    78 78
    79 79 
     This plugin provides the following configurable warnings:

  • buildstream/plugins/sources/local.py
    ... ... @@ -29,11 +29,11 @@ local - stage local files and directories
    29 29 
        # Specify the local source kind
    30 30 
        kind: local
    31 31
    32 
    -   # Optionally specify a relative staging directory
    33 
    -   # directory: path/to/stage
    34 
    -
    35 32 
        # Specify the project relative path to a file or directory
    36 33 
        path: files/somefile.txt
    34 
    +
    35 
    +See :ref:`built-in functionality doumentation <core_source_builtins>` for
    36 
    +details on common configuration options for sources.
    37 37 
     """
    38 38
    39 39 
     import os

  • buildstream/plugins/sources/ostree.py
    ... ... @@ -29,9 +29,6 @@ ostree - stage files from an OSTree repository
    29 29 
        # Specify the ostree source kind
    30 30 
        kind: ostree
    31 31
    32 
    -   # Optionally specify a relative staging directory
    33 
    -   # directory: path/to/stage
    34 
    -
    35 32 
        # Specify the repository url, using an alias defined
    36 33 
        # in your project configuration is recommended.
    37 34 
        url: upstream:runtime
    ... ... @@ -48,6 +45,9 @@ ostree - stage files from an OSTree repository
    48 45 
        # For signed ostree repositories, specify a local project relative
    49 46 
        # path to the public verifying GPG key for this remote.
    50 47 
        gpg-key: keys/runtime.gpg
    48 
    +
    49 
    +See :ref:`built-in functionality doumentation <core_source_builtins>` for
    50 
    +details on common configuration options for sources.
    51 51 
     """
    52 52
    53 53 
     import os

  • buildstream/plugins/sources/patch.py
    ... ... @@ -37,12 +37,11 @@ patch - apply locally stored patches
    37 37 
        # Specify the project relative path to a patch file
    38 38 
        path: files/somefile.diff
    39 39
    40 
    -   # Optionally specify the root directory for the patch
    41 
    -   # directory: path/to/stage
    42 
    -
    43 40 
        # Optionally specify the strip level, defaults to 1
    44 41 
        strip-level: 1
    45 42
    43 
    +See :ref:`built-in functionality doumentation <core_source_builtins>` for
    44 
    +details on common configuration options for sources.
    46 45 
     """
    47 46
    48 47 
     import os

  • buildstream/plugins/sources/pip.py
    ... ... @@ -54,19 +54,18 @@ Downloaded tarballs will be stored in a directory called ".bst_pip_downloads".
    54 54 
        packages:
    55 55 
        - flake8
    56 56
    57 
    -   # Optionally specify a relative staging directory
    58 
    -   directory: path/to/stage
    59 
    -
    60 57 
        # Specify the ref. It is a list of strings of format
    61 58 
        # "<package-name>==<version>", separated by "\\n".
    62 59 
        # Usually this will be contents of a requirements.txt file where all
    63 60 
        # package versions have been frozen.
    64 61 
        ref: "flake8==3.5.0\\nmccabe==0.6.1\\npkg-resources==0.0.0\\npycodestyle==2.3.1\\npyflakes==1.6.0"
    65 62
    63 
    +See :ref:`built-in functionality doumentation <core_source_builtins>` for
    64 
    +details on common configuration options for sources.
    65 
    +
    66 66 
     .. note::
    67 67
    68 68 
        The ``pip`` plugin is available since :ref:`format version 16 <project_format_version>`
    69 
    -
    70 69 
     """
    71 70
    72 71 
     import errno

  • buildstream/plugins/sources/remote.py
    ... ... @@ -28,9 +28,6 @@ remote - stage files from remote urls
    28 28 
        # Specify the remote source kind
    29 29 
        kind: remote
    30 30
    31 
    -   # Optionally specify a relative staging directory
    32 
    -   # directory: path/to/stage
    33 
    -
    34 31 
        # Optionally specify a relative staging filename.
    35 32 
        # If not specified, the basename of the url will be used.
    36 33 
        # filename: customfilename
    ... ... @@ -47,12 +44,12 @@ remote - stage files from remote urls
    47 44 
        # Specify the ref. It's a sha256sum of the file you download.
    48 45 
        ref: 6c9f6f68a131ec6381da82f2bff978083ed7f4f7991d931bfa767b7965ebc94b
    49 46
    50 
    -
    47 
    +See :ref:`built-in functionality doumentation <core_source_builtins>` for
    48 
    +details on common configuration options for sources.
    51 49
    52 50 
     .. note::
    53 51
    54 52 
        The ``remote`` plugin is available since :ref:`format version 10 <project_format_version>`
    55 
    -
    56 53 
     """
    57 54 
     import os
    58 55 
     from buildstream import SourceError, utils

  • buildstream/plugins/sources/tar.py
    ... ... @@ -32,9 +32,6 @@ tar - stage files from tar archives
    32 32 
        # Specify the tar source kind
    33 33 
        kind: tar
    34 34
    35 
    -   # Optionally specify a relative staging directory
    36 
    -   # directory: path/to/stage
    37 
    -
    38 35 
        # Specify the tar url. Using an alias defined in your project
    39 36 
        # configuration is encouraged. 'bst track' will update the
    40 37 
        # sha256sum in 'ref' to the downloaded file's sha256sum.
    ... ... @@ -53,6 +50,9 @@ tar - stage files from tar archives
    53 50 
        # To extract the root of the tarball directly, this can be set
    54 51 
        # to an empty string.
    55 52 
        base-dir: '*'
    53 
    +
    54 
    +See :ref:`built-in functionality doumentation <core_source_builtins>` for
    55 
    +details on common configuration options for sources.
    56 56 
     """
    57 57
    58 58 
     import os

  • buildstream/plugins/sources/zip.py
    ... ... @@ -28,9 +28,6 @@ zip - stage files from zip archives
    28 28 
        # Specify the zip source kind
    29 29 
        kind: zip
    30 30
    31 
    -   # Optionally specify a relative staging directory
    32 
    -   # directory: path/to/stage
    33 
    -
    34 31 
        # Specify the zip url. Using an alias defined in your project
    35 32 
        # configuration is encouraged. 'bst track' will update the
    36 33 
        # sha256sum in 'ref' to the downloaded file's sha256sum.
    ... ... @@ -50,11 +47,13 @@ zip - stage files from zip archives
    50 47 
        # to an empty string.
    51 48 
        base-dir: '*'
    52 49
    50 
    +See :ref:`built-in functionality doumentation <core_source_builtins>` for
    51 
    +details on common configuration options for sources.
    52 
    +
    53 53 
     .. attention::
    54 54
    55 55 
        File permissions are not preserved. All extracted directories have
    56 56 
        permissions 0755 and all extracted files have permissions 0644.
    57 
    -
    58 57 
     """
    59 58
    60 59 
     import os

  • buildstream/sandbox/_sandboxremote.py
    ... ... @@ -19,6 +19,7 @@
    19 19 
     #        Jim MacArthur <jim macarthur codethink co uk>
    20 20
    21 21 
     import os
    22 
    +import signal
    22 23 
     from urllib.parse import urlparse
    23 24
    24 25 
     import grpc
    ... ... @@ -26,8 +27,10 @@ import grpc
    26 27 
     from . import Sandbox
    27 28 
     from ..storage._filebaseddirectory import FileBasedDirectory
    28 29 
     from ..storage._casbaseddirectory import CasBasedDirectory
    30 
    +from .. import _signals
    29 31 
     from .._protos.build.bazel.remote.execution.v2 import remote_execution_pb2, remote_execution_pb2_grpc
    30 32 
     from .._protos.google.rpc import code_pb2
    33 
    +from .._protos.google.longrunning import operations_pb2, operations_pb2_grpc
    31 34
    32 35
    33 36 
     class SandboxError(Exception):
    ... ... @@ -54,6 +57,7 @@ class SandboxRemote(Sandbox):
    54 57 
                                    "Only plain HTTP is currenlty supported (no HTTPS).")
    55 58
    56 59 
             self.server_url = '{}:{}'.format(url.hostname, url.port)
    60 
    +        self.operation_name = None
    57 61
    58 62 
         def run_remote_command(self, command, input_root_digest, working_directory, environment):
    59 63 
             # Sends an execution request to the remote execution server.
    ... ... @@ -104,11 +108,15 @@ class SandboxRemote(Sandbox):
    104 108 
                         request = remote_execution_pb2.WaitExecutionRequest(name=running_operation.name)
    105 109 
                         operation_iterator = stub.WaitExecution(request)
    106 110
    107 
    -                for operation in operation_iterator:
    108 
    -                    if operation.done:
    109 
    -                        return operation
    110 
    -                    else:
    111 
    -                        last_operation = operation
    111 
    +                with _signals.terminator(self.cancel_operation):
    112 
    +                    with _signals.blocked([signal.SIGTERM], ignore=False):
    113 
    +                        operation = next(operation_iterator)
    114 
    +                        self.operation_name = operation.name
    115 
    +                    for operation in operation_iterator:
    116 
    +                        if operation.done:
    117 
    +                            return operation
    118 
    +                        else:
    119 
    +                            last_operation = operation
    112 120 
                 except grpc.RpcError as e:
    113 121 
                     status_code = e.code()
    114 122 
                     if status_code == grpc.StatusCode.UNAVAILABLE:
    ... ... @@ -135,12 +143,23 @@ class SandboxRemote(Sandbox):
    135 143 
                     return None
    136 144 
                 elif operation.done:
    137 145 
                     return operation
    138 
    -
    139 146 
                 while operation is not None and not operation.done:
    140 147 
                     operation = __run_remote_command(stub, running_operation=operation)
    141 148
    142 149 
             return operation
    143 150
    151 
    +    def cancel_operation(self):
    152 
    +        channel = grpc.insecure_channel(self.server_url)
    153 
    +        stub = operations_pb2_grpc.OperationsStub(channel)
    154 
    +        request = operations_pb2.CancelOperationRequest(
    155 
    +            name="/{}".format(self.operation_name))
    156 
    +
    157 
    +        try:
    158 
    +            stub.CancelOperation(request)
    159 
    +        except grpc.RpcError as e:
    160 
    +            if e.code() != grpc.StatusCode.UNIMPLEMENTED:
    161 
    +                raise SandboxError("{} ({})".format(e.details(), e.code().name))
    162 
    +
    144 163 
         def process_job_output(self, output_directories, output_files):
    145 164 
             # Reads the remote execution server response to an execution request.
    146 165 
             #

  • buildstream/source.py
    ... ... @@ -20,6 +20,8 @@
    20 20 
     Source - Base source class
    21 21 
     ==========================
    22 22
    23 
    +.. _core_source_builtins:
    24 
    +
    23 25 
     Built-in functionality
    24 26 
     ----------------------
    25 27

    • —
    View it on GitLab.
    You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.

    [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]