[Notes] [Git][BuildStream/buildstream][507-some-log-lines-appear-to-be-duplicates] 24 commits: README.md: Minor corrections



Title: GitLab

Tristan Van Berkom pushed to branch 507-some-log-lines-appear-to-be-duplicates at BuildStream / buildstream

Commits:

25 changed files:

Changes:

  • .gitignore
    ... ... @@ -15,6 +15,7 @@ tmp
    15 15
     .coverage
    
    16 16
     .coverage.*
    
    17 17
     .cache
    
    18
    +.pytest_cache/
    
    18 19
     *.bst/
    
    19 20
     
    
    20 21
     # Pycache, in case buildstream is ran directly from within the source
    

  • NEWS
    ... ... @@ -5,6 +5,10 @@ buildstream 1.3.1
    5 5
       o Add a `--tar` option to `bst checkout` which allows a tarball to be
    
    6 6
         created from the artifact contents.
    
    7 7
     
    
    8
    +  o Fetching and tracking will consult mirrors defined in project config,
    
    9
    +    and the preferred mirror to fetch from can be defined in the command
    
    10
    +    line or user config.
    
    11
    +
    
    8 12
     =================
    
    9 13
     buildstream 1.1.4
    
    10 14
     =================
    

  • README.rst
    ... ... @@ -25,7 +25,7 @@ BuildStream offers the following advantages:
    25 25
     
    
    26 26
     * **Declarative build instructions/definitions**
    
    27 27
     
    
    28
    -  BuildStream provides a a flexible and extensible framework for the modelling
    
    28
    +  BuildStream provides a flexible and extensible framework for the modelling
    
    29 29
       of software build pipelines in a declarative YAML format, which allows you to
    
    30 30
       manipulate filesystem data in a controlled, reproducible sandboxed environment.
    
    31 31
     
    
    ... ... @@ -61,25 +61,29 @@ How does BuildStream work?
    61 61
     ==========================
    
    62 62
     BuildStream operates on a set of YAML files (.bst files), as follows:
    
    63 63
     
    
    64
    -* loads the YAML files which describe the target(s) and all dependencies
    
    65
    -* evaluates the version information and build instructions to calculate a build
    
    64
    +* Loads the YAML files which describe the target(s) and all dependencies.
    
    65
    +* Evaluates the version information and build instructions to calculate a build
    
    66 66
       graph for the target(s) and all dependencies and unique cache-keys for each
    
    67
    -  element
    
    68
    -* retrieves elements from cache if they are already built, or builds them in a
    
    69
    -  sandboxed environment using the instructions declared in the .bst files
    
    70
    -* transforms/configures and/or deploys the resulting target(s) based on the
    
    67
    +  element.
    
    68
    +* Retrieves previously built elements (artifacts) from a local/remote cache, or
    
    69
    +  builds the elements in a sandboxed environment using the instructions declared
    
    70
    +  in the .bst files.
    
    71
    +* Transforms/configures and/or deploys the resulting target(s) based on the
    
    71 72
       instructions declared in the .bst files.
    
    72 73
     
    
    73 74
     
    
    74 75
     How can I get started?
    
    75 76
     ======================
    
    76
    -The easiest way to get started is to explore some existing .bst files, for example:
    
    77
    +To start using BuildStream, first,
    
    78
    +`install <https://buildstream.gitlab.io/buildstream/main_install.html>`_
    
    79
    +BuildStream onto your machine and then follow our
    
    80
    +`tutorial <https://buildstream.gitlab.io/buildstream/using_tutorial.html>`_.
    
    81
    +
    
    82
    +We also recommend exploring some existing BuildStream projects:
    
    77 83
     
    
    78 84
     * https://gitlab.gnome.org/GNOME/gnome-build-meta/
    
    79 85
     * https://gitlab.com/freedesktop-sdk/freedesktop-sdk
    
    80 86
     * https://gitlab.com/baserock/definitions
    
    81
    -* https://gitlab.com/BuildStream/buildstream-examples/tree/master/build-x86image
    
    82
    -* https://gitlab.com/BuildStream/buildstream-examples/tree/master/netsurf-flatpak
    
    83 87
     
    
    84 88
     If you have any questions please ask on our `#buildstream <irc://irc.gnome.org/buildstream>`_ channel in `irc.gnome.org <irc://irc.gnome.org>`_
    
    85 89
     

  • buildstream/__init__.py
    ... ... @@ -29,7 +29,7 @@ if "_BST_COMPLETION" not in os.environ:
    29 29
         from .utils import UtilError, ProgramNotFoundError
    
    30 30
         from .sandbox import Sandbox, SandboxFlags
    
    31 31
         from .plugin import Plugin
    
    32
    -    from .source import Source, SourceError, Consistency
    
    32
    +    from .source import Source, SourceError, Consistency, SourceFetcher
    
    33 33
         from .element import Element, ElementError, Scope
    
    34 34
         from .buildelement import BuildElement
    
    35 35
         from .scriptelement import ScriptElement

  • buildstream/_context.py
    ... ... @@ -197,29 +197,55 @@ class Context():
    197 197
                                 "\nValid values are, for example: 800M 10G 1T 50%\n"
    
    198 198
                                 .format(str(e))) from e
    
    199 199
     
    
    200
    -        # If we are asked not to set a quota, we set it to the maximum
    
    201
    -        # disk space available minus a headroom of 2GB, such that we
    
    202
    -        # at least try to avoid raising Exceptions.
    
    200
    +        # Headroom intended to give BuildStream a bit of leeway.
    
    201
    +        # This acts as the minimum size of cache_quota and also
    
    202
    +        # is taken from the user requested cache_quota.
    
    203 203
             #
    
    204
    -        # Of course, we might still end up running out during a build
    
    205
    -        # if we end up writing more than 2G, but hey, this stuff is
    
    206
    -        # already really fuzzy.
    
    207
    -        #
    
    208
    -        if cache_quota is None:
    
    209
    -            stat = os.statvfs(artifactdir_volume)
    
    210
    -            # Again, the artifact directory may not yet have been
    
    211
    -            # created
    
    212
    -            if not os.path.exists(self.artifactdir):
    
    213
    -                cache_size = 0
    
    214
    -            else:
    
    215
    -                cache_size = utils._get_dir_size(self.artifactdir)
    
    216
    -            cache_quota = cache_size + stat.f_bsize * stat.f_bavail
    
    217
    -
    
    218 204
             if 'BST_TEST_SUITE' in os.environ:
    
    219 205
                 headroom = 0
    
    220 206
             else:
    
    221 207
                 headroom = 2e9
    
    222 208
     
    
    209
    +        stat = os.statvfs(artifactdir_volume)
    
    210
    +        available_space = (stat.f_bsize * stat.f_bavail)
    
    211
    +
    
    212
    +        # Again, the artifact directory may not yet have been created yet
    
    213
    +        #
    
    214
    +        if not os.path.exists(self.artifactdir):
    
    215
    +            cache_size = 0
    
    216
    +        else:
    
    217
    +            cache_size = utils._get_dir_size(self.artifactdir)
    
    218
    +
    
    219
    +        # Ensure system has enough storage for the cache_quota
    
    220
    +        #
    
    221
    +        # If cache_quota is none, set it to the maximum it could possibly be.
    
    222
    +        #
    
    223
    +        # Also check that cache_quota is atleast as large as our headroom.
    
    224
    +        #
    
    225
    +        if cache_quota is None:  # Infinity, set to max system storage
    
    226
    +            cache_quota = cache_size + available_space
    
    227
    +        if cache_quota < headroom:  # Check minimum
    
    228
    +            raise LoadError(LoadErrorReason.INVALID_DATA,
    
    229
    +                            "Invalid cache quota ({}): ".format(utils._pretty_size(cache_quota)) +
    
    230
    +                            "BuildStream requires a minimum cache quota of 2G.")
    
    231
    +        elif cache_quota > cache_size + available_space:  # Check maximum
    
    232
    +            raise LoadError(LoadErrorReason.INVALID_DATA,
    
    233
    +                            ("Your system does not have enough available " +
    
    234
    +                             "space to support the cache quota specified.\n" +
    
    235
    +                             "You currently have:\n" +
    
    236
    +                             "- {used} of cache in use at {local_cache_path}\n" +
    
    237
    +                             "- {available} of available system storage").format(
    
    238
    +                                 used=utils._pretty_size(cache_size),
    
    239
    +                                 local_cache_path=self.artifactdir,
    
    240
    +                                 available=utils._pretty_size(available_space)))
    
    241
    +
    
    242
    +        # Place a slight headroom (2e9 (2GB) on the cache_quota) into
    
    243
    +        # cache_quota to try and avoid exceptions.
    
    244
    +        #
    
    245
    +        # Of course, we might still end up running out during a build
    
    246
    +        # if we end up writing more than 2G, but hey, this stuff is
    
    247
    +        # already really fuzzy.
    
    248
    +        #
    
    223 249
             self.cache_quota = cache_quota - headroom
    
    224 250
             self.cache_lower_threshold = self.cache_quota / 2
    
    225 251
     
    
    ... ... @@ -259,7 +285,7 @@ class Context():
    259 285
             # Shallow validation of overrides, parts of buildstream which rely
    
    260 286
             # on the overrides are expected to validate elsewhere.
    
    261 287
             for _, overrides in _yaml.node_items(self._project_overrides):
    
    262
    -            _yaml.node_validate(overrides, ['artifacts', 'options', 'strict'])
    
    288
    +            _yaml.node_validate(overrides, ['artifacts', 'options', 'strict', 'default-mirror'])
    
    263 289
     
    
    264 290
             profile_end(Topics.LOAD_CONTEXT, 'load')
    
    265 291
     
    

  • buildstream/_frontend/app.py
    ... ... @@ -202,7 +202,8 @@ class App():
    202 202
             # Load the Project
    
    203 203
             #
    
    204 204
             try:
    
    205
    -            self.project = Project(directory, self.context, cli_options=self._main_options['option'])
    
    205
    +            self.project = Project(directory, self.context, cli_options=self._main_options['option'],
    
    206
    +                                   default_mirror=self._main_options.get('default_mirror'))
    
    206 207
             except LoadError as e:
    
    207 208
     
    
    208 209
                 # Let's automatically start a `bst init` session in this case
    

  • buildstream/_frontend/cli.py
    ... ... @@ -217,6 +217,8 @@ def print_version(ctx, param, value):
    217 217
                   help="Elements must be rebuilt when their dependencies have changed")
    
    218 218
     @click.option('--option', '-o', type=click.Tuple([str, str]), multiple=True, metavar='OPTION VALUE',
    
    219 219
                   help="Specify a project option")
    
    220
    +@click.option('--default-mirror', default=None,
    
    221
    +              help="The mirror to fetch from first, before attempting other mirrors")
    
    220 222
     @click.pass_context
    
    221 223
     def cli(context, **kwargs):
    
    222 224
         """Build and manipulate BuildStream projects
    

  • buildstream/_loader/loader.py
    ... ... @@ -513,7 +513,7 @@ class Loader():
    513 513
                     if self._fetch_subprojects:
    
    514 514
                         if ticker:
    
    515 515
                             ticker(filename, 'Fetching subproject from {} source'.format(source.get_kind()))
    
    516
    -                    source.fetch()
    
    516
    +                    source._fetch()
    
    517 517
                     else:
    
    518 518
                         detail = "Try fetching the project with `bst fetch {}`".format(filename)
    
    519 519
                         raise LoadError(LoadErrorReason.SUBPROJECT_FETCH_NEEDED,
    

  • buildstream/_project.py
    ... ... @@ -19,7 +19,7 @@
    19 19
     
    
    20 20
     import os
    
    21 21
     import multiprocessing  # for cpu_count()
    
    22
    -from collections import Mapping
    
    22
    +from collections import Mapping, OrderedDict
    
    23 23
     from pluginbase import PluginBase
    
    24 24
     from . import utils
    
    25 25
     from . import _cachekey
    
    ... ... @@ -35,9 +35,6 @@ from ._projectrefs import ProjectRefs, ProjectRefStorage
    35 35
     from ._versions import BST_FORMAT_VERSION
    
    36 36
     
    
    37 37
     
    
    38
    -# The separator we use for user specified aliases
    
    39
    -_ALIAS_SEPARATOR = ':'
    
    40
    -
    
    41 38
     # Project Configuration file
    
    42 39
     _PROJECT_CONF_FILE = 'project.conf'
    
    43 40
     
    
    ... ... @@ -70,7 +67,7 @@ class HostMount():
    70 67
     #
    
    71 68
     class Project():
    
    72 69
     
    
    73
    -    def __init__(self, directory, context, *, junction=None, cli_options=None):
    
    70
    +    def __init__(self, directory, context, *, junction=None, cli_options=None, default_mirror=None):
    
    74 71
     
    
    75 72
             # The project name
    
    76 73
             self.name = None
    
    ... ... @@ -94,6 +91,8 @@ class Project():
    94 91
             self.base_env_nocache = None             # The base nocache mask (list) for the environment
    
    95 92
             self.element_overrides = {}              # Element specific configurations
    
    96 93
             self.source_overrides = {}               # Source specific configurations
    
    94
    +        self.mirrors = OrderedDict()             # contains dicts of alias-mappings to URIs.
    
    95
    +        self.default_mirror = default_mirror     # The name of the preferred mirror.
    
    97 96
     
    
    98 97
             #
    
    99 98
             # Private Members
    
    ... ... @@ -133,8 +132,8 @@ class Project():
    133 132
         # fully qualified urls based on the shorthand which is allowed
    
    134 133
         # to be specified in the YAML
    
    135 134
         def translate_url(self, url):
    
    136
    -        if url and _ALIAS_SEPARATOR in url:
    
    137
    -            url_alias, url_body = url.split(_ALIAS_SEPARATOR, 1)
    
    135
    +        if url and utils._ALIAS_SEPARATOR in url:
    
    136
    +            url_alias, url_body = url.split(utils._ALIAS_SEPARATOR, 1)
    
    138 137
                 alias_url = self._aliases.get(url_alias)
    
    139 138
                 if alias_url:
    
    140 139
                     url = alias_url + url_body
    
    ... ... @@ -202,6 +201,36 @@ class Project():
    202 201
             self._assert_plugin_format(source, version)
    
    203 202
             return source
    
    204 203
     
    
    204
    +    # get_alias_uri()
    
    205
    +    #
    
    206
    +    # Returns the URI for a given alias, if it exists
    
    207
    +    #
    
    208
    +    # Args:
    
    209
    +    #    alias (str): The alias.
    
    210
    +    #
    
    211
    +    # Returns:
    
    212
    +    #    str: The URI for the given alias; or None: if there is no URI for
    
    213
    +    #         that alias.
    
    214
    +    def get_alias_uri(self, alias):
    
    215
    +        return self._aliases.get(alias)
    
    216
    +
    
    217
    +    # get_alias_uris()
    
    218
    +    #
    
    219
    +    # Returns a list of every URI to replace an alias with
    
    220
    +    def get_alias_uris(self, alias):
    
    221
    +        if not alias or alias not in self._aliases:
    
    222
    +            return [None]
    
    223
    +
    
    224
    +        mirror_list = []
    
    225
    +        for key, alias_mapping in self.mirrors.items():
    
    226
    +            if alias in alias_mapping:
    
    227
    +                if key == self.default_mirror:
    
    228
    +                    mirror_list = alias_mapping[alias] + mirror_list
    
    229
    +                else:
    
    230
    +                    mirror_list += alias_mapping[alias]
    
    231
    +        mirror_list.append(self._aliases[alias])
    
    232
    +        return mirror_list
    
    233
    +
    
    205 234
         # _load():
    
    206 235
         #
    
    207 236
         # Loads the project configuration file in the project directory.
    
    ... ... @@ -249,7 +278,7 @@ class Project():
    249 278
                 'aliases', 'name',
    
    250 279
                 'artifacts', 'options',
    
    251 280
                 'fail-on-overlap', 'shell',
    
    252
    -            'ref-storage', 'sandbox'
    
    281
    +            'ref-storage', 'sandbox', 'mirrors',
    
    253 282
             ])
    
    254 283
     
    
    255 284
             # The project name, element path and option declarations
    
    ... ... @@ -290,6 +319,10 @@ class Project():
    290 319
             #
    
    291 320
             self.options.process_node(config)
    
    292 321
     
    
    322
    +        # Override default_mirror if not set by command-line
    
    323
    +        if not self.default_mirror:
    
    324
    +            self.default_mirror = _yaml.node_get(overrides, str, 'default-mirror', default_value=None)
    
    325
    +
    
    293 326
             #
    
    294 327
             # Now all YAML composition is done, from here on we just load
    
    295 328
             # the values from our loaded configuration dictionary.
    
    ... ... @@ -414,6 +447,21 @@ class Project():
    414 447
     
    
    415 448
                 self._shell_host_files.append(mount)
    
    416 449
     
    
    450
    +        mirrors = _yaml.node_get(config, list, 'mirrors', default_value=[])
    
    451
    +        for mirror in mirrors:
    
    452
    +            allowed_mirror_fields = [
    
    453
    +                'name', 'aliases'
    
    454
    +            ]
    
    455
    +            _yaml.node_validate(mirror, allowed_mirror_fields)
    
    456
    +            mirror_name = _yaml.node_get(mirror, str, 'name')
    
    457
    +            alias_mappings = {}
    
    458
    +            for alias_mapping, uris in _yaml.node_items(mirror['aliases']):
    
    459
    +                assert isinstance(uris, list)
    
    460
    +                alias_mappings[alias_mapping] = list(uris)
    
    461
    +            self.mirrors[mirror_name] = alias_mappings
    
    462
    +            if not self.default_mirror:
    
    463
    +                self.default_mirror = mirror_name
    
    464
    +
    
    417 465
         # _assert_plugin_format()
    
    418 466
         #
    
    419 467
         # Helper to raise a PluginError if the loaded plugin is of a lesser version then
    

  • buildstream/_versions.py
    ... ... @@ -23,7 +23,7 @@
    23 23
     # This version is bumped whenever enhancements are made
    
    24 24
     # to the `project.conf` format or the core element format.
    
    25 25
     #
    
    26
    -BST_FORMAT_VERSION = 10
    
    26
    +BST_FORMAT_VERSION = 11
    
    27 27
     
    
    28 28
     
    
    29 29
     # The base BuildStream artifact version
    

  • buildstream/buildelement.py
    1 1
     #
    
    2 2
     #  Copyright (C) 2016 Codethink Limited
    
    3
    +#  Copyright (C) 2018 Bloomberg Finance LP
    
    3 4
     #
    
    4 5
     #  This program is free software; you can redistribute it and/or
    
    5 6
     #  modify it under the terms of the GNU Lesser General Public
    
    ... ... @@ -204,8 +205,9 @@ class BuildElement(Element):
    204 205
         def prepare(self, sandbox):
    
    205 206
             commands = self.__commands['configure-commands']
    
    206 207
             if commands:
    
    207
    -            for cmd in commands:
    
    208
    -                self.__run_command(sandbox, cmd, 'configure-commands')
    
    208
    +            with self.timed_activity("Running configure-commands"):
    
    209
    +                for cmd in commands:
    
    210
    +                    self.__run_command(sandbox, cmd, 'configure-commands')
    
    209 211
     
    
    210 212
         def generate_script(self):
    
    211 213
             script = ""
    
    ... ... @@ -231,13 +233,12 @@ class BuildElement(Element):
    231 233
             return commands
    
    232 234
     
    
    233 235
         def __run_command(self, sandbox, cmd, cmd_name):
    
    234
    -        with self.timed_activity("Running {}".format(cmd_name)):
    
    235
    -            self.status("Running {}".format(cmd_name), detail=cmd)
    
    236
    -
    
    237
    -            # Note the -e switch to 'sh' means to exit with an error
    
    238
    -            # if any untested command fails.
    
    239
    -            #
    
    240
    -            exitcode = sandbox.run(['sh', '-c', '-e', cmd + '\n'],
    
    241
    -                                   SandboxFlags.ROOT_READ_ONLY)
    
    242
    -            if exitcode != 0:
    
    243
    -                raise ElementError("Command '{}' failed with exitcode {}".format(cmd, exitcode))
    236
    +        self.status("Running {}".format(cmd_name), detail=cmd)
    
    237
    +
    
    238
    +        # Note the -e switch to 'sh' means to exit with an error
    
    239
    +        # if any untested command fails.
    
    240
    +        #
    
    241
    +        exitcode = sandbox.run(['sh', '-c', '-e', cmd + '\n'],
    
    242
    +                               SandboxFlags.ROOT_READ_ONLY)
    
    243
    +        if exitcode != 0:
    
    244
    +            raise ElementError("Command '{}' failed with exitcode {}".format(cmd, exitcode))

  • buildstream/plugins/sources/bzr.py
    ... ... @@ -102,7 +102,7 @@ class BzrSource(Source):
    102 102
         def track(self):
    
    103 103
             with self.timed_activity("Tracking {}".format(self.url),
    
    104 104
                                      silent_nested=True):
    
    105
    -            self._ensure_mirror()
    
    105
    +            self._ensure_mirror(skip_ref_check=True)
    
    106 106
                 ret, out = self.check_output([self.host_bzr, "version-info",
    
    107 107
                                               "--custom", "--template={revno}",
    
    108 108
                                               self._get_branch_dir()],
    
    ... ... @@ -214,7 +214,7 @@ class BzrSource(Source):
    214 214
                 yield repodir
    
    215 215
                 self._atomic_replace_mirrordir(repodir)
    
    216 216
     
    
    217
    -    def _ensure_mirror(self):
    
    217
    +    def _ensure_mirror(self, skip_ref_check=False):
    
    218 218
             with self._atomic_repodir() as repodir:
    
    219 219
                 # Initialize repo if no metadata
    
    220 220
                 bzr_metadata_dir = os.path.join(repodir, ".bzr")
    
    ... ... @@ -223,18 +223,21 @@ class BzrSource(Source):
    223 223
                               fail="Failed to initialize bzr repository")
    
    224 224
     
    
    225 225
                 branch_dir = os.path.join(repodir, self.tracking)
    
    226
    +            branch_url = self.url + "/" + self.tracking
    
    226 227
                 if not os.path.exists(branch_dir):
    
    227 228
                     # `bzr branch` the branch if it doesn't exist
    
    228 229
                     # to get the upstream code
    
    229
    -                branch_url = self.url + "/" + self.tracking
    
    230 230
                     self.call([self.host_bzr, "branch", branch_url, branch_dir],
    
    231 231
                               fail="Failed to branch from {} to {}".format(branch_url, branch_dir))
    
    232 232
     
    
    233 233
                 else:
    
    234 234
                     # `bzr pull` the branch if it does exist
    
    235 235
                     # to get any changes to the upstream code
    
    236
    -                self.call([self.host_bzr, "pull", "--directory={}".format(branch_dir)],
    
    236
    +                self.call([self.host_bzr, "pull", "--directory={}".format(branch_dir), branch_url],
    
    237 237
                               fail="Failed to pull new changes for {}".format(branch_dir))
    
    238
    +        if not skip_ref_check and not self._check_ref():
    
    239
    +            raise SourceError("Failed to ensure ref '{}' was mirrored".format(self.ref),
    
    240
    +                              reason="ref-not-mirrored")
    
    238 241
     
    
    239 242
     
    
    240 243
     def setup():
    

  • buildstream/plugins/sources/git.py
    ... ... @@ -78,7 +78,7 @@ from io import StringIO
    78 78
     
    
    79 79
     from configparser import RawConfigParser
    
    80 80
     
    
    81
    -from buildstream import Source, SourceError, Consistency
    
    81
    +from buildstream import Source, SourceError, Consistency, SourceFetcher
    
    82 82
     from buildstream import utils
    
    83 83
     
    
    84 84
     GIT_MODULES = '.gitmodules'
    
    ... ... @@ -88,18 +88,20 @@ GIT_MODULES = '.gitmodules'
    88 88
     # for the primary git source and also for each submodule it
    
    89 89
     # might have at a given time
    
    90 90
     #
    
    91
    -class GitMirror():
    
    91
    +class GitMirror(SourceFetcher):
    
    92 92
     
    
    93 93
         def __init__(self, source, path, url, ref):
    
    94 94
     
    
    95
    +        super().__init__()
    
    95 96
             self.source = source
    
    96 97
             self.path = path
    
    97
    -        self.url = source.translate_url(url)
    
    98
    +        self.url = url
    
    98 99
             self.ref = ref
    
    99
    -        self.mirror = os.path.join(source.get_mirror_directory(), utils.url_directory_name(self.url))
    
    100
    +        self.mirror = os.path.join(source.get_mirror_directory(), utils.url_directory_name(url))
    
    101
    +        self.mark_download_url(url)
    
    100 102
     
    
    101 103
         # Ensures that the mirror exists
    
    102
    -    def ensure(self):
    
    104
    +    def ensure(self, alias_override=None):
    
    103 105
     
    
    104 106
             # Unfortunately, git does not know how to only clone just a specific ref,
    
    105 107
             # so we have to download all of those gigs even if we only need a couple
    
    ... ... @@ -112,22 +114,47 @@ class GitMirror():
    112 114
                 # system configured tmpdir is not on the same partition.
    
    113 115
                 #
    
    114 116
                 with self.source.tempdir() as tmpdir:
    
    115
    -                self.source.call([self.source.host_git, 'clone', '--mirror', '-n', self.url, tmpdir],
    
    116
    -                                 fail="Failed to clone git repository {}".format(self.url),
    
    117
    +                url = self.source.translate_url(self.url, alias_override=alias_override)
    
    118
    +                self.source.call([self.source.host_git, 'clone', '--mirror', '-n', url, tmpdir],
    
    119
    +                                 fail="Failed to clone git repository {}".format(url),
    
    117 120
                                      fail_temporarily=True)
    
    118 121
     
    
    119 122
                     try:
    
    120 123
                         shutil.move(tmpdir, self.mirror)
    
    121 124
                     except (shutil.Error, OSError) as e:
    
    122 125
                         raise SourceError("{}: Failed to move cloned git repository {} from '{}' to '{}'"
    
    123
    -                                      .format(self.source, self.url, tmpdir, self.mirror)) from e
    
    126
    +                                      .format(self.source, url, tmpdir, self.mirror)) from e
    
    127
    +
    
    128
    +    def _fetch(self, alias_override=None):
    
    129
    +        url = self.source.translate_url(self.url, alias_override=alias_override)
    
    130
    +
    
    131
    +        if alias_override:
    
    132
    +            remote_name = utils.url_directory_name(alias_override)
    
    133
    +            _, remotes = self.source.check_output(
    
    134
    +                [self.source.host_git, 'remote'],
    
    135
    +                fail="Failed to retrieve list of remotes in {}".format(self.mirror),
    
    136
    +                cwd=self.mirror
    
    137
    +            )
    
    138
    +            if remote_name not in remotes:
    
    139
    +                self.source.call(
    
    140
    +                    [self.source.host_git, 'remote', 'add', remote_name, url],
    
    141
    +                    fail="Failed to add remote {} with url {}".format(remote_name, url),
    
    142
    +                    cwd=self.mirror
    
    143
    +                )
    
    144
    +        else:
    
    145
    +            remote_name = "origin"
    
    124 146
     
    
    125
    -    def fetch(self):
    
    126
    -        self.source.call([self.source.host_git, 'fetch', 'origin', '--prune'],
    
    127
    -                         fail="Failed to fetch from remote git repository: {}".format(self.url),
    
    147
    +        self.source.call([self.source.host_git, 'fetch', remote_name, '--prune'],
    
    148
    +                         fail="Failed to fetch from remote git repository: {}".format(url),
    
    128 149
                              fail_temporarily=True,
    
    129 150
                              cwd=self.mirror)
    
    130 151
     
    
    152
    +    def fetch(self, alias_override=None):
    
    153
    +        self.ensure(alias_override)
    
    154
    +        if not self.has_ref():
    
    155
    +            self._fetch(alias_override)
    
    156
    +        self.assert_ref()
    
    157
    +
    
    131 158
         def has_ref(self):
    
    132 159
             if not self.ref:
    
    133 160
                 return False
    
    ... ... @@ -171,13 +198,14 @@ class GitMirror():
    171 198
     
    
    172 199
         def init_workspace(self, directory):
    
    173 200
             fullpath = os.path.join(directory, self.path)
    
    201
    +        url = self.source.translate_url(self.url)
    
    174 202
     
    
    175 203
             self.source.call([self.source.host_git, 'clone', '--no-checkout', self.mirror, fullpath],
    
    176 204
                              fail="Failed to clone git mirror {} in directory: {}".format(self.mirror, fullpath),
    
    177 205
                              fail_temporarily=True)
    
    178 206
     
    
    179
    -        self.source.call([self.source.host_git, 'remote', 'set-url', 'origin', self.url],
    
    180
    -                         fail='Failed to add remote origin "{}"'.format(self.url),
    
    207
    +        self.source.call([self.source.host_git, 'remote', 'set-url', 'origin', url],
    
    208
    +                         fail='Failed to add remote origin "{}"'.format(url),
    
    181 209
                              cwd=fullpath)
    
    182 210
     
    
    183 211
             self.source.call([self.source.host_git, 'checkout', '--force', self.ref],
    
    ... ... @@ -277,6 +305,8 @@ class GitSource(Source):
    277 305
                     checkout = self.node_get_member(submodule, bool, 'checkout')
    
    278 306
                     self.submodule_checkout_overrides[path] = checkout
    
    279 307
     
    
    308
    +        self.mark_download_url(self.original_url)
    
    309
    +
    
    280 310
         def preflight(self):
    
    281 311
             # Check if git is installed, get the binary at the same time
    
    282 312
             self.host_git = utils.get_host_tool('git')
    
    ... ... @@ -328,31 +358,13 @@ class GitSource(Source):
    328 358
                                      .format(self.tracking, self.mirror.url),
    
    329 359
                                      silent_nested=True):
    
    330 360
                 self.mirror.ensure()
    
    331
    -            self.mirror.fetch()
    
    361
    +            self.mirror._fetch()
    
    332 362
     
    
    333 363
                 # Update self.mirror.ref and node.ref from the self.tracking branch
    
    334 364
                 ret = self.mirror.latest_commit(self.tracking)
    
    335 365
     
    
    336 366
             return ret
    
    337 367
     
    
    338
    -    def fetch(self):
    
    339
    -
    
    340
    -        with self.timed_activity("Fetching {}".format(self.mirror.url), silent_nested=True):
    
    341
    -
    
    342
    -            # Here we are only interested in ensuring that our mirror contains
    
    343
    -            # the self.mirror.ref commit.
    
    344
    -            self.mirror.ensure()
    
    345
    -            if not self.mirror.has_ref():
    
    346
    -                self.mirror.fetch()
    
    347
    -
    
    348
    -            self.mirror.assert_ref()
    
    349
    -
    
    350
    -            # Here after performing any fetches, we need to also ensure that
    
    351
    -            # we've cached the desired refs in our mirrors of submodules.
    
    352
    -            #
    
    353
    -            self.refresh_submodules()
    
    354
    -            self.fetch_submodules()
    
    355
    -
    
    356 368
         def init_workspace(self, directory):
    
    357 369
             # XXX: may wish to refactor this as some code dupe with stage()
    
    358 370
             self.refresh_submodules()
    
    ... ... @@ -384,6 +396,10 @@ class GitSource(Source):
    384 396
                     if checkout:
    
    385 397
                         mirror.stage(directory)
    
    386 398
     
    
    399
    +    def get_source_fetchers(self):
    
    400
    +        self.refresh_submodules()
    
    401
    +        return [self.mirror] + self.submodules
    
    402
    +
    
    387 403
         ###########################################################
    
    388 404
         #                     Local Functions                     #
    
    389 405
         ###########################################################
    
    ... ... @@ -405,6 +421,7 @@ class GitSource(Source):
    405 421
         # Assumes that we have our mirror and we have the ref which we point to
    
    406 422
         #
    
    407 423
         def refresh_submodules(self):
    
    424
    +        self.mirror.ensure()
    
    408 425
             submodules = []
    
    409 426
     
    
    410 427
             # XXX Here we should issue a warning if either:
    
    ... ... @@ -426,19 +443,6 @@ class GitSource(Source):
    426 443
     
    
    427 444
             self.submodules = submodules
    
    428 445
     
    
    429
    -    # Ensures that we have mirrored git repositories for all
    
    430
    -    # the submodules existing at the given commit of the main git source.
    
    431
    -    #
    
    432
    -    # Also ensure that these mirrors have the required commits
    
    433
    -    # referred to at the given commit of the main git source.
    
    434
    -    #
    
    435
    -    def fetch_submodules(self):
    
    436
    -        for mirror in self.submodules:
    
    437
    -            mirror.ensure()
    
    438
    -            if not mirror.has_ref():
    
    439
    -                mirror.fetch()
    
    440
    -                mirror.assert_ref()
    
    441
    -
    
    442 446
     
    
    443 447
     # Plugin entry point
    
    444 448
     def setup():
    

  • buildstream/source.py
    ... ... @@ -65,6 +65,33 @@ these methods are mandatory to implement.
    65 65
     
    
    66 66
       **Optional**: If left unimplemented, this will default to calling
    
    67 67
       :func:`Source.stage() <buildstream.source.Source.stage>`
    
    68
    +
    
    69
    +* :func:`Source.get_source_fetchers() <buildstream.source.Source.get_source_fetchers>`
    
    70
    +
    
    71
    +  Get the objects that are used for fetching.
    
    72
    +
    
    73
    +  **Optional**: This only needs to be implemented for sources that need to
    
    74
    +  download from multiple URLs while fetching (e.g. a git repo and its
    
    75
    +  submodules). For details on how to define a SourceFetcher, see
    
    76
    +  :ref:`SourceFetcher <core_source_fetcher>`.
    
    77
    +
    
    78
    +
    
    79
    +.. _core_source_fetcher:
    
    80
    +
    
    81
    +SourceFetcher - Object for fetching individual URLs
    
    82
    +===================================================
    
    83
    +
    
    84
    +
    
    85
    +Abstract Methods
    
    86
    +----------------
    
    87
    +SourceFetchers expose the following abstract methods. Unless explicitly
    
    88
    +mentioned, these methods are mandatory to implement.
    
    89
    +
    
    90
    +* :func:`SourceFetcher.fetch() <buildstream.source.SourceFetcher.fetch>`
    
    91
    +
    
    92
    +  Fetches the URL associated with this SourceFetcher, optionally taking an
    
    93
    +  alias override.
    
    94
    +
    
    68 95
     """
    
    69 96
     
    
    70 97
     import os
    
    ... ... @@ -114,6 +141,63 @@ class SourceError(BstError):
    114 141
             super().__init__(message, detail=detail, domain=ErrorDomain.SOURCE, reason=reason, temporary=temporary)
    
    115 142
     
    
    116 143
     
    
    144
    +class SourceFetcher():
    
    145
    +    """SourceFetcher()
    
    146
    +
    
    147
    +    This interface exists so that a source that downloads from multiple
    
    148
    +    places (e.g. a git source with submodules) has a consistent interface for
    
    149
    +    fetching and substituting aliases.
    
    150
    +
    
    151
    +    *Since: 1.4*
    
    152
    +    """
    
    153
    +    def __init__(self):
    
    154
    +        self.__alias = None
    
    155
    +
    
    156
    +    #############################################################
    
    157
    +    #                      Abstract Methods                     #
    
    158
    +    #############################################################
    
    159
    +    def fetch(self, alias_override=None):
    
    160
    +        """Fetch remote sources and mirror them locally, ensuring at least
    
    161
    +        that the specific reference is cached locally.
    
    162
    +
    
    163
    +        Args:
    
    164
    +           alias_override (str): The alias to use instead of the default one
    
    165
    +               defined by the :ref:`aliases <project_source_aliases>` field
    
    166
    +               in the project's config.
    
    167
    +
    
    168
    +        Raises:
    
    169
    +           :class:`.SourceError`
    
    170
    +
    
    171
    +        Implementors should raise :class:`.SourceError` if the there is some
    
    172
    +        network error or if the source reference could not be matched.
    
    173
    +        """
    
    174
    +        raise ImplError("Source fetcher '{}' does not implement fetch()".format(type(self)))
    
    175
    +
    
    176
    +    #############################################################
    
    177
    +    #                       Public Methods                      #
    
    178
    +    #############################################################
    
    179
    +    def mark_download_url(self, url):
    
    180
    +        """Identifies the URL that this SourceFetcher uses to download
    
    181
    +
    
    182
    +        This must be called during the fetcher's initialization
    
    183
    +
    
    184
    +        Args:
    
    185
    +           url (str): The url used to download.
    
    186
    +        """
    
    187
    +        # Not guaranteed to be a valid alias yet.
    
    188
    +        # Ensuring it's a valid alias currently happens in Project.get_alias_uris
    
    189
    +        alias, _ = url.split(utils._ALIAS_SEPARATOR, 1)
    
    190
    +        self.__alias = alias
    
    191
    +
    
    192
    +    #############################################################
    
    193
    +    #            Private Methods used in BuildStream            #
    
    194
    +    #############################################################
    
    195
    +
    
    196
    +    # Returns the alias used by this fetcher
    
    197
    +    def _get_alias(self):
    
    198
    +        return self.__alias
    
    199
    +
    
    200
    +
    
    117 201
     class Source(Plugin):
    
    118 202
         """Source()
    
    119 203
     
    
    ... ... @@ -125,7 +209,7 @@ class Source(Plugin):
    125 209
         __defaults = {}          # The defaults from the project
    
    126 210
         __defaults_set = False   # Flag, in case there are not defaults at all
    
    127 211
     
    
    128
    -    def __init__(self, context, project, meta):
    
    212
    +    def __init__(self, context, project, meta, *, alias_override=None):
    
    129 213
             provenance = _yaml.node_get_provenance(meta.config)
    
    130 214
             super().__init__("{}-{}".format(meta.element_name, meta.element_index),
    
    131 215
                              context, project, provenance, "source")
    
    ... ... @@ -135,6 +219,11 @@ class Source(Plugin):
    135 219
             self.__element_kind = meta.element_kind         # The kind of the element owning this source
    
    136 220
             self.__directory = meta.directory               # Staging relative directory
    
    137 221
             self.__consistency = Consistency.INCONSISTENT   # Cached consistency state
    
    222
    +        self.__alias_override = alias_override          # Tuple of alias and its override to use instead
    
    223
    +        self.__expected_alias = None                    # A hacky way to store the first alias used
    
    224
    +
    
    225
    +        # FIXME: Reconstruct a MetaSource from a Source instead of storing it.
    
    226
    +        self.__meta = meta                              # MetaSource stored so we can copy this source later.
    
    138 227
     
    
    139 228
             # Collect the composited element configuration and
    
    140 229
             # ask the element to configure itself.
    
    ... ... @@ -284,6 +373,36 @@ class Source(Plugin):
    284 373
             """
    
    285 374
             self.stage(directory)
    
    286 375
     
    
    376
    +    def mark_download_url(self, url):
    
    377
    +        """Identifies the URL that this Source uses to download
    
    378
    +
    
    379
    +        This must be called during :func:`~buildstream.plugin.Plugin.configure` if
    
    380
    +        :func:`~buildstream.source.Source.translate_url` is not called.
    
    381
    +
    
    382
    +        Args:
    
    383
    +           url (str): The url used to download
    
    384
    +
    
    385
    +        *Since: 1.4*
    
    386
    +        """
    
    387
    +        alias, _ = url.split(utils._ALIAS_SEPARATOR, 1)
    
    388
    +        self.__expected_alias = alias
    
    389
    +
    
    390
    +    def get_source_fetchers(self):
    
    391
    +        """Get the objects that are used for fetching
    
    392
    +
    
    393
    +        If this source doesn't download from multiple URLs,
    
    394
    +        returning None and falling back on the default behaviour
    
    395
    +        is recommended.
    
    396
    +
    
    397
    +        Returns:
    
    398
    +           list: A list of SourceFetchers. If SourceFetchers are not supported,
    
    399
    +                 this will be an empty list.
    
    400
    +
    
    401
    +        *Since: 1.4*
    
    402
    +        """
    
    403
    +
    
    404
    +        return []
    
    405
    +
    
    287 406
         #############################################################
    
    288 407
         #                       Public Methods                      #
    
    289 408
         #############################################################
    
    ... ... @@ -300,18 +419,42 @@ class Source(Plugin):
    300 419
             os.makedirs(directory, exist_ok=True)
    
    301 420
             return directory
    
    302 421
     
    
    303
    -    def translate_url(self, url):
    
    422
    +    def translate_url(self, url, *, alias_override=None):
    
    304 423
             """Translates the given url which may be specified with an alias
    
    305 424
             into a fully qualified url.
    
    306 425
     
    
    307 426
             Args:
    
    308 427
                url (str): A url, which may be using an alias
    
    428
    +           alias_override (str): Optionally, an URI to override the alias with. (*Since: 1.4*)
    
    309 429
     
    
    310 430
             Returns:
    
    311 431
                str: The fully qualified url, with aliases resolved
    
    312 432
             """
    
    313
    -        project = self._get_project()
    
    314
    -        return project.translate_url(url)
    
    433
    +        # Alias overriding can happen explicitly (by command-line) or
    
    434
    +        # implicitly (the Source being constructed with an __alias_override).
    
    435
    +        if alias_override or self.__alias_override:
    
    436
    +            url_alias, url_body = url.split(utils._ALIAS_SEPARATOR, 1)
    
    437
    +            if url_alias:
    
    438
    +                if alias_override:
    
    439
    +                    url = alias_override + url_body
    
    440
    +                else:
    
    441
    +                    # Implicit alias overrides may only be done for one
    
    442
    +                    # specific alias, so that sources that fetch from multiple
    
    443
    +                    # URLs and use different aliases default to only overriding
    
    444
    +                    # one alias, rather than getting confused.
    
    445
    +                    override_alias = self.__alias_override[0]
    
    446
    +                    override_url = self.__alias_override[1]
    
    447
    +                    if url_alias == override_alias:
    
    448
    +                        url = override_url + url_body
    
    449
    +            return url
    
    450
    +        else:
    
    451
    +            # Sneakily store the alias if it hasn't already been stored
    
    452
    +            if not self.__expected_alias and url and utils._ALIAS_SEPARATOR in url:
    
    453
    +                url_alias, _ = url.split(utils._ALIAS_SEPARATOR, 1)
    
    454
    +                self.__expected_alias = url_alias
    
    455
    +
    
    456
    +            project = self._get_project()
    
    457
    +            return project.translate_url(url)
    
    315 458
     
    
    316 459
         def get_project_directory(self):
    
    317 460
             """Fetch the project base directory
    
    ... ... @@ -375,7 +518,45 @@ class Source(Plugin):
    375 518
         # Wrapper function around plugin provided fetch method
    
    376 519
         #
    
    377 520
         def _fetch(self):
    
    378
    -        self.fetch()
    
    521
    +        project = self._get_project()
    
    522
    +        source_fetchers = self.get_source_fetchers()
    
    523
    +        if source_fetchers:
    
    524
    +            for fetcher in source_fetchers:
    
    525
    +                alias = fetcher._get_alias()
    
    526
    +                success = False
    
    527
    +                for uri in project.get_alias_uris(alias):
    
    528
    +                    try:
    
    529
    +                        fetcher.fetch(uri)
    
    530
    +                    # FIXME: Need to consider temporary vs. permanent failures,
    
    531
    +                    #        and how this works with retries.
    
    532
    +                    except BstError as e:
    
    533
    +                        last_error = e
    
    534
    +                        continue
    
    535
    +                    success = True
    
    536
    +                    break
    
    537
    +                if not success:
    
    538
    +                    raise last_error
    
    539
    +        else:
    
    540
    +            alias = self._get_alias()
    
    541
    +            if not project.mirrors or not alias:
    
    542
    +                self.fetch()
    
    543
    +                return
    
    544
    +
    
    545
    +            context = self._get_context()
    
    546
    +            source_kind = type(self)
    
    547
    +            for uri in project.get_alias_uris(alias):
    
    548
    +                new_source = source_kind(context, project, self.__meta,
    
    549
    +                                         alias_override=(alias, uri))
    
    550
    +                new_source._preflight()
    
    551
    +                try:
    
    552
    +                    new_source.fetch()
    
    553
    +                # FIXME: Need to consider temporary vs. permanent failures,
    
    554
    +                #        and how this works with retries.
    
    555
    +                except BstError as e:
    
    556
    +                    last_error = e
    
    557
    +                    continue
    
    558
    +                return
    
    559
    +            raise last_error
    
    379 560
     
    
    380 561
         # Wrapper for stage() api which gives the source
    
    381 562
         # plugin a fully constructed path considering the
    
    ... ... @@ -582,7 +763,7 @@ class Source(Plugin):
    582 763
         # Wrapper for track()
    
    583 764
         #
    
    584 765
         def _track(self):
    
    585
    -        new_ref = self.track()
    
    766
    +        new_ref = self.__do_track()
    
    586 767
             current_ref = self.get_ref()
    
    587 768
     
    
    588 769
             if new_ref is None:
    
    ... ... @@ -594,10 +775,48 @@ class Source(Plugin):
    594 775
     
    
    595 776
             return new_ref
    
    596 777
     
    
    778
    +    # Returns the alias if it's defined in the project
    
    779
    +    def _get_alias(self):
    
    780
    +        alias = self.__expected_alias
    
    781
    +        project = self._get_project()
    
    782
    +        if project.get_alias_uri(alias):
    
    783
    +            # The alias must already be defined in the project's aliases
    
    784
    +            # otherwise http://foo gets treated like it contains an alias
    
    785
    +            return alias
    
    786
    +        else:
    
    787
    +            return None
    
    788
    +
    
    597 789
         #############################################################
    
    598 790
         #                   Local Private Methods                   #
    
    599 791
         #############################################################
    
    600 792
     
    
    793
    +    # Tries to call track for every mirror, stopping once it succeeds
    
    794
    +    def __do_track(self):
    
    795
    +        project = self._get_project()
    
    796
    +        # If there are no mirrors, or no aliases to replace, there's nothing to do here.
    
    797
    +        alias = self._get_alias()
    
    798
    +        if not project.mirrors or not alias:
    
    799
    +            return self.track()
    
    800
    +
    
    801
    +        context = self._get_context()
    
    802
    +        source_kind = type(self)
    
    803
    +
    
    804
    +        # NOTE: We are assuming here that tracking only requires substituting the
    
    805
    +        #       first alias used
    
    806
    +        for uri in reversed(project.get_alias_uris(alias)):
    
    807
    +            new_source = source_kind(context, project, self.__meta,
    
    808
    +                                     alias_override=(alias, uri))
    
    809
    +            new_source._preflight()
    
    810
    +            try:
    
    811
    +                ref = new_source.track()
    
    812
    +            # FIXME: Need to consider temporary vs. permanent failures,
    
    813
    +            #        and how this works with retries.
    
    814
    +            except BstError as e:
    
    815
    +                last_error = e
    
    816
    +                continue
    
    817
    +            return ref
    
    818
    +        raise last_error
    
    819
    +
    
    601 820
         # Ensures a fully constructed path and returns it
    
    602 821
         def __ensure_directory(self, directory):
    
    603 822
     
    

  • buildstream/utils.py
    ... ... @@ -42,6 +42,10 @@ from . import _signals
    42 42
     from ._exceptions import BstError, ErrorDomain
    
    43 43
     
    
    44 44
     
    
    45
    +# The separator we use for user specified aliases
    
    46
    +_ALIAS_SEPARATOR = ':'
    
    47
    +
    
    48
    +
    
    45 49
     class UtilError(BstError):
    
    46 50
         """Raised by utility functions when system calls fail.
    
    47 51
     
    
    ... ... @@ -608,6 +612,27 @@ def _parse_size(size, volume):
    608 612
         return int(num) * 1024**units.index(unit)
    
    609 613
     
    
    610 614
     
    
    615
    +# _pretty_size()
    
    616
    +#
    
    617
    +# Converts a number of bytes into a string representation in KB, MB, GB, TB
    
    618
    +# represented as K, M, G, T etc.
    
    619
    +#
    
    620
    +# Args:
    
    621
    +#   size (int): The size to convert in bytes.
    
    622
    +#   dec_places (int): The number of decimal places to output to.
    
    623
    +#
    
    624
    +# Returns:
    
    625
    +#   (str): The string representation of the number of bytes in the largest
    
    626
    +def _pretty_size(size, dec_places=0):
    
    627
    +    psize = size
    
    628
    +    unit = 'B'
    
    629
    +    for unit in ('B', 'K', 'M', 'G', 'T'):
    
    630
    +        if psize < 1024:
    
    631
    +            break
    
    632
    +        else:
    
    633
    +            psize /= 1024
    
    634
    +    return "{size:g}{unit}".format(size=round(psize, dec_places), unit=unit)
    
    635
    +
    
    611 636
     # A sentinel to be used as a default argument for functions that need
    
    612 637
     # to distinguish between a kwarg set to None and an unset kwarg.
    
    613 638
     _sentinel = object()
    

  • doc/source/examples/git-mirror.rst
    1
    +
    
    2
    +
    
    3
    +Creating and using a git mirror
    
    4
    +'''''''''''''''''''''''''''''''
    
    5
    +This is an example of how to create a git mirror using git's
    
    6
    +`git-http-backend <https://git-scm.com/docs/git-http-backend>`_ and
    
    7
    +`lighttpd <https://redmine.lighttpd.net/projects/1/wiki/TutorialConfiguration>`_.
    
    8
    +
    
    9
    +
    
    10
    +Prerequisites
    
    11
    +=============
    
    12
    +You will need git installed, and git-http-backend must be present. It is assumed
    
    13
    +that the git-http-backend binary exists at `/usr/lib/git-core/git-http-backend`.
    
    14
    +
    
    15
    +You will need `lighttpd` installed, and at the bare minimum has the modules
    
    16
    +`mod_alias`, `mod_cgi`, and `mod_setenv`.
    
    17
    +
    
    18
    +I will be using gnome-modulesets as an example, which can be cloned from
    
    19
    +`http://gnome7.codethink.co.uk/gnome-modulesets.git`.
    
    20
    +
    
    21
    +
    
    22
    +Starting a git http server
    
    23
    +==========================
    
    24
    +
    
    25
    +
    
    26
    +1. Set up a directory containing mirrors
    
    27
    +----------------------------------------
    
    28
    +Choose a suitable directory to hold your mirrors, e.g. `/var/www/git`.
    
    29
    +
    
    30
    +Place the git repositories you want to use as mirrors in the mirror dir, e.g.
    
    31
    +``git clone --mirror http://git.gnome.org/browse/yelp-xsl /var/www/git/yelp-xsl.git``.
    
    32
    +
    
    33
    +
    
    34
    +2. Configure lighttpd
    
    35
    +---------------------
    
    36
    +Write out a lighttpd.conf as follows:
    
    37
    +
    
    38
    +::
    
    39
    +
    
    40
    +   server.document-root = "/var/www/git/" 
    
    41
    +   server.port = 3000
    
    42
    +   server.modules = (
    
    43
    +        "mod_alias",
    
    44
    +        "mod_cgi",
    
    45
    +        "mod_setenv",
    
    46
    +   )
    
    47
    +   
    
    48
    +   alias.url += ( "/git" => "/usr/lib/git-core/git-http-backend" )
    
    49
    +   $HTTP["url"] =~ "^/git" {
    
    50
    +        cgi.assign = ("" => "")
    
    51
    +        setenv.add-environment = (
    
    52
    +                "GIT_PROJECT_ROOT" => "/var/www/git",
    
    53
    +                "GIT_HTTP_EXPORT_ALL" => ""
    
    54
    +        )
    
    55
    +   }
    
    56
    +
    
    57
    +.. note::
    
    58
    +
    
    59
    +   If you have your mirrors in another directory, replace /var/www/git/ with that directory.
    
    60
    +
    
    61
    +
    
    62
    +3. Start lighttpd
    
    63
    +-----------------
    
    64
    +lighttpd can be invoked with the command-line ``lighttpd -D -f lighttpd.conf``.
    
    65
    +
    
    66
    +
    
    67
    +4. Test that you can fetch from it
    
    68
    +----------------------------------
    
    69
    +We can then clone the mirrored repo using git via http with
    
    70
    +``git clone http://127.0.0.1:3000/git/yelp-xsl``.
    
    71
    +
    
    72
    +.. note::
    
    73
    +
    
    74
    +   If you have set server.port to something other than the default, you will
    
    75
    +   need to replace the '3000' in the command-line.
    
    76
    +
    
    77
    +
    
    78
    +5. Configure the project to use the mirror
    
    79
    +------------------------------------------
    
    80
    +To add this local http server as a mirror, add the following to the project.conf:
    
    81
    +
    
    82
    +.. code:: yaml
    
    83
    +
    
    84
    +   mirrors:
    
    85
    +   - name: local-mirror
    
    86
    +     aliases:
    
    87
    +       git_gnome_org:
    
    88
    +       - http://127.0.0.1:3000/git/
    
    89
    +
    
    90
    +
    
    91
    +6. Test that the mirror works
    
    92
    +-----------------------------
    
    93
    +We can make buildstream use the mirror by setting the alias to an invalid URL, e.g.
    
    94
    +
    
    95
    +.. code:: yaml
    
    96
    +
    
    97
    +   aliases:
    
    98
    +     git_gnome_org: https://www.example.com/invalid/url/
    
    99
    +
    
    100
    +Now, if you build an element that uses the source you placed in the mirror
    
    101
    +(e.g. ``bst build core-deps/yelp-xsl.bst``), you will see that it uses your mirror.
    
    102
    +
    
    103
    +
    
    104
    +.. _lighttpd_git_tar_conf:
    
    105
    +
    
    106
    +Bonus: lighttpd conf for git and tar
    
    107
    +====================================
    
    108
    +For those who have also used the :ref:`tar-mirror tutorial <using_tar_mirror>`,
    
    109
    +a combined lighttpd.conf is below:
    
    110
    +
    
    111
    +::
    
    112
    +
    
    113
    +   server.document-root = "/var/www/"
    
    114
    +   server.port = 3000
    
    115
    +   server.modules = (
    
    116
    +           "mod_alias",
    
    117
    +           "mod_cgi",
    
    118
    +           "mod_setenv",
    
    119
    +   )
    
    120
    +   
    
    121
    +   alias.url += ( "/git" => "/usr/lib/git-core/git-http-backend" )
    
    122
    +   $HTTP["url"] =~ "^/git" {
    
    123
    +           cgi.assign = ("" => "")
    
    124
    +           setenv.add-environment = (
    
    125
    +                   "GIT_PROJECT_ROOT" => "/var/www/git",
    
    126
    +                   "GIT_HTTP_EXPORT_ALL" => ""
    
    127
    +           )
    
    128
    +   } else $HTTP["url"] =~ "^/tar" {
    
    129
    +           dir-listing.activate = "enable"
    
    130
    +   }
    
    131
    +
    
    132
    +
    
    133
    +Further reading
    
    134
    +===============
    
    135
    +If this mirror isn't being used exclusively in a secure network, it is strongly
    
    136
    +recommended you `use SSL <https://redmine.lighttpd.net/projects/1/wiki/HowToSimpleSSL>`_.
    
    137
    +
    
    138
    +This is the bare minimum required to set up a git mirror. A large, public project
    
    139
    +would prefer to set it up using the
    
    140
    +`git protocol <https://git-scm.com/book/en/v1/Git-on-the-Server-Git-Daemon>`_,
    
    141
    +and a security-conscious project would be configured to use
    
    142
    +`git over SSH <https://git-scm.com/book/en/v1/Git-on-the-Server-Getting-Git-on-a-Server#Small-Setups>`_.
    
    143
    +
    
    144
    +Lighttpd is documented on `its wiki <https://redmine.lighttpd.net/projects/lighttpd/wiki>`_.

  • doc/source/examples/tar-mirror.rst
    1
    +
    
    2
    +
    
    3
    +.. _using_tar_mirror:
    
    4
    +
    
    5
    +Creating and using a tar mirror
    
    6
    +'''''''''''''''''''''''''''''''
    
    7
    +This is an example of how to create a tar mirror using 
    
    8
    +`lighttpd <https://redmine.lighttpd.net/projects/1/wiki/TutorialConfiguration>`_.
    
    9
    +
    
    10
    +
    
    11
    +Prerequisites
    
    12
    +=============
    
    13
    +You will need `lighttpd` installed.
    
    14
    +
    
    15
    +
    
    16
    +I will be using gnome-modulesets as an example, which can be cloned from
    
    17
    +`http://gnome7.codethink.co.uk/gnome-modulesets.git`.
    
    18
    +
    
    19
    +
    
    20
    +Starting a tar server
    
    21
    +=====================
    
    22
    +
    
    23
    +
    
    24
    +1. Set up a directory containing mirrors
    
    25
    +----------------------------------------
    
    26
    +Choose a suitable directory to hold your mirrored tar files, e.g. `/var/www/tar`.
    
    27
    +
    
    28
    +Place the tar files you want to use as mirrors in your mirror dir, e.g.
    
    29
    +
    
    30
    +.. code::
    
    31
    +
    
    32
    +   mkdir -p /var/www/tar/gettext
    
    33
    +   wget -O /var/www/tar/gettext/gettext-0.19.8.1.tar.xz https://ftp.gnu.org/gnu/gettext/gettext-0.19.8.1.tar.xz
    
    34
    +
    
    35
    +
    
    36
    +2. Configure lighttpd
    
    37
    +---------------------
    
    38
    +Write out a lighttpd.conf as follows:
    
    39
    +
    
    40
    +::
    
    41
    +
    
    42
    +   server.document-root = "/var/www/tar/" 
    
    43
    +   server.port = 3000
    
    44
    +   
    
    45
    +   dir-listing.activate = "enable"
    
    46
    +
    
    47
    +.. note::
    
    48
    +
    
    49
    +   If you have your mirrors in another directory, replace /var/www/tar/ with that directory.
    
    50
    +
    
    51
    +.. note::
    
    52
    +
    
    53
    +   An example lighttpd.conf that works for both git and tar services is available
    
    54
    +   :ref:`here <lighttpd_git_tar_conf>`
    
    55
    +
    
    56
    +
    
    57
    +3. Start lighttpd
    
    58
    +-----------------
    
    59
    +lighttpd can be invoked with the command-line ``lighttpd -D -f lighttpd.conf``.
    
    60
    +
    
    61
    +
    
    62
    +4. Test that you can fetch from it
    
    63
    +----------------------------------
    
    64
    +We can then download the mirrored file with ``wget 127.0.0.1:3000/tar/gettext/gettext-0.19.8.1.tar.xz``.
    
    65
    +
    
    66
    +.. note::
    
    67
    +
    
    68
    +   If you have set server.port to something other than the default, you will need
    
    69
    +   to replace the '3000' in the command-line.
    
    70
    +
    
    71
    +
    
    72
    +5. Configure the project to use the mirror
    
    73
    +------------------------------------------
    
    74
    +To add this local http server as a mirror, add the following to the project.conf:
    
    75
    +
    
    76
    +.. code:: yaml
    
    77
    +
    
    78
    +   mirrors:
    
    79
    +   - name: local-mirror
    
    80
    +     aliases:
    
    81
    +       ftp_gnu_org:
    
    82
    +       - http://127.0.0.1:3000/tar/
    
    83
    +
    
    84
    +
    
    85
    +6. Test that the mirror works
    
    86
    +-----------------------------
    
    87
    +We can make buildstream use the mirror by setting the alias to an invalid URL, e.g.
    
    88
    +
    
    89
    +.. code:: yaml
    
    90
    +
    
    91
    +   aliases:
    
    92
    +     ftp_gnu_org: https://www.example.com/invalid/url/
    
    93
    +
    
    94
    +Now, if you build an element that uses the source you placed in the mirror
    
    95
    +(e.g. ``bst build core-deps/gettext.bst``), you will see that it uses your mirror.
    
    96
    +
    
    97
    +
    
    98
    +Further reading
    
    99
    +===============
    
    100
    +If this mirror isn't being used exclusively in a secure network, it is strongly
    
    101
    +recommended you `use SSL <https://redmine.lighttpd.net/projects/1/wiki/HowToSimpleSSL>`_.
    
    102
    +
    
    103
    +Lighttpd is documented on `its wiki <https://redmine.lighttpd.net/projects/lighttpd/wiki>`_.

  • doc/source/format_project.rst
    ... ... @@ -198,6 +198,43 @@ You can also specify a list of caches here; earlier entries in the list
    198 198
     will have higher priority than later ones.
    
    199 199
     
    
    200 200
     
    
    201
    +.. _project_essentials_mirrors:
    
    202
    +
    
    203
    +Mirrors
    
    204
    +~~~~~~~
    
    205
    +A list of mirrors can be defined that couple a location to a mapping of aliases to a
    
    206
    +list of URIs, e.g.
    
    207
    +
    
    208
    +.. code:: yaml
    
    209
    +
    
    210
    +  mirrors:
    
    211
    +  - name: middle-earth
    
    212
    +    aliases:
    
    213
    +      foo:
    
    214
    +      - http://www.middle-earth.com/foo/1
    
    215
    +      - http://www.middle-earth.com/foo/2
    
    216
    +      bar:
    
    217
    +      - http://www.middle-earth.com/bar/1
    
    218
    +      - http://www.middle-earth.com/bar/2
    
    219
    +  - name: oz
    
    220
    +    aliases:
    
    221
    +      foo:
    
    222
    +      - http://www.oz.com/foo
    
    223
    +      bar:
    
    224
    +      - http://www.oz.com/bar
    
    225
    +
    
    226
    +The order that the mirrors (and the URIs therein) are consulted is in the order
    
    227
    +they are defined when fetching, and in reverse-order when tracking.
    
    228
    +
    
    229
    +A default mirror to consult first can be defined via
    
    230
    +:ref:`user config <config_default_mirror>`, or the command-line argument
    
    231
    +:ref:`--default-mirror <invoking_bst>`.
    
    232
    +
    
    233
    +.. note::
    
    234
    +
    
    235
    +   The ``mirrors`` field is available since :ref:`format version 11 <project_format_version>`
    
    236
    +
    
    237
    +
    
    201 238
     .. _project_plugins:
    
    202 239
     
    
    203 240
     External plugins
    

  • doc/source/main_install.rst
    1 1
     Install
    
    2 2
     =======
    
    3
    -This section covers how to install BuildStream onto your machine, how to run BuildStream inside a docker image and also how to configure an artifact server.
    
    3
    +This section covers how to install BuildStream onto your machine, how to run
    
    4
    +BuildStream inside a docker image and also how to configure an artifact server.
    
    4 5
     
    
    6
    +.. note::
    
    7
    +
    
    8
    +   BuildStream is not currently supported natively on macOS and Windows. Windows
    
    9
    +   and macOS users should refer to :ref:`docker`.
    
    5 10
     
    
    6 11
     .. toctree::
    
    7 12
        :maxdepth: 2
    

  • doc/source/using_config.rst
    ... ... @@ -89,6 +89,27 @@ modifying some low level component.
    89 89
        the ``--strict`` and ``--no-strict`` command line options.
    
    90 90
     
    
    91 91
     
    
    92
    +.. _config_default_mirror:
    
    93
    +
    
    94
    +Default Mirror
    
    95
    +~~~~~~~~~~~~~~
    
    96
    +When using :ref:`mirrors <project_essentials_mirrors>`, a default mirror can
    
    97
    +be defined to be fetched first.
    
    98
    +The default mirror is defined by its name, e.g.
    
    99
    +
    
    100
    +.. code:: yaml
    
    101
    +
    
    102
    +  projects:
    
    103
    +    project-name:
    
    104
    +      default-mirror: oz
    
    105
    +
    
    106
    +
    
    107
    +.. note::
    
    108
    +
    
    109
    +   It is possible to override this at invocation time using the
    
    110
    +   ``--default-mirror`` command-line option.
    
    111
    +
    
    112
    +
    
    92 113
     Default configuration
    
    93 114
     ---------------------
    
    94 115
     The default BuildStream configuration is specified here for reference:
    

  • doc/source/using_examples.rst
    ... ... @@ -10,3 +10,5 @@ maintained and work as expected.
    10 10
        :maxdepth: 1
    
    11 11
     
    
    12 12
        examples/flatpak-autotools
    
    13
    +   examples/tar-mirror
    
    14
    +   examples/git-mirror

  • tests/completions/completions.py
    ... ... @@ -27,6 +27,7 @@ MAIN_OPTIONS = [
    27 27
         "--colors ",
    
    28 28
         "--config ",
    
    29 29
         "--debug ",
    
    30
    +    "--default-mirror ",
    
    30 31
         "--directory ",
    
    31 32
         "--error-lines ",
    
    32 33
         "--fetchers ",
    

  • tests/frontend/mirror.py
    1
    +import os
    
    2
    +import pytest
    
    3
    +
    
    4
    +from tests.testutils import cli, create_repo, ALL_REPO_KINDS
    
    5
    +
    
    6
    +from buildstream import _yaml
    
    7
    +
    
    8
    +
    
    9
    +# Project directory
    
    10
    +TOP_DIR = os.path.dirname(os.path.realpath(__file__))
    
    11
    +DATA_DIR = os.path.join(TOP_DIR, 'project')
    
    12
    +
    
    13
    +
    
    14
    +def generate_element(output_file):
    
    15
    +    element = {
    
    16
    +        'kind': 'import',
    
    17
    +        'sources': [
    
    18
    +            {
    
    19
    +                'kind': 'fetch_source',
    
    20
    +                "output-text": output_file,
    
    21
    +                "urls": ["foo:repo1", "bar:repo2"],
    
    22
    +                "fetch-succeeds": {
    
    23
    +                    "FOO/repo1": True,
    
    24
    +                    "BAR/repo2": False,
    
    25
    +                    "OOF/repo1": False,
    
    26
    +                    "RAB/repo2": True,
    
    27
    +                    "OFO/repo1": False,
    
    28
    +                    "RBA/repo2": False,
    
    29
    +                    "ooF/repo1": False,
    
    30
    +                    "raB/repo2": False,
    
    31
    +                }
    
    32
    +            }
    
    33
    +        ]
    
    34
    +    }
    
    35
    +    return element
    
    36
    +
    
    37
    +
    
    38
    +def generate_project():
    
    39
    +    project = {
    
    40
    +        'name': 'test',
    
    41
    +        'element-path': 'elements',
    
    42
    +        'aliases': {
    
    43
    +            'foo': 'FOO/',
    
    44
    +            'bar': 'BAR/',
    
    45
    +        },
    
    46
    +        'mirrors': [
    
    47
    +            {
    
    48
    +                'name': 'middle-earth',
    
    49
    +                'aliases': {
    
    50
    +                    'foo': ['OOF/'],
    
    51
    +                    'bar': ['RAB/'],
    
    52
    +                },
    
    53
    +            },
    
    54
    +            {
    
    55
    +                'name': 'arrakis',
    
    56
    +                'aliases': {
    
    57
    +                    'foo': ['OFO/'],
    
    58
    +                    'bar': ['RBA/'],
    
    59
    +                },
    
    60
    +            },
    
    61
    +            {
    
    62
    +                'name': 'oz',
    
    63
    +                'aliases': {
    
    64
    +                    'foo': ['ooF/'],
    
    65
    +                    'bar': ['raB/'],
    
    66
    +                }
    
    67
    +            },
    
    68
    +        ],
    
    69
    +        'plugins': [
    
    70
    +            {
    
    71
    +                'origin': 'local',
    
    72
    +                'path': 'sources',
    
    73
    +                'sources': {
    
    74
    +                    'fetch_source': 0
    
    75
    +                }
    
    76
    +            }
    
    77
    +        ]
    
    78
    +    }
    
    79
    +    return project
    
    80
    +
    
    81
    +
    
    82
    +@pytest.mark.datafiles(DATA_DIR)
    
    83
    +@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
    
    84
    +def test_mirror_fetch(cli, tmpdir, datafiles, kind):
    
    85
    +    bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
    
    86
    +    dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
    
    87
    +    upstream_repodir = os.path.join(str(tmpdir), 'upstream')
    
    88
    +    mirror_repodir = os.path.join(str(tmpdir), 'mirror')
    
    89
    +    project_dir = os.path.join(str(tmpdir), 'project')
    
    90
    +    os.makedirs(project_dir)
    
    91
    +    element_dir = os.path.join(project_dir, 'elements')
    
    92
    +
    
    93
    +    # Create repo objects of the upstream and mirror
    
    94
    +    upstream_repo = create_repo(kind, upstream_repodir)
    
    95
    +    upstream_ref = upstream_repo.create(bin_files_path)
    
    96
    +    mirror_repo = upstream_repo.copy(mirror_repodir)
    
    97
    +    mirror_ref = upstream_ref
    
    98
    +    upstream_ref = upstream_repo.create(dev_files_path)
    
    99
    +
    
    100
    +    element = {
    
    101
    +        'kind': 'import',
    
    102
    +        'sources': [
    
    103
    +            upstream_repo.source_config(ref=upstream_ref)
    
    104
    +        ]
    
    105
    +    }
    
    106
    +    element_name = 'test.bst'
    
    107
    +    element_path = os.path.join(element_dir, element_name)
    
    108
    +    full_repo = element['sources'][0]['url']
    
    109
    +    upstream_map, repo_name = os.path.split(full_repo)
    
    110
    +    alias = 'foo-' + kind
    
    111
    +    aliased_repo = alias + ':' + repo_name
    
    112
    +    element['sources'][0]['url'] = aliased_repo
    
    113
    +    full_mirror = mirror_repo.source_config()['url']
    
    114
    +    mirror_map, _ = os.path.split(full_mirror)
    
    115
    +    os.makedirs(element_dir)
    
    116
    +    _yaml.dump(element, element_path)
    
    117
    +
    
    118
    +    project = {
    
    119
    +        'name': 'test',
    
    120
    +        'element-path': 'elements',
    
    121
    +        'aliases': {
    
    122
    +            alias: upstream_map + "/"
    
    123
    +        },
    
    124
    +        'mirrors': [
    
    125
    +            {
    
    126
    +                'name': 'middle-earth',
    
    127
    +                'aliases': {
    
    128
    +                    alias: [mirror_map + "/"],
    
    129
    +                },
    
    130
    +            },
    
    131
    +        ]
    
    132
    +    }
    
    133
    +    project_file = os.path.join(project_dir, 'project.conf')
    
    134
    +    _yaml.dump(project, project_file)
    
    135
    +
    
    136
    +    # No obvious ways of checking that the mirror has been fetched
    
    137
    +    # But at least we can be sure it succeeds
    
    138
    +    result = cli.run(project=project_dir, args=['fetch', element_name])
    
    139
    +    result.assert_success()
    
    140
    +
    
    141
    +
    
    142
    +@pytest.mark.datafiles(DATA_DIR)
    
    143
    +def test_mirror_fetch_multi(cli, tmpdir, datafiles):
    
    144
    +    output_file = os.path.join(str(tmpdir), "output.txt")
    
    145
    +    project_dir = str(tmpdir)
    
    146
    +    element_dir = os.path.join(project_dir, 'elements')
    
    147
    +    os.makedirs(element_dir, exist_ok=True)
    
    148
    +    element_name = "test.bst"
    
    149
    +    element_path = os.path.join(element_dir, element_name)
    
    150
    +    element = generate_element(output_file)
    
    151
    +    _yaml.dump(element, element_path)
    
    152
    +
    
    153
    +    project_file = os.path.join(project_dir, 'project.conf')
    
    154
    +    project = generate_project()
    
    155
    +    _yaml.dump(project, project_file)
    
    156
    +
    
    157
    +    result = cli.run(project=project_dir, args=['fetch', element_name])
    
    158
    +    result.assert_success()
    
    159
    +    with open(output_file) as f:
    
    160
    +        contents = f.read()
    
    161
    +        assert "Fetch foo:repo1 succeeded from FOO/repo1" in contents
    
    162
    +        assert "Fetch bar:repo2 succeeded from RAB/repo2" in contents
    
    163
    +
    
    164
    +
    
    165
    +@pytest.mark.datafiles(DATA_DIR)
    
    166
    +def test_mirror_fetch_default_cmdline(cli, tmpdir, datafiles):
    
    167
    +    output_file = os.path.join(str(tmpdir), "output.txt")
    
    168
    +    project_dir = str(tmpdir)
    
    169
    +    element_dir = os.path.join(project_dir, 'elements')
    
    170
    +    os.makedirs(element_dir, exist_ok=True)
    
    171
    +    element_name = "test.bst"
    
    172
    +    element_path = os.path.join(element_dir, element_name)
    
    173
    +    element = generate_element(output_file)
    
    174
    +    _yaml.dump(element, element_path)
    
    175
    +
    
    176
    +    project_file = os.path.join(project_dir, 'project.conf')
    
    177
    +    project = generate_project()
    
    178
    +    _yaml.dump(project, project_file)
    
    179
    +
    
    180
    +    result = cli.run(project=project_dir, args=['--default-mirror', 'arrakis', 'fetch', element_name])
    
    181
    +    result.assert_success()
    
    182
    +    with open(output_file) as f:
    
    183
    +        contents = f.read()
    
    184
    +        print(contents)
    
    185
    +        # Success if fetching from arrakis' mirror happened before middle-earth's
    
    186
    +        arrakis_str = "OFO/repo1"
    
    187
    +        arrakis_pos = contents.find(arrakis_str)
    
    188
    +        assert arrakis_pos != -1, "'{}' wasn't found".format(arrakis_str)
    
    189
    +        me_str = "OOF/repo1"
    
    190
    +        me_pos = contents.find(me_str)
    
    191
    +        assert me_pos != -1, "'{}' wasn't found".format(me_str)
    
    192
    +        assert arrakis_pos < me_pos, "'{}' wasn't found before '{}'".format(arrakis_str, me_str)
    
    193
    +
    
    194
    +
    
    195
    +@pytest.mark.datafiles(DATA_DIR)
    
    196
    +def test_mirror_fetch_default_userconfig(cli, tmpdir, datafiles):
    
    197
    +    output_file = os.path.join(str(tmpdir), "output.txt")
    
    198
    +    project_dir = str(tmpdir)
    
    199
    +    element_dir = os.path.join(project_dir, 'elements')
    
    200
    +    os.makedirs(element_dir, exist_ok=True)
    
    201
    +    element_name = "test.bst"
    
    202
    +    element_path = os.path.join(element_dir, element_name)
    
    203
    +    element = generate_element(output_file)
    
    204
    +    _yaml.dump(element, element_path)
    
    205
    +
    
    206
    +    project_file = os.path.join(project_dir, 'project.conf')
    
    207
    +    project = generate_project()
    
    208
    +    _yaml.dump(project, project_file)
    
    209
    +
    
    210
    +    userconfig = {
    
    211
    +        'projects': {
    
    212
    +            'test': {
    
    213
    +                'default-mirror': 'oz'
    
    214
    +            }
    
    215
    +        }
    
    216
    +    }
    
    217
    +    cli.configure(userconfig)
    
    218
    +
    
    219
    +    result = cli.run(project=project_dir, args=['fetch', element_name])
    
    220
    +    result.assert_success()
    
    221
    +    with open(output_file) as f:
    
    222
    +        contents = f.read()
    
    223
    +        print(contents)
    
    224
    +        # Success if fetching from Oz' mirror happened before middle-earth's
    
    225
    +        oz_str = "ooF/repo1"
    
    226
    +        oz_pos = contents.find(oz_str)
    
    227
    +        assert oz_pos != -1, "'{}' wasn't found".format(oz_str)
    
    228
    +        me_str = "OOF/repo1"
    
    229
    +        me_pos = contents.find(me_str)
    
    230
    +        assert me_pos != -1, "'{}' wasn't found".format(me_str)
    
    231
    +        assert oz_pos < me_pos, "'{}' wasn't found before '{}'".format(oz_str, me_str)
    
    232
    +
    
    233
    +
    
    234
    +@pytest.mark.datafiles(DATA_DIR)
    
    235
    +def test_mirror_fetch_default_cmdline_overrides_config(cli, tmpdir, datafiles):
    
    236
    +    output_file = os.path.join(str(tmpdir), "output.txt")
    
    237
    +    project_dir = str(tmpdir)
    
    238
    +    element_dir = os.path.join(project_dir, 'elements')
    
    239
    +    os.makedirs(element_dir, exist_ok=True)
    
    240
    +    element_name = "test.bst"
    
    241
    +    element_path = os.path.join(element_dir, element_name)
    
    242
    +    element = generate_element(output_file)
    
    243
    +    _yaml.dump(element, element_path)
    
    244
    +
    
    245
    +    project_file = os.path.join(project_dir, 'project.conf')
    
    246
    +    project = generate_project()
    
    247
    +    _yaml.dump(project, project_file)
    
    248
    +
    
    249
    +    userconfig = {
    
    250
    +        'projects': {
    
    251
    +            'test': {
    
    252
    +                'default-mirror': 'oz'
    
    253
    +            }
    
    254
    +        }
    
    255
    +    }
    
    256
    +    cli.configure(userconfig)
    
    257
    +
    
    258
    +    result = cli.run(project=project_dir, args=['--default-mirror', 'arrakis', 'fetch', element_name])
    
    259
    +    result.assert_success()
    
    260
    +    with open(output_file) as f:
    
    261
    +        contents = f.read()
    
    262
    +        print(contents)
    
    263
    +        # Success if fetching from arrakis' mirror happened before middle-earth's
    
    264
    +        arrakis_str = "OFO/repo1"
    
    265
    +        arrakis_pos = contents.find(arrakis_str)
    
    266
    +        assert arrakis_pos != -1, "'{}' wasn't found".format(arrakis_str)
    
    267
    +        me_str = "OOF/repo1"
    
    268
    +        me_pos = contents.find(me_str)
    
    269
    +        assert me_pos != -1, "'{}' wasn't found".format(me_str)
    
    270
    +        assert arrakis_pos < me_pos, "'{}' wasn't found before '{}'".format(arrakis_str, me_str)
    
    271
    +
    
    272
    +
    
    273
    +@pytest.mark.datafiles(DATA_DIR)
    
    274
    +@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
    
    275
    +def test_mirror_track_upstream_present(cli, tmpdir, datafiles, kind):
    
    276
    +    bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
    
    277
    +    dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
    
    278
    +    upstream_repodir = os.path.join(str(tmpdir), 'upstream')
    
    279
    +    mirror_repodir = os.path.join(str(tmpdir), 'mirror')
    
    280
    +    project_dir = os.path.join(str(tmpdir), 'project')
    
    281
    +    os.makedirs(project_dir)
    
    282
    +    element_dir = os.path.join(project_dir, 'elements')
    
    283
    +
    
    284
    +    # Create repo objects of the upstream and mirror
    
    285
    +    upstream_repo = create_repo(kind, upstream_repodir)
    
    286
    +    upstream_ref = upstream_repo.create(bin_files_path)
    
    287
    +    mirror_repo = upstream_repo.copy(mirror_repodir)
    
    288
    +    mirror_ref = upstream_ref
    
    289
    +    upstream_ref = upstream_repo.create(dev_files_path)
    
    290
    +
    
    291
    +    element = {
    
    292
    +        'kind': 'import',
    
    293
    +        'sources': [
    
    294
    +            upstream_repo.source_config(ref=upstream_ref)
    
    295
    +        ]
    
    296
    +    }
    
    297
    +
    
    298
    +    element['sources'][0]
    
    299
    +    element_name = 'test.bst'
    
    300
    +    element_path = os.path.join(element_dir, element_name)
    
    301
    +    full_repo = element['sources'][0]['url']
    
    302
    +    upstream_map, repo_name = os.path.split(full_repo)
    
    303
    +    alias = 'foo-' + kind
    
    304
    +    aliased_repo = alias + ':' + repo_name
    
    305
    +    element['sources'][0]['url'] = aliased_repo
    
    306
    +    full_mirror = mirror_repo.source_config()['url']
    
    307
    +    mirror_map, _ = os.path.split(full_mirror)
    
    308
    +    os.makedirs(element_dir)
    
    309
    +    _yaml.dump(element, element_path)
    
    310
    +
    
    311
    +    project = {
    
    312
    +        'name': 'test',
    
    313
    +        'element-path': 'elements',
    
    314
    +        'aliases': {
    
    315
    +            alias: upstream_map + "/"
    
    316
    +        },
    
    317
    +        'mirrors': [
    
    318
    +            {
    
    319
    +                'name': 'middle-earth',
    
    320
    +                'aliases': {
    
    321
    +                    alias: [mirror_map + "/"],
    
    322
    +                },
    
    323
    +            },
    
    324
    +        ]
    
    325
    +    }
    
    326
    +    project_file = os.path.join(project_dir, 'project.conf')
    
    327
    +    _yaml.dump(project, project_file)
    
    328
    +
    
    329
    +    result = cli.run(project=project_dir, args=['track', element_name])
    
    330
    +    result.assert_success()
    
    331
    +
    
    332
    +    # Tracking tries upstream first. Check the ref is from upstream.
    
    333
    +    new_element = _yaml.load(element_path)
    
    334
    +    source = new_element['sources'][0]
    
    335
    +    if 'ref' in source:
    
    336
    +        assert source['ref'] == upstream_ref
    
    337
    +
    
    338
    +
    
    339
    +@pytest.mark.datafiles(DATA_DIR)
    
    340
    +@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
    
    341
    +def test_mirror_track_upstream_absent(cli, tmpdir, datafiles, kind):
    
    342
    +    bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
    
    343
    +    dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
    
    344
    +    upstream_repodir = os.path.join(str(tmpdir), 'upstream')
    
    345
    +    mirror_repodir = os.path.join(str(tmpdir), 'mirror')
    
    346
    +    project_dir = os.path.join(str(tmpdir), 'project')
    
    347
    +    os.makedirs(project_dir)
    
    348
    +    element_dir = os.path.join(project_dir, 'elements')
    
    349
    +
    
    350
    +    # Create repo objects of the upstream and mirror
    
    351
    +    upstream_repo = create_repo(kind, upstream_repodir)
    
    352
    +    upstream_ref = upstream_repo.create(bin_files_path)
    
    353
    +    mirror_repo = upstream_repo.copy(mirror_repodir)
    
    354
    +    mirror_ref = upstream_ref
    
    355
    +    upstream_ref = upstream_repo.create(dev_files_path)
    
    356
    +
    
    357
    +    element = {
    
    358
    +        'kind': 'import',
    
    359
    +        'sources': [
    
    360
    +            upstream_repo.source_config(ref=upstream_ref)
    
    361
    +        ]
    
    362
    +    }
    
    363
    +
    
    364
    +    element['sources'][0]
    
    365
    +    element_name = 'test.bst'
    
    366
    +    element_path = os.path.join(element_dir, element_name)
    
    367
    +    full_repo = element['sources'][0]['url']
    
    368
    +    upstream_map, repo_name = os.path.split(full_repo)
    
    369
    +    alias = 'foo-' + kind
    
    370
    +    aliased_repo = alias + ':' + repo_name
    
    371
    +    element['sources'][0]['url'] = aliased_repo
    
    372
    +    full_mirror = mirror_repo.source_config()['url']
    
    373
    +    mirror_map, _ = os.path.split(full_mirror)
    
    374
    +    os.makedirs(element_dir)
    
    375
    +    _yaml.dump(element, element_path)
    
    376
    +
    
    377
    +    project = {
    
    378
    +        'name': 'test',
    
    379
    +        'element-path': 'elements',
    
    380
    +        'aliases': {
    
    381
    +            alias: 'http://www.example.com/'
    
    382
    +        },
    
    383
    +        'mirrors': [
    
    384
    +            {
    
    385
    +                'name': 'middle-earth',
    
    386
    +                'aliases': {
    
    387
    +                    alias: [mirror_map + "/"],
    
    388
    +                },
    
    389
    +            },
    
    390
    +        ]
    
    391
    +    }
    
    392
    +    project_file = os.path.join(project_dir, 'project.conf')
    
    393
    +    _yaml.dump(project, project_file)
    
    394
    +
    
    395
    +    result = cli.run(project=project_dir, args=['track', element_name])
    
    396
    +    result.assert_success()
    
    397
    +
    
    398
    +    # Check that tracking fell back to the mirror
    
    399
    +    new_element = _yaml.load(element_path)
    
    400
    +    source = new_element['sources'][0]
    
    401
    +    if 'ref' in source:
    
    402
    +        assert source['ref'] == mirror_ref

  • tests/frontend/project/sources/fetch_source.py
    1
    +import os
    
    2
    +import sys
    
    3
    +
    
    4
    +from buildstream import Source, Consistency, SourceError, SourceFetcher
    
    5
    +
    
    6
    +# Expected config
    
    7
    +# sources:
    
    8
    +# - output-text: $FILE
    
    9
    +#   urls:
    
    10
    +#   - foo:bar
    
    11
    +#   - baz:quux
    
    12
    +#   fetch-succeeds:
    
    13
    +#     Foo/bar: true
    
    14
    +#     ooF/bar: false
    
    15
    +
    
    16
    +
    
    17
    +class FetchFetcher(SourceFetcher):
    
    18
    +    def __init__(self, source, url):
    
    19
    +        super().__init__()
    
    20
    +        self.source = source
    
    21
    +        self.original_url = url
    
    22
    +        self.mark_download_url(url)
    
    23
    +
    
    24
    +    def fetch(self, alias_override=None):
    
    25
    +        url = self.source.translate_url(self.original_url, alias_override=alias_override)
    
    26
    +        with open(self.source.output_file, "a") as f:
    
    27
    +            success = url in self.source.fetch_succeeds and self.source.fetch_succeeds[url]
    
    28
    +            message = "Fetch {} {} from {}\n".format(self.original_url,
    
    29
    +                                                     "succeeded" if success else "failed",
    
    30
    +                                                     url)
    
    31
    +            f.write(message)
    
    32
    +            if not success:
    
    33
    +                raise SourceError("Failed to fetch {}".format(url))
    
    34
    +
    
    35
    +
    
    36
    +class FetchSource(Source):
    
    37
    +    # Read config to know which URLs to fetch
    
    38
    +    def configure(self, node):
    
    39
    +        self.original_urls = self.node_get_member(node, list, 'urls')
    
    40
    +        self.fetchers = [FetchFetcher(self, url) for url in self.original_urls]
    
    41
    +        self.output_file = self.node_get_member(node, str, 'output-text')
    
    42
    +        self.fetch_succeeds = {}
    
    43
    +        if 'fetch-succeeds' in node:
    
    44
    +            self.fetch_succeeds = {x[0]: x[1] for x in self.node_items(node['fetch-succeeds'])}
    
    45
    +
    
    46
    +    def get_source_fetchers(self):
    
    47
    +        return self.fetchers
    
    48
    +
    
    49
    +    def preflight(self):
    
    50
    +        output_dir = os.path.dirname(self.output_file)
    
    51
    +        if not os.path.exists(output_dir):
    
    52
    +            raise SourceError("Directory '{}' does not exist".format(output_dir))
    
    53
    +
    
    54
    +    def fetch(self):
    
    55
    +        for fetcher in self.fetchers:
    
    56
    +            fetcher.fetch()
    
    57
    +
    
    58
    +    def get_unique_key(self):
    
    59
    +        return {"urls": self.original_urls, "output_file": self.output_file}
    
    60
    +
    
    61
    +    def get_consistency(self):
    
    62
    +        if not os.path.exists(self.output_file):
    
    63
    +            return Consistency.RESOLVED
    
    64
    +
    
    65
    +        with open(self.output_file, "r") as f:
    
    66
    +            contents = f.read()
    
    67
    +            for url in self.original_urls:
    
    68
    +                if url not in contents:
    
    69
    +                    return Consistency.RESOLVED
    
    70
    +
    
    71
    +        return Consistency.CACHED
    
    72
    +
    
    73
    +    # We dont have a ref, we're a local file...
    
    74
    +    def load_ref(self, node):
    
    75
    +        pass
    
    76
    +
    
    77
    +    def get_ref(self):
    
    78
    +        return None  # pragma: nocover
    
    79
    +
    
    80
    +    def set_ref(self, ref, node):
    
    81
    +        pass  # pragma: nocover
    
    82
    +
    
    83
    +
    
    84
    +def setup():
    
    85
    +    return FetchSource

  • tests/testutils/repo/repo.py
    ... ... @@ -22,7 +22,7 @@ class Repo():
    22 22
             # The directory the actual repo will be stored in
    
    23 23
             self.repo = os.path.join(self.directory, subdir)
    
    24 24
     
    
    25
    -        os.makedirs(self.repo)
    
    25
    +        os.makedirs(self.repo, exist_ok=True)
    
    26 26
     
    
    27 27
         # create():
    
    28 28
         #
    
    ... ... @@ -69,3 +69,22 @@ class Repo():
    69 69
                     shutil.copytree(src_path, dest_path)
    
    70 70
                 else:
    
    71 71
                     shutil.copy2(src_path, dest_path)
    
    72
    +
    
    73
    +    # copy():
    
    74
    +    #
    
    75
    +    # Creates a copy of this repository in the specified
    
    76
    +    # destination.
    
    77
    +    #
    
    78
    +    # Args:
    
    79
    +    #    dest (str): The destination directory
    
    80
    +    #
    
    81
    +    # Returns:
    
    82
    +    #    (Repo): A Repo object for the new repository.
    
    83
    +    def copy(self, dest):
    
    84
    +        subdir = self.repo[len(self.directory):].lstrip(os.sep)
    
    85
    +        new_dir = os.path.join(dest, subdir)
    
    86
    +        os.makedirs(new_dir, exist_ok=True)
    
    87
    +        self.copy_directory(self.repo, new_dir)
    
    88
    +        repo_type = type(self)
    
    89
    +        new_repo = repo_type(dest, subdir)
    
    90
    +        return new_repo



  • [Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]