GNOME.org

[Notes] [Git][BuildStream/buildstream][328-support-for-downloading-sources-from-mirrors] 22 commits: doc/source/install_linux_distro.rst: buildstream doesn't depend on ostree or pygobject anymore

From: Jonathan Maw <gitlab mg gitlab com>
To: buildstream-notifications-list gnome org
Subject: [Notes] [Git][BuildStream/buildstream][328-support-for-downloading-sources-from-mirrors] 22 commits: doc/source/install_linux_distro.rst: buildstream doesn't depend on ostree or pygobject anymore
Date: Wed, 25 Jul 2018 16:47:51 +0000

Title: GitLab

Jonathan Maw pushed to branch 328-support-for-downloading-sources-from-mirrors at BuildStream / buildstream

Commits:

60290f31

by Javier Jardón at 2018-07-23T15:29:37Z

doc/source/install_linux_distro.rst: buildstream doesn't depend on ostree or pygobject anymore

de162e43

by Javier Jardón at 2018-07-24T11:58:16Z

Merge branch 'jjardon/ostree_repo' into 'master'

doc/source/install_linux_distro.rst: buildstream doesn't hard depend on ostree or pygobject anymore

Closes #492

See merge request BuildStream/buildstream!558

6106d657

by Josh Smith at 2018-07-25T10:50:14Z

cascache.py: Remove misleading info message

This addresses issue #501.

4a637d63

by Jonathan Maw at 2018-07-25T12:05:19Z

Merge branch 'Qinusty/501' into 'master'

Remove misleading info message

See merge request BuildStream/buildstream!567

a0ad985f

by Phil Dawson at 2018-07-25T13:03:23Z

tests/frontend/show.py: Add test case for maximum recursion depth being exceeded

Add test to ensure gracefull handling of exception thrown while loading
a pipeline due the python's max recursion depth being exceeded.

This is part of the work for issue #203

95b4eae4

by Phil Dawson at 2018-07-25T13:03:23Z

app.py: Handle exception thrown when recursion limit is exceeded

Because the RecursionError exception was introduced in Python 3.5, until we
drop support for for Python 3.4, we must use RuntimeError.

889bf238

by Jim MacArthur at 2018-07-25T13:43:21Z

Merge branch 'phil/203-BuildStream-crashes-when-dependency-tree-too-deep' into 'master'

Phil/203 BuildStream crashes when dependency tree too deep

See merge request BuildStream/buildstream!512

54fee6c8

by Phillip Smyth at 2018-07-25T14:15:36Z

_stream.py: Added functionality for workspace open -f

tests/frontend/workspace.py: Added tests

f62b6cb7

by Jonathan Maw at 2018-07-25T15:01:33Z

Merge branch 'bst_workspace_open_force_does_nothing' into 'master'

_stream.py: Added functionality for workspace open -f

See merge request BuildStream/buildstream!549

9961cd65
by Jonathan Maw at 2018-07-25T16:25:10Z
```
project: Parse and store mirrors
```

a75928e9

by Jonathan Maw at 2018-07-25T16:25:10Z

Move _ALIAS_SEPARATOR into utils

The separator is useful in source files other than _project.py

0cfccfd8

by Jonathan Maw at 2018-07-25T16:34:54Z

source: Store the url aliases or use an override

This is part of a later plan to implement mirroring without forcing
everyone to update their sources. We use the expected calls to
Source.translate_url() when running Source.configure() to extract the
aliases from the URL. Multiple aliases must be extracted because
sources exist that may fetch from multiple aliases (for example, git
submodules)

Later, we want to substitute another URI where the alias normally reads
from the project - We accomplish this by re-instantiating the Source
with the alias overrides passed as an argument to the constructor.

28f18e08

by Jonathan Maw at 2018-07-25T16:35:40Z

source: When fetching, try to fetch from mirrors first

**KLUDGE WARNING**: This involves making the source store its "meta"
object so that it's possible to create a copy of the source inside the
fetch queue, instead of back when the pipeline was being loaded.

This adds the SourceFetcher class, which is intended for sources that
fetch from multiple URLs (e.g. the git source and its submodules)

Fix when fetching

aca8b29e

by Jonathan Maw at 2018-07-25T16:35:40Z

Set default mirror via command-line or user config

In user config (buildstream.conf), it is set with the "default-mirror"
field.

On the command-line, it is set with "--default-mirror"

ffbf293d
by Jonathan Maw at 2018-07-25T16:35:40Z
```
Add mirrored source tracking
```

cece104e

by Jonathan Maw at 2018-07-25T16:35:40Z

bzr.py: Improve mirror support

This fixes:
* Bzr repositories pulling from the branch they were created with.
* Bzr's _ensure_mirror() not actually checking that it successfully
  mirrored the ref.

5feb2110

by Jonathan Maw at 2018-07-25T16:35:40Z

git.py: Improve mirror support

When implementing fetching from mirrors, I encountered some problems
with the git source:

1. The mirror URL was using translate_url()'s output, so if a different
   alias was used, then fetching from the mirror would go to a different
   directory, and be inaccessible.
2. After fixing that, fetching was unable to pull from a URL other than
   the one used at repository creation, meaning it wouldn't actually
   pull from the mirror.

3cce90ba

by Jonathan Maw at 2018-07-25T16:35:40Z

testutils: Add a helper to copy a testutils repo

This is helpful if you want to test what happens when you have one repo
that has diverged from another. By copying the repo you're sure they
start with shared history.

This is especially useful when mirroring.

ff3eccf0

by Jonathan Maw at 2018-07-25T16:35:40Z

tests: Add mirrored fetching and tracking tests

6127f954
by Jonathan Maw at 2018-07-25T16:35:40Z
```
doc: Add documentation for source mirroring
```
7ebadccb
by Jonathan Maw at 2018-07-25T16:35:40Z
```
NEWS: Add item for source mirroring
```

c290fec3

by Jonathan Maw at 2018-07-25T16:35:40Z

doc: Add tutorials for setting up git and tar mirrors

25 changed files:

NEWS
buildstream/__init__.py
buildstream/_artifactcache/cascache.py
buildstream/_context.py
buildstream/_frontend/app.py
buildstream/_frontend/cli.py
buildstream/_loader/loader.py
buildstream/_project.py
buildstream/_stream.py
buildstream/plugins/sources/bzr.py
buildstream/plugins/sources/git.py
buildstream/source.py
buildstream/utils.py
+ doc/source/examples/git-mirror.rst
+ doc/source/examples/tar-mirror.rst
doc/source/format_project.rst
doc/source/install_linux_distro.rst
doc/source/using_config.rst
doc/source/using_examples.rst
tests/completions/completions.py
+ tests/frontend/mirror.py
+ tests/frontend/project/sources/fetch_source.py
tests/frontend/show.py
tests/frontend/workspace.py
tests/testutils/repo/repo.py

Changes:

NEWS

@@ -5,6 +5,10 @@ buildstream 1.3.1
    o Add a `--tar` option to `bst checkout` which allows a tarball to be
      created from the artifact contents.
 +  o Fetching and tracking will consult mirrors defined in project config,
 +    and the preferred mirror to fetch from can be defined in the command
 +    line or user config.
++
  =================
  buildstream 1.1.4
  =================

buildstream/__init__.py

@@ -29,7 +29,7 @@ if "_BST_COMPLETION" not in os.environ:
      from .utils import UtilError, ProgramNotFoundError
      from .sandbox import Sandbox, SandboxFlags
      from .plugin import Plugin
 -    from .source import Source, SourceError, Consistency
 +    from .source import Source, SourceError, Consistency, SourceFetcher
      from .element import Element, ElementError, Scope
      from .buildelement import BuildElement
      from .scriptelement import ScriptElement

buildstream/_artifactcache/cascache.py

@@ -240,7 +240,6 @@ class CASCache(ArtifactCache):
              except grpc.RpcError as e:
                  if e.code() != grpc.StatusCode.NOT_FOUND:
 -                    element.info("{} not found at remote {}".format(element._get_brief_display_key(), remote.spec.url))
                      raise
          return False

buildstream/_context.py

@@ -259,7 +259,7 @@ class Context():
          # Shallow validation of overrides, parts of buildstream which rely
          # on the overrides are expected to validate elsewhere.
          for _, overrides in _yaml.node_items(self._project_overrides):
 -            _yaml.node_validate(overrides, ['artifacts', 'options', 'strict'])
 +            _yaml.node_validate(overrides, ['artifacts', 'options', 'strict', 'default-mirror'])
          profile_end(Topics.LOAD_CONTEXT, 'load')

buildstream/_frontend/app.py

@@ -202,7 +202,8 @@ class App():
          # Load the Project
+         #
          try:
 -            self.project = Project(directory, self.context, cli_options=self._main_options['option'])
 +            self.project = Project(directory, self.context, cli_options=self._main_options['option'],
 +                                   default_mirror=self._main_options.get('default_mirror'))
          except LoadError as e:
              # Let's automatically start a `bst init` session in this case
@@ -270,6 +271,10 @@ class App():
              # Exit with the error
              self._error_exit(e)
 +        except RecursionError:
 +            click.echo("RecursionError: Depency depth is too large. Maximum recursion depth exceeded.",
 +                       err=True)
 +            sys.exit(-1)
          else:
              # No exceptions occurred, print session time and summary

buildstream/_frontend/cli.py

@@ -217,6 +217,8 @@ def print_version(ctx, param, value):
                help="Elements must be rebuilt when their dependencies have changed")
  @click.option('--option', '-o', type=click.Tuple([str, str]), multiple=True, metavar='OPTION VALUE',
                help="Specify a project option")
 +@click.option('--default-mirror', default=None,
 +              help="The mirror to fetch from first, before attempting other mirrors")
  @click.pass_context
  def cli(context, **kwargs):
      """Build and manipulate BuildStream projects

buildstream/_loader/loader.py

@@ -513,7 +513,7 @@ class Loader():
                  if self._fetch_subprojects:
                      if ticker:
                          ticker(filename, 'Fetching subproject from {} source'.format(source.get_kind()))
 -                    source.fetch()
 +                    source._fetch()
                  else:
                      detail = "Try fetching the project with `bst fetch {}`".format(filename)
                      raise LoadError(LoadErrorReason.SUBPROJECT_FETCH_NEEDED,

buildstream/_project.py

@@ -19,7 +19,7 @@
  import os
  import multiprocessing  # for cpu_count()
 -from collections import Mapping
 +from collections import Mapping, OrderedDict
  from pluginbase import PluginBase
  from . import utils
  from . import _cachekey
@@ -35,9 +35,6 @@ from ._projectrefs import ProjectRefs, ProjectRefStorage
  from ._versions import BST_FORMAT_VERSION
 -# The separator we use for user specified aliases
 -_ALIAS_SEPARATOR = ':'
+-
  # Project Configuration file
  _PROJECT_CONF_FILE = 'project.conf'
@@ -70,7 +67,7 @@ class HostMount():
+ #
  class Project():
 -    def __init__(self, directory, context, *, junction=None, cli_options=None):
 +    def __init__(self, directory, context, *, junction=None, cli_options=None, default_mirror=None):
          # The project name
          self.name = None
@@ -94,6 +91,8 @@ class Project():
          self.base_env_nocache = None             # The base nocache mask (list) for the environment
          self.element_overrides = {}              # Element specific configurations
          self.source_overrides = {}               # Source specific configurations
 +        self.mirrors = OrderedDict()             # contains dicts of alias-mappings to URIs.
 +        self.default_mirror = default_mirror     # The name of the preferred mirror.
+         #
          # Private Members
@@ -133,8 +132,8 @@ class Project():
      # fully qualified urls based on the shorthand which is allowed
      # to be specified in the YAML
      def translate_url(self, url):
 -        if url and _ALIAS_SEPARATOR in url:
 -            url_alias, url_body = url.split(_ALIAS_SEPARATOR, 1)
 +        if url and utils._ALIAS_SEPARATOR in url:
 +            url_alias, url_body = url.split(utils._ALIAS_SEPARATOR, 1)
              alias_url = self._aliases.get(url_alias)
              if alias_url:
                  url = alias_url + url_body
@@ -202,6 +201,36 @@ class Project():
          self._assert_plugin_format(source, version)
          return source
 +    # get_alias_uri()
 +    #
 +    # Returns the URI for a given alias, if it exists
 +    #
 +    # Args:
 +    #    alias (str): The alias.
 +    #
 +    # Returns:
 +    #    str: The URI for the given alias; or None: if there is no URI for
 +    #         that alias.
 +    def get_alias_uri(self, alias):
 +        return self._aliases.get(alias)
++
 +    # get_alias_uris()
 +    #
 +    # Returns a list of every URI to replace an alias with
 +    def get_alias_uris(self, alias):
 +        if not alias or alias not in self._aliases:
 +            return [None]
++
 +        mirror_list = []
 +        for key, alias_mapping in self.mirrors.items():
 +            if alias in alias_mapping:
 +                if key == self.default_mirror:
 +                    mirror_list = alias_mapping[alias] + mirror_list
 +                else:
 +                    mirror_list += alias_mapping[alias]
 +        mirror_list.append(self._aliases[alias])
 +        return mirror_list
++
      # _load():
+     #
      # Loads the project configuration file in the project directory.
@@ -249,7 +278,7 @@ class Project():
              'aliases', 'name',
              'artifacts', 'options',
              'fail-on-overlap', 'shell',
 -            'ref-storage', 'sandbox'
 +            'ref-storage', 'sandbox', 'mirrors',
          ])
          # The project name, element path and option declarations
@@ -290,6 +319,10 @@ class Project():
+         #
          self.options.process_node(config)
 +        # Override default_mirror if not set by command-line
 +        if not self.default_mirror:
 +            self.default_mirror = _yaml.node_get(overrides, str, 'default-mirror', default_value=None)
++
+         #
          # Now all YAML composition is done, from here on we just load
          # the values from our loaded configuration dictionary.
@@ -414,6 +447,21 @@ class Project():
              self._shell_host_files.append(mount)
 +        mirrors = _yaml.node_get(config, list, 'mirrors', default_value=[])
 +        for mirror in mirrors:
 +            allowed_mirror_fields = [
 +                'name', 'aliases'
 +            ]
 +            _yaml.node_validate(mirror, allowed_mirror_fields)
 +            mirror_name = _yaml.node_get(mirror, str, 'name')
 +            alias_mappings = {}
 +            for alias_mapping, uris in _yaml.node_items(mirror['aliases']):
 +                assert isinstance(uris, list)
 +                alias_mappings[alias_mapping] = list(uris)
 +            self.mirrors[mirror_name] = alias_mappings
 +            if not self.default_mirror:
 +                self.default_mirror = mirror_name
++
      # _assert_plugin_format()
+     #
      # Helper to raise a PluginError if the loaded plugin is of a lesser version then

buildstream/_stream.py

@@ -476,7 +476,7 @@ class Stream():
          # Check for workspace config
          workspace = workspaces.get_workspace(target._get_full_name())
 -        if workspace:
 +        if workspace and not force:
              raise StreamError("Workspace '{}' is already defined at: {}"
                                .format(target.name, workspace.path))
@@ -495,6 +495,10 @@ class Stream():
                                "fetch the latest version of the " +
                                "source.")
 +        if workspace:
 +            workspaces.delete_workspace(target._get_full_name())
 +            workspaces.save_config()
 +            shutil.rmtree(directory)
          try:
              os.makedirs(directory, exist_ok=True)
          except OSError as e:

buildstream/plugins/sources/bzr.py

@@ -102,7 +102,7 @@ class BzrSource(Source):
      def track(self):
          with self.timed_activity("Tracking {}".format(self.url),
                                   silent_nested=True):
 -            self._ensure_mirror()
 +            self._ensure_mirror(skip_ref_check=True)
              ret, out = self.check_output([self.host_bzr, "version-info",
                                            "--custom", "--template={revno}",
                                            self._get_branch_dir()],
@@ -214,7 +214,7 @@ class BzrSource(Source):
              yield repodir
              self._atomic_replace_mirrordir(repodir)
 -    def _ensure_mirror(self):
 +    def _ensure_mirror(self, skip_ref_check=False):
          with self._atomic_repodir() as repodir:
              # Initialize repo if no metadata
              bzr_metadata_dir = os.path.join(repodir, ".bzr")
@@ -223,18 +223,21 @@ class BzrSource(Source):
                            fail="Failed to initialize bzr repository")
              branch_dir = os.path.join(repodir, self.tracking)
 +            branch_url = self.url + "/" + self.tracking
              if not os.path.exists(branch_dir):
                  # `bzr branch` the branch if it doesn't exist
                  # to get the upstream code
 -                branch_url = self.url + "/" + self.tracking
                  self.call([self.host_bzr, "branch", branch_url, branch_dir],
                            fail="Failed to branch from {} to {}".format(branch_url, branch_dir))
              else:
                  # `bzr pull` the branch if it does exist
                  # to get any changes to the upstream code
 -                self.call([self.host_bzr, "pull", "--directory={}".format(branch_dir)],
 +                self.call([self.host_bzr, "pull", "--directory={}".format(branch_dir), branch_url],
                            fail="Failed to pull new changes for {}".format(branch_dir))
 +        if not skip_ref_check and not self._check_ref():
 +            raise SourceError("Failed to ensure ref '{}' was mirrored".format(self.ref),
 +                              reason="ref-not-mirrored")
  def setup():

buildstream/plugins/sources/git.py

@@ -78,7 +78,7 @@ from io import StringIO
  from configparser import RawConfigParser
 -from buildstream import Source, SourceError, Consistency
 +from buildstream import Source, SourceError, Consistency, SourceFetcher
  from buildstream import utils
  GIT_MODULES = '.gitmodules'
@@ -88,18 +88,22 @@ GIT_MODULES = '.gitmodules'
  # for the primary git source and also for each submodule it
  # might have at a given time
+ #
 -class GitMirror():
 +class GitMirror(SourceFetcher):
      def __init__(self, source, path, url, ref):
 +        super().__init__()
          self.source = source
          self.path = path
 -        self.url = source.translate_url(url)
 +        self.url = url
          self.ref = ref
 -        self.mirror = os.path.join(source.get_mirror_directory(), utils.url_directory_name(self.url))
 +        self.mirror = os.path.join(source.get_mirror_directory(), utils.url_directory_name(url))
++
 +        alias, _ = self.source.split_aliased_url(self.url)
 +        self.set_alias(alias)
      # Ensures that the mirror exists
 -    def ensure(self):
 +    def ensure(self, alias_override=None):
          # Unfortunately, git does not know how to only clone just a specific ref,
          # so we have to download all of those gigs even if we only need a couple
@@ -112,20 +116,49 @@ class GitMirror():
              # system configured tmpdir is not on the same partition.
+             #
              with self.source.tempdir() as tmpdir:
 -                self.source.call([self.source.host_git, 'clone', '--mirror', '-n', self.url, tmpdir],
 -                                 fail="Failed to clone git repository {}".format(self.url))
 +                url = self.source.translate_url(self.url, alias_override)
 +                self.source.call([self.source.host_git, 'clone', '--mirror', '-n', url, tmpdir],
 +                                 fail="Failed to clone git repository {}".format(url))
                  try:
                      shutil.move(tmpdir, self.mirror)
                  except (shutil.Error, OSError) as e:
                      raise SourceError("{}: Failed to move cloned git repository {} from '{}' to '{}'"
 -                                      .format(self.source, self.url, tmpdir, self.mirror)) from e
 +                                      .format(self.source, url, tmpdir, self.mirror)) from e
++
 +    def _fetch(self, alias_override=None):
 +        url = self.source.translate_url(self.url, alias_override)
++
 +        if alias_override:
 +            remote_name = utils.url_directory_name(alias_override)
 +            _, remotes = self.source.check_output(
 +                [self.source.host_git, 'remote'],
 +                fail="Failed to retrieve list of remotes in {}".format(self.mirror),
 +                cwd=self.mirror
 +            )
 +            if remote_name not in remotes:
 +                self.source.call(
 +                    [self.source.host_git, 'remote', 'add', remote_name, url],
 +                    fail="Failed to add remote {} with url {}".format(remote_name, url),
 +                    cwd=self.mirror
 +                )
 +        else:
 +            remote_name = "origin"
 -    def fetch(self):
 -        self.source.call([self.source.host_git, 'fetch', 'origin', '--prune'],
 -                         fail="Failed to fetch from remote git repository: {}".format(self.url),
 +        self.source.call([self.source.host_git, 'fetch', remote_name, '--prune'],
 +                         fail="Failed to fetch from remote git repository: {}".format(url),
                           cwd=self.mirror)
 +    def fetch(self, alias_override=None):
 +        self.ensure(alias_override)
 +        if not self.has_ref():
 +            self._fetch(alias_override)
 +        self.assert_ref()
++
 +    def get_alias(self):
 +        alias, _ = self.source.split_aliased_url(self.url)
 +        return alias
++
      def has_ref(self):
          if not self.ref:
              return False
@@ -168,12 +201,13 @@ class GitMirror():
      def init_workspace(self, directory):
          fullpath = os.path.join(directory, self.path)
 +        url = self.source.translate_url(self.url)
          self.source.call([self.source.host_git, 'clone', '--no-checkout', self.mirror, fullpath],
                           fail="Failed to clone git mirror {} in directory: {}".format(self.mirror, fullpath))
 -        self.source.call([self.source.host_git, 'remote', 'set-url', 'origin', self.url],
 -                         fail='Failed to add remote origin "{}"'.format(self.url),
 +        self.source.call([self.source.host_git, 'remote', 'set-url', 'origin', url],
 +                         fail='Failed to add remote origin "{}"'.format(url),
                           cwd=fullpath)
          self.source.call([self.source.host_git, 'checkout', '--force', self.ref],
@@ -273,6 +307,10 @@ class GitSource(Source):
                  checkout = self.node_get_member(submodule, bool, 'checkout')
                  self.submodule_checkout_overrides[path] = checkout
 +        # Set the alias, because it's not implicitly set by a translate_url call.
 +        alias, _ = self.split_aliased_url(self.original_url)
 +        self.set_alias(alias)
++
      def preflight(self):
          # Check if git is installed, get the binary at the same time
          self.host_git = utils.get_host_tool('git')
@@ -324,31 +362,13 @@ class GitSource(Source):
                                   .format(self.tracking, self.mirror.url),
                                   silent_nested=True):
              self.mirror.ensure()
 -            self.mirror.fetch()
 +            self.mirror._fetch()
              # Update self.mirror.ref and node.ref from the self.tracking branch
              ret = self.mirror.latest_commit(self.tracking)
          return ret
 -    def fetch(self):
+-
 -        with self.timed_activity("Fetching {}".format(self.mirror.url), silent_nested=True):
+-
 -            # Here we are only interested in ensuring that our mirror contains
 -            # the self.mirror.ref commit.
 -            self.mirror.ensure()
 -            if not self.mirror.has_ref():
 -                self.mirror.fetch()
+-
 -            self.mirror.assert_ref()
+-
 -            # Here after performing any fetches, we need to also ensure that
 -            # we've cached the desired refs in our mirrors of submodules.
 -            #
 -            self.refresh_submodules()
 -            self.fetch_submodules()
+-
      def init_workspace(self, directory):
          # XXX: may wish to refactor this as some code dupe with stage()
          self.refresh_submodules()
@@ -380,6 +400,10 @@ class GitSource(Source):
                  if checkout:
                      mirror.stage(directory)
 +    def get_source_fetchers(self):
 +        self.refresh_submodules()
 +        return [self.mirror] + self.submodules
++
      ###########################################################
      #                     Local Functions                     #
      ###########################################################
@@ -401,6 +425,7 @@ class GitSource(Source):
      # Assumes that we have our mirror and we have the ref which we point to
+     #
      def refresh_submodules(self):
 +        self.mirror.ensure()
          submodules = []
          # XXX Here we should issue a warning if either:
@@ -422,19 +447,6 @@ class GitSource(Source):
          self.submodules = submodules
 -    # Ensures that we have mirrored git repositories for all
 -    # the submodules existing at the given commit of the main git source.
 -    #
 -    # Also ensure that these mirrors have the required commits
 -    # referred to at the given commit of the main git source.
 -    #
 -    def fetch_submodules(self):
 -        for mirror in self.submodules:
 -            mirror.ensure()
 -            if not mirror.has_ref():
 -                mirror.fetch()
 -                mirror.assert_ref()
+-
  # Plugin entry point
  def setup():

buildstream/source.py

@@ -65,6 +65,33 @@ these methods are mandatory to implement.
    **Optional**: If left unimplemented, this will default to calling
    :func:`Source.stage() <buildstream.source.Source.stage>`
++
 +* :func:`Source.get_source_fetchers() <buildstream.source.Source.get_source_fetchers>`
++
 +  Get the objects that are used for fetching.
++
 +  **Optional**: This only needs to be implemented for sources that need to
 +  download from multiple URLs while fetching (e.g. a git repo and its
 +  submodules). For details on how to define a SourceFetcher, see
 +  :ref:`SourceFetcher <core_source_fetcher>`.
++
++
 +.. _core_source_fetcher:
++
 +SourceFetcher - Object for fetching individual URLs
 +===================================================
++
++
 +Abstract Methods
 +----------------
 +SourceFetchers expose the following abstract methods. Unless explicitly
 +mentioned, these methods are mandatory to implement.
++
 +* :func:`SourceFetcher.fetch() <buildstream.source.SourceFetcher.fetch>`
++
 +  Fetches the URL associated with this SourceFetcher, optionally taking an
 +  alias override.
++
  """
  import os
@@ -113,6 +140,63 @@ class SourceError(BstError):
          super().__init__(message, detail=detail, domain=ErrorDomain.SOURCE, reason=reason)
 +class SourceFetcher():
 +    """SourceFetcher()
++
 +    This interface exists so that a source that downloads from multiple
 +    places (e.g. a git source with submodules) has a consistent interface for
 +    fetching and substituting aliases.
++
 +    *Since: 1.4*
 +    """
 +    def __init__(self):
 +        self.__alias = None
++
 +    #############################################################
 +    #                      Abstract Methods                     #
 +    #############################################################
 +    def fetch(self, alias_override=None):
 +        """Fetch remote sources and mirror them locally, ensuring at least
 +        that the specific reference is cached locally.
++
 +        Args:
 +           alias_override (str): The alias to use instead of the default one
 +               defined by the :ref:`aliases <project_source_aliases>` field
 +               in the project's config.
++
 +        Raises:
 +           :class:`.SourceError`
++
 +        Implementors should raise :class:`.SourceError` if the there is some
 +        network error or if the source reference could not be matched.
 +        """
 +        raise ImplError("Source fetcher '{}' does not implement fetch()".format(type(self)))
++
 +    #############################################################
 +    #                       Public Methods                      #
 +    #############################################################
 +    def set_alias(self, alias):
 +        """Sets the alias used by this fetcher
++
 +        This must be called during the fetcher's initialization, so that the
 +        correct alias_override can be provided to
 +        :func:`fetch()<buildstream.source.SourceFetcher.fetch>`.
++
 +        Args:
 +           alias (str): The alias component of the SourceFetcher's URL, as
 +                        described by :ref:`Source aliases<project_source_aliases>`.
 +        """
 +        self.__alias = alias
++
 +    #############################################################
 +    #            Private Methods used in BuildStream            #
 +    #############################################################
++
 +    # Returns the alias used by this fetcher
 +    def _get_alias(self):
 +        return self.__alias
++
++
  class Source(Plugin):
      """Source()
@@ -124,7 +208,7 @@ class Source(Plugin):
      __defaults = {}          # The defaults from the project
      __defaults_set = False   # Flag, in case there are not defaults at all
 -    def __init__(self, context, project, meta):
 +    def __init__(self, context, project, meta, *, alias_override=None):
          provenance = _yaml.node_get_provenance(meta.config)
          super().__init__("{}-{}".format(meta.element_name, meta.element_index),
                           context, project, provenance, "source")
@@ -134,6 +218,11 @@ class Source(Plugin):
          self.__element_kind = meta.element_kind         # The kind of the element owning this source
          self.__directory = meta.directory               # Staging relative directory
          self.__consistency = Consistency.INCONSISTENT   # Cached consistency state
 +        self.__alias_override = alias_override          # Tuple of alias and its override to use instead
 +        self.__expected_alias = None                    # A hacky way to store the first alias used
++
 +        # FIXME: Reconstruct a MetaSource from a Source instead of storing it.
 +        self.__meta = meta                              # MetaSource stored so we can copy this source later.
          # Collect the composited element configuration and
          # ask the element to configure itself.
@@ -283,6 +372,34 @@ class Source(Plugin):
          """
          self.stage(directory)
 +    def set_alias(self, alias):
 +        """Set the alias of the source's URL
++
 +        A source's alias should be set using set_alias if it does not call
 +        :func:`~buildstream.source.Source.translate_url` during
 +        :func:`~buildstream.plugin.Plugin.configure`.
++
 +        Args:
 +           (str) alias: An URL alias
 +        """
 +        self.__expected_alias = alias
++
 +    def get_source_fetchers(self):
 +        """Get the objects that are used for fetching
++
 +        If this source doesn't download from multiple URLs,
 +        returning None and falling back on the default behaviour
 +        is recommended.
++
 +        Returns:
 +           list: A list of SourceFetchers. If SourceFetchers are not supported,
 +                 this will be an empty list.
++
 +        *Since: 1.4*
 +        """
++
 +        return []
++
      #############################################################
      #                       Public Methods                      #
      #############################################################
@@ -299,18 +416,42 @@ class Source(Plugin):
          os.makedirs(directory, exist_ok=True)
          return directory
 -    def translate_url(self, url):
 +    def translate_url(self, url, alias_override=None):
          """Translates the given url which may be specified with an alias
          into a fully qualified url.
          Args:
             url (str): A url, which may be using an alias
 +           alias_override (str): Optionally, an URI to override the alias with.
          Returns:
             str: The fully qualified url, with aliases resolved
          """
 -        project = self._get_project()
 -        return project.translate_url(url)
 +        # Alias overriding can happen explicitly (by command-line) or
 +        # implicitly (the Source being constructed with an __alias_override).
 +        if alias_override or self.__alias_override:
 +            url_alias, url_body = self.split_aliased_url(url)
 +            if url_alias:
 +                if alias_override:
 +                    url = alias_override + url_body
 +                else:
 +                    # Implicit alias overrides may only be done for one
 +                    # specific alias, so that sources that fetch from multiple
 +                    # URLs and use different aliases default to only overriding
 +                    # one alias, rather than getting confused.
 +                    override_alias = self.__alias_override[0]
 +                    override_url = self.__alias_override[1]
 +                    if url_alias == override_alias:
 +                        url = override_url + url_body
 +            return url
 +        else:
 +            # Sneakily store the alias if it hasn't already been stored
 +            if not self.__expected_alias and url and utils._ALIAS_SEPARATOR in url:
 +                url_alias, _ = url.split(utils._ALIAS_SEPARATOR, 1)
 +                self.__expected_alias = url_alias
++
 +            project = self._get_project()
 +            return project.translate_url(url)
      def get_project_directory(self):
          """Fetch the project base directory
@@ -340,6 +481,19 @@ class Source(Plugin):
          with utils._tempdir(dir=mirrordir) as tempdir:
              yield tempdir
 +    def split_aliased_url(self, url):
 +        """Helper for splitting an URL into its alias and body
++
 +        Returns:
 +           (str): The alias component of the URL, or None if there is no alias
 +           (str): The body of the URL, or None if there is no alias
 +        """
 +        if utils._ALIAS_SEPARATOR in url:
 +            alias, body = url.split(utils._ALIAS_SEPARATOR, 1)
 +            return alias, body
 +        else:
 +            return None, None
++
      #############################################################
      #            Private Methods used in BuildStream            #
      #############################################################
@@ -374,7 +528,45 @@ class Source(Plugin):
      # Wrapper function around plugin provided fetch method
+     #
      def _fetch(self):
 -        self.fetch()
 +        project = self._get_project()
 +        source_fetchers = self.get_source_fetchers()
 +        if source_fetchers:
 +            for fetcher in source_fetchers:
 +                alias = fetcher._get_alias()
 +                success = False
 +                for uri in project.get_alias_uris(alias):
 +                    try:
 +                        fetcher.fetch(uri)
 +                    # FIXME: We need a way to distinguish between serious errors and
 +                    # the host being unreachable
 +                    except BstError as e:
 +                        last_error = e
 +                        continue
 +                    success = True
 +                    break
 +                if not success:
 +                    raise last_error
 +        else:
 +            alias = self._get_alias()
 +            if not project.mirrors or not alias:
 +                self.fetch()
 +                return
++
 +            context = self._get_context()
 +            source_kind = type(self)
 +            for uri in project.get_alias_uris(alias):
 +                new_source = source_kind(context, project, self.__meta,
 +                                         alias_override=(alias, uri))
 +                new_source._preflight()
 +                try:
 +                    new_source.fetch()
 +                # FIXME: We need a way to distinguish between serious errors and
 +                # the host being unreachable
 +                except BstError as e:
 +                    last_error = e
 +                    continue
 +                return
 +            raise last_error
      # Wrapper for stage() api which gives the source
      # plugin a fully constructed path considering the
@@ -581,7 +773,7 @@ class Source(Plugin):
      # Wrapper for track()
+     #
      def _track(self):
 -        new_ref = self.track()
 +        new_ref = self.__do_track()
          current_ref = self.get_ref()
          if new_ref is None:
@@ -593,10 +785,48 @@ class Source(Plugin):
          return new_ref
 +    # Returns the alias if it's defined in the project
 +    def _get_alias(self):
 +        alias = self.__expected_alias
 +        project = self._get_project()
 +        if project.get_alias_uri(alias):
 +            # The alias must already be defined in the project's aliases
 +            # otherwise http://foo gets treated like it contains an alias
 +            return alias
 +        else:
 +            return None
++
      #############################################################
      #                   Local Private Methods                   #
      #############################################################
 +    # Tries to call track for every mirror, stopping once it succeeds
 +    def __do_track(self):
 +        project = self._get_project()
 +        # If there are no mirrors, or no aliases to replace, there's nothing to do here.
 +        alias = self._get_alias()
 +        if not project.mirrors or not alias:
 +            return self.track()
++
 +        context = self._get_context()
 +        source_kind = type(self)
++
 +        # NOTE: We are assuming here that tracking only requires substituting the
 +        #       first alias used
 +        for uri in reversed(project.get_alias_uris(alias)):
 +            new_source = source_kind(context, project, self.__meta,
 +                                     alias_override=(alias, uri))
 +            new_source._preflight()
 +            try:
 +                ref = new_source.track()
 +            # FIXME: We need a way to distinguish between serious errors and
 +            # the host being unreachable
 +            except BstError as e:
 +                last_error = e
 +                continue
 +            return ref
 +        raise last_error
++
      # Ensures a fully constructed path and returns it
      def __ensure_directory(self, directory):

buildstream/utils.py

@@ -42,6 +42,10 @@ from . import _signals
  from ._exceptions import BstError, ErrorDomain
 +# The separator we use for user specified aliases
 +_ALIAS_SEPARATOR = ':'
++
++
  class UtilError(BstError):
      """Raised by utility functions when system calls fail.

doc/source/examples/git-mirror.rst

++
++
 +Creating and using a git mirror
 +'''''''''''''''''''''''''''''''
 +This is an example of how to create a git mirror using git's
 +`git-http-backend <https://git-scm.com/docs/git-http-backend>`_ and
 +`lighttpd <https://redmine.lighttpd.net/projects/1/wiki/TutorialConfiguration>`_.
++
++
 +Prerequisites
 +=============
 +You will need git installed, and git-http-backend must be present. It is assumed
 +that the git-http-backend binary exists at `/usr/lib/git-core/git-http-backend`.
++
 +You will need `lighttpd` installed, and at the bare minimum has the modules
 +`mod_alias`, `mod_cgi`, and `mod_setenv`.
++
 +I will be using gnome-modulesets as an example, which can be cloned from
 +`http://gnome7.codethink.co.uk/gnome-modulesets.git`.
++
++
 +Starting a git http server
 +==========================
++
++
 +1. Set up a directory containing mirrors
 +----------------------------------------
 +Choose a suitable directory to hold your mirrors, e.g. `/var/www/git`.
++
 +Place the git repositories you want to use as mirrors in the mirror dir, e.g.
 +``git clone --mirror http://git.gnome.org/browse/yelp-xsl /var/www/git/yelp-xsl.git``.
++
++
 +2. Configure lighttpd
 +---------------------
 +Write out a lighttpd.conf as follows:
++
 +::
++
 +   server.document-root = "/var/www/git/"
 +   server.port = 3000
 +   server.modules = (
 +        "mod_alias",
 +        "mod_cgi",
 +        "mod_setenv",
 +   )
++
 +   alias.url += ( "/git" => "/usr/lib/git-core/git-http-backend" )
 +   $HTTP["url"] =~ "^/git" {
 +        cgi.assign = ("" => "")
 +        setenv.add-environment = (
 +                "GIT_PROJECT_ROOT" => "/var/www/git",
 +                "GIT_HTTP_EXPORT_ALL" => ""
 +        )
 +   }
++
 +.. note::
++
 +   If you have your mirrors in another directory, replace /var/www/git/ with that directory.
++
++
 +3. Start lighttpd
 +-----------------
 +lighttpd can be invoked with the command-line ``lighttpd -D -f lighttpd.conf``.
++
++
 +4. Test that you can fetch from it
 +----------------------------------
 +We can then clone the mirrored repo using git via http with
 +``git clone http://127.0.0.1:3000/git/yelp-xsl``.
++
 +.. note::
++
 +   If you have set server.port to something other than the default, you will
 +   need to replace the '3000' in the command-line.
++
++
 +5. Configure the project to use the mirror
 +------------------------------------------
 +To add this local http server as a mirror, add the following to the project.conf:
++
 +.. code:: yaml
++
 +   mirrors:
 +   - name: local-mirror
 +     aliases:
 +       git_gnome_org:
 +       - http://127.0.0.1:3000/git/
++
++
 +6. Test that the mirror works
 +-----------------------------
 +We can make buildstream use the mirror by setting the alias to an invalid URL, e.g.
++
 +.. code:: yaml
++
 +   aliases:
 +     git_gnome_org: https://www.example.com/invalid/url/
++
 +Now, if you build an element that uses the source you placed in the mirror
 +(e.g. ``bst build core-deps/yelp-xsl.bst``), you will see that it uses your mirror.
++
++
 +.. _lighttpd_git_tar_conf:
++
 +Bonus: lighttpd conf for git and tar
 +====================================
 +For those who have also used the :ref:`tar-mirror tutorial <using_tar_mirror>`,
 +a combined lighttpd.conf is below:
++
 +::
++
 +   server.document-root = "/var/www/"
 +   server.port = 3000
 +   server.modules = (
 +           "mod_alias",
 +           "mod_cgi",
 +           "mod_setenv",
 +   )
++
 +   alias.url += ( "/git" => "/usr/lib/git-core/git-http-backend" )
 +   $HTTP["url"] =~ "^/git" {
 +           cgi.assign = ("" => "")
 +           setenv.add-environment = (
 +                   "GIT_PROJECT_ROOT" => "/var/www/git",
 +                   "GIT_HTTP_EXPORT_ALL" => ""
 +           )
 +   } else $HTTP["url"] =~ "^/tar" {
 +           dir-listing.activate = "enable"
 +   }
++
++
 +Further reading
 +===============
 +If this mirror isn't being used exclusively in a secure network, it is strongly
 +recommended you `use SSL <https://redmine.lighttpd.net/projects/1/wiki/HowToSimpleSSL>`_.
++
 +This is the bare minimum required to set up a git mirror. A large, public project
 +would prefer to set it up using the
 +`git protocol <https://git-scm.com/book/en/v1/Git-on-the-Server-Git-Daemon>`_,
 +and a security-conscious project would be configured to use
 +`git over SSH <https://git-scm.com/book/en/v1/Git-on-the-Server-Getting-Git-on-a-Server#Small-Setups>`_.
++
 +Lighttpd is documented on `its wiki <https://redmine.lighttpd.net/projects/lighttpd/wiki>`_.

doc/source/examples/tar-mirror.rst

++
++
 +.. _using_tar_mirror:
++
 +Creating and using a tar mirror
 +'''''''''''''''''''''''''''''''
 +This is an example of how to create a tar mirror using
 +`lighttpd <https://redmine.lighttpd.net/projects/1/wiki/TutorialConfiguration>`_.
++
++
 +Prerequisites
 +=============
 +You will need `lighttpd` installed.
++
++
 +I will be using gnome-modulesets as an example, which can be cloned from
 +`http://gnome7.codethink.co.uk/gnome-modulesets.git`.
++
++
 +Starting a tar server
 +=====================
++
++
 +1. Set up a directory containing mirrors
 +----------------------------------------
 +Choose a suitable directory to hold your mirrored tar files, e.g. `/var/www/tar`.
++
 +Place the tar files you want to use as mirrors in your mirror dir, e.g.
++
 +.. code::
++
 +   mkdir -p /var/www/tar/gettext
 +   wget -O /var/www/tar/gettext/gettext-0.19.8.1.tar.xz https://ftp.gnu.org/gnu/gettext/gettext-0.19.8.1.tar.xz
++
++
 +2. Configure lighttpd
 +---------------------
 +Write out a lighttpd.conf as follows:
++
 +::
++
 +   server.document-root = "/var/www/tar/"
 +   server.port = 3000
++
 +   dir-listing.activate = "enable"
++
 +.. note::
++
 +   If you have your mirrors in another directory, replace /var/www/tar/ with that directory.
++
 +.. note::
++
 +   An example lighttpd.conf that works for both git and tar services is available
 +   :ref:`here <lighttpd_git_tar_conf>`
++
++
 +3. Start lighttpd
 +-----------------
 +lighttpd can be invoked with the command-line ``lighttpd -D -f lighttpd.conf``.
++
++
 +4. Test that you can fetch from it
 +----------------------------------
 +We can then download the mirrored file with ``wget 127.0.0.1:3000/tar/gettext/gettext-0.19.8.1.tar.xz``.
++
 +.. note::
++
 +   If you have set server.port to something other than the default, you will need
 +   to replace the '3000' in the command-line.
++
++
 +5. Configure the project to use the mirror
 +------------------------------------------
 +To add this local http server as a mirror, add the following to the project.conf:
++
 +.. code:: yaml
++
 +   mirrors:
 +   - name: local-mirror
 +     aliases:
 +       ftp_gnu_org:
 +       - http://127.0.0.1:3000/tar/
++
++
 +6. Test that the mirror works
 +-----------------------------
 +We can make buildstream use the mirror by setting the alias to an invalid URL, e.g.
++
 +.. code:: yaml
++
 +   aliases:
 +     ftp_gnu_org: https://www.example.com/invalid/url/
++
 +Now, if you build an element that uses the source you placed in the mirror
 +(e.g. ``bst build core-deps/gettext.bst``), you will see that it uses your mirror.
++
++
 +Further reading
 +===============
 +If this mirror isn't being used exclusively in a secure network, it is strongly
 +recommended you `use SSL <https://redmine.lighttpd.net/projects/1/wiki/HowToSimpleSSL>`_.
++
 +Lighttpd is documented on `its wiki <https://redmine.lighttpd.net/projects/lighttpd/wiki>`_.

doc/source/format_project.rst

@@ -198,6 +198,39 @@ You can also specify a list of caches here; earlier entries in the list
  will have higher priority than later ones.
 +.. _project_essentials_mirrors:
++
 +Mirrors
 +~~~~~~~
 +A list of mirrors can be defined that couple a location to a mapping of aliases to a
 +list of URIs, e.g.
++
 +.. code:: yaml
++
 +  mirrors:
 +  - name: middle-earth
 +    aliases:
 +      foo:
 +      - http://www.middle-earth.com/foo/1
 +      - http://www.middle-earth.com/foo/2
 +      bar:
 +      - http://www.middle-earth.com/bar/1
 +      - http://www.middle-earth.com/bar/2
 +  - name: oz
 +    aliases:
 +      foo:
 +      - http://www.oz.com/foo
 +      bar:
 +      - http://www.oz.com/bar
++
 +The order that the mirrors (and the URIs therein) are consulted is in the order
 +they are defined when fetching, and in reverse-order when tracking.
++
 +A default mirror to consult first can be defined via
 +:ref:`user config <config_default_mirror>`, or the command-line argument
 +:ref:`--default-mirror <invoking_bst>`.
++
++
  .. _project_plugins:
  External plugins

doc/source/install_linux_distro.rst

@@ -6,10 +6,8 @@ Installing BuildStream on a Linux distro
  BuildStream requires the following base system requirements:
  * python3 >= 3.5
 -* libostree >= v2017.8 with introspection data
  * bubblewrap >= 0.1.2
  * fuse2
 -* PyGObject introspection bindings
  BuildStream also depends on the host tools for the :mod:`Source <buildstream.source>` plugins.
  Refer to the respective :ref:`source plugin <plugins_sources>` documentation for host tool
@@ -20,6 +18,7 @@ The default plugins with extra host dependencies are:
  * bzr
  * deb
  * git
 +* ostree
  * patch
  * tar
@@ -52,18 +51,24 @@ Arch Linux
  Install the dependencies with::
    sudo pacman -S \
 -      python fuse2 ostree bubblewrap python-gobject \
 +      python fuse2 bubblewrap \
        python-pip git
  For the default plugins::
    sudo pacman -S \
 -      lzip git bzr patch python-arpy
 +      bzr git lzip ostree patch python-arpy python-gobject
  Debian
  ++++++
 +Install the dependencies with::
++
 +  sudo apt-get install \
 +      python3 fuse bubblewrap \
 +      python3-pip python3-dev git
 +For the default plugins:
  Stretch
  ^^^^^^^
@@ -78,32 +83,20 @@ And then running::
    sudo apt-get update
 -At this point you should be able to get the system requirements with::
 +At this point you should be able to get the system requirements for the default plugins with::
    sudo apt-get install \
 -      python3 fuse ostree gir1.2-ostree-1.0 bubblewrap python3-gi \
 -      python3-pip python3-dev git
 +      bzr git lzip patch python3-arpy python3-gi
    sudo apt-get install -t stretch-backports \
        gir1.2-ostree-1.0 ostree
 -For the default plugins::
+-
 -  sudo apt-get install \
 -      lzip git bzr patch python3-arpy
+-
  Buster or Sid
  ^^^^^^^^^^^^^
  For debian unstable or testing, only the following line should be enough
 -to get the base system requirements installed::
+-
 -  sudo apt-get install \
 -      python3 fuse ostree gir1.2-ostree-1.0 bubblewrap python3-gi \
 -      python3-pip python3-dev git
+-
 -For the default plugins::
 +to get the system requirements for the default plugins installed::
    sudo apt-get install \
 -      lzip git bzr patch python3-arpy
 +      lzip gir1.2-ostree-1.0 git bzr ostree patch python3-arpy python3-gi
  Fedora
@@ -112,13 +105,13 @@ For recent fedora systems, the following line should get you the system
  requirements you need::
    dnf install -y \
 -      python3 fuse ostree bubblewrap python3-gobject \
 +      python3 fuse bubblewrap \
        python3-pip python3-devel git
  For the default plugins::
    dnf install -y \
 -      lzip git bzr patch python3-arpy
 +      bzr git lzip patch ostree python3-arpy python3-gobject
  Installing

doc/source/using_config.rst

@@ -89,6 +89,28 @@ modifying some low level component.
     the ``--strict`` and ``--no-strict`` command line options.
 +.. _config_default_mirror:
++
 +Default Mirror
 +~~~~~~~~~~~~~~
++
 +When using :ref:`mirrors <project_essentials_mirrors>`, a default mirror can
 +be defined to be fetched first.
 +The default mirror is defined by its name, e.g.
++
 +.. code:: yaml
++
 +  projects:
 +    project-name:
 +      default-mirror: oz
++
++
 +.. note::
++
 +   It is possible to override this at invocation time using the
 +   ``--default-mirror`` command-line option.
++
++
  Default configuration
  ---------------------
  The default BuildStream configuration is specified here for reference:

doc/source/using_examples.rst

@@ -10,3 +10,5 @@ maintained and work as expected.
     :maxdepth: 1
     examples/flatpak-autotools
 +   examples/tar-mirror
 +   examples/git-mirror

tests/completions/completions.py

@@ -27,6 +27,7 @@ MAIN_OPTIONS = [
      "--colors ",
      "--config ",
      "--debug ",
 +    "--default-mirror ",
      "--directory ",
      "--error-lines ",
      "--fetchers ",

tests/frontend/mirror.py

 +import os
 +import pytest
++
 +from tests.testutils import cli, create_repo, ALL_REPO_KINDS
++
 +from buildstream import _yaml
++
++
 +# Project directory
 +TOP_DIR = os.path.dirname(os.path.realpath(__file__))
 +DATA_DIR = os.path.join(TOP_DIR, 'project')
++
++
 +def generate_element(output_file):
 +    element = {
 +        'kind': 'import',
 +        'sources': [
 +            {
 +                'kind': 'fetch_source',
 +                "output-text": output_file,
 +                "urls": ["foo:repo1", "bar:repo2"],
 +                "fetch-succeeds": {
 +                    "FOO/repo1": True,
 +                    "BAR/repo2": False,
 +                    "OOF/repo1": False,
 +                    "RAB/repo2": True,
 +                    "OFO/repo1": False,
 +                    "RBA/repo2": False,
 +                    "ooF/repo1": False,
 +                    "raB/repo2": False,
 +                }
 +            }
 +        ]
 +    }
 +    return element
++
++
 +def generate_project():
 +    project = {
 +        'name': 'test',
 +        'element-path': 'elements',
 +        'aliases': {
 +            'foo': 'FOO/',
 +            'bar': 'BAR/',
 +        },
 +        'mirrors': [
 +            {
 +                'name': 'middle-earth',
 +                'aliases': {
 +                    'foo': ['OOF/'],
 +                    'bar': ['RAB/'],
 +                },
 +            },
 +            {
 +                'name': 'arrakis',
 +                'aliases': {
 +                    'foo': ['OFO/'],
 +                    'bar': ['RBA/'],
 +                },
 +            },
 +            {
 +                'name': 'oz',
 +                'aliases': {
 +                    'foo': ['ooF/'],
 +                    'bar': ['raB/'],
 +                }
 +            },
 +        ],
 +        'plugins': [
 +            {
 +                'origin': 'local',
 +                'path': 'sources',
 +                'sources': {
 +                    'fetch_source': 0
 +                }
 +            }
 +        ]
 +    }
 +    return project
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
 +def test_mirror_fetch(cli, tmpdir, datafiles, kind):
 +    bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
 +    dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
 +    upstream_repodir = os.path.join(str(tmpdir), 'upstream')
 +    mirror_repodir = os.path.join(str(tmpdir), 'mirror')
 +    project_dir = os.path.join(str(tmpdir), 'project')
 +    os.makedirs(project_dir)
 +    element_dir = os.path.join(project_dir, 'elements')
++
 +    # Create repo objects of the upstream and mirror
 +    upstream_repo = create_repo(kind, upstream_repodir)
 +    upstream_ref = upstream_repo.create(bin_files_path)
 +    mirror_repo = upstream_repo.copy(mirror_repodir)
 +    mirror_ref = upstream_ref
 +    upstream_ref = upstream_repo.create(dev_files_path)
++
 +    element = {
 +        'kind': 'import',
 +        'sources': [
 +            upstream_repo.source_config(ref=upstream_ref)
 +        ]
 +    }
 +    element_name = 'test.bst'
 +    element_path = os.path.join(element_dir, element_name)
 +    full_repo = element['sources'][0]['url']
 +    upstream_map, repo_name = os.path.split(full_repo)
 +    alias = 'foo-' + kind
 +    aliased_repo = alias + ':' + repo_name
 +    element['sources'][0]['url'] = aliased_repo
 +    full_mirror = mirror_repo.source_config()['url']
 +    mirror_map, _ = os.path.split(full_mirror)
 +    os.makedirs(element_dir)
 +    _yaml.dump(element, element_path)
++
 +    project = {
 +        'name': 'test',
 +        'element-path': 'elements',
 +        'aliases': {
 +            alias: upstream_map + "/"
 +        },
 +        'mirrors': [
 +            {
 +                'name': 'middle-earth',
 +                'aliases': {
 +                    alias: [mirror_map + "/"],
 +                },
 +            },
 +        ]
 +    }
 +    project_file = os.path.join(project_dir, 'project.conf')
 +    _yaml.dump(project, project_file)
++
 +    # No obvious ways of checking that the mirror has been fetched
 +    # But at least we can be sure it succeeds
 +    result = cli.run(project=project_dir, args=['fetch', element_name])
 +    result.assert_success()
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_mirror_fetch_multi(cli, tmpdir, datafiles):
 +    output_file = os.path.join(str(tmpdir), "output.txt")
 +    project_dir = str(tmpdir)
 +    element_dir = os.path.join(project_dir, 'elements')
 +    os.makedirs(element_dir, exist_ok=True)
 +    element_name = "test.bst"
 +    element_path = os.path.join(element_dir, element_name)
 +    element = generate_element(output_file)
 +    _yaml.dump(element, element_path)
++
 +    project_file = os.path.join(project_dir, 'project.conf')
 +    project = generate_project()
 +    _yaml.dump(project, project_file)
++
 +    result = cli.run(project=project_dir, args=['fetch', element_name])
 +    result.assert_success()
 +    with open(output_file) as f:
 +        contents = f.read()
 +        assert "Fetch foo:repo1 succeeded from FOO/repo1" in contents
 +        assert "Fetch bar:repo2 succeeded from RAB/repo2" in contents
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_mirror_fetch_default_cmdline(cli, tmpdir, datafiles):
 +    output_file = os.path.join(str(tmpdir), "output.txt")
 +    project_dir = str(tmpdir)
 +    element_dir = os.path.join(project_dir, 'elements')
 +    os.makedirs(element_dir, exist_ok=True)
 +    element_name = "test.bst"
 +    element_path = os.path.join(element_dir, element_name)
 +    element = generate_element(output_file)
 +    _yaml.dump(element, element_path)
++
 +    project_file = os.path.join(project_dir, 'project.conf')
 +    project = generate_project()
 +    _yaml.dump(project, project_file)
++
 +    result = cli.run(project=project_dir, args=['--default-mirror', 'arrakis', 'fetch', element_name])
 +    result.assert_success()
 +    with open(output_file) as f:
 +        contents = f.read()
 +        print(contents)
 +        # Success if fetching from arrakis' mirror happened before middle-earth's
 +        arrakis_str = "OFO/repo1"
 +        arrakis_pos = contents.find(arrakis_str)
 +        assert arrakis_pos != -1, "'{}' wasn't found".format(arrakis_str)
 +        me_str = "OOF/repo1"
 +        me_pos = contents.find(me_str)
 +        assert me_pos != -1, "'{}' wasn't found".format(me_str)
 +        assert arrakis_pos < me_pos, "'{}' wasn't found before '{}'".format(arrakis_str, me_str)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_mirror_fetch_default_userconfig(cli, tmpdir, datafiles):
 +    output_file = os.path.join(str(tmpdir), "output.txt")
 +    project_dir = str(tmpdir)
 +    element_dir = os.path.join(project_dir, 'elements')
 +    os.makedirs(element_dir, exist_ok=True)
 +    element_name = "test.bst"
 +    element_path = os.path.join(element_dir, element_name)
 +    element = generate_element(output_file)
 +    _yaml.dump(element, element_path)
++
 +    project_file = os.path.join(project_dir, 'project.conf')
 +    project = generate_project()
 +    _yaml.dump(project, project_file)
++
 +    userconfig = {
 +        'projects': {
 +            'test': {
 +                'default-mirror': 'oz'
 +            }
 +        }
 +    }
 +    cli.configure(userconfig)
++
 +    result = cli.run(project=project_dir, args=['fetch', element_name])
 +    result.assert_success()
 +    with open(output_file) as f:
 +        contents = f.read()
 +        print(contents)
 +        # Success if fetching from Oz' mirror happened before middle-earth's
 +        oz_str = "ooF/repo1"
 +        oz_pos = contents.find(oz_str)
 +        assert oz_pos != -1, "'{}' wasn't found".format(oz_str)
 +        me_str = "OOF/repo1"
 +        me_pos = contents.find(me_str)
 +        assert me_pos != -1, "'{}' wasn't found".format(me_str)
 +        assert oz_pos < me_pos, "'{}' wasn't found before '{}'".format(oz_str, me_str)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_mirror_fetch_default_cmdline_overrides_config(cli, tmpdir, datafiles):
 +    output_file = os.path.join(str(tmpdir), "output.txt")
 +    project_dir = str(tmpdir)
 +    element_dir = os.path.join(project_dir, 'elements')
 +    os.makedirs(element_dir, exist_ok=True)
 +    element_name = "test.bst"
 +    element_path = os.path.join(element_dir, element_name)
 +    element = generate_element(output_file)
 +    _yaml.dump(element, element_path)
++
 +    project_file = os.path.join(project_dir, 'project.conf')
 +    project = generate_project()
 +    _yaml.dump(project, project_file)
++
 +    userconfig = {
 +        'projects': {
 +            'test': {
 +                'default-mirror': 'oz'
 +            }
 +        }
 +    }
 +    cli.configure(userconfig)
++
 +    result = cli.run(project=project_dir, args=['--default-mirror', 'arrakis', 'fetch', element_name])
 +    result.assert_success()
 +    with open(output_file) as f:
 +        contents = f.read()
 +        print(contents)
 +        # Success if fetching from arrakis' mirror happened before middle-earth's
 +        arrakis_str = "OFO/repo1"
 +        arrakis_pos = contents.find(arrakis_str)
 +        assert arrakis_pos != -1, "'{}' wasn't found".format(arrakis_str)
 +        me_str = "OOF/repo1"
 +        me_pos = contents.find(me_str)
 +        assert me_pos != -1, "'{}' wasn't found".format(me_str)
 +        assert arrakis_pos < me_pos, "'{}' wasn't found before '{}'".format(arrakis_str, me_str)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
 +def test_mirror_track_upstream_present(cli, tmpdir, datafiles, kind):
 +    bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
 +    dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
 +    upstream_repodir = os.path.join(str(tmpdir), 'upstream')
 +    mirror_repodir = os.path.join(str(tmpdir), 'mirror')
 +    project_dir = os.path.join(str(tmpdir), 'project')
 +    os.makedirs(project_dir)
 +    element_dir = os.path.join(project_dir, 'elements')
++
 +    # Create repo objects of the upstream and mirror
 +    upstream_repo = create_repo(kind, upstream_repodir)
 +    upstream_ref = upstream_repo.create(bin_files_path)
 +    mirror_repo = upstream_repo.copy(mirror_repodir)
 +    mirror_ref = upstream_ref
 +    upstream_ref = upstream_repo.create(dev_files_path)
++
 +    element = {
 +        'kind': 'import',
 +        'sources': [
 +            upstream_repo.source_config(ref=upstream_ref)
 +        ]
 +    }
++
 +    element['sources'][0]
 +    element_name = 'test.bst'
 +    element_path = os.path.join(element_dir, element_name)
 +    full_repo = element['sources'][0]['url']
 +    upstream_map, repo_name = os.path.split(full_repo)
 +    alias = 'foo-' + kind
 +    aliased_repo = alias + ':' + repo_name
 +    element['sources'][0]['url'] = aliased_repo
 +    full_mirror = mirror_repo.source_config()['url']
 +    mirror_map, _ = os.path.split(full_mirror)
 +    os.makedirs(element_dir)
 +    _yaml.dump(element, element_path)
++
 +    project = {
 +        'name': 'test',
 +        'element-path': 'elements',
 +        'aliases': {
 +            alias: upstream_map + "/"
 +        },
 +        'mirrors': [
 +            {
 +                'name': 'middle-earth',
 +                'aliases': {
 +                    alias: [mirror_map + "/"],
 +                },
 +            },
 +        ]
 +    }
 +    project_file = os.path.join(project_dir, 'project.conf')
 +    _yaml.dump(project, project_file)
++
 +    result = cli.run(project=project_dir, args=['track', element_name])
 +    result.assert_success()
++
 +    # Tracking tries upstream first. Check the ref is from upstream.
 +    new_element = _yaml.load(element_path)
 +    source = new_element['sources'][0]
 +    if 'ref' in source:
 +        assert source['ref'] == upstream_ref
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
 +def test_mirror_track_upstream_absent(cli, tmpdir, datafiles, kind):
 +    bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
 +    dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
 +    upstream_repodir = os.path.join(str(tmpdir), 'upstream')
 +    mirror_repodir = os.path.join(str(tmpdir), 'mirror')
 +    project_dir = os.path.join(str(tmpdir), 'project')
 +    os.makedirs(project_dir)
 +    element_dir = os.path.join(project_dir, 'elements')
++
 +    # Create repo objects of the upstream and mirror
 +    upstream_repo = create_repo(kind, upstream_repodir)
 +    upstream_ref = upstream_repo.create(bin_files_path)
 +    mirror_repo = upstream_repo.copy(mirror_repodir)
 +    mirror_ref = upstream_ref
 +    upstream_ref = upstream_repo.create(dev_files_path)
++
 +    element = {
 +        'kind': 'import',
 +        'sources': [
 +            upstream_repo.source_config(ref=upstream_ref)
 +        ]
 +    }
++
 +    element['sources'][0]
 +    element_name = 'test.bst'
 +    element_path = os.path.join(element_dir, element_name)
 +    full_repo = element['sources'][0]['url']
 +    upstream_map, repo_name = os.path.split(full_repo)
 +    alias = 'foo-' + kind
 +    aliased_repo = alias + ':' + repo_name
 +    element['sources'][0]['url'] = aliased_repo
 +    full_mirror = mirror_repo.source_config()['url']
 +    mirror_map, _ = os.path.split(full_mirror)
 +    os.makedirs(element_dir)
 +    _yaml.dump(element, element_path)
++
 +    project = {
 +        'name': 'test',
 +        'element-path': 'elements',
 +        'aliases': {
 +            alias: 'http://www.example.com/'
 +        },
 +        'mirrors': [
 +            {
 +                'name': 'middle-earth',
 +                'aliases': {
 +                    alias: [mirror_map + "/"],
 +                },
 +            },
 +        ]
 +    }
 +    project_file = os.path.join(project_dir, 'project.conf')
 +    _yaml.dump(project, project_file)
++
 +    result = cli.run(project=project_dir, args=['track', element_name])
 +    result.assert_success()
++
 +    # Check that tracking fell back to the mirror
 +    new_element = _yaml.load(element_path)
 +    source = new_element['sources'][0]
 +    if 'ref' in source:
 +        assert source['ref'] == mirror_ref

tests/frontend/project/sources/fetch_source.py

 +import os
 +import sys
++
 +from buildstream import Source, Consistency, SourceError, SourceFetcher
++
 +# Expected config
 +# sources:
 +# - output-text: $FILE
 +#   urls:
 +#   - foo:bar
 +#   - baz:quux
 +#   fetch-succeeds:
 +#     Foo/bar: true
 +#     ooF/bar: false
++
++
 +class FetchFetcher(SourceFetcher):
 +    def __init__(self, source, url):
 +        super().__init__()
 +        self.source = source
 +        self.original_url = url
++
 +        alias, _ = self.source.split_aliased_url(self.original_url)
 +        self.set_alias(alias)
++
 +    def fetch(self, alias_override=None):
 +        url = self.source.translate_url(self.original_url, alias_override)
 +        with open(self.source.output_file, "a") as f:
 +            success = url in self.source.fetch_succeeds and self.source.fetch_succeeds[url]
 +            message = "Fetch {} {} from {}\n".format(self.original_url,
 +                                                     "succeeded" if success else "failed",
 +                                                     url)
 +            f.write(message)
 +            if not success:
 +                raise SourceError("Failed to fetch {}".format(url))
++
++
 +class FetchSource(Source):
 +    # Read config to know which URLs to fetch
 +    def configure(self, node):
 +        self.original_urls = self.node_get_member(node, list, 'urls')
 +        self.fetchers = [FetchFetcher(self, url) for url in self.original_urls]
 +        self.output_file = self.node_get_member(node, str, 'output-text')
 +        self.fetch_succeeds = {}
 +        if 'fetch-succeeds' in node:
 +            self.fetch_succeeds = {x[0]: x[1] for x in self.node_items(node['fetch-succeeds'])}
++
 +    def get_source_fetchers(self):
 +        return self.fetchers
++
 +    def preflight(self):
 +        output_dir = os.path.dirname(self.output_file)
 +        if not os.path.exists(output_dir):
 +            raise SourceError("Directory '{}' does not exist".format(output_dir))
++
 +    def fetch(self):
 +        for fetcher in self.fetchers:
 +            fetcher.fetch()
++
 +    def get_unique_key(self):
 +        return {"urls": self.original_urls, "output_file": self.output_file}
++
 +    def get_consistency(self):
 +        if not os.path.exists(self.output_file):
 +            return Consistency.RESOLVED
++
 +        with open(self.output_file, "r") as f:
 +            contents = f.read()
 +            for url in self.original_urls:
 +                if url not in contents:
 +                    return Consistency.RESOLVED
++
 +        return Consistency.CACHED
++
 +    # We dont have a ref, we're a local file...
 +    def load_ref(self, node):
 +        pass
++
 +    def get_ref(self):
 +        return None  # pragma: nocover
++
 +    def set_ref(self, ref, node):
 +        pass  # pragma: nocover
++
++
 +def setup():
 +    return FetchSource

tests/frontend/show.py

  import os
 -import pytest
 +import sys
 +import shutil
  import itertools
 +import pytest
  from tests.testutils import cli
+-
  from buildstream import _yaml
  from buildstream._exceptions import ErrorDomain, LoadErrorReason
@@ -232,3 +233,58 @@ def test_fetched_junction(cli, tmpdir, datafiles, element_name):
      results = result.output.strip().splitlines()
      assert 'junction.bst:import-etc.bst-buildable' in results
++
++
 +###############################################################
 +#                   Testing recursion depth                   #
 +###############################################################
 +@pytest.mark.parametrize("dependency_depth", [100, 500, 1200])
 +def test_exceed_max_recursion_depth(cli, tmpdir, dependency_depth):
 +    project_name = "recursion-test"
 +    path = str(tmpdir)
 +    project_path = os.path.join(path, project_name)
++
 +    def setup_test():
 +        """
 +        Creates a bst project with dependencydepth + 1 elements, each of which
 +        depends of the previous element to be created. Each element created
 +        is of type import and has an empty source file.
 +        """
 +        os.mkdir(project_path)
++
 +        result = cli.run(project=project_path, silent=True,
 +                         args=['init', '--project-name', project_name])
 +        result.assert_success()
++
 +        sourcefiles_path = os.path.join(project_path, "files")
 +        os.mkdir(sourcefiles_path)
++
 +        element_path = os.path.join(project_path, "elements")
 +        for i in range(0, dependency_depth + 1):
 +            element = {
 +                'kind': 'import',
 +                'sources': [{'kind': 'local',
 +                             'path': 'files/source{}'.format(str(i))}],
 +                'depends': ['element{}.bst'.format(str(i - 1))]
 +            }
 +            if i == 0:
 +                del element['depends']
 +            _yaml.dump(element, os.path.join(element_path, "element{}.bst".format(str(i))))
++
 +            source = os.path.join(sourcefiles_path, "source{}".format(str(i)))
 +            open(source, 'x').close()
 +            assert os.path.exists(source)
++
 +    setup_test()
 +    result = cli.run(project=project_path, silent=True,
 +                     args=['show', "element{}.bst".format(str(dependency_depth))])
++
 +    recursion_limit = sys.getrecursionlimit()
 +    if dependency_depth <= recursion_limit:
 +        result.assert_success()
 +    else:
 +        #  Assert exception is thown and handled
 +        assert not result.unhandled_exception
 +        assert result.exit_code == -1
++
 +    shutil.rmtree(project_path)

tests/frontend/workspace.py

@@ -123,6 +123,58 @@ def test_open_force(cli, tmpdir, datafiles, kind):
      result.assert_success()
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize("kind", repo_kinds)
 +def test_open_force_open(cli, tmpdir, datafiles, kind):
 +    element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, kind, False)
++
 +    # Assert the workspace dir exists
 +    assert os.path.exists(workspace)
++
 +    # Now open the workspace again with --force, this should happily succeed
 +    result = cli.run(project=project, args=[
 +        'workspace', 'open', '--force', element_name, workspace
 +    ])
 +    result.assert_success()
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize("kind", repo_kinds)
 +def test_open_force_different_workspace(cli, tmpdir, datafiles, kind):
 +    element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, kind, False, "-alpha")
++
 +    # Assert the workspace dir exists
 +    assert os.path.exists(workspace)
++
 +    hello_path = os.path.join(workspace, 'usr', 'bin', 'hello')
 +    hello1_path = os.path.join(workspace, 'usr', 'bin', 'hello1')
++
 +    tmpdir = os.path.join(str(tmpdir), "-beta")
 +    shutil.move(hello_path, hello1_path)
 +    element_name2, project2, workspace2 = open_workspace(cli, tmpdir, datafiles, kind, False, "-beta")
++
 +    # Assert the workspace dir exists
 +    assert os.path.exists(workspace2)
++
 +    # Assert that workspace 1 contains the modified file
 +    assert os.path.exists(hello1_path)
++
 +    # Assert that workspace 2 contains the unmodified file
 +    assert os.path.exists(os.path.join(workspace2, 'usr', 'bin', 'hello'))
++
 +    # Now open the workspace again with --force, this should happily succeed
 +    result = cli.run(project=project, args=[
 +        'workspace', 'open', '--force', element_name2, workspace
 +    ])
++
 +    # Assert that the file in workspace 1 has been replaced
 +    # With the file from workspace 2
 +    assert os.path.exists(hello_path)
 +    assert not os.path.exists(hello1_path)
++
 +    result.assert_success()
++
++
  @pytest.mark.datafiles(DATA_DIR)
  @pytest.mark.parametrize("kind", repo_kinds)
  def test_close(cli, tmpdir, datafiles, kind):

tests/testutils/repo/repo.py

@@ -22,7 +22,7 @@ class Repo():
          # The directory the actual repo will be stored in
          self.repo = os.path.join(self.directory, subdir)
 -        os.makedirs(self.repo)
 +        os.makedirs(self.repo, exist_ok=True)
      # create():
+     #
@@ -69,3 +69,22 @@ class Repo():
                  shutil.copytree(src_path, dest_path)
              else:
                  shutil.copy2(src_path, dest_path)
++
 +    # copy():
 +    #
 +    # Creates a copy of this repository in the specified
 +    # destination.
 +    #
 +    # Args:
 +    #    dest (str): The destination directory
 +    #
 +    # Returns:
 +    #    (Repo): A Repo object for the new repository.
 +    def copy(self, dest):
 +        subdir = self.repo[len(self.directory):].lstrip(os.sep)
 +        new_dir = os.path.join(dest, subdir)
 +        os.makedirs(new_dir, exist_ok=True)
 +        self.copy_directory(self.repo, new_dir)
 +        repo_type = type(self)
 +        new_repo = repo_type(dest, subdir)
 +        return new_repo

[Date Prev][Date Next] [Thread Prev][Thread Next] [Thread Index] [Date Index] [Author Index]