[Notes] [Git][BuildStream/buildstream][gokcennurlu/remote_url_override

Gökçen Nurlu pushed to branch gokcennurlu/remote_url_override_push_error at BuildStream / buildstream

Commits:

e29aea36

by William Salmon at 2018-12-19T13:23:19Z

Basic options for shell --build to use buildtrees

Fixes issue #740

898a23a5

by Will Salmon at 2018-12-19T14:35:41Z

Merge branch 'willsalmon/shellBuildTrees' into 'master'

Shell --build has optional buildtrees

Closes #740

See merge request BuildStream/buildstream!986

a2f1d879
by Javier Jardón at 2018-12-19T15:36:11Z
```
README.rst: Add license badge
```

aae5e4b3

by Javier Jardón at 2018-12-19T16:09:48Z

Merge branch 'jjardon/license_badge' into 'master'

README.rst: Add license badge

See merge request BuildStream/buildstream!1014

2b767fe8

by Jürg Billeter at 2018-12-20T10:06:11Z

Move fetch logic from FetchQueue to Element

The queue shouldn't need to know about individual sources. This is in
line with _track() and _get_consistency().

7a102144

by Jürg Billeter at 2018-12-20T10:06:11Z

element.py: Do not call fetch() for cached sources

b325989e

by Jürg Billeter at 2018-12-20T10:07:20Z

tests/sources: Test that fetch() is not called for cached sources

77d8ad45

by Jürg Billeter at 2018-12-20T10:42:39Z

Merge branch 'juerg/fetch' into 'master'

Do not call fetch() for cached sources

See merge request BuildStream/buildstream!992

9835b7f1

by Chandan Singh at 2018-12-20T12:50:02Z

Expose base class for Git source plugins

Add a `_GitSourceBase` class to act as the base class for deriving
source plugins that work with Git. The first user of this base class is
the `git` source plugin that's in core at the moment.

The contents of this base class is almost identical to the existing `GitSource`
class. The two notable differences are:

- the private methods now have leading underscores, as some were missing
  it originally
- the name of the class

Note that we are exposing a private member here as we expect it to move to a
separate package soon. See the following discussion for more details:
https://gitlab.com/BuildStream/buildstream/issues/739#note_124819869

Fixes #739.

14da6955

by Chandan Singh at 2018-12-20T13:34:10Z

Merge branch 'chandan/abstract-git-source' into 'master'

Expose base class for Git source plugins

Closes #739

See merge request BuildStream/buildstream!1019

7368f569

by Angelos Evripiotis at 2018-12-20T13:59:23Z

_project::_find_project_dir: fix error message

Now that we are also looking for WORKSPACE_PROJECT_FILE, we should add
it to the error message. While we're there, also mention the directory
we are looking in - this could help scripters be clear about what went
wrong.

The new error message looks like this:

    Error loading project: None of ['project.conf', '.bstproject.yaml']
    found in '/src/temp/blah' or any of its parent directories

46efc91d

by Angelos Evripiotis at 2018-12-20T13:59:23Z

BREAK: remove auto-init behaviour

In the event that the project could not be found, stop BuildStream from
asking if the user would like to create a new project. Exit with error
instead, and give a hint to the user in case they're new.

As proposed on the mailing list here:
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00082.html

The new interaction looks like this:

    $ bst show nonsuch.bst
    No project found. You can create a new project like so:

        bst init

    Error loading project: None of ['project.conf', '.bstproject.yaml']
    found in '/src/temp/blah' or any of its parent directories

Fixes #826

e0c575c4

by Angelos Evripiotis at 2018-12-20T14:37:38Z

Merge branch 'aevri/rm-autoinit' into 'master'

BREAK: remove auto-init behaviour

Closes #826

See merge request BuildStream/buildstream!1015

c05d8b4f

by Chandan Singh at 2018-12-20T15:53:12Z

tests/frontend/logging.py: Fix regex Deprecation Warning

Use raw strings for regex searches, which is the preferred way to do
regular expressions in Python.

Without this patch, currently we get the following warnings:

```
tests/frontend/logging.py:44
  /builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:44: DeprecationWarning: invalid escape sequence \[
    m = re.search("\[\d\d:\d\d:\d\d\]\[\]\[\] SUCCESS Checking sources", result.stderr)

tests/frontend/logging.py:80
  /builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:80: DeprecationWarning: invalid escape sequence \d
    m = re.search("\d\d:\d\d:\d\d,\d\d:\d\d:\d\d.\d{6},\d\d:\d\d:\d\d,,,SUCCESS,Checking sources", result.stderr)
```

e8055a56

by Chandan Singh at 2018-12-20T15:53:12Z

buildstream/utils.py: Fix regex Deprecation Warning

Specify flags at the start of the _expression_ as per the recommendation
of the standard library.

Without this patch, we currently get the following warning:

```
tests/examples/junctions.py::test_open_cross_junction_workspace
  /builds/BuildStream/buildstream/dist/buildstream/buildstream/utils.py:213: DeprecationWarning: Flags not at the start of the _expression_ '\\/[^/]*\\Z(?ms)'
    regexer = re.compile(_expression_)
```

cd4889af

by Chandan Singh at 2018-12-20T16:34:08Z

Merge branch 'chandan/fix-warning' into 'master'

Fix Deprecation warnings from regex module

See merge request BuildStream/buildstream!1010

ac995236

by Tom Pollard at 2018-12-20T17:18:02Z

_stream.py: fix _buildtree_pull_required logic

c3153dea

by Tom Pollard at 2018-12-21T10:10:58Z

Merge branch 'tpollard/fixbuildtreereq' into 'master'

_stream.py: fix _buildtree_pull_required logic

See merge request BuildStream/buildstream!1021

d5f3c723

by Gökçen Nurlu at 2018-12-21T12:31:37Z

Set ArtifactCache for push/pull correctly when `--remote` is used

This change makes `ArtifactCache.setup_remotes()` use a boolean parameter, set
by `Stream.push()` or `Stream.pull()`, to configure the overriden `remote_url`
correctly, instead of for `push` by default.

This fixes #798.

df73c09f

by Gökçen Nurlu at 2018-12-21T12:31:37Z

Fix type of artifact_remote_url in _load() docs

fc47c148

by Gökçen Nurlu at 2018-12-21T12:31:37Z

Refactor setup_remotes() to remove --remote overriding logic

This change simplifies the responsibilities of `setup_remotes()` by extracting
"getting the default remotes" logic to its own function and moving "--remote"
related logic to _load(). The latter, especially, helps to remove the need of
passing the "remote_url" string and "can_push?" boolean to `setup_remotes()`.

25 changed files:

NEWS
README.rst
buildstream/__init__.py
buildstream/_artifactcache/artifactcache.py
buildstream/_context.py
buildstream/_frontend/app.py
buildstream/_frontend/cli.py
+ buildstream/_gitsourcebase.py
buildstream/_project.py
buildstream/_scheduler/queues/fetchqueue.py
buildstream/_stream.py
buildstream/data/userconfig.yaml
buildstream/element.py
buildstream/plugins/sources/git.py
buildstream/sandbox/sandbox.py
buildstream/utils.py
tests/artifactcache/pull.py
tests/artifactcache/push.py
tests/frontend/logging.py
tests/integration/build-tree.py
+ tests/sources/no-fetch-cached/files/file
+ tests/sources/no-fetch-cached/plugins/sources/always_cached.py
+ tests/sources/no-fetch-cached/project.conf
+ tests/sources/no_fetch_cached.py
tests/testutils/runcli.py

Changes:

NEWS

@@ -30,6 +30,12 @@ buildstream 1.3.1
      make changes to their .bst files if they are expecting these environment
      variables to be set.
 +  o BREAKING CHANGE: The 'auto-init' functionality has been removed. This would
 +    offer to create a project in the event that bst was run against a directory
 +    without a project, to be friendly to new users. It has been replaced with
 +    an error message and a hint instead, to avoid bothering folks that just
 +    made a mistake.
++
    o Failed builds are included in the cache as well.
      `bst checkout` will provide anything in `%{install-root}`.
      A build including cached fails will cause any dependant elements
@@ -67,8 +73,8 @@ buildstream 1.3.1
      instead of just a specially-formatted build-root with a `root` and `scratch`
      subdirectory.
 -  o The buildstream.conf file learned new 'prompt.auto-init',
 -    'prompt.really-workspace-close-remove-dir', and
 +  o The buildstream.conf file learned new
 +    'prompt.really-workspace-close-remove-dir' and
      'prompt.really-workspace-reset-hard' options. These allow users to suppress
      certain confirmation prompts, e.g. double-checking that the user meant to
      run the command as typed.

README.rst

@@ -16,6 +16,9 @@ About
  .. image:: https://img.shields.io/pypi/v/BuildStream.svg
     :target: https://pypi.org/project/BuildStream
 +.. image:: https://app.fossa.io/api/projects/git%2Bgitlab.com%2FBuildStream%2Fbuildstream.svg?type=shield
 +   :target: https://app.fossa.io/projects/git%2Bgitlab.com%2FBuildStream%2Fbuildstream?ref=badge_shield
++
  What is BuildStream?
  ====================

buildstream/__init__.py

@@ -34,3 +34,8 @@ if "_BST_COMPLETION" not in os.environ:
      from .element import Element, ElementError
      from .buildelement import BuildElement
      from .scriptelement import ScriptElement
++
 +    # XXX We are exposing a private member here as we expect it to move to a
 +    # separate package soon. See the following discussion for more details:
 +    # https://gitlab.com/BuildStream/buildstream/issues/739#note_124819869
 +    from ._gitsourcebase import _GitSourceBase

buildstream/_artifactcache/artifactcache.py

@@ -110,36 +110,42 @@ class ArtifactCache():
          # assume project and element names are not allowed to contain slashes
          return '{0}/{1}/{2}'.format(project.name, element_name, key)
 +    # get_remotes_from_projects()
 +    #
 +    # Generates list artifact caches based on project configuration
 +    #
 +    # Returns:
 +    #    (list of (list of ArtifactCacheSpec, Project)): Configurations each are
 +    #        ready to be consumed by `self._set_remotes()`
 +    #
 +    # This requires that all of the projects which are to be processed in the session
 +    # have already been loaded and are observable in the Context.
 +    #
 +    def get_remotes_from_projects(self):
 +        return [
 +            (_configured_remote_artifact_cache_specs(self.context, prj), prj)
 +            for prj in self.context.get_projects()
 +        ]
++
      # setup_remotes():
+     #
      # Sets up which remotes to use
+     #
      # Args:
 -    #    use_config (bool): Whether to use project configuration
 -    #    remote_url (str): Remote artifact cache URL
 +    #    remotes (list of (list of ArtifactCacheSpec, Project)): Configurations which are
 +    #        ready to be consumed by `self._set_remotes()`
+     #
      # This requires that all of the projects which are to be processed in the session
      # have already been loaded and are observable in the Context.
+     #
 -    def setup_remotes(self, *, use_config=False, remote_url=None):
+-
 +    def setup_remotes(self, *, remotes=None):
          # Ensure we do not double-initialise since this can be expensive
          assert not self._remotes_setup
          self._remotes_setup = True
 -        # Initialize remote artifact caches. We allow the commandline to override
 -        # the user config in some cases (for example `bst push --remote=...`).
 -        has_remote_caches = False
 -        if remote_url:
 -            self._set_remotes([ArtifactCacheSpec(remote_url, push=True)])
 -            has_remote_caches = True
 -        if use_config:
 -            for project in self.context.get_projects():
 -                artifact_caches = _configured_remote_artifact_cache_specs(self.context, project)
 -                if artifact_caches:  # artifact_caches is a list of ArtifactCacheSpec instances
 -                    self._set_remotes(artifact_caches, project=project)
 -                    has_remote_caches = True
 -        if has_remote_caches:
 +        if remotes:
 +            for caches, project in remotes:
 +                self._set_remotes(caches, project=project)
              self._initialize_remotes()
      # specs_from_config_node()

buildstream/_context.py

@@ -117,10 +117,6 @@ class Context():
          # Whether or not to attempt to pull build trees globally
          self.pull_buildtrees = None
 -        # Boolean, whether to offer to create a project for the user, if we are
 -        # invoked outside of a directory where we can resolve the project.
 -        self.prompt_auto_init = None
+-
          # Boolean, whether we double-check with the user that they meant to
          # remove a workspace directory.
          self.prompt_workspace_close_remove_dir = None
@@ -258,12 +254,10 @@ class Context():
          prompt = _yaml.node_get(
              defaults, Mapping, 'prompt')
          _yaml.node_validate(prompt, [
 -            'auto-init', 'really-workspace-close-remove-dir',
 +            'really-workspace-close-remove-dir',
              'really-workspace-close-project-inaccessible',
              'really-workspace-reset-hard',
          ])
 -        self.prompt_auto_init = _node_get_option_str(
 -            prompt, 'auto-init', ['ask', 'no']) == 'ask'
          self.prompt_workspace_close_remove_dir = _node_get_option_str(
              prompt, 'really-workspace-close-remove-dir', ['ask', 'yes']) == 'ask'
          self.prompt_workspace_close_project_inaccessible = _node_get_option_str(

buildstream/_frontend/app.py

@@ -219,13 +219,13 @@ class App():
                                     default_mirror=self._main_options.get('default_mirror'))
          except LoadError as e:
 -            # Let's automatically start a `bst init` session in this case
 -            if e.reason == LoadErrorReason.MISSING_PROJECT_CONF and self.interactive:
 -                click.echo("A project was not detected in the directory: {}".format(directory), err=True)
 -                if self.context.prompt_auto_init:
 -                    click.echo("", err=True)
 -                    if click.confirm("Would you like to create a new project here?"):
 -                        self.init_project(None)
 +            # Help users that are new to BuildStream by suggesting 'init'.
 +            # We don't want to slow down users that just made a mistake, so
 +            # don't stop them with an offer to create a project for them.
 +            if e.reason == LoadErrorReason.MISSING_PROJECT_CONF:
 +                click.echo("No project found. You can create a new project like so:", err=True)
 +                click.echo("", err=True)
 +                click.echo("    bst init", err=True)
              self._error_exit(e, "Error loading project")

buildstream/_frontend/cli.py

@@ -527,11 +527,14 @@ def show(app, elements, deps, except_, order, format_):
                help="Mount a file or directory into the sandbox")
  @click.option('--isolate', is_flag=True, default=False,
                help='Create an isolated build sandbox')
 +@click.option('--use-buildtree', '-t', 'cli_buildtree', type=click.Choice(['ask', 'try', 'always', 'never']),
 +              default='ask',
 +              help='Defaults to ask but if set to always the function will fail if a build tree is not available')
  @click.argument('element', required=False,
                  type=click.Path(readable=False))
  @click.argument('command', type=click.STRING, nargs=-1)
  @click.pass_obj
 -def shell(app, element, sysroot, mount, isolate, build_, command):
 +def shell(app, element, sysroot, mount, isolate, build_, cli_buildtree, command):
      """Run a command in the target element's sandbox environment
      This will stage a temporary sysroot for running the target
@@ -557,6 +560,8 @@ def shell(app, element, sysroot, mount, isolate, build_, command):
      else:
          scope = Scope.RUN
 +    use_buildtree = False
++
      with app.initialized():
          if not element:
              element = app.context.guess_element()
@@ -570,12 +575,30 @@ def shell(app, element, sysroot, mount, isolate, build_, command):
              HostMount(path, host_path)
              for host_path, path in mount
+         ]
++
 +        cached = element._cached_buildtree()
 +        if cli_buildtree == "always":
 +            if cached:
 +                use_buildtree = True
 +            else:
 +                raise AppError("No buildtree is cached but the use buildtree option was specified")
 +        elif cli_buildtree == "never":
 +            pass
 +        elif cli_buildtree == "try":
 +            use_buildtree = cached
 +        else:
 +            if app.interactive and cached:
 +                use_buildtree = bool(click.confirm('Do you want to use the cached buildtree?'))
 +        if use_buildtree and not element._cached_success():
 +            click.echo("Warning: using a buildtree from a failed build.")
++
          try:
              exitcode = app.stream.shell(element, scope, prompt,
                                          directory=sysroot,
                                          mounts=mounts,
                                          isolate=isolate,
 -                                        command=command)
 +                                        command=command,
 +                                        usebuildtree=use_buildtree)
          except BstError as e:
              raise AppError("Error launching shell: {}".format(e), detail=e.detail) from e

buildstream/_gitsourcebase.py

 +#
 +#  Copyright (C) 2016 Codethink Limited
 +#  Copyright (C) 2018 Bloomberg Finance LP
 +#
 +#  This program is free software; you can redistribute it and/or
 +#  modify it under the terms of the GNU Lesser General Public
 +#  License as published by the Free Software Foundation; either
 +#  version 2 of the License, or (at your option) any later version.
 +#
 +#  This library is distributed in the hope that it will be useful,
 +#  but WITHOUT ANY WARRANTY; without even the implied warranty of
 +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	 See the GNU
 +#  Lesser General Public License for more details.
 +#
 +#  You should have received a copy of the GNU Lesser General Public
 +#  License along with this library. If not, see <http://www.gnu.org/licenses/>.
 +#
 +#  Authors:
 +#        Tristan Van Berkom <tristan vanberkom codethink co uk>
 +#        Chandan Singh <csingh43 bloomberg net>
++
 +"""Abstract base class for source implementations that work with a Git repository"""
++
 +import os
 +import re
 +import shutil
 +from collections.abc import Mapping
 +from io import StringIO
 +from tempfile import TemporaryFile
++
 +from configparser import RawConfigParser
++
 +from buildstream import Source, SourceError, Consistency, SourceFetcher, CoreWarnings
 +from buildstream import utils
 +from buildstream.utils import move_atomic, DirectoryExistsError
++
 +GIT_MODULES = '.gitmodules'
++
 +# Warnings
 +WARN_INCONSISTENT_SUBMODULE = "inconsistent-submodule"
 +WARN_UNLISTED_SUBMODULE = "unlisted-submodule"
 +WARN_INVALID_SUBMODULE = "invalid-submodule"
++
++
 +# Because of handling of submodules, we maintain a GitMirror
 +# for the primary git source and also for each submodule it
 +# might have at a given time
 +#
 +class GitMirror(SourceFetcher):
++
 +    def __init__(self, source, path, url, ref, *, primary=False, tags=[]):
++
 +        super().__init__()
 +        self.source = source
 +        self.path = path
 +        self.url = url
 +        self.ref = ref
 +        self.tags = tags
 +        self.primary = primary
 +        self.mirror = os.path.join(source.get_mirror_directory(), utils.url_directory_name(url))
 +        self.mark_download_url(url)
++
 +    # Ensures that the mirror exists
 +    def ensure(self, alias_override=None):
++
 +        # Unfortunately, git does not know how to only clone just a specific ref,
 +        # so we have to download all of those gigs even if we only need a couple
 +        # of bytes.
 +        if not os.path.exists(self.mirror):
++
 +            # Do the initial clone in a tmpdir just because we want an atomic move
 +            # after a long standing clone which could fail overtime, for now do
 +            # this directly in our git directory, eliminating the chances that the
 +            # system configured tmpdir is not on the same partition.
 +            #
 +            with self.source.tempdir() as tmpdir:
 +                url = self.source.translate_url(self.url, alias_override=alias_override,
 +                                                primary=self.primary)
 +                self.source.call([self.source.host_git, 'clone', '--mirror', '-n', url, tmpdir],
 +                                 fail="Failed to clone git repository {}".format(url),
 +                                 fail_temporarily=True)
++
 +                try:
 +                    move_atomic(tmpdir, self.mirror)
 +                except DirectoryExistsError:
 +                    # Another process was quicker to download this repository.
 +                    # Let's discard our own
 +                    self.source.status("{}: Discarding duplicate clone of {}"
 +                                       .format(self.source, url))
 +                except OSError as e:
 +                    raise SourceError("{}: Failed to move cloned git repository {} from '{}' to '{}': {}"
 +                                      .format(self.source, url, tmpdir, self.mirror, e)) from e
++
 +    def _fetch(self, alias_override=None):
 +        url = self.source.translate_url(self.url,
 +                                        alias_override=alias_override,
 +                                        primary=self.primary)
++
 +        if alias_override:
 +            remote_name = utils.url_directory_name(alias_override)
 +            _, remotes = self.source.check_output(
 +                [self.source.host_git, 'remote'],
 +                fail="Failed to retrieve list of remotes in {}".format(self.mirror),
 +                cwd=self.mirror
 +            )
 +            if remote_name not in remotes:
 +                self.source.call(
 +                    [self.source.host_git, 'remote', 'add', remote_name, url],
 +                    fail="Failed to add remote {} with url {}".format(remote_name, url),
 +                    cwd=self.mirror
 +                )
 +        else:
 +            remote_name = "origin"
++
 +        self.source.call([self.source.host_git, 'fetch', remote_name, '--prune', '--force', '--tags'],
 +                         fail="Failed to fetch from remote git repository: {}".format(url),
 +                         fail_temporarily=True,
 +                         cwd=self.mirror)
++
 +    def fetch(self, alias_override=None):
 +        # Resolve the URL for the message
 +        resolved_url = self.source.translate_url(self.url,
 +                                                 alias_override=alias_override,
 +                                                 primary=self.primary)
++
 +        with self.source.timed_activity("Fetching from {}"
 +                                        .format(resolved_url),
 +                                        silent_nested=True):
 +            self.ensure(alias_override)
 +            if not self.has_ref():
 +                self._fetch(alias_override)
 +            self.assert_ref()
++
 +    def has_ref(self):
 +        if not self.ref:
 +            return False
++
 +        # If the mirror doesnt exist, we also dont have the ref
 +        if not os.path.exists(self.mirror):
 +            return False
++
 +        # Check if the ref is really there
 +        rc = self.source.call([self.source.host_git, 'cat-file', '-t', self.ref], cwd=self.mirror)
 +        return rc == 0
++
 +    def assert_ref(self):
 +        if not self.has_ref():
 +            raise SourceError("{}: expected ref '{}' was not found in git repository: '{}'"
 +                              .format(self.source, self.ref, self.url))
++
 +    def latest_commit_with_tags(self, tracking, track_tags=False):
 +        _, output = self.source.check_output(
 +            [self.source.host_git, 'rev-parse', tracking],
 +            fail="Unable to find commit for specified branch name '{}'".format(tracking),
 +            cwd=self.mirror)
 +        ref = output.rstrip('\n')
++
 +        if self.source.ref_format == 'git-describe':
 +            # Prefix the ref with the closest tag, if available,
 +            # to make the ref human readable
 +            exit_code, output = self.source.check_output(
 +                [self.source.host_git, 'describe', '--tags', '--abbrev=40', '--long', ref],
 +                cwd=self.mirror)
 +            if exit_code == 0:
 +                ref = output.rstrip('\n')
++
 +        if not track_tags:
 +            return ref, []
++
 +        tags = set()
 +        for options in [[], ['--first-parent'], ['--tags'], ['--tags', '--first-parent']]:
 +            exit_code, output = self.source.check_output(
 +                [self.source.host_git, 'describe', '--abbrev=0', ref] + options,
 +                cwd=self.mirror)
 +            if exit_code == 0:
 +                tag = output.strip()
 +                _, commit_ref = self.source.check_output(
 +                    [self.source.host_git, 'rev-parse', tag + '^{commit}'],
 +                    fail="Unable to resolve tag '{}'".format(tag),
 +                    cwd=self.mirror)
 +                exit_code = self.source.call(
 +                    [self.source.host_git, 'cat-file', 'tag', tag],
 +                    cwd=self.mirror)
 +                annotated = (exit_code == 0)
++
 +                tags.add((tag, commit_ref.strip(), annotated))
++
 +        return ref, list(tags)
++
 +    def stage(self, directory):
 +        fullpath = os.path.join(directory, self.path)
++
 +        # Using --shared here avoids copying the objects into the checkout, in any
 +        # case we're just checking out a specific commit and then removing the .git/
 +        # directory.
 +        self.source.call([self.source.host_git, 'clone', '--no-checkout', '--shared', self.mirror, fullpath],
 +                         fail="Failed to create git mirror {} in directory: {}".format(self.mirror, fullpath),
 +                         fail_temporarily=True)
++
 +        self.source.call([self.source.host_git, 'checkout', '--force', self.ref],
 +                         fail="Failed to checkout git ref {}".format(self.ref),
 +                         cwd=fullpath)
++
 +        # Remove .git dir
 +        shutil.rmtree(os.path.join(fullpath, ".git"))
++
 +        self._rebuild_git(fullpath)
++
 +    def init_workspace(self, directory):
 +        fullpath = os.path.join(directory, self.path)
 +        url = self.source.translate_url(self.url)
++
 +        self.source.call([self.source.host_git, 'clone', '--no-checkout', self.mirror, fullpath],
 +                         fail="Failed to clone git mirror {} in directory: {}".format(self.mirror, fullpath),
 +                         fail_temporarily=True)
++
 +        self.source.call([self.source.host_git, 'remote', 'set-url', 'origin', url],
 +                         fail='Failed to add remote origin "{}"'.format(url),
 +                         cwd=fullpath)
++
 +        self.source.call([self.source.host_git, 'checkout', '--force', self.ref],
 +                         fail="Failed to checkout git ref {}".format(self.ref),
 +                         cwd=fullpath)
++
 +    # List the submodules (path/url tuples) present at the given ref of this repo
 +    def submodule_list(self):
 +        modules = "{}:{}".format(self.ref, GIT_MODULES)
 +        exit_code, output = self.source.check_output(
 +            [self.source.host_git, 'show', modules], cwd=self.mirror)
++
 +        # If git show reports error code 128 here, we take it to mean there is
 +        # no .gitmodules file to display for the given revision.
 +        if exit_code == 128:
 +            return
 +        elif exit_code != 0:
 +            raise SourceError(
 +                "{plugin}: Failed to show gitmodules at ref {ref}".format(
 +                    plugin=self, ref=self.ref))
++
 +        content = '\n'.join([l.strip() for l in output.splitlines()])
++
 +        io = StringIO(content)
 +        parser = RawConfigParser()
 +        parser.read_file(io)
++
 +        for section in parser.sections():
 +            # validate section name against the 'submodule "foo"' pattern
 +            if re.match(r'submodule "(.*)"', section):
 +                path = parser.get(section, 'path')
 +                url = parser.get(section, 'url')
++
 +                yield (path, url)
++
 +    # Fetch the ref which this mirror requires its submodule to have,
 +    # at the given ref of this mirror.
 +    def submodule_ref(self, submodule, ref=None):
 +        if not ref:
 +            ref = self.ref
++
 +        # list objects in the parent repo tree to find the commit
 +        # object that corresponds to the submodule
 +        _, output = self.source.check_output([self.source.host_git, 'ls-tree', ref, submodule],
 +                                             fail="ls-tree failed for commit {} and submodule: {}".format(
 +                                                 ref, submodule),
 +                                             cwd=self.mirror)
++
 +        # read the commit hash from the output
 +        fields = output.split()
 +        if len(fields) >= 2 and fields[1] == 'commit':
 +            submodule_commit = output.split()[2]
++
 +            # fail if the commit hash is invalid
 +            if len(submodule_commit) != 40:
 +                raise SourceError("{}: Error reading commit information for submodule '{}'"
 +                                  .format(self.source, submodule))
++
 +            return submodule_commit
++
 +        else:
 +            detail = "The submodule '{}' is defined either in the BuildStream source\n".format(submodule) + \
 +                     "definition, or in a .gitmodules file. But the submodule was never added to the\n" + \
 +                     "underlying git repository with `git submodule add`."
++
 +            self.source.warn("{}: Ignoring inconsistent submodule '{}'"
 +                             .format(self.source, submodule), detail=detail,
 +                             warning_token=WARN_INCONSISTENT_SUBMODULE)
++
 +            return None
++
 +    def _rebuild_git(self, fullpath):
 +        if not self.tags:
 +            return
++
 +        with self.source.tempdir() as tmpdir:
 +            included = set()
 +            shallow = set()
 +            for _, commit_ref, _ in self.tags:
++
 +                _, out = self.source.check_output([self.source.host_git, 'rev-list',
 +                                                   '--boundary', '{}..{}'.format(commit_ref, self.ref)],
 +                                                  fail="Failed to get git history {}..{} in directory: {}"
 +                                                  .format(commit_ref, self.ref, fullpath),
 +                                                  fail_temporarily=True,
 +                                                  cwd=self.mirror)
 +                for line in out.splitlines():
 +                    rev = line.lstrip('-')
 +                    if line[0] == '-':
 +                        shallow.add(rev)
 +                    else:
 +                        included.add(rev)
++
 +            shallow -= included
 +            included |= shallow
++
 +            self.source.call([self.source.host_git, 'init'],
 +                             fail="Cannot initialize git repository: {}".format(fullpath),
 +                             cwd=fullpath)
++
 +            for rev in included:
 +                with TemporaryFile(dir=tmpdir) as commit_file:
 +                    self.source.call([self.source.host_git, 'cat-file', 'commit', rev],
 +                                     stdout=commit_file,
 +                                     fail="Failed to get commit {}".format(rev),
 +                                     cwd=self.mirror)
 +                    commit_file.seek(0, 0)
 +                    self.source.call([self.source.host_git, 'hash-object', '-w', '-t', 'commit', '--stdin'],
 +                                     stdin=commit_file,
 +                                     fail="Failed to add commit object {}".format(rev),
 +                                     cwd=fullpath)
++
 +            with open(os.path.join(fullpath, '.git', 'shallow'), 'w') as shallow_file:
 +                for rev in shallow:
 +                    shallow_file.write('{}\n'.format(rev))
++
 +            for tag, commit_ref, annotated in self.tags:
 +                if annotated:
 +                    with TemporaryFile(dir=tmpdir) as tag_file:
 +                        tag_data = 'object {}\ntype commit\ntag {}\n'.format(commit_ref, tag)
 +                        tag_file.write(tag_data.encode('ascii'))
 +                        tag_file.seek(0, 0)
 +                        _, tag_ref = self.source.check_output(
 +                            [self.source.host_git, 'hash-object', '-w', '-t',
 +                             'tag', '--stdin'],
 +                            stdin=tag_file,
 +                            fail="Failed to add tag object {}".format(tag),
 +                            cwd=fullpath)
++
 +                    self.source.call([self.source.host_git, 'tag', tag, tag_ref.strip()],
 +                                     fail="Failed to tag: {}".format(tag),
 +                                     cwd=fullpath)
 +                else:
 +                    self.source.call([self.source.host_git, 'tag', tag, commit_ref],
 +                                     fail="Failed to tag: {}".format(tag),
 +                                     cwd=fullpath)
++
 +            with open(os.path.join(fullpath, '.git', 'HEAD'), 'w') as head:
 +                self.source.call([self.source.host_git, 'rev-parse', self.ref],
 +                                 stdout=head,
 +                                 fail="Failed to parse commit {}".format(self.ref),
 +                                 cwd=self.mirror)
++
++
 +class _GitSourceBase(Source):
 +    # pylint: disable=attribute-defined-outside-init
++
 +    def configure(self, node):
 +        ref = self.node_get_member(node, str, 'ref', None)
++
 +        config_keys = ['url', 'track', 'ref', 'submodules',
 +                       'checkout-submodules', 'ref-format',
 +                       'track-tags', 'tags']
 +        self.node_validate(node, config_keys + Source.COMMON_CONFIG_KEYS)
++
 +        tags_node = self.node_get_member(node, list, 'tags', [])
 +        for tag_node in tags_node:
 +            self.node_validate(tag_node, ['tag', 'commit', 'annotated'])
++
 +        tags = self._load_tags(node)
 +        self.track_tags = self.node_get_member(node, bool, 'track-tags', False)
++
 +        self.original_url = self.node_get_member(node, str, 'url')
 +        self.mirror = GitMirror(self, '', self.original_url, ref, tags=tags, primary=True)
 +        self.tracking = self.node_get_member(node, str, 'track', None)
++
 +        self.ref_format = self.node_get_member(node, str, 'ref-format', 'sha1')
 +        if self.ref_format not in ['sha1', 'git-describe']:
 +            provenance = self.node_provenance(node, member_name='ref-format')
 +            raise SourceError("{}: Unexpected value for ref-format: {}".format(provenance, self.ref_format))
++
 +        # At this point we now know if the source has a ref and/or a track.
 +        # If it is missing both then we will be unable to track or build.
 +        if self.mirror.ref is None and self.tracking is None:
 +            raise SourceError("{}: Git sources require a ref and/or track".format(self),
 +                              reason="missing-track-and-ref")
++
 +        self.checkout_submodules = self.node_get_member(node, bool, 'checkout-submodules', True)
 +        self.submodules = []
++
 +        # Parse a dict of submodule overrides, stored in the submodule_overrides
 +        # and submodule_checkout_overrides dictionaries.
 +        self.submodule_overrides = {}
 +        self.submodule_checkout_overrides = {}
 +        modules = self.node_get_member(node, Mapping, 'submodules', {})
 +        for path, _ in self.node_items(modules):
 +            submodule = self.node_get_member(modules, Mapping, path)
 +            url = self.node_get_member(submodule, str, 'url', None)
++
 +            # Make sure to mark all URLs that are specified in the configuration
 +            if url:
 +                self.mark_download_url(url, primary=False)
++
 +            self.submodule_overrides[path] = url
 +            if 'checkout' in submodule:
 +                checkout = self.node_get_member(submodule, bool, 'checkout')
 +                self.submodule_checkout_overrides[path] = checkout
++
 +        self.mark_download_url(self.original_url)
++
 +    def preflight(self):
 +        # Check if git is installed, get the binary at the same time
 +        self.host_git = utils.get_host_tool('git')
++
 +    def get_unique_key(self):
 +        # Here we want to encode the local name of the repository and
 +        # the ref, if the user changes the alias to fetch the same sources
 +        # from another location, it should not affect the cache key.
 +        key = [self.original_url, self.mirror.ref]
 +        if self.mirror.tags:
 +            tags = {tag: (commit, annotated) for tag, commit, annotated in self.mirror.tags}
 +            key.append({'tags': tags})
++
 +        # Only modify the cache key with checkout_submodules if it's something
 +        # other than the default behaviour.
 +        if self.checkout_submodules is False:
 +            key.append({"checkout_submodules": self.checkout_submodules})
++
 +        # We want the cache key to change if the source was
 +        # configured differently, and submodules count.
 +        if self.submodule_overrides:
 +            key.append(self.submodule_overrides)
++
 +        if self.submodule_checkout_overrides:
 +            key.append({"submodule_checkout_overrides": self.submodule_checkout_overrides})
++
 +        return key
++
 +    def get_consistency(self):
 +        if self._have_all_refs():
 +            return Consistency.CACHED
 +        elif self.mirror.ref is not None:
 +            return Consistency.RESOLVED
 +        return Consistency.INCONSISTENT
++
 +    def load_ref(self, node):
 +        self.mirror.ref = self.node_get_member(node, str, 'ref', None)
 +        self.mirror.tags = self._load_tags(node)
++
 +    def get_ref(self):
 +        return self.mirror.ref, self.mirror.tags
++
 +    def set_ref(self, ref_data, node):
 +        if not ref_data:
 +            self.mirror.ref = None
 +            if 'ref' in node:
 +                del node['ref']
 +            self.mirror.tags = []
 +            if 'tags' in node:
 +                del node['tags']
 +        else:
 +            ref, tags = ref_data
 +            node['ref'] = self.mirror.ref = ref
 +            self.mirror.tags = tags
 +            if tags:
 +                node['tags'] = []
 +                for tag, commit_ref, annotated in tags:
 +                    data = {'tag': tag,
 +                            'commit': commit_ref,
 +                            'annotated': annotated}
 +                    node['tags'].append(data)
 +            else:
 +                if 'tags' in node:
 +                    del node['tags']
++
 +    def track(self):
++
 +        # If self.tracking is not specified it's not an error, just silently return
 +        if not self.tracking:
 +            # Is there a better way to check if a ref is given.
 +            if self.mirror.ref is None:
 +                detail = 'Without a tracking branch ref can not be updated. Please ' + \
 +                         'provide a ref or a track.'
 +                raise SourceError("{}: No track or ref".format(self),
 +                                  detail=detail, reason="track-attempt-no-track")
 +            return None
++
 +        # Resolve the URL for the message
 +        resolved_url = self.translate_url(self.mirror.url)
 +        with self.timed_activity("Tracking {} from {}"
 +                                 .format(self.tracking, resolved_url),
 +                                 silent_nested=True):
 +            self.mirror.ensure()
 +            self.mirror._fetch()
++
 +            # Update self.mirror.ref and node.ref from the self.tracking branch
 +            ret = self.mirror.latest_commit_with_tags(self.tracking, self.track_tags)
++
 +        return ret
++
 +    def init_workspace(self, directory):
 +        # XXX: may wish to refactor this as some code dupe with stage()
 +        self._refresh_submodules()
++
 +        with self.timed_activity('Setting up workspace "{}"'.format(directory), silent_nested=True):
 +            self.mirror.init_workspace(directory)
 +            for mirror in self.submodules:
 +                mirror.init_workspace(directory)
++
 +    def stage(self, directory):
++
 +        # Need to refresh submodule list here again, because
 +        # it's possible that we did not load in the main process
 +        # with submodules present (source needed fetching) and
 +        # we may not know about the submodule yet come time to build.
 +        #
 +        self._refresh_submodules()
++
 +        # Stage the main repo in the specified directory
 +        #
 +        with self.timed_activity("Staging {}".format(self.mirror.url), silent_nested=True):
 +            self.mirror.stage(directory)
 +            for mirror in self.submodules:
 +                mirror.stage(directory)
++
 +    def get_source_fetchers(self):
 +        yield self.mirror
 +        self._refresh_submodules()
 +        for submodule in self.submodules:
 +            yield submodule
++
 +    def validate_cache(self):
 +        discovered_submodules = {}
 +        unlisted_submodules = []
 +        invalid_submodules = []
++
 +        for path, url in self.mirror.submodule_list():
 +            discovered_submodules[path] = url
 +            if self._ignore_submodule(path):
 +                continue
++
 +            override_url = self.submodule_overrides.get(path)
 +            if not override_url:
 +                unlisted_submodules.append((path, url))
++
 +        # Warn about submodules which are explicitly configured but do not exist
 +        for path, url in self.submodule_overrides.items():
 +            if path not in discovered_submodules:
 +                invalid_submodules.append((path, url))
++
 +        if invalid_submodules:
 +            detail = []
 +            for path, url in invalid_submodules:
 +                detail.append("  Submodule URL '{}' at path '{}'".format(url, path))
++
 +            self.warn("{}: Invalid submodules specified".format(self),
 +                      warning_token=WARN_INVALID_SUBMODULE,
 +                      detail="The following submodules are specified in the source "
 +                      "description but do not exist according to the repository\n\n" +
 +                      "\n".join(detail))
++
 +        # Warn about submodules which exist but have not been explicitly configured
 +        if unlisted_submodules:
 +            detail = []
 +            for path, url in unlisted_submodules:
 +                detail.append("  Submodule URL '{}' at path '{}'".format(url, path))
++
 +            self.warn("{}: Unlisted submodules exist".format(self),
 +                      warning_token=WARN_UNLISTED_SUBMODULE,
 +                      detail="The following submodules exist but are not specified " +
 +                      "in the source description\n\n" +
 +                      "\n".join(detail))
++
 +        # Assert that the ref exists in the track tag/branch, if track has been specified.
 +        ref_in_track = False
 +        if self.tracking:
 +            _, branch = self.check_output([self.host_git, 'branch', '--list', self.tracking,
 +                                           '--contains', self.mirror.ref],
 +                                          cwd=self.mirror.mirror)
 +            if branch:
 +                ref_in_track = True
 +            else:
 +                _, tag = self.check_output([self.host_git, 'tag', '--list', self.tracking,
 +                                            '--contains', self.mirror.ref],
 +                                           cwd=self.mirror.mirror)
 +                if tag:
 +                    ref_in_track = True
++
 +            if not ref_in_track:
 +                detail = "The ref provided for the element does not exist locally " + \
 +                         "in the provided track branch / tag '{}'.\n".format(self.tracking) + \
 +                         "You may wish to track the element to update the ref from '{}' ".format(self.tracking) + \
 +                         "with `bst track`,\n" + \
 +                         "or examine the upstream at '{}' for the specific ref.".format(self.mirror.url)
++
 +                self.warn("{}: expected ref '{}' was not found in given track '{}' for staged repository: '{}'\n"
 +                          .format(self, self.mirror.ref, self.tracking, self.mirror.url),
 +                          detail=detail, warning_token=CoreWarnings.REF_NOT_IN_TRACK)
++
 +    ###########################################################
 +    #                     Local Functions                     #
 +    ###########################################################
++
 +    def _have_all_refs(self):
 +        if not self.mirror.has_ref():
 +            return False
++
 +        self._refresh_submodules()
 +        for mirror in self.submodules:
 +            if not os.path.exists(mirror.mirror):
 +                return False
 +            if not mirror.has_ref():
 +                return False
++
 +        return True
++
 +    # Refreshes the GitMirror objects for submodules
 +    #
 +    # Assumes that we have our mirror and we have the ref which we point to
 +    #
 +    def _refresh_submodules(self):
 +        self.mirror.ensure()
 +        submodules = []
++
 +        for path, url in self.mirror.submodule_list():
++
 +            # Completely ignore submodules which are disabled for checkout
 +            if self._ignore_submodule(path):
 +                continue
++
 +            # Allow configuration to override the upstream
 +            # location of the submodules.
 +            override_url = self.submodule_overrides.get(path)
 +            if override_url:
 +                url = override_url
++
 +            ref = self.mirror.submodule_ref(path)
 +            if ref is not None:
 +                mirror = GitMirror(self, path, url, ref)
 +                submodules.append(mirror)
++
 +        self.submodules = submodules
++
 +    def _load_tags(self, node):
 +        tags = []
 +        tags_node = self.node_get_member(node, list, 'tags', [])
 +        for tag_node in tags_node:
 +            tag = self.node_get_member(tag_node, str, 'tag')
 +            commit_ref = self.node_get_member(tag_node, str, 'commit')
 +            annotated = self.node_get_member(tag_node, bool, 'annotated')
 +            tags.append((tag, commit_ref, annotated))
 +        return tags
++
 +    # Checks whether the plugin configuration has explicitly
 +    # configured this submodule to be ignored
 +    def _ignore_submodule(self, path):
 +        try:
 +            checkout = self.submodule_checkout_overrides[path]
 +        except KeyError:
 +            checkout = self.checkout_submodules
++
 +        return not checkout

buildstream/_project.py

@@ -677,8 +677,9 @@ class Project():
+     #
      def _find_project_dir(self, directory):
          workspace_element = None
 +        config_filenames = [_PROJECT_CONF_FILE, WORKSPACE_PROJECT_FILE]
          found_directory, filename = utils._search_upward_for_files(
 -            directory, [_PROJECT_CONF_FILE, WORKSPACE_PROJECT_FILE]
 +            directory, config_filenames
+         )
          if filename == _PROJECT_CONF_FILE:
              project_directory = found_directory
@@ -691,8 +692,8 @@ class Project():
          else:
              raise LoadError(
                  LoadErrorReason.MISSING_PROJECT_CONF,
 -                '{} not found in current directory or any of its parent directories'
 -                .format(_PROJECT_CONF_FILE))
 +                "None of {names} found in '{path}' or any of its parent directories"
 +                .format(names=config_filenames, path=directory))
          return project_directory, workspace_element

buildstream/_scheduler/queues/fetchqueue.py

@@ -40,10 +40,7 @@ class FetchQueue(Queue):
          self._skip_cached = skip_cached
      def process(self, element):
 -        previous_sources = []
 -        for source in element.sources():
 -            source._fetch(previous_sources)
 -            previous_sources.append(source)
 +        element._fetch()
      def status(self, element):
          # state of dependencies may have changed, recalculate element state

buildstream/_stream.py

@@ -28,6 +28,7 @@ import tarfile
  import tempfile
  from contextlib import contextmanager, suppress
 +from ._artifactcache import ArtifactCacheSpec
  from ._exceptions import StreamError, ImplError, BstError, set_last_task_error
  from ._message import Message, MessageType
  from ._scheduler import Scheduler, SchedStatus, TrackQueue, FetchQueue, BuildQueue, PullQueue, PushQueue
@@ -124,6 +125,7 @@ class Stream():
      #    mounts (list of HostMount): Additional directories to mount into the sandbox
      #    isolate (bool): Whether to isolate the environment like we do in builds
      #    command (list): An argv to launch in the sandbox, or None
 +    #    usebuildtree (bool): Wheather to use a buildtree as the source.
+     #
      # Returns:
      #    (int): The exit code of the launched shell
@@ -132,7 +134,8 @@ class Stream():
                directory=None,
                mounts=None,
                isolate=False,
 -              command=None):
 +              command=None,
 +              usebuildtree=False):
          # Assert we have everything we need built, unless the directory is specified
          # in which case we just blindly trust the directory, using the element
@@ -147,7 +150,8 @@ class Stream():
                  raise StreamError("Elements need to be built or downloaded before staging a shell environment",
                                    detail="\n".join(missing_deps))
 -        return element._shell(scope, directory, mounts=mounts, isolate=isolate, prompt=prompt, command=command)
 +        return element._shell(scope, directory, mounts=mounts, isolate=isolate, prompt=prompt, command=command,
 +                              usebuildtree=usebuildtree)
      # build()
+     #
@@ -305,6 +309,7 @@ class Stream():
                                   selection=selection,
                                   use_artifact_config=use_config,
                                   artifact_remote_url=remote,
 +                                 artifact_remote_can_push=False,
                                   fetch_subprojects=True)
          if not self._artifacts.has_fetch_remotes():
@@ -343,6 +348,7 @@ class Stream():
                                   selection=selection,
                                   use_artifact_config=use_config,
                                   artifact_remote_url=remote,
 +                                 artifact_remote_can_push=True,
                                   fetch_subprojects=True)
          if not self._artifacts.has_push_remotes():
@@ -849,7 +855,8 @@ class Stream():
      #    track_except_targets (list of str): Specified targets to except from fetching
      #    track_cross_junctions (bool): Whether tracking should cross junction boundaries
      #    use_artifact_config (bool): Whether to initialize artifacts with the config
 -    #    artifact_remote_url (bool): A remote url for initializing the artifacts
 +    #    artifact_remote_url (str): A remote url for initializing the artifacts
 +    #    artifact_remote_can_push (bool): Whether `artifact_remote_url` can be used to push
      #    fetch_subprojects (bool): Whether to fetch subprojects while loading
+     #
      # Returns:
@@ -864,6 +871,7 @@ class Stream():
                track_cross_junctions=False,
                use_artifact_config=False,
                artifact_remote_url=None,
 +              artifact_remote_can_push=False,
                fetch_subprojects=False,
                dynamic_plan=False):
@@ -927,12 +935,20 @@ class Stream():
              self._pipeline.resolve_elements(track_selected)
              return [], track_selected
 -        # ArtifactCache.setup_remotes expects all projects to be fully loaded
 -        for project in self._context.get_projects():
 -            project.ensure_fully_loaded()
+-
 +        if use_artifact_config:
 +            # ArtifactCache.get_remotes_from_projects expects all projects to be
 +            # fully loaded
 +            for project in self._context.get_projects():
 +                project.ensure_fully_loaded()
 +            remotes = self._artifacts.get_remotes_from_projects()
 +        else:
 +            # Build the ArtifactCacheSpec instance based on `--remote`
 +            remotes = [(
 +                [ArtifactCacheSpec(artifact_remote_url, push=artifact_remote_can_push)],
 +                None
 +            )]
          # Connect to remote caches, this needs to be done before resolving element state
 -        self._artifacts.setup_remotes(use_config=use_artifact_config, remote_url=artifact_remote_url)
 +        self._artifacts.setup_remotes(remotes=remotes)
          # Now move on to loading primary selection.
+         #
@@ -1260,7 +1276,7 @@ class Stream():
          required_list = []
          # If context is set to not pull buildtrees, or no fetch remotes, return empty list
 -        if not (self._context.pull_buildtrees or self._artifacts.has_fetch_remotes()):
 +        if not self._context.pull_buildtrees or not self._artifacts.has_fetch_remotes():
              return required_list
          for element in elements:

buildstream/data/userconfig.yaml

@@ -112,14 +112,6 @@ logging:
+ #
  prompt:
 -  # Whether to create a project with 'bst init' if we are invoked outside of a
 -  # directory where we can resolve the project.
 -  #
 -  #  ask - Prompt the user to choose.
 -  #  no  - Never create the project.
 -  #
 -  auto-init: ask
+-
    # Whether to really proceed with 'bst workspace close --remove-dir' removing
    # a workspace directory, potentially losing changes.
+   #

buildstream/element.py

@@ -1338,11 +1338,12 @@ class Element(Plugin):
      # is used to stage things by the `bst checkout` codepath
+     #
      @contextmanager
 -    def _prepare_sandbox(self, scope, directory, shell=False, integrate=True):
 +    def _prepare_sandbox(self, scope, directory, shell=False, integrate=True, usebuildtree=False):
          # bst shell and bst checkout require a local sandbox.
          bare_directory = True if directory else False
          with self.__sandbox(directory, config=self.__sandbox_config, allow_remote=False,
                              bare_directory=bare_directory) as sandbox:
 +            sandbox._usebuildtree = usebuildtree
              # Configure always comes first, and we need it.
              self.__configure_sandbox(sandbox)
@@ -1386,7 +1387,7 @@ class Element(Plugin):
          # Stage all sources that need to be copied
          sandbox_vroot = sandbox.get_virtual_directory()
          host_vdirectory = sandbox_vroot.descend(directory.lstrip(os.sep).split(os.sep), create=True)
 -        self._stage_sources_at(host_vdirectory, mount_workspaces=mount_workspaces)
 +        self._stage_sources_at(host_vdirectory, mount_workspaces=mount_workspaces, usebuildtree=sandbox._usebuildtree)
      # _stage_sources_at():
+     #
@@ -1395,10 +1396,10 @@ class Element(Plugin):
      # Args:
      #     vdirectory (:class:`.storage.Directory`): A virtual directory object to stage sources into.
      #     mount_workspaces (bool): mount workspaces if True, copy otherwise
 +    #     usebuildtree (bool): use a the elements build tree as its source.
+     #
 -    def _stage_sources_at(self, vdirectory, mount_workspaces=True):
 +    def _stage_sources_at(self, vdirectory, mount_workspaces=True, usebuildtree=False):
          with self.timed_activity("Staging sources", silent_nested=True):
+-
              if not isinstance(vdirectory, Directory):
                  vdirectory = FileBasedDirectory(vdirectory)
              if not vdirectory.is_empty():
@@ -1420,7 +1421,7 @@ class Element(Plugin):
                                                   .format(workspace.get_absolute_path())):
                              workspace.stage(temp_staging_directory)
                  # Check if we have a cached buildtree to use
 -                elif self._cached_buildtree():
 +                elif usebuildtree:
                      artifact_base, _ = self.__extract()
                      import_dir = os.path.join(artifact_base, 'buildtree')
                  else:
@@ -1850,13 +1851,15 @@ class Element(Plugin):
      #    isolate (bool): Whether to isolate the environment like we do in builds
      #    prompt (str): A suitable prompt string for PS1
      #    command (list): An argv to launch in the sandbox
 +    #    usebuildtree (bool): Use the buildtree as its source
+     #
      # Returns: Exit code
+     #
      # If directory is not specified, one will be staged using scope
 -    def _shell(self, scope=None, directory=None, *, mounts=None, isolate=False, prompt=None, command=None):
 +    def _shell(self, scope=None, directory=None, *, mounts=None, isolate=False, prompt=None, command=None,
 +               usebuildtree=False):
 -        with self._prepare_sandbox(scope, directory, shell=True) as sandbox:
 +        with self._prepare_sandbox(scope, directory, shell=True, usebuildtree=usebuildtree) as sandbox:
              environment = self.get_environment()
              environment = copy.copy(environment)
              flags = SandboxFlags.INTERACTIVE | SandboxFlags.ROOT_READ_ONLY
@@ -2019,6 +2022,20 @@ class Element(Plugin):
          return True
 +    # _fetch()
 +    #
 +    # Fetch the element's sources.
 +    #
 +    # Raises:
 +    #    SourceError: If one of the element sources has an error
 +    #
 +    def _fetch(self):
 +        previous_sources = []
 +        for source in self.sources():
 +            if source._get_consistency() < Consistency.CACHED:
 +                source._fetch(previous_sources)
 +            previous_sources.append(source)
++
      #############################################################
      #                   Private Local Methods                   #
      #############################################################

buildstream/plugins/sources/git.py

@@ -156,652 +156,11 @@ This plugin also utilises the following configurable :class:`core warnings <buil
    found in the provided track in the element's git repository.
  """
 -import os
 -import re
 -import shutil
 -from collections.abc import Mapping
 -from io import StringIO
 -from tempfile import TemporaryFile
 +from buildstream import _GitSourceBase
 -from configparser import RawConfigParser
 -from buildstream import Source, SourceError, Consistency, SourceFetcher, CoreWarnings
 -from buildstream import utils
 -from buildstream.utils import move_atomic, DirectoryExistsError
+-
 -GIT_MODULES = '.gitmodules'
+-
 -# Warnings
 -WARN_INCONSISTENT_SUBMODULE = "inconsistent-submodule"
 -WARN_UNLISTED_SUBMODULE = "unlisted-submodule"
 -WARN_INVALID_SUBMODULE = "invalid-submodule"
+-
+-
 -# Because of handling of submodules, we maintain a GitMirror
 -# for the primary git source and also for each submodule it
 -# might have at a given time
 -#
 -class GitMirror(SourceFetcher):
+-
 -    def __init__(self, source, path, url, ref, *, primary=False, tags=[]):
+-
 -        super().__init__()
 -        self.source = source
 -        self.path = path
 -        self.url = url
 -        self.ref = ref
 -        self.tags = tags
 -        self.primary = primary
 -        self.mirror = os.path.join(source.get_mirror_directory(), utils.url_directory_name(url))
 -        self.mark_download_url(url)
+-
 -    # Ensures that the mirror exists
 -    def ensure(self, alias_override=None):
+-
 -        # Unfortunately, git does not know how to only clone just a specific ref,
 -        # so we have to download all of those gigs even if we only need a couple
 -        # of bytes.
 -        if not os.path.exists(self.mirror):
+-
 -            # Do the initial clone in a tmpdir just because we want an atomic move
 -            # after a long standing clone which could fail overtime, for now do
 -            # this directly in our git directory, eliminating the chances that the
 -            # system configured tmpdir is not on the same partition.
 -            #
 -            with self.source.tempdir() as tmpdir:
 -                url = self.source.translate_url(self.url, alias_override=alias_override,
 -                                                primary=self.primary)
 -                self.source.call([self.source.host_git, 'clone', '--mirror', '-n', url, tmpdir],
 -                                 fail="Failed to clone git repository {}".format(url),
 -                                 fail_temporarily=True)
+-
 -                try:
 -                    move_atomic(tmpdir, self.mirror)
 -                except DirectoryExistsError:
 -                    # Another process was quicker to download this repository.
 -                    # Let's discard our own
 -                    self.source.status("{}: Discarding duplicate clone of {}"
 -                                       .format(self.source, url))
 -                except OSError as e:
 -                    raise SourceError("{}: Failed to move cloned git repository {} from '{}' to '{}': {}"
 -                                      .format(self.source, url, tmpdir, self.mirror, e)) from e
+-
 -    def _fetch(self, alias_override=None):
 -        url = self.source.translate_url(self.url,
 -                                        alias_override=alias_override,
 -                                        primary=self.primary)
+-
 -        if alias_override:
 -            remote_name = utils.url_directory_name(alias_override)
 -            _, remotes = self.source.check_output(
 -                [self.source.host_git, 'remote'],
 -                fail="Failed to retrieve list of remotes in {}".format(self.mirror),
 -                cwd=self.mirror
 -            )
 -            if remote_name not in remotes:
 -                self.source.call(
 -                    [self.source.host_git, 'remote', 'add', remote_name, url],
 -                    fail="Failed to add remote {} with url {}".format(remote_name, url),
 -                    cwd=self.mirror
 -                )
 -        else:
 -            remote_name = "origin"
+-
 -        self.source.call([self.source.host_git, 'fetch', remote_name, '--prune', '--force', '--tags'],
 -                         fail="Failed to fetch from remote git repository: {}".format(url),
 -                         fail_temporarily=True,
 -                         cwd=self.mirror)
+-
 -    def fetch(self, alias_override=None):
 -        # Resolve the URL for the message
 -        resolved_url = self.source.translate_url(self.url,
 -                                                 alias_override=alias_override,
 -                                                 primary=self.primary)
+-
 -        with self.source.timed_activity("Fetching from {}"
 -                                        .format(resolved_url),
 -                                        silent_nested=True):
 -            self.ensure(alias_override)
 -            if not self.has_ref():
 -                self._fetch(alias_override)
 -            self.assert_ref()
+-
 -    def has_ref(self):
 -        if not self.ref:
 -            return False
+-
 -        # If the mirror doesnt exist, we also dont have the ref
 -        if not os.path.exists(self.mirror):
 -            return False
+-
 -        # Check if the ref is really there
 -        rc = self.source.call([self.source.host_git, 'cat-file', '-t', self.ref], cwd=self.mirror)
 -        return rc == 0
+-
 -    def assert_ref(self):
 -        if not self.has_ref():
 -            raise SourceError("{}: expected ref '{}' was not found in git repository: '{}'"
 -                              .format(self.source, self.ref, self.url))
+-
 -    def latest_commit_with_tags(self, tracking, track_tags=False):
 -        _, output = self.source.check_output(
 -            [self.source.host_git, 'rev-parse', tracking],
 -            fail="Unable to find commit for specified branch name '{}'".format(tracking),
 -            cwd=self.mirror)
 -        ref = output.rstrip('\n')
+-
 -        if self.source.ref_format == 'git-describe':
 -            # Prefix the ref with the closest tag, if available,
 -            # to make the ref human readable
 -            exit_code, output = self.source.check_output(
 -                [self.source.host_git, 'describe', '--tags', '--abbrev=40', '--long', ref],
 -                cwd=self.mirror)
 -            if exit_code == 0:
 -                ref = output.rstrip('\n')
+-
 -        if not track_tags:
 -            return ref, []
+-
 -        tags = set()
 -        for options in [[], ['--first-parent'], ['--tags'], ['--tags', '--first-parent']]:
 -            exit_code, output = self.source.check_output(
 -                [self.source.host_git, 'describe', '--abbrev=0', ref] + options,
 -                cwd=self.mirror)
 -            if exit_code == 0:
 -                tag = output.strip()
 -                _, commit_ref = self.source.check_output(
 -                    [self.source.host_git, 'rev-parse', tag + '^{commit}'],
 -                    fail="Unable to resolve tag '{}'".format(tag),
 -                    cwd=self.mirror)
 -                exit_code = self.source.call(
 -                    [self.source.host_git, 'cat-file', 'tag', tag],
 -                    cwd=self.mirror)
 -                annotated = (exit_code == 0)
+-
 -                tags.add((tag, commit_ref.strip(), annotated))
+-
 -        return ref, list(tags)
+-
 -    def stage(self, directory):
 -        fullpath = os.path.join(directory, self.path)
+-
 -        # Using --shared here avoids copying the objects into the checkout, in any
 -        # case we're just checking out a specific commit and then removing the .git/
 -        # directory.
 -        self.source.call([self.source.host_git, 'clone', '--no-checkout', '--shared', self.mirror, fullpath],
 -                         fail="Failed to create git mirror {} in directory: {}".format(self.mirror, fullpath),
 -                         fail_temporarily=True)
+-
 -        self.source.call([self.source.host_git, 'checkout', '--force', self.ref],
 -                         fail="Failed to checkout git ref {}".format(self.ref),
 -                         cwd=fullpath)
+-
 -        # Remove .git dir
 -        shutil.rmtree(os.path.join(fullpath, ".git"))
+-
 -        self._rebuild_git(fullpath)
+-
 -    def init_workspace(self, directory):
 -        fullpath = os.path.join(directory, self.path)
 -        url = self.source.translate_url(self.url)
+-
 -        self.source.call([self.source.host_git, 'clone', '--no-checkout', self.mirror, fullpath],
 -                         fail="Failed to clone git mirror {} in directory: {}".format(self.mirror, fullpath),
 -                         fail_temporarily=True)
+-
 -        self.source.call([self.source.host_git, 'remote', 'set-url', 'origin', url],
 -                         fail='Failed to add remote origin "{}"'.format(url),
 -                         cwd=fullpath)
+-
 -        self.source.call([self.source.host_git, 'checkout', '--force', self.ref],
 -                         fail="Failed to checkout git ref {}".format(self.ref),
 -                         cwd=fullpath)
+-
 -    # List the submodules (path/url tuples) present at the given ref of this repo
 -    def submodule_list(self):
 -        modules = "{}:{}".format(self.ref, GIT_MODULES)
 -        exit_code, output = self.source.check_output(
 -            [self.source.host_git, 'show', modules], cwd=self.mirror)
+-
 -        # If git show reports error code 128 here, we take it to mean there is
 -        # no .gitmodules file to display for the given revision.
 -        if exit_code == 128:
 -            return
 -        elif exit_code != 0:
 -            raise SourceError(
 -                "{plugin}: Failed to show gitmodules at ref {ref}".format(
 -                    plugin=self, ref=self.ref))
+-
 -        content = '\n'.join([l.strip() for l in output.splitlines()])
+-
 -        io = StringIO(content)
 -        parser = RawConfigParser()
 -        parser.read_file(io)
+-
 -        for section in parser.sections():
 -            # validate section name against the 'submodule "foo"' pattern
 -            if re.match(r'submodule "(.*)"', section):
 -                path = parser.get(section, 'path')
 -                url = parser.get(section, 'url')
+-
 -                yield (path, url)
+-
 -    # Fetch the ref which this mirror requires its submodule to have,
 -    # at the given ref of this mirror.
 -    def submodule_ref(self, submodule, ref=None):
 -        if not ref:
 -            ref = self.ref
+-
 -        # list objects in the parent repo tree to find the commit
 -        # object that corresponds to the submodule
 -        _, output = self.source.check_output([self.source.host_git, 'ls-tree', ref, submodule],
 -                                             fail="ls-tree failed for commit {} and submodule: {}".format(
 -                                                 ref, submodule),
 -                                             cwd=self.mirror)
+-
 -        # read the commit hash from the output
 -        fields = output.split()
 -        if len(fields) >= 2 and fields[1] == 'commit':
 -            submodule_commit = output.split()[2]
+-
 -            # fail if the commit hash is invalid
 -            if len(submodule_commit) != 40:
 -                raise SourceError("{}: Error reading commit information for submodule '{}'"
 -                                  .format(self.source, submodule))
+-
 -            return submodule_commit
+-
 -        else:
 -            detail = "The submodule '{}' is defined either in the BuildStream source\n".format(submodule) + \
 -                     "definition, or in a .gitmodules file. But the submodule was never added to the\n" + \
 -                     "underlying git repository with `git submodule add`."
+-
 -            self.source.warn("{}: Ignoring inconsistent submodule '{}'"
 -                             .format(self.source, submodule), detail=detail,
 -                             warning_token=WARN_INCONSISTENT_SUBMODULE)
+-
 -            return None
+-
 -    def _rebuild_git(self, fullpath):
 -        if not self.tags:
 -            return
+-
 -        with self.source.tempdir() as tmpdir:
 -            included = set()
 -            shallow = set()
 -            for _, commit_ref, _ in self.tags:
+-
 -                _, out = self.source.check_output([self.source.host_git, 'rev-list',
 -                                                   '--boundary', '{}..{}'.format(commit_ref, self.ref)],
 -                                                  fail="Failed to get git history {}..{} in directory: {}"
 -                                                  .format(commit_ref, self.ref, fullpath),
 -                                                  fail_temporarily=True,
 -                                                  cwd=self.mirror)
 -                for line in out.splitlines():
 -                    rev = line.lstrip('-')
 -                    if line[0] == '-':
 -                        shallow.add(rev)
 -                    else:
 -                        included.add(rev)
+-
 -            shallow -= included
 -            included |= shallow
+-
 -            self.source.call([self.source.host_git, 'init'],
 -                             fail="Cannot initialize git repository: {}".format(fullpath),
 -                             cwd=fullpath)
+-
 -            for rev in included:
 -                with TemporaryFile(dir=tmpdir) as commit_file:
 -                    self.source.call([self.source.host_git, 'cat-file', 'commit', rev],
 -                                     stdout=commit_file,
 -                                     fail="Failed to get commit {}".format(rev),
 -                                     cwd=self.mirror)
 -                    commit_file.seek(0, 0)
 -                    self.source.call([self.source.host_git, 'hash-object', '-w', '-t', 'commit', '--stdin'],
 -                                     stdin=commit_file,
 -                                     fail="Failed to add commit object {}".format(rev),
 -                                     cwd=fullpath)
+-
 -            with open(os.path.join(fullpath, '.git', 'shallow'), 'w') as shallow_file:
 -                for rev in shallow:
 -                    shallow_file.write('{}\n'.format(rev))
+-
 -            for tag, commit_ref, annotated in self.tags:
 -                if annotated:
 -                    with TemporaryFile(dir=tmpdir) as tag_file:
 -                        tag_data = 'object {}\ntype commit\ntag {}\n'.format(commit_ref, tag)
 -                        tag_file.write(tag_data.encode('ascii'))
 -                        tag_file.seek(0, 0)
 -                        _, tag_ref = self.source.check_output(
 -                            [self.source.host_git, 'hash-object', '-w', '-t',
 -                             'tag', '--stdin'],
 -                            stdin=tag_file,
 -                            fail="Failed to add tag object {}".format(tag),
 -                            cwd=fullpath)
+-
 -                    self.source.call([self.source.host_git, 'tag', tag, tag_ref.strip()],
 -                                     fail="Failed to tag: {}".format(tag),
 -                                     cwd=fullpath)
 -                else:
 -                    self.source.call([self.source.host_git, 'tag', tag, commit_ref],
 -                                     fail="Failed to tag: {}".format(tag),
 -                                     cwd=fullpath)
+-
 -            with open(os.path.join(fullpath, '.git', 'HEAD'), 'w') as head:
 -                self.source.call([self.source.host_git, 'rev-parse', self.ref],
 -                                 stdout=head,
 -                                 fail="Failed to parse commit {}".format(self.ref),
 -                                 cwd=self.mirror)
+-
+-
 -class GitSource(Source):
 -    # pylint: disable=attribute-defined-outside-init
+-
 -    def configure(self, node):
 -        ref = self.node_get_member(node, str, 'ref', None)
+-
 -        config_keys = ['url', 'track', 'ref', 'submodules',
 -                       'checkout-submodules', 'ref-format',
 -                       'track-tags', 'tags']
 -        self.node_validate(node, config_keys + Source.COMMON_CONFIG_KEYS)
+-
 -        tags_node = self.node_get_member(node, list, 'tags', [])
 -        for tag_node in tags_node:
 -            self.node_validate(tag_node, ['tag', 'commit', 'annotated'])
+-
 -        tags = self._load_tags(node)
 -        self.track_tags = self.node_get_member(node, bool, 'track-tags', False)
+-
 -        self.original_url = self.node_get_member(node, str, 'url')
 -        self.mirror = GitMirror(self, '', self.original_url, ref, tags=tags, primary=True)
 -        self.tracking = self.node_get_member(node, str, 'track', None)
+-
 -        self.ref_format = self.node_get_member(node, str, 'ref-format', 'sha1')
 -        if self.ref_format not in ['sha1', 'git-describe']:
 -            provenance = self.node_provenance(node, member_name='ref-format')
 -            raise SourceError("{}: Unexpected value for ref-format: {}".format(provenance, self.ref_format))
+-
 -        # At this point we now know if the source has a ref and/or a track.
 -        # If it is missing both then we will be unable to track or build.
 -        if self.mirror.ref is None and self.tracking is None:
 -            raise SourceError("{}: Git sources require a ref and/or track".format(self),
 -                              reason="missing-track-and-ref")
+-
 -        self.checkout_submodules = self.node_get_member(node, bool, 'checkout-submodules', True)
 -        self.submodules = []
+-
 -        # Parse a dict of submodule overrides, stored in the submodule_overrides
 -        # and submodule_checkout_overrides dictionaries.
 -        self.submodule_overrides = {}
 -        self.submodule_checkout_overrides = {}
 -        modules = self.node_get_member(node, Mapping, 'submodules', {})
 -        for path, _ in self.node_items(modules):
 -            submodule = self.node_get_member(modules, Mapping, path)
 -            url = self.node_get_member(submodule, str, 'url', None)
+-
 -            # Make sure to mark all URLs that are specified in the configuration
 -            if url:
 -                self.mark_download_url(url, primary=False)
+-
 -            self.submodule_overrides[path] = url
 -            if 'checkout' in submodule:
 -                checkout = self.node_get_member(submodule, bool, 'checkout')
 -                self.submodule_checkout_overrides[path] = checkout
+-
 -        self.mark_download_url(self.original_url)
+-
 -    def preflight(self):
 -        # Check if git is installed, get the binary at the same time
 -        self.host_git = utils.get_host_tool('git')
+-
 -    def get_unique_key(self):
 -        # Here we want to encode the local name of the repository and
 -        # the ref, if the user changes the alias to fetch the same sources
 -        # from another location, it should not affect the cache key.
 -        key = [self.original_url, self.mirror.ref]
 -        if self.mirror.tags:
 -            tags = {tag: (commit, annotated) for tag, commit, annotated in self.mirror.tags}
 -            key.append({'tags': tags})
+-
 -        # Only modify the cache key with checkout_submodules if it's something
 -        # other than the default behaviour.
 -        if self.checkout_submodules is False:
 -            key.append({"checkout_submodules": self.checkout_submodules})
+-
 -        # We want the cache key to change if the source was
 -        # configured differently, and submodules count.
 -        if self.submodule_overrides:
 -            key.append(self.submodule_overrides)
+-
 -        if self.submodule_checkout_overrides:
 -            key.append({"submodule_checkout_overrides": self.submodule_checkout_overrides})
+-
 -        return key
+-
 -    def get_consistency(self):
 -        if self.have_all_refs():
 -            return Consistency.CACHED
 -        elif self.mirror.ref is not None:
 -            return Consistency.RESOLVED
 -        return Consistency.INCONSISTENT
+-
 -    def load_ref(self, node):
 -        self.mirror.ref = self.node_get_member(node, str, 'ref', None)
 -        self.mirror.tags = self._load_tags(node)
+-
 -    def get_ref(self):
 -        return self.mirror.ref, self.mirror.tags
+-
 -    def set_ref(self, ref_data, node):
 -        if not ref_data:
 -            self.mirror.ref = None
 -            if 'ref' in node:
 -                del node['ref']
 -            self.mirror.tags = []
 -            if 'tags' in node:
 -                del node['tags']
 -        else:
 -            ref, tags = ref_data
 -            node['ref'] = self.mirror.ref = ref
 -            self.mirror.tags = tags
 -            if tags:
 -                node['tags'] = []
 -                for tag, commit_ref, annotated in tags:
 -                    data = {'tag': tag,
 -                            'commit': commit_ref,
 -                            'annotated': annotated}
 -                    node['tags'].append(data)
 -            else:
 -                if 'tags' in node:
 -                    del node['tags']
+-
 -    def track(self):
+-
 -        # If self.tracking is not specified it's not an error, just silently return
 -        if not self.tracking:
 -            # Is there a better way to check if a ref is given.
 -            if self.mirror.ref is None:
 -                detail = 'Without a tracking branch ref can not be updated. Please ' + \
 -                         'provide a ref or a track.'
 -                raise SourceError("{}: No track or ref".format(self),
 -                                  detail=detail, reason="track-attempt-no-track")
 -            return None
+-
 -        # Resolve the URL for the message
 -        resolved_url = self.translate_url(self.mirror.url)
 -        with self.timed_activity("Tracking {} from {}"
 -                                 .format(self.tracking, resolved_url),
 -                                 silent_nested=True):
 -            self.mirror.ensure()
 -            self.mirror._fetch()
+-
 -            # Update self.mirror.ref and node.ref from the self.tracking branch
 -            ret = self.mirror.latest_commit_with_tags(self.tracking, self.track_tags)
+-
 -        return ret
+-
 -    def init_workspace(self, directory):
 -        # XXX: may wish to refactor this as some code dupe with stage()
 -        self.refresh_submodules()
+-
 -        with self.timed_activity('Setting up workspace "{}"'.format(directory), silent_nested=True):
 -            self.mirror.init_workspace(directory)
 -            for mirror in self.submodules:
 -                mirror.init_workspace(directory)
+-
 -    def stage(self, directory):
+-
 -        # Need to refresh submodule list here again, because
 -        # it's possible that we did not load in the main process
 -        # with submodules present (source needed fetching) and
 -        # we may not know about the submodule yet come time to build.
 -        #
 -        self.refresh_submodules()
+-
 -        # Stage the main repo in the specified directory
 -        #
 -        with self.timed_activity("Staging {}".format(self.mirror.url), silent_nested=True):
 -            self.mirror.stage(directory)
 -            for mirror in self.submodules:
 -                mirror.stage(directory)
+-
 -    def get_source_fetchers(self):
 -        yield self.mirror
 -        self.refresh_submodules()
 -        for submodule in self.submodules:
 -            yield submodule
+-
 -    def validate_cache(self):
 -        discovered_submodules = {}
 -        unlisted_submodules = []
 -        invalid_submodules = []
+-
 -        for path, url in self.mirror.submodule_list():
 -            discovered_submodules[path] = url
 -            if self.ignore_submodule(path):
 -                continue
+-
 -            override_url = self.submodule_overrides.get(path)
 -            if not override_url:
 -                unlisted_submodules.append((path, url))
+-
 -        # Warn about submodules which are explicitly configured but do not exist
 -        for path, url in self.submodule_overrides.items():
 -            if path not in discovered_submodules:
 -                invalid_submodules.append((path, url))
+-
 -        if invalid_submodules:
 -            detail = []
 -            for path, url in invalid_submodules:
 -                detail.append("  Submodule URL '{}' at path '{}'".format(url, path))
+-
 -            self.warn("{}: Invalid submodules specified".format(self),
 -                      warning_token=WARN_INVALID_SUBMODULE,
 -                      detail="The following submodules are specified in the source "
 -                      "description but do not exist according to the repository\n\n" +
 -                      "\n".join(detail))
+-
 -        # Warn about submodules which exist but have not been explicitly configured
 -        if unlisted_submodules:
 -            detail = []
 -            for path, url in unlisted_submodules:
 -                detail.append("  Submodule URL '{}' at path '{}'".format(url, path))
+-
 -            self.warn("{}: Unlisted submodules exist".format(self),
 -                      warning_token=WARN_UNLISTED_SUBMODULE,
 -                      detail="The following submodules exist but are not specified " +
 -                      "in the source description\n\n" +
 -                      "\n".join(detail))
+-
 -        # Assert that the ref exists in the track tag/branch, if track has been specified.
 -        ref_in_track = False
 -        if self.tracking:
 -            _, branch = self.check_output([self.host_git, 'branch', '--list', self.tracking,
 -                                           '--contains', self.mirror.ref],
 -                                          cwd=self.mirror.mirror)
 -            if branch:
 -                ref_in_track = True
 -            else:
 -                _, tag = self.check_output([self.host_git, 'tag', '--list', self.tracking,
 -                                            '--contains', self.mirror.ref],
 -                                           cwd=self.mirror.mirror)
 -                if tag:
 -                    ref_in_track = True
+-
 -            if not ref_in_track:
 -                detail = "The ref provided for the element does not exist locally " + \
 -                         "in the provided track branch / tag '{}'.\n".format(self.tracking) + \
 -                         "You may wish to track the element to update the ref from '{}' ".format(self.tracking) + \
 -                         "with `bst track`,\n" + \
 -                         "or examine the upstream at '{}' for the specific ref.".format(self.mirror.url)
+-
 -                self.warn("{}: expected ref '{}' was not found in given track '{}' for staged repository: '{}'\n"
 -                          .format(self, self.mirror.ref, self.tracking, self.mirror.url),
 -                          detail=detail, warning_token=CoreWarnings.REF_NOT_IN_TRACK)
+-
 -    ###########################################################
 -    #                     Local Functions                     #
 -    ###########################################################
 -    def have_all_refs(self):
 -        if not self.mirror.has_ref():
 -            return False
+-
 -        self.refresh_submodules()
 -        for mirror in self.submodules:
 -            if not os.path.exists(mirror.mirror):
 -                return False
 -            if not mirror.has_ref():
 -                return False
+-
 -        return True
+-
 -    # Refreshes the GitMirror objects for submodules
 -    #
 -    # Assumes that we have our mirror and we have the ref which we point to
 -    #
 -    def refresh_submodules(self):
 -        self.mirror.ensure()
 -        submodules = []
+-
 -        for path, url in self.mirror.submodule_list():
+-
 -            # Completely ignore submodules which are disabled for checkout
 -            if self.ignore_submodule(path):
 -                continue
+-
 -            # Allow configuration to override the upstream
 -            # location of the submodules.
 -            override_url = self.submodule_overrides.get(path)
 -            if override_url:
 -                url = override_url
+-
 -            ref = self.mirror.submodule_ref(path)
 -            if ref is not None:
 -                mirror = GitMirror(self, path, url, ref)
 -                submodules.append(mirror)
+-
 -        self.submodules = submodules
+-
 -    def _load_tags(self, node):
 -        tags = []
 -        tags_node = self.node_get_member(node, list, 'tags', [])
 -        for tag_node in tags_node:
 -            tag = self.node_get_member(tag_node, str, 'tag')
 -            commit_ref = self.node_get_member(tag_node, str, 'commit')
 -            annotated = self.node_get_member(tag_node, bool, 'annotated')
 -            tags.append((tag, commit_ref, annotated))
 -        return tags
+-
 -    # Checks whether the plugin configuration has explicitly
 -    # configured this submodule to be ignored
 -    def ignore_submodule(self, path):
 -        try:
 -            checkout = self.submodule_checkout_overrides[path]
 -        except KeyError:
 -            checkout = self.checkout_submodules
+-
 -        return not checkout
 +class GitSource(_GitSourceBase):
 +    pass
  # Plugin entry point

buildstream/sandbox/sandbox.py

@@ -146,6 +146,7 @@ class Sandbox():
          self._output_directory = None
          self._vdir = None
 +        self._usebuildtree = False
          # This is set if anyone requests access to the underlying
          # directory via get_directory.

buildstream/utils.py

@@ -1168,7 +1168,7 @@ def _call(*popenargs, terminate=False, **kwargs):
+ #
  def _glob2re(pat):
      i, n = 0, len(pat)
 -    res = ''
 +    res = '(?ms)'
      while i < n:
          c = pat[i]
          i = i + 1
@@ -1205,7 +1205,7 @@ def _glob2re(pat):
                  res = '{}[{}]'.format(res, stuff)
          else:
              res = res + re.escape(c)
 -    return res + r'\Z(?ms)'
 +    return res + r'\Z'
  # _deduplicate()

tests/artifactcache/pull.py

@@ -146,7 +146,8 @@ def _test_pull(user_config_file, project_dir, artifact_dir,
      element = project.load_elements([element_name])[0]
      # Manually setup the CAS remote
 -    cas.setup_remotes(use_config=True)
 +    remotes = cas.get_remotes_from_projects()
 +    cas.setup_remotes(remotes=remotes)
      if cas.has_push_remotes(element=element):
          # Push the element's artifact
@@ -284,7 +285,8 @@ def _test_push_tree(user_config_file, project_dir, artifact_dir, artifact_digest
      cas = artifactcache.cas
      # Manually setup the CAS remote
 -    artifactcache.setup_remotes(use_config=True)
 +    remotes = artifactcache.get_remotes_from_projects()
 +    artifactcache.setup_remotes(remotes=remotes)
      if artifactcache.has_push_remotes():
          directory = remote_execution_pb2.Directory()
@@ -319,7 +321,8 @@ def _test_pull_tree(user_config_file, project_dir, artifact_dir, artifact_digest
      cas = context.artifactcache
      # Manually setup the CAS remote
 -    cas.setup_remotes(use_config=True)
 +    remotes = cas.get_remotes_from_projects()
 +    cas.setup_remotes(remotes=remotes)
      if cas.has_push_remotes():
          # Pull the artifact using the Tree object

tests/artifactcache/push.py

@@ -125,8 +125,8 @@ def _test_push(user_config_file, project_dir, artifact_dir,
      element = project.load_elements([element_name])[0]
      # Manually setup the CAS remote
 -    cas.setup_remotes(use_config=True)
 -    cas.initialize_remotes()
 +    remotes = cas.get_remotes_from_projects()
 +    cas.setup_remotes(remotes=remotes)
      if cas.has_push_remotes(element=element):
          # Push the element's artifact
@@ -185,8 +185,8 @@ def test_push_directory(cli, tmpdir, datafiles):
          assert artifactcache.contains(element, element_key)
          # Manually setup the CAS remote
 -        artifactcache.setup_remotes(use_config=True)
 -        artifactcache.initialize_remotes()
 +        remotes = artifactcache.get_remotes_from_projects()
 +        artifactcache.setup_remotes(remotes=remotes)
          assert artifactcache.has_push_remotes(element=element)
          # Recreate the CasBasedDirectory object from the cached artifact
@@ -231,8 +231,8 @@ def _test_push_directory(user_config_file, project_dir, artifact_dir, artifact_d
      cas = context.artifactcache
      # Manually setup the CAS remote
 -    cas.setup_remotes(use_config=True)
 -    cas.initialize_remotes()
 +    remotes = cas.get_remotes_from_projects()
 +    cas.setup_remotes(remotes=remotes)
      if cas.has_push_remotes():
          # Create a CasBasedDirectory from local CAS cache content
@@ -307,8 +307,8 @@ def _test_push_message(user_config_file, project_dir, artifact_dir, queue):
      cas = context.artifactcache
      # Manually setup the CAS remote
 -    cas.setup_remotes(use_config=True)
 -    cas.initialize_remotes()
 +    remotes = cas.get_remotes_from_projects()
 +    cas.setup_remotes(remotes=remotes)
      if cas.has_push_remotes():
          # Create an example message object

tests/frontend/logging.py

@@ -41,7 +41,7 @@ def test_default_logging(cli, tmpdir, datafiles):
      result = cli.run(project=project, args=['source', 'fetch', element_name])
      result.assert_success()
 -    m = re.search("\[\d\d:\d\d:\d\d\]\[\]\[\] SUCCESS Checking sources", result.stderr)
 +    m = re.search(r"\[\d\d:\d\d:\d\d\]\[\]\[\] SUCCESS Checking sources", result.stderr)
      assert(m is not None)
@@ -77,7 +77,7 @@ def test_custom_logging(cli, tmpdir, datafiles):
      result = cli.run(project=project, args=['source', 'fetch', element_name])
      result.assert_success()
 -    m = re.search("\d\d:\d\d:\d\d,\d\d:\d\d:\d\d.\d{6},\d\d:\d\d:\d\d,,,SUCCESS,Checking sources", result.stderr)
 +    m = re.search(r"\d\d:\d\d:\d\d,\d\d:\d\d:\d\d.\d{6},\d\d:\d\d:\d\d,,,SUCCESS,Checking sources", result.stderr)
      assert(m is not None)

tests/integration/build-tree.py

@@ -19,7 +19,9 @@ DATA_DIR = os.path.join(
  @pytest.mark.datafiles(DATA_DIR)
  @pytest.mark.skipif(IS_LINUX and not HAVE_BWRAP, reason='Only available with bubblewrap on Linux')
  def test_buildtree_staged(cli_integration, tmpdir, datafiles):
 -    # i.e. tests that cached build trees are staged by `bst shell --build`
 +    # We can only test the non interacitve case
 +    # The non interactive case defaults to not using buildtrees
 +    # for `bst shell --build`
      project = os.path.join(datafiles.dirname, datafiles.basename)
      element_name = 'build-shell/buildtree.bst'
@@ -27,15 +29,67 @@ def test_buildtree_staged(cli_integration, tmpdir, datafiles):
      res.assert_success()
      res = cli_integration.run(project=project, args=[
 -        'shell', '--build', element_name, '--', 'grep', '-q', 'Hi', 'test'
 +        'shell', '--build', element_name, '--', 'cat', 'test'
 +    ])
 +    res.assert_shell_error()
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.skipif(IS_LINUX and not HAVE_BWRAP, reason='Only available with bubblewrap on Linux')
 +def test_buildtree_staged_forced_true(cli_integration, tmpdir, datafiles):
 +    # Test that if we ask for a build tree it is there.
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    element_name = 'build-shell/buildtree.bst'
++
 +    res = cli_integration.run(project=project, args=['build', element_name])
 +    res.assert_success()
++
 +    res = cli_integration.run(project=project, args=[
 +        'shell', '--build', '--use-buildtree', 'always', element_name, '--', 'cat', 'test'
 +    ])
 +    res.assert_success()
 +    assert 'Hi' in res.output
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.skipif(IS_LINUX and not HAVE_BWRAP, reason='Only available with bubblewrap on Linux')
 +def test_buildtree_staged_if_available(cli_integration, tmpdir, datafiles):
 +    # Test that a build tree can be correctly detected.
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    element_name = 'build-shell/buildtree.bst'
++
 +    res = cli_integration.run(project=project, args=['build', element_name])
 +    res.assert_success()
++
 +    res = cli_integration.run(project=project, args=[
 +        'shell', '--build', '--use-buildtree', 'try', element_name, '--', 'cat', 'test'
      ])
      res.assert_success()
 +    assert 'Hi' in res.output
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.skipif(IS_LINUX and not HAVE_BWRAP, reason='Only available with bubblewrap on Linux')
 +def test_buildtree_staged_forced_false(cli_integration, tmpdir, datafiles):
 +    # Test that if we ask not to have a build tree it is not there
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    element_name = 'build-shell/buildtree.bst'
++
 +    res = cli_integration.run(project=project, args=['build', element_name])
 +    res.assert_success()
++
 +    res = cli_integration.run(project=project, args=[
 +        'shell', '--build', '--use-buildtree', 'never', element_name, '--', 'cat', 'test'
 +    ])
 +    res.assert_shell_error()
++
 +    assert 'Hi' not in res.output
  @pytest.mark.datafiles(DATA_DIR)
  @pytest.mark.skipif(IS_LINUX and not HAVE_BWRAP, reason='Only available with bubblewrap on Linux')
  def test_buildtree_from_failure(cli_integration, tmpdir, datafiles):
 -    # i.e. test that on a build failure, we can still shell into it
 +    # Test that we can use a build tree after a failure
      project = os.path.join(datafiles.dirname, datafiles.basename)
      element_name = 'build-shell/buildtree-fail.bst'
@@ -44,9 +98,10 @@ def test_buildtree_from_failure(cli_integration, tmpdir, datafiles):
      # Assert that file has expected contents
      res = cli_integration.run(project=project, args=[
 -        'shell', '--build', element_name, '--', 'cat', 'test'
 +        'shell', '--build', element_name, '--use-buildtree', 'always', '--', 'cat', 'test'
      ])
      res.assert_success()
 +    assert "Warning: using a buildtree from a failed build" in res.output
      assert 'Hi' in res.output
@@ -80,6 +135,65 @@ def test_buildtree_pulled(cli, tmpdir, datafiles):
          # Check it's using the cached build tree
          res = cli.run(project=project, args=[
 -            'shell', '--build', element_name, '--', 'grep', '-q', 'Hi', 'test'
 +            'shell', '--build', element_name, '--use-buildtree', 'always', '--', 'cat', 'test'
          ])
          res.assert_success()
++
++
 +# This test checks for correct behaviour if a buildtree is not present.
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.skipif(IS_LINUX and not HAVE_BWRAP, reason='Only available with bubblewrap on Linux')
 +def test_buildtree_options(cli, tmpdir, datafiles):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    element_name = 'build-shell/buildtree.bst'
++
 +    with create_artifact_share(os.path.join(str(tmpdir), 'artifactshare')) as share:
 +        # Build the element to push it to cache
 +        cli.configure({
 +            'artifacts': {'url': share.repo, 'push': True}
 +        })
 +        result = cli.run(project=project, args=['build', element_name])
 +        result.assert_success()
 +        assert cli.get_element_state(project, element_name) == 'cached'
++
 +        # Discard the cache
 +        cli.configure({
 +            'artifacts': {'url': share.repo, 'push': True},
 +            'artifactdir': os.path.join(cli.directory, 'artifacts2')
 +        })
 +        assert cli.get_element_state(project, element_name) != 'cached'
++
 +        # Pull from cache, but do not include buildtrees.
 +        result = cli.run(project=project, args=['pull', '--deps', 'all', element_name])
 +        result.assert_success()
++
 +        # The above is the simplest way I know to create a local cache without any buildtrees.
++
 +        # Check it's not using the cached build tree
 +        res = cli.run(project=project, args=[
 +            'shell', '--build', element_name, '--use-buildtree', 'never', '--', 'cat', 'test'
 +        ])
 +        res.assert_shell_error()
 +        assert 'Hi' not in res.output
++
 +        # Check it's not correctly handling the lack of buildtree
 +        res = cli.run(project=project, args=[
 +            'shell', '--build', element_name, '--use-buildtree', 'try', '--', 'cat', 'test'
 +        ])
 +        res.assert_shell_error()
 +        assert 'Hi' not in res.output
++
 +        # Check it's not using the cached build tree, default is to ask, and fall back to not
 +        # for non interactive behavior
 +        res = cli.run(project=project, args=[
 +            'shell', '--build', element_name, '--', 'cat', 'test'
 +        ])
 +        res.assert_shell_error()
 +        assert 'Hi' not in res.output
++
 +        # Check it's using the cached build tree
 +        res = cli.run(project=project, args=[
 +            'shell', '--build', element_name, '--use-buildtree', 'always', '--', 'cat', 'test'
 +        ])
 +        res.assert_main_error(ErrorDomain.PROG_NOT_FOUND, None)
 +        assert 'Hi' not in res.output

tests/sources/no-fetch-cached/files/file

+Hello World!

tests/sources/no-fetch-cached/plugins/sources/always_cached.py

 +"""
 +always_cached
 +=============
++
 +This is a test source plugin that is always cached.
 +Used to test that BuildStream core does not call fetch() for cached sources.
++
 +"""
++
 +from buildstream import Consistency, Source
++
++
 +class AlwaysCachedSource(Source):
++
 +    def configure(self, node):
 +        pass
++
 +    def preflight(self):
 +        pass
++
 +    def get_unique_key(self):
 +        return None
++
 +    def get_consistency(self):
 +        return Consistency.CACHED
++
 +    def load_ref(self, node):
 +        pass
++
 +    def get_ref(self):
 +        return None
++
 +    def set_ref(self, ref, node):
 +        pass
++
 +    def fetch(self):
 +        # Source is always cached, so fetch() should never be called
 +        assert False
++
 +    def stage(self, directory):
 +        pass
++
++
 +def setup():
 +    return AlwaysCachedSource

tests/sources/no-fetch-cached/project.conf

 +# Project with local source plugins
 +name: no-fetch-cached
++
 +plugins:
 +- origin: local
 +  path: plugins/sources
 +  sources:
 +    always_cached: 0

tests/sources/no_fetch_cached.py

 +import os
 +import pytest
++
 +from buildstream import _yaml
++
 +from tests.testutils import cli, create_repo
 +from tests.testutils.site import HAVE_GIT
++
 +DATA_DIR = os.path.join(
 +    os.path.dirname(os.path.realpath(__file__)),
 +    'no-fetch-cached'
 +)
++
++
 +##################################################################
 +#                              Tests                             #
 +##################################################################
 +# Test that fetch() is not called for cached sources
 +@pytest.mark.skipif(HAVE_GIT is False, reason="git is not available")
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_no_fetch_cached(cli, tmpdir, datafiles):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
++
 +    # Create the repo from 'files' subdir
 +    repo = create_repo('git', str(tmpdir))
 +    ref = repo.create(os.path.join(project, 'files'))
++
 +    # Write out test target with a cached and a non-cached source
 +    element = {
 +        'kind': 'import',
 +        'sources': [
 +            repo.source_config(ref=ref),
 +            {
 +                'kind': 'always_cached'
 +            }
 +        ]
 +    }
 +    _yaml.dump(element, os.path.join(project, 'target.bst'))
++
 +    # Test fetch of target with a cached and a non-cached source
 +    result = cli.run(project=project, args=[
 +        'source', 'fetch', 'target.bst'
 +    ])
 +    result.assert_success()

tests/testutils/runcli.py

@@ -153,6 +153,20 @@ class Result():
          assert self.task_error_domain == error_domain, fail_message
          assert self.task_error_reason == error_reason, fail_message
 +    # assert_shell_error()
 +    #
 +    # Asserts that the buildstream created a shell and that the task in the
 +    # shell failed.
 +    #
 +    # Args:
 +    #    fail_message (str): An optional message to override the automatic
 +    #                        assertion error messages
 +    # Raises:
 +    #    (AssertionError): If any of the assertions fail
 +    #
 +    def assert_shell_error(self, fail_message=''):
 +        assert self.exit_code == 1, fail_message
++
      # get_tracked_elements()
+     #
      # Produces a list of element names on which tracking occurred

[Notes] [Git][BuildStream/buildstream][gokcennurlu/remote_url_override_push_error] 21 commits: Basic options for shell --build to use buildtrees

Gökçen Nurlu pushed to branch gokcennurlu/remote_url_override_push_error at BuildStream / buildstream

Commits:

25 changed files:

Changes: