[Notes] [Git][BuildStream/buildstream][chandan/bst-docker-import] 25 com

Chandan Singh pushed to branch chandan/bst-docker-import at BuildStream / buildstream

Commits:

26e1a3c7

by Jürg Billeter at 2018-10-01T14:58:06Z

_artifactcache/casserver.py: Implement BatchUpdateBlobs

Fixes #676.

f47895c0

by Jürg Billeter at 2018-10-01T14:58:06Z

_artifactcache/cascache.py: Use BatchUpdateBlobs

This uses BatchUpdateBlobs instead of individual blob upload to speed up
artifact pushing, if the server supports it.

Fixes #677.

cf00c0a1

by Jürg Billeter at 2018-10-01T15:32:30Z

Merge branch 'juerg/cas-batch' into 'master'

CAS: Implement BatchUpdateBlobs support

Closes #677 and #676

See merge request BuildStream/buildstream!839

5f4ae90b

by Jürg Billeter at 2018-10-02T06:34:02Z

sandbox/_sandboxremote.py: Use helpers to set up cwd and env

This matches the other sandbox implementations.

0458bc4e

by Jürg Billeter at 2018-10-02T07:08:35Z

Merge branch 'juerg/sandbox' into 'master'

sandbox/_sandboxremote.py: Use helpers to set up cwd and env

See merge request BuildStream/buildstream!841

d5b396e0

by Phillip Smyth at 2018-10-02T16:40:18Z

Forcing CI to install Sphinx 1.7.9

.gitlab-ci.yml: forced pip3 install Sphinx 1.7.9
The theme we use in our docs "sphinx_rtd_theme"
Does not support the latest version of Sphinx (>1.8)
This results in search functionality failing

dae842fd

by Phillip Smyth at 2018-10-02T17:14:09Z

Merge branch 'docs_Search_not_working' into 'master'

Fixing: Search functionality in the Documentation has stopped working

Closes #681

See merge request BuildStream/buildstream!848

8dc16d3f

by Jürg Billeter at 2018-10-03T05:08:21Z

_platform/linux.py: Accept all configs for dummy sandbox

If Sandbox.run() is never called, sandbox config is irrelevant. If it is
called, the build will fail anyway with the dummy sandbox.

This matches the logic in the Darwin platform.

66446fc3

by Jürg Billeter at 2018-10-03T05:36:38Z

Merge branch 'juerg/dummy-sandbox' into 'master'

_platform/linux.py: Accept all configs for dummy sandbox

See merge request BuildStream/buildstream!843

29c19bea

by Tristan Van Berkom at 2018-10-03T07:33:48Z

git.py: Added timed activity around fetching, displaying the URL being fetched.

This will take care of silencing the status messages while checking submodules.

b645881c

by Tristan Van Berkom at 2018-10-03T07:33:48Z

source.py: Silence messages while consuming source fetchers

The source fetchers might be a list or a generator, when it is
a generator (like the git source does), then we want to ensure that
we silence the status messages which might occur as a result of
consuming a source fetcher from the generator.

This fixes the logs to be less verbose.

c9437616

by Tristan Van Berkom at 2018-10-03T08:07:15Z

Merge branch 'tristan/fix-status-messages' into 'master'

fix status messages

See merge request BuildStream/buildstream!845

11320fe2

by Tristan Van Berkom at 2018-10-03T09:33:39Z

_scheduler: Fix bookkeeping of terminated jobs

 * Enhanced the base Job class to bookkeep which jobs have
   been terminated, and to consider them as `skipped` when
   asked via the `skipped` property.

 * Enhanced the base Queue class to bookkeep the job statuses
   more carefully.

This fixes #479

91271964

by Tristan Van Berkom at 2018-10-03T09:59:40Z

Merge branch 'tristan/fix-terminated-jobs' into 'master'

_scheduler: Fix bookkeeping of terminated jobs

Closes #479

See merge request BuildStream/buildstream!850

3bf895d2

by Jonathan Maw at 2018-10-03T11:48:48Z

tests: Add regression test for mirroring with project.refs

e4969807

by Jonathan Maw at 2018-10-03T12:48:07Z

Merge branch 'jonathan/source-mirror-project-refs' into 'master'

tests: Add regression test for mirroring with project.refs

See merge request BuildStream/buildstream!823

a0814399

by Tristan Van Berkom at 2018-10-03T13:05:52Z

_scheduler/scheduler.py: Ignore interrupt events while terminating.

For some reason, we now receive a SIGINT from the main loop even
when the SIGINT occurred with the handler disconnected in an interactive
prompt. This patch simply ignores any received SIGINT events from the
main loop in the case that we are already in the process of terminating.

This fixes issue #693

0a1f8e3c

by Tristan Van Berkom at 2018-10-03T13:42:20Z

Merge branch 'tristan/fix-double-terminate-prompt' into 'master'

_scheduler/scheduler.py: Ignore interrupt events while terminating.

Closes #693

See merge request BuildStream/buildstream!852

11161f99

by Tristan Van Berkom at 2018-10-03T13:44:02Z

utils.py: Give save_file_atomic() a tempdir argument

Allow callers to decide where the temporary file will be created.

3e797bb9

by Tristan Van Berkom at 2018-10-03T13:44:02Z

_artifactcache/cascache.py: Don't create temporary files in the CAS storage

Use the designated tempdir when creating refs, we expect that temporary
files are not created in the storage directory ever, they should be
only ever created in the designated temporary directory.

This fixes race conditions where utils._get_dir_size() throws an
unhandled exception when attempting to stat the file which inadvertantly
disappears.

d9020e43

by Tristan Van Berkom at 2018-10-03T13:44:02Z

utils.py: Document _get_dir_size() expectations.

This function assumes that files do not disappear while
walking the given directory.

3e5ff5a9

by Tristan Van Berkom at 2018-10-03T14:09:51Z

Merge branch 'tristan/fix-cache-size-race' into 'master'

fix cache size race

See merge request BuildStream/buildstream!854

e8179c34

by Jim MacArthur at 2018-10-03T16:06:59Z

element.py: Put the temporary staging directory in artifactdir

3cf38c8e

by Jim MacArthur at 2018-10-03T16:44:02Z

Merge branch 'jmac/temporaries-inside-cachedir' into 'master'

Move the temporary staging directory to artifactdir

See merge request BuildStream/buildstream!856

4aaa4cb6

by Chandan Singh at 2018-10-03T21:31:29Z

Add contrib script to generate Docker images from bst checkout

This script can be useful to generate a Docker image from `bst checkout`
of an element as a single command. While this script does not eliminate
the need for a proper Docker/OCI element plugin that would probably also
support layering, it provides a cheap way to export the element into a
Docker image.

Changes:

.gitlab-ci.yml

@@ -145,7 +145,8 @@ docs:
    stage: test
    script:
    - export BST_SOURCE_CACHE="$(pwd)/cache/integration-cache/sources"
 -  - pip3 install sphinx
 +  # Currently sphinx_rtd_theme does not support Sphinx >1.8, this breaks search functionality
 +  - pip3 install sphinx==1.7.9
    - pip3 install sphinx-click
    - pip3 install sphinx_rtd_theme
    - cd dist && ./unpack.sh && cd buildstream

buildstream/_artifactcache/cascache.py

@@ -506,7 +506,7 @@ class CASCache(ArtifactCache):
      def set_ref(self, ref, tree):
          refpath = self._refpath(ref)
          os.makedirs(os.path.dirname(refpath), exist_ok=True)
 -        with utils.save_file_atomic(refpath, 'wb') as f:
 +        with utils.save_file_atomic(refpath, 'wb', tempdir=self.tmpdir) as f:
              f.write(tree.SerializeToString())
      # resolve_ref():
@@ -1048,10 +1048,29 @@ class CASCache(ArtifactCache):
                  missing_blobs[d.hash] = d
          # Upload any blobs missing on the server
 -        for blob_digest in missing_blobs.values():
 -            with open(self.objpath(blob_digest), 'rb') as f:
 -                assert os.fstat(f.fileno()).st_size == blob_digest.size_bytes
 -                self._send_blob(remote, blob_digest, f, u_uid=u_uid)
 +        self._send_blobs(remote, missing_blobs.values(), u_uid)
++
 +    def _send_blobs(self, remote, digests, u_uid=uuid.uuid4()):
 +        batch = _CASBatchUpdate(remote)
++
 +        for digest in digests:
 +            with open(self.objpath(digest), 'rb') as f:
 +                assert os.fstat(f.fileno()).st_size == digest.size_bytes
++
 +                if (digest.size_bytes >= remote.max_batch_total_size_bytes or
 +                        not remote.batch_update_supported):
 +                    # Too large for batch request, upload in independent request.
 +                    self._send_blob(remote, digest, f, u_uid=u_uid)
 +                else:
 +                    if not batch.add(digest, f):
 +                        # Not enough space left in batch request.
 +                        # Complete pending batch first.
 +                        batch.send()
 +                        batch = _CASBatchUpdate(remote)
 +                        batch.add(digest, f)
++
 +        # Send final batch
 +        batch.send()
  # Represents a single remote CAS cache.
@@ -1126,6 +1145,17 @@ class _CASRemote():
                  if e.code() != grpc.StatusCode.UNIMPLEMENTED:
                      raise
 +            # Check whether the server supports BatchUpdateBlobs()
 +            self.batch_update_supported = False
 +            try:
 +                request = remote_execution_pb2.BatchUpdateBlobsRequest()
 +                response = self.cas.BatchUpdateBlobs(request)
 +                self.batch_update_supported = True
 +            except grpc.RpcError as e:
 +                if (e.code() != grpc.StatusCode.UNIMPLEMENTED and
 +                        e.code() != grpc.StatusCode.PERMISSION_DENIED):
 +                    raise
++
              self._initialized = True
@@ -1173,6 +1203,46 @@ class _CASBatchRead():
              yield (response.digest, response.data)
 +# Represents a batch of blobs queued for upload.
 +#
 +class _CASBatchUpdate():
 +    def __init__(self, remote):
 +        self._remote = remote
 +        self._max_total_size_bytes = remote.max_batch_total_size_bytes
 +        self._request = remote_execution_pb2.BatchUpdateBlobsRequest()
 +        self._size = 0
 +        self._sent = False
++
 +    def add(self, digest, stream):
 +        assert not self._sent
++
 +        new_batch_size = self._size + digest.size_bytes
 +        if new_batch_size > self._max_total_size_bytes:
 +            # Not enough space left in current batch
 +            return False
++
 +        blob_request = self._request.requests.add()
 +        blob_request.digest.hash = digest.hash
 +        blob_request.digest.size_bytes = digest.size_bytes
 +        blob_request.data = stream.read(digest.size_bytes)
 +        self._size = new_batch_size
 +        return True
++
 +    def send(self):
 +        assert not self._sent
 +        self._sent = True
++
 +        if len(self._request.requests) == 0:
 +            return
++
 +        batch_response = self._remote.cas.BatchUpdateBlobs(self._request)
++
 +        for response in batch_response.responses:
 +            if response.status.code != grpc.StatusCode.OK.value[0]:
 +                raise ArtifactError("Failed to upload blob {}: {}".format(
 +                    response.digest.hash, response.status.code))
++
++
  def _grouper(iterable, n):
      while True:
          try:

buildstream/_artifactcache/casserver.py

@@ -68,7 +68,7 @@ def create_server(repo, *, enable_push):
          _ByteStreamServicer(artifactcache, enable_push=enable_push), server)
      remote_execution_pb2_grpc.add_ContentAddressableStorageServicer_to_server(
 -        _ContentAddressableStorageServicer(artifactcache), server)
 +        _ContentAddressableStorageServicer(artifactcache, enable_push=enable_push), server)
      remote_execution_pb2_grpc.add_CapabilitiesServicer_to_server(
          _CapabilitiesServicer(), server)
@@ -222,9 +222,10 @@ class _ByteStreamServicer(bytestream_pb2_grpc.ByteStreamServicer):
  class _ContentAddressableStorageServicer(remote_execution_pb2_grpc.ContentAddressableStorageServicer):
 -    def __init__(self, cas):
 +    def __init__(self, cas, *, enable_push):
          super().__init__()
          self.cas = cas
 +        self.enable_push = enable_push
      def FindMissingBlobs(self, request, context):
          response = remote_execution_pb2.FindMissingBlobsResponse()
@@ -260,6 +261,46 @@ class _ContentAddressableStorageServicer(remote_execution_pb2_grpc.ContentAddres
          return response
 +    def BatchUpdateBlobs(self, request, context):
 +        response = remote_execution_pb2.BatchUpdateBlobsResponse()
++
 +        if not self.enable_push:
 +            context.set_code(grpc.StatusCode.PERMISSION_DENIED)
 +            return response
++
 +        batch_size = 0
++
 +        for blob_request in request.requests:
 +            digest = blob_request.digest
++
 +            batch_size += digest.size_bytes
 +            if batch_size > _MAX_PAYLOAD_BYTES:
 +                context.set_code(grpc.StatusCode.INVALID_ARGUMENT)
 +                return response
++
 +            blob_response = response.responses.add()
 +            blob_response.digest.hash = digest.hash
 +            blob_response.digest.size_bytes = digest.size_bytes
++
 +            if len(blob_request.data) != digest.size_bytes:
 +                blob_response.status.code = grpc.StatusCode.FAILED_PRECONDITION
 +                continue
++
 +            try:
 +                _clean_up_cache(self.cas, digest.size_bytes)
++
 +                with tempfile.NamedTemporaryFile(dir=self.cas.tmpdir) as out:
 +                    out.write(blob_request.data)
 +                    out.flush()
 +                    server_digest = self.cas.add_object(path=out.name)
 +                    if server_digest.hash != digest.hash:
 +                        blob_response.status.code = grpc.StatusCode.FAILED_PRECONDITION
++
 +            except ArtifactTooLargeException:
 +                blob_response.status.code = grpc.StatusCode.RESOURCE_EXHAUSTED
++
 +        return response
++
  class _CapabilitiesServicer(remote_execution_pb2_grpc.CapabilitiesServicer):
      def GetCapabilities(self, request, context):

buildstream/_platform/linux.py

@@ -55,6 +55,10 @@ class Linux(Platform):
              return SandboxBwrap(*args, **kwargs)
      def check_sandbox_config(self, config):
 +        if not self._local_sandbox_available():
 +            # Accept all sandbox configs as it's irrelevant with the dummy sandbox (no Sandbox.run).
 +            return True
++
          if self._user_ns_available:
              # User namespace support allows arbitrary build UID/GID settings.
              return True

buildstream/_scheduler/jobs/job.py

@@ -119,6 +119,8 @@ class Job():
          self._result = None                    # Return value of child action in the parent
          self._tries = 0                        # Try count, for retryable jobs
          self._skipped_flag = False             # Indicate whether the job was skipped.
 +        self._terminated = False               # Whether this job has been explicitly terminated
++
          # If False, a retry will not be attempted regardless of whether _tries is less than _max_retries.
+         #
          self._retry_flag = True
@@ -190,6 +192,8 @@ class Job():
          # Terminate the process using multiprocessing API pathway
          self._process.terminate()
 +        self._terminated = True
++
      # terminate_wait()
+     #
      # Wait for terminated jobs to complete
@@ -273,18 +277,22 @@ class Job():
      # running the integration commands).
+     #
      # Args:
 -    #     (int): The plugin identifier for this task
 +    #     task_id (int): The plugin identifier for this task
+     #
      def set_task_id(self, task_id):
          self._task_id = task_id
      # skipped
+     #
 +    # This will evaluate to True if the job was skipped
 +    # during processing, or if it was forcefully terminated.
 +    #
      # Returns:
 -    #    bool: True if the job was skipped while processing.
 +    #    (bool): Whether the job should appear as skipped
 +    #
      @property
      def skipped(self):
 -        return self._skipped_flag
 +        return self._skipped_flag or self._terminated
      #######################################################
      #                  Abstract Methods                   #

buildstream/_scheduler/queues/queue.py

@@ -326,16 +326,20 @@ class Queue():
                            detail=traceback.format_exc())
              self.failed_elements.append(element)
          else:
+-
 -            # No exception occured, handle the success/failure state in the normal way
+             #
 +            # No exception occured in post processing
 +            #
++
 +            # All jobs get placed on the done queue for later processing.
              self._done_queue.append(job)
 -            if success:
 -                if not job.skipped:
 -                    self.processed_elements.append(element)
 -                else:
 -                    self.skipped_elements.append(element)
 +            # A Job can be skipped whether or not it has failed,
 +            # we want to only bookkeep them as processed or failed
 +            # if they are not skipped.
 +            if job.skipped:
 +                self.skipped_elements.append(element)
 +            elif success:
 +                self.processed_elements.append(element)
              else:
                  self.failed_elements.append(element)

buildstream/_scheduler/scheduler.py

@@ -387,6 +387,15 @@ class Scheduler():
      # A loop registered event callback for keyboard interrupts
+     #
      def _interrupt_event(self):
++
 +        # FIXME: This should not be needed, but for some reason we receive an
 +        #        additional SIGINT event when the user hits ^C a second time
 +        #        to inform us that they really intend to terminate; even though
 +        #        we have disconnected our handlers at this time.
 +        #
 +        if self.terminated:
 +            return
++
          # Leave this to the frontend to decide, if no
          # interrrupt callback was specified, then just terminate.
          if self._interrupt_callback:

buildstream/element.py

@@ -1379,10 +1379,10 @@ class Element(Plugin):
              if not vdirectory.is_empty():
                  raise ElementError("Staging directory '{}' is not empty".format(vdirectory))
 -            # While mkdtemp is advertised as using the TMP environment variable, it
 -            # doesn't, so this explicit extraction is necesasry.
 -            tmp_prefix = os.environ.get("TMP", None)
 -            temp_staging_directory = tempfile.mkdtemp(prefix=tmp_prefix)
 +            # It's advantageous to have this temporary directory on
 +            # the same filing system as the rest of our cache.
 +            temp_staging_location = os.path.join(self._get_context().artifactdir, "staging_temp")
 +            temp_staging_directory = tempfile.mkdtemp(prefix=temp_staging_location)
              try:
                  workspace = self._get_workspace()

buildstream/plugins/sources/git.py

@@ -184,10 +184,18 @@ class GitMirror(SourceFetcher):
                           cwd=self.mirror)
      def fetch(self, alias_override=None):
 -        self.ensure(alias_override)
 -        if not self.has_ref():
 -            self._fetch(alias_override)
 -        self.assert_ref()
 +        # Resolve the URL for the message
 +        resolved_url = self.source.translate_url(self.url,
 +                                                 alias_override=alias_override,
 +                                                 primary=self.primary)
++
 +        with self.source.timed_activity("Fetching from {}"
 +                                        .format(resolved_url),
 +                                        silent_nested=True):
 +            self.ensure(alias_override)
 +            if not self.has_ref():
 +                self._fetch(alias_override)
 +            self.assert_ref()
      def has_ref(self):
          if not self.ref:

buildstream/sandbox/_sandboxremote.py

@@ -177,15 +177,11 @@ class SandboxRemote(Sandbox):
          if not cascache.verify_digest_pushed(self._get_project(), upload_vdir.ref):
              raise SandboxError("Failed to verify that source has been pushed to the remote artifact cache.")
 -        # Set up environment and working directory
 -        if cwd is None:
 -            cwd = self._get_work_directory()
+-
 -        if cwd is None:
 -            cwd = '/'
+-
 -        if env is None:
 -            env = self._get_environment()
 +        # Fallback to the sandbox default settings for
 +        # the cwd and env.
 +        #
 +        cwd = self._get_work_directory(cwd=cwd)
 +        env = self._get_environment(cwd=cwd, env=env)
          # We want command args as a list of strings
          if isinstance(command, str):

buildstream/source.py

@@ -965,28 +965,48 @@ class Source(Plugin):
      # Tries to call fetch for every mirror, stopping once it succeeds
      def __do_fetch(self, **kwargs):
          project = self._get_project()
 -        source_fetchers = self.get_source_fetchers()
 +        context = self._get_context()
++
 +        # Silence the STATUS messages which might happen as a result
 +        # of checking the source fetchers.
 +        with context.silence():
 +            source_fetchers = self.get_source_fetchers()
          # Use the source fetchers if they are provided
+         #
          if source_fetchers:
 -            for fetcher in source_fetchers:
 -                alias = fetcher._get_alias()
 -                for uri in project.get_alias_uris(alias, first_pass=self.__first_pass):
 -                    try:
 -                        fetcher.fetch(uri)
 -                    # FIXME: Need to consider temporary vs. permanent failures,
 -                    #        and how this works with retries.
 -                    except BstError as e:
 -                        last_error = e
 -                        continue
+-
 -                    # No error, we're done with this fetcher
 -                    break
 -                else:
 -                    # No break occurred, raise the last detected error
 -                    raise last_error
 +            # Use a contorted loop here, this is to allow us to
 +            # silence the messages which can result from consuming
 +            # the items of source_fetchers, if it happens to be a generator.
 +            #
 +            source_fetchers = iter(source_fetchers)
 +            try:
++
 +                while True:
++
 +                    with context.silence():
 +                        fetcher = next(source_fetchers)
++
 +                    alias = fetcher._get_alias()
 +                    for uri in project.get_alias_uris(alias, first_pass=self.__first_pass):
 +                        try:
 +                            fetcher.fetch(uri)
 +                        # FIXME: Need to consider temporary vs. permanent failures,
 +                        #        and how this works with retries.
 +                        except BstError as e:
 +                            last_error = e
 +                            continue
++
 +                        # No error, we're done with this fetcher
 +                        break
++
 +                    else:
 +                        # No break occurred, raise the last detected error
 +                        raise last_error
++
 +            except StopIteration:
 +                pass
          # Default codepath is to reinstantiate the Source
+         #

buildstream/utils.py

@@ -502,7 +502,7 @@ def get_bst_version():
  @contextmanager
  def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
 -                     errors=None, newline=None, closefd=True, opener=None):
 +                     errors=None, newline=None, closefd=True, opener=None, tempdir=None):
      """Save a file with a temporary name and rename it into place when ready.
      This is a context manager which is meant for saving data to files.
@@ -529,8 +529,9 @@ def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
      # https://bugs.python.org/issue8604
      assert os.path.isabs(filename), "The utils.save_file_atomic() parameter ``filename`` must be an absolute path"
 -    dirname = os.path.dirname(filename)
 -    fd, tempname = tempfile.mkstemp(dir=dirname)
 +    if tempdir is None:
 +        tempdir = os.path.dirname(filename)
 +    fd, tempname = tempfile.mkstemp(dir=tempdir)
      os.close(fd)
      f = open(tempname, mode=mode, buffering=buffering, encoding=encoding,
@@ -562,6 +563,9 @@ def save_file_atomic(filename, mode='w', *, buffering=-1, encoding=None,
+ #
  # Get the disk usage of a given directory in bytes.
+ #
 +# This function assumes that files do not inadvertantly
 +# disappear while this function is running.
 +#
  # Arguments:
  #     (str) The path whose size to check.
+ #

contrib/bst-docker-import

 +#!/bin/bash
 +#
 +#  Copyright 2018 Bloomberg Finance LP
 +#
 +#  This program is free software; you can redistribute it and/or
 +#  modify it under the terms of the GNU Lesser General Public
 +#  License as published by the Free Software Foundation; either
 +#  version 2 of the License, or (at your option) any later version.
 +#
 +#  This library is distributed in the hope that it will be useful,
 +#  but WITHOUT ANY WARRANTY; without even the implied warranty of
 +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
 +#  Lesser General Public License for more details.
 +#
 +#  You should have received a copy of the GNU Lesser General Public
 +#  License along with this library. If not, see <http://www.gnu.org/licenses/>.
 +#
 +#  Authors:
 +#        Chadnan Singh <csingh43 bloomberg net>
++
 +# This is a helper script to generate Docker images using checkouts of
 +# BuildStream elements.
++
 +usage() {
 +    cat <<EOF
++
 +USAGE: $(basename "$0") [-c BST_CMD] [-m MESSAGE] [-t TAG] [-h] ELEMENT
++
 +Create a Docker image from bst checkout of an element.
++
 +OPTIONS:
 +    -c BST_CMD    Path to BuildStream command (default: bst).
 +    -m MESSAGE    Commit message for the imported image.
 +    -t TAG        Tag of the imported image.
 +    -h            Print this help text and exit.
++
 +EXAMPLES:
++
 +    # Import hello.bst as a Docker image with tag "bst-hello" and message "hello"
 +    $(basename "$0") -m hello -t bst-hello hello.bst
++
 +    # Import hello.bst as a Docker image with tag "bst-hello" using bst-here
 +    $(basename "$0") -c bst-here -t bst-hello hello.bst
++
 +EOF
 +    exit "$1"
 +}
++
 +die() {
 +    echo "FATAL: $1" >&2
 +    exit 1
 +}
++
 +bst_cmd=bst
 +docker_import_cmd="docker import"
 +docker_image_tag=
++
 +while getopts c:m:t:h arg
 +do
 +    case $arg in
 +    c)
 +        bst_cmd="$OPTARG"
 +        ;;
 +    m)
 +        docker_import_cmd="$docker_import_cmd -m $OPTARG"
 +        ;;
 +    t)
 +        docker_image_tag="$OPTARG"
 +        ;;
 +    h)
 +        usage 0
 +        ;;
 +    \?)
 +        usage 1
 +    esac
 +done
++
 +shift $((OPTIND-1))
 +if [[ "$#" != 1 ]]; then
 +    echo "$0: No element specified" >&2
 +    usage 1
 +fi
 +element="$1"
++
 +# Dump to a temporary file in the current directory.
 +# NOTE: We use current directory to try to ensure compatibility with scripts
 +# like bst-here, assuming that the current working directory is mounted
 +# inside the container.
++
 +checkout_tar="bst-checkout-$(basename "$element")-$RANDOM.tar"
++
 +echo "INFO: Checking out $element ..." >&2
 +$bst_cmd checkout --tar "$element" "$checkout_tar" || die "Failed to checkout $element"
 +echo "INFO: Successfully checked out $element" >&2
++
 +echo "INFO: Importing Docker image ..."
 +$docker_import_cmd "$checkout_tar" "$docker_image_tag" || die "Failed to import Docker image from tarball"
 +echo "INFO: Successfully import Docker image $docker_image_tag"
++
 +echo "INFO: Cleaning up ..."
 +rm "$checkout_tar" || die "Failed to remove $checkout_tar"
 +echo "INFO: Clean up finished"

tests/frontend/mirror.py

@@ -139,6 +139,82 @@ def test_mirror_fetch(cli, tmpdir, datafiles, kind):
      result.assert_success()
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize("ref_storage", [("inline"), ("project.refs")])
 +@pytest.mark.parametrize("mirror", [("no-mirror"), ("mirror"), ("unrelated-mirror")])
 +def test_mirror_fetch_ref_storage(cli, tmpdir, datafiles, ref_storage, mirror):
 +    bin_files_path = os.path.join(str(datafiles), 'files', 'bin-files', 'usr')
 +    dev_files_path = os.path.join(str(datafiles), 'files', 'dev-files', 'usr')
 +    upstream_repodir = os.path.join(str(tmpdir), 'upstream')
 +    mirror_repodir = os.path.join(str(tmpdir), 'mirror')
 +    project_dir = os.path.join(str(tmpdir), 'project')
 +    os.makedirs(project_dir)
 +    element_dir = os.path.join(project_dir, 'elements')
++
 +    # Create repo objects of the upstream and mirror
 +    upstream_repo = create_repo('tar', upstream_repodir)
 +    upstream_ref = upstream_repo.create(bin_files_path)
 +    mirror_repo = upstream_repo.copy(mirror_repodir)
 +    mirror_ref = upstream_ref
 +    upstream_ref = upstream_repo.create(dev_files_path)
++
 +    element = {
 +        'kind': 'import',
 +        'sources': [
 +            upstream_repo.source_config(ref=upstream_ref if ref_storage == 'inline' else None)
 +        ]
 +    }
 +    element_name = 'test.bst'
 +    element_path = os.path.join(element_dir, element_name)
 +    full_repo = element['sources'][0]['url']
 +    upstream_map, repo_name = os.path.split(full_repo)
 +    alias = 'foo'
 +    aliased_repo = alias + ':' + repo_name
 +    element['sources'][0]['url'] = aliased_repo
 +    full_mirror = mirror_repo.source_config()['url']
 +    mirror_map, _ = os.path.split(full_mirror)
 +    os.makedirs(element_dir)
 +    _yaml.dump(element, element_path)
++
 +    if ref_storage == 'project.refs':
 +        # Manually set project.refs to avoid caching the repo prematurely
 +        project_refs = {'projects': {
 +            'test': {
 +                element_name: [
 +                    {'ref': upstream_ref}
 +                ]
 +            }
 +        }}
 +        project_refs_path = os.path.join(project_dir, 'project.refs')
 +        _yaml.dump(project_refs, project_refs_path)
++
 +    project = {
 +        'name': 'test',
 +        'element-path': 'elements',
 +        'aliases': {
 +            alias: upstream_map + "/"
 +        },
 +        'ref-storage': ref_storage
 +    }
 +    if mirror != 'no-mirror':
 +        mirror_data = [{
 +            'name': 'middle-earth',
 +            'aliases': {alias: [mirror_map + '/']}
 +        }]
 +        if mirror == 'unrelated-mirror':
 +            mirror_data.insert(0, {
 +                'name': 'narnia',
 +                'aliases': {'frob': ['http://www.example.com/repo']}
 +            })
 +        project['mirrors'] = mirror_data
++
 +    project_file = os.path.join(project_dir, 'project.conf')
 +    _yaml.dump(project, project_file)
++
 +    result = cli.run(project=project_dir, args=['fetch', element_name])
 +    result.assert_success()
++
++
  @pytest.mark.datafiles(DATA_DIR)
  @pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS])
  def test_mirror_fetch_upstream_absent(cli, tmpdir, datafiles, kind):

[Notes] [Git][BuildStream/buildstream][chandan/bst-docker-import] 25 commits: _artifactcache/casserver.py: Implement BatchUpdateBlobs

Chandan Singh pushed to branch chandan/bst-docker-import at BuildStream / buildstream

Commits:

14 changed files:

Changes: