[Notes] [Git][BuildStream/buildstream][bschubert/fix-unbound-variable-ex

Benjamin Schubert pushed to branch bschubert/fix-unbound-variable-exception at BuildStream / buildstream

Commits:

d55b9e39

by Phil Dawson at 2018-12-12T13:55:19Z

Add --tar option to source-checkout command

This commit is part of the work towards #672

733aab53

by Phil Dawson at 2018-12-12T13:55:19Z

Add --force / -f option to source-checkout command

c2efeba0

by Phil Dawson at 2018-12-12T14:43:40Z

Add option to source-checkout command to generate build scripts

85c61894

by Phil Dawson at 2018-12-12T14:43:40Z

Remove source bundle command

This is part of the work towards #672

a322d5c0

by Phil Dawson at 2018-12-12T14:43:40Z

NEWS: Add entry about the removal of source-bundle command

ec909605

by Phil Dawson at 2018-12-12T15:45:42Z

Merge branch 'phil/source-checkout-options' into 'master'

Retire bst source bundle command

Closes #672

See merge request BuildStream/buildstream!959

3697a611

by Richard Maw at 2018-12-12T16:31:38Z

cli: Add support for auto-completing artifact ref names

b3dceb16
by Richard Maw at 2018-12-12T16:32:41Z
```
cli: Add artifact command group
```
ba08a0cd
by Richard Maw at 2018-12-12T16:32:41Z
```
cli: Add artifact log command
```

70fb9554

by Richard Maw at 2018-12-12T16:32:41Z

tests: Add tests for artifact group commands

f773e746
by Richard Maw at 2018-12-12T16:33:02Z
```
NEWS: Mention `bst artifact log`
```

b6528441

by richardmaw-codethink at 2018-12-12T18:00:59Z

Merge branch 'richardmaw/artifact-log' into 'master'

Add artifact log command

See merge request BuildStream/buildstream!920

d03bf316

by Benjamin Schubert at 2018-12-13T10:24:11Z

Mock storage space checks for tests.

Fix #530

- Extract free space computation in a function for easier
mocking
- Mock space computation during cache quota tests
- Mock cache size during cachque quota tests
- Add two more tests when the configuration would require
to much storage space

a116f576

by Tristan Van Berkom at 2018-12-13T10:58:46Z

Merge branch 'BenjaminSchubert/fix-quota-tests' into 'master'

Mock storage space checks for tests.

Closes #530

See merge request BuildStream/buildstream!702

180fa774

by Benjamin Schubert at 2018-12-13T12:05:15Z

element.py: fix unbounded variable in nested python exception

As per https://docs.python.org/3.7/reference/compound_stmts.html#except
variables used in the "except" clause are erased at the end of the
block.

We had a conflict with a nested exception using the same variable as
before.

This renames the nested variable to avoid the clash

13 changed files:

NEWS
buildstream/_artifactcache/artifactcache.py
buildstream/_frontend/cli.py
buildstream/_stream.py
buildstream/element.py
doc/source/using_commands.rst
tests/artifactcache/expiry.py
tests/completions/completions.py
tests/frontend/help.py
tests/frontend/source_checkout.py
tests/frontend/source_bundle.py → tests/integration/artifact.py
− tests/testutils/mock_os.py
tests/utils/misc.py

Changes:

NEWS

@@ -2,6 +2,14 @@
  buildstream 1.3.1
  =================
 +  o Added `bst artifact log` subcommand for viewing build logs.
++
 +  o BREAKING CHANGE: The bst source-bundle command has been removed. The
 +    functionality it provided has been replaced by the `--include-build-scripts`
 +    option of the `bst source-checkout` command. To produce a tarball containing
 +    an element's sources and generated build scripts you can do the command
 +    `bst source-checkout --include-build-scripts --tar foo.bst some-file.tar`
++
    o BREAKING CHANGE: Default strip-commands have been removed as they are too
      specific. Recommendation if you are building in Linux is to use the
      ones being used in freedesktop-sdk project, for example

buildstream/_artifactcache/artifactcache.py

@@ -874,9 +874,7 @@ class ArtifactCache():
                              "\nValid values are, for example: 800M 10G 1T 50%\n"
                              .format(str(e))) from e
 -        stat = os.statvfs(artifactdir_volume)
 -        available_space = (stat.f_bsize * stat.f_bavail)
+-
 +        available_space, total_size = self._get_volume_space_info_for(artifactdir_volume)
          cache_size = self.get_cache_size()
          # Ensure system has enough storage for the cache_quota
@@ -893,7 +891,7 @@ class ArtifactCache():
                              "BuildStream requires a minimum cache quota of 2G.")
          elif cache_quota > cache_size + available_space:  # Check maximum
              if '%' in self.context.config_cache_quota:
 -                available = (available_space / (stat.f_blocks * stat.f_bsize)) * 100
 +                available = (available_space / total_size) * 100
                  available = '{}% of total disk space'.format(round(available, 1))
              else:
                  available = utils._pretty_size(available_space)
@@ -919,6 +917,20 @@ class ArtifactCache():
          self._cache_quota = cache_quota - headroom
          self._cache_lower_threshold = self._cache_quota / 2
 +    # _get_volume_space_info_for
 +    #
 +    # Get the available space and total space for the given volume
 +    #
 +    # Args:
 +    #     volume: volume for which to get the size
 +    #
 +    # Returns:
 +    #     A tuple containing first the availabe number of bytes on the requested
 +    #     volume, then the total number of bytes of the volume.
 +    def _get_volume_space_info_for(self, volume):
 +        stat = os.statvfs(volume)
 +        return stat.f_bsize * stat.f_bavail, stat.f_bsize * stat.f_blocks
++
  # _configured_remote_artifact_cache_specs():
+ #

buildstream/_frontend/cli.py

  import os
  import sys
 +from contextlib import ExitStack
 +from fnmatch import fnmatch
 +from tempfile import TemporaryDirectory
  import click
  from .. import _yaml
@@ -107,6 +110,23 @@ def complete_target(args, incomplete):
      return complete_list
 +def complete_artifact(args, incomplete):
 +    from .._context import Context
 +    ctx = Context()
++
 +    config = None
 +    for i, arg in enumerate(args):
 +        if arg in ('-c', '--config'):
 +            config = args[i + 1]
 +    ctx.load(config)
++
 +    # element targets are valid artifact names
 +    complete_list = complete_target(args, incomplete)
 +    complete_list.extend(ref for ref in ctx.artifactcache.cas.list_refs() if ref.startswith(incomplete))
++
 +    return complete_list
++
++
  def override_completions(cmd, cmd_param, args, incomplete):
      """
      :param cmd_param: command definition
@@ -121,13 +141,15 @@ def override_completions(cmd, cmd_param, args, incomplete):
      # We can't easily extend click's data structures without
      # modifying click itself, so just do some weak special casing
      # right here and select which parameters we want to handle specially.
 -    if isinstance(cmd_param.type, click.Path) and \
 -       (cmd_param.name == 'elements' or
 -        cmd_param.name == 'element' or
 -        cmd_param.name == 'except_' or
 -        cmd_param.opts == ['--track'] or
 -        cmd_param.opts == ['--track-except']):
 -        return complete_target(args, incomplete)
 +    if isinstance(cmd_param.type, click.Path):
 +        if (cmd_param.name == 'elements' or
 +                cmd_param.name == 'element' or
 +                cmd_param.name == 'except_' or
 +                cmd_param.opts == ['--track'] or
 +                cmd_param.opts == ['--track-except']):
 +            return complete_target(args, incomplete)
 +        if cmd_param.name == 'artifacts':
 +            return complete_artifact(args, incomplete)
      raise CompleteUnhandled()
@@ -725,6 +747,8 @@ def checkout(app, element, location, force, deps, integrate, hardlinks, tar):
  #                  Source Checkout Command                      #
  ##################################################################
  @cli.command(name='source-checkout', short_help='Checkout sources for an element')
 +@click.option('--force', '-f', default=False, is_flag=True,
 +              help="Allow files to be overwritten")
  @click.option('--except', 'except_', multiple=True,
                type=click.Path(readable=False),
                help="Except certain dependencies")
@@ -733,11 +757,15 @@ def checkout(app, element, location, force, deps, integrate, hardlinks, tar):
                help='The dependencies whose sources to checkout (default: none)')
  @click.option('--fetch', 'fetch_', default=False, is_flag=True,
                help='Fetch elements if they are not fetched')
 -@click.argument('element', required=False,
 -                type=click.Path(readable=False))
 +@click.option('--tar', 'tar', default=False, is_flag=True,
 +              help='Create a tarball from the element\'s sources instead of a '
 +                   'file tree.')
 +@click.option('--include-build-scripts', 'build_scripts', is_flag=True)
 +@click.argument('element', required=False, type=click.Path(readable=False))
  @click.argument('location', type=click.Path(), required=False)
  @click.pass_obj
 -def source_checkout(app, element, location, deps, fetch_, except_):
 +def source_checkout(app, element, location, force, deps, fetch_, except_,
 +                    tar, build_scripts):
      """Checkout sources of an element to the specified location
      """
      if not element and not location:
@@ -757,9 +785,12 @@ def source_checkout(app, element, location, deps, fetch_, except_):
          app.stream.source_checkout(element,
                                     location=location,
 +                                   force=force,
                                     deps=deps,
                                     fetch=fetch_,
 -                                   except_targets=except_)
 +                                   except_targets=except_,
 +                                   tar=tar,
 +                                   include_build_scripts=build_scripts)
  ##################################################################
@@ -908,32 +939,99 @@ def workspace_list(app):
          app.stream.workspace_list()
 -##################################################################
 -#                     Source Bundle Command                      #
 -##################################################################
 -@cli.command(name="source-bundle", short_help="Produce a build bundle to be manually executed")
 -@click.option('--except', 'except_', multiple=True,
 -              type=click.Path(readable=False),
 -              help="Elements to except from the tarball")
 -@click.option('--compression', default='gz',
 -              type=click.Choice(['none', 'gz', 'bz2', 'xz']),
 -              help="Compress the tar file using the given algorithm.")
 -@click.option('--track', 'track_', default=False, is_flag=True,
 -              help="Track new source references before bundling")
 -@click.option('--force', '-f', default=False, is_flag=True,
 -              help="Overwrite an existing tarball")
 -@click.option('--directory', default=os.getcwd(),
 -              help="The directory to write the tarball to")
 -@click.argument('element',
 -                type=click.Path(readable=False))
 +#############################################################
 +#                     Artifact Commands                     #
 +#############################################################
 +def _classify_artifacts(names, cas, project_directory):
 +    element_targets = []
 +    artifact_refs = []
 +    element_globs = []
 +    artifact_globs = []
++
 +    for name in names:
 +        if name.endswith('.bst'):
 +            if any(c in "*?[" for c in name):
 +                element_globs.append(name)
 +            else:
 +                element_targets.append(name)
 +        else:
 +            if any(c in "*?[" for c in name):
 +                artifact_globs.append(name)
 +            else:
 +                artifact_refs.append(name)
++
 +    if element_globs:
 +        for dirpath, _, filenames in os.walk(project_directory):
 +            for filename in filenames:
 +                element_path = os.path.join(dirpath, filename).lstrip(project_directory).lstrip('/')
 +                if any(fnmatch(element_path, glob) for glob in element_globs):
 +                    element_targets.append(element_path)
++
 +    if artifact_globs:
 +        artifact_refs.extend(ref for ref in cas.list_refs()
 +                             if any(fnmatch(ref, glob) for glob in artifact_globs))
++
 +    return element_targets, artifact_refs
++
++
 +@cli.group(short_help="Manipulate cached artifacts")
 +def artifact():
 +    """Manipulate cached artifacts"""
 +    pass
++
++
 +################################################################
 +#                     Artifact Log Command                     #
 +################################################################
 +@artifact.command(name='log', short_help="Show logs of an artifact")
 +@click.argument('artifacts', type=click.Path(), nargs=-1)
  @click.pass_obj
 -def source_bundle(app, element, force, directory,
 -                  track_, compression, except_):
 -    """Produce a source bundle to be manually executed
 -    """
 -    with app.initialized():
 -        app.stream.source_bundle(element, directory,
 -                                 track_first=track_,
 -                                 force=force,
 -                                 compression=compression,
 -                                 except_targets=except_)
 +def artifact_log(app, artifacts):
 +    """Show logs of all artifacts"""
 +    from .._exceptions import CASError
 +    from .._message import MessageType
 +    from .._pipeline import PipelineSelection
 +    from ..storage._casbaseddirectory import CasBasedDirectory
++
 +    with ExitStack() as stack:
 +        stack.enter_context(app.initialized())
 +        cache = app.context.artifactcache
++
 +        elements, artifacts = _classify_artifacts(artifacts, cache.cas,
 +                                                  app.project.directory)
++
 +        vdirs = []
 +        extractdirs = []
 +        if artifacts:
 +            for ref in artifacts:
 +                try:
 +                    cache_id = cache.cas.resolve_ref(ref, update_mtime=True)
 +                    vdir = CasBasedDirectory(cache.cas, cache_id)
 +                    vdirs.append(vdir)
 +                except CASError as e:
 +                    app._message(MessageType.WARN, "Artifact {} is not cached".format(ref), detail=str(e))
 +                    continue
 +        if elements:
 +            elements = app.stream.load_selection(elements, selection=PipelineSelection.NONE)
 +            for element in elements:
 +                if not element._cached():
 +                    app._message(MessageType.WARN, "Element {} is not cached".format(element))
 +                    continue
 +                ref = cache.get_artifact_fullname(element, element._get_cache_key())
 +                cache_id = cache.cas.resolve_ref(ref, update_mtime=True)
 +                vdir = CasBasedDirectory(cache.cas, cache_id)
 +                vdirs.append(vdir)
++
 +        for vdir in vdirs:
 +            # NOTE: If reading the logs feels unresponsive, here would be a good place to provide progress information.
 +            logsdir = vdir.descend(["logs"])
 +            td = stack.enter_context(TemporaryDirectory())
 +            logsdir.export_files(td, can_link=True)
 +            extractdirs.append(td)
++
 +        for extractdir in extractdirs:
 +            for log in (os.path.join(extractdir, log) for log in os.listdir(extractdir)):
 +                # NOTE: Should click gain the ability to pass files to the pager this can be optimised.
 +                with open(log) as f:
 +                    data = f.read()
 +                    click.echo_via_pager(data)

buildstream/_stream.py

@@ -25,8 +25,8 @@ import stat
  import shlex
  import shutil
  import tarfile
 -from contextlib import contextmanager
 -from tempfile import TemporaryDirectory
 +import tempfile
 +from contextlib import contextmanager, suppress
  from ._exceptions import StreamError, ImplError, BstError, set_last_task_error
  from ._message import Message, MessageType
@@ -449,11 +449,14 @@ class Stream():
+     #
      def source_checkout(self, target, *,
                          location=None,
 +                        force=False,
                          deps='none',
                          fetch=False,
 -                        except_targets=()):
 +                        except_targets=(),
 +                        tar=False,
 +                        include_build_scripts=False):
 -        self._check_location_writable(location)
 +        self._check_location_writable(location, force=force, tar=tar)
          elements, _ = self._load((target,), (),
                                   selection=deps,
@@ -467,7 +470,8 @@ class Stream():
          # Stage all sources determined by scope
          try:
 -            self._write_element_sources(location, elements)
 +            self._source_checkout(elements, location, force, deps,
 +                                  fetch, tar, include_build_scripts)
          except BstError as e:
              raise StreamError("Error while writing sources"
                                ": '{}'".format(e), detail=e.detail, reason=e.reason) from e
@@ -728,87 +732,6 @@ class Stream():
              'workspaces': workspaces
          })
 -    # source_bundle()
 -    #
 -    # Create a host buildable tarball bundle for the given target.
 -    #
 -    # Args:
 -    #    target (str): The target element to bundle
 -    #    directory (str): The directory to output the tarball
 -    #    track_first (bool): Track new source references before bundling
 -    #    compression (str): The compression type to use
 -    #    force (bool): Overwrite an existing tarball
 -    #
 -    def source_bundle(self, target, directory, *,
 -                      track_first=False,
 -                      force=False,
 -                      compression="gz",
 -                      except_targets=()):
+-
 -        if track_first:
 -            track_targets = (target,)
 -        else:
 -            track_targets = ()
+-
 -        elements, track_elements = self._load((target,), track_targets,
 -                                              selection=PipelineSelection.ALL,
 -                                              except_targets=except_targets,
 -                                              track_selection=PipelineSelection.ALL,
 -                                              fetch_subprojects=True)
+-
 -        # source-bundle only supports one target
 -        target = self.targets[0]
+-
 -        self._message(MessageType.INFO, "Bundling sources for target {}".format(target.name))
+-
 -        # Find the correct filename for the compression algorithm
 -        tar_location = os.path.join(directory, target.normal_name + ".tar")
 -        if compression != "none":
 -            tar_location += "." + compression
+-
 -        # Attempt writing a file to generate a good error message
 -        # early
 -        #
 -        # FIXME: A bit hackish
 -        try:
 -            open(tar_location, mode="x")
 -            os.remove(tar_location)
 -        except IOError as e:
 -            raise StreamError("Cannot write to {0}: {1}"
 -                              .format(tar_location, e)) from e
+-
 -        # Fetch and possibly track first
 -        #
 -        self._fetch(elements, track_elements=track_elements)
+-
 -        # We don't use the scheduler for this as it is almost entirely IO
 -        # bound.
+-
 -        # Create a temporary directory to build the source tree in
 -        builddir = self._context.builddir
 -        os.makedirs(builddir, exist_ok=True)
 -        prefix = "{}-".format(target.normal_name)
+-
 -        with TemporaryDirectory(prefix=prefix, dir=builddir) as tempdir:
 -            source_directory = os.path.join(tempdir, 'source')
 -            try:
 -                os.makedirs(source_directory)
 -            except OSError as e:
 -                raise StreamError("Failed to create directory: {}"
 -                                  .format(e)) from e
+-
 -            # Any elements that don't implement _write_script
 -            # should not be included in the later stages.
 -            elements = [
 -                element for element in elements
 -                if self._write_element_script(source_directory, element)
 -            ]
+-
 -            self._write_element_sources(os.path.join(tempdir, "source"), elements)
 -            self._write_build_script(tempdir, elements)
 -            self._collect_sources(tempdir, tar_location,
 -                                  target.normal_name, compression)
+-
      # redirect_element_names()
+     #
      # Takes a list of element names and returns a list where elements have been
@@ -1189,6 +1112,54 @@ class Stream():
          sandbox_vroot.export_files(directory, can_link=True, can_destroy=True)
 +    # Helper function for source_checkout()
 +    def _source_checkout(self, elements,
 +                         location=None,
 +                         force=False,
 +                         deps='none',
 +                         fetch=False,
 +                         tar=False,
 +                         include_build_scripts=False):
 +        location = os.path.abspath(location)
 +        location_parent = os.path.abspath(os.path.join(location, ".."))
++
 +        # Stage all our sources in a temporary directory. The this
 +        # directory can be used to either construct a tarball or moved
 +        # to the final desired location.
 +        temp_source_dir = tempfile.TemporaryDirectory(dir=location_parent)
 +        try:
 +            self._write_element_sources(temp_source_dir.name, elements)
 +            if include_build_scripts:
 +                self._write_build_scripts(temp_source_dir.name, elements)
 +            if tar:
 +                self._create_tarball(temp_source_dir.name, location)
 +            else:
 +                self._move_directory(temp_source_dir.name, location, force)
 +        except OSError as e:
 +            raise StreamError("Failed to checkout sources to {}: {}"
 +                              .format(location, e)) from e
 +        finally:
 +            with suppress(FileNotFoundError):
 +                temp_source_dir.cleanup()
++
 +    # Move a directory src to dest. This will work across devices and
 +    # may optionaly overwrite existing files.
 +    def _move_directory(self, src, dest, force=False):
 +        def is_empty_dir(path):
 +            return os.path.isdir(dest) and not os.listdir(dest)
++
 +        try:
 +            os.rename(src, dest)
 +            return
 +        except OSError:
 +            pass
++
 +        if force or is_empty_dir(dest):
 +            try:
 +                utils.link_files(src, dest)
 +            except utils.UtilError as e:
 +                raise StreamError("Failed to move directory: {}".format(e)) from e
++
      # Write the element build script to the given directory
      def _write_element_script(self, directory, element):
          try:
@@ -1205,8 +1176,28 @@ class Stream():
                  os.makedirs(element_source_dir)
                  element._stage_sources_at(element_source_dir, mount_workspaces=False)
 +    # Create a tarball from the content of directory
 +    def _create_tarball(self, directory, tar_name):
 +        try:
 +            with utils.save_file_atomic(tar_name, mode='wb') as f:
 +                # This TarFile does not need to be explicitly closed
 +                # as the underlying file object will be closed be the
 +                # save_file_atomic contect manager
 +                tarball = tarfile.open(fileobj=f, mode='w')
 +                for item in os.listdir(str(directory)):
 +                    file_to_add = os.path.join(directory, item)
 +                    tarball.add(file_to_add, arcname=item)
 +        except OSError as e:
 +            raise StreamError("Failed to create tar archive: {}".format(e)) from e
++
 +    # Write all the build_scripts for elements in the directory location
 +    def _write_build_scripts(self, location, elements):
 +        for element in elements:
 +            self._write_element_script(location, element)
 +        self._write_master_build_script(location, elements)
++
      # Write a master build script to the sandbox
 -    def _write_build_script(self, directory, elements):
 +    def _write_master_build_script(self, directory, elements):
          module_string = ""
          for element in elements:

buildstream/element.py

@@ -1612,9 +1612,9 @@ class Element(Plugin):
                          sandbox_vpath = sandbox_vroot.descend(path_components)
                          try:
                              sandbox_vpath.import_files(workspace.get_absolute_path())
 -                        except UtilError as e:
 +                        except UtilError as e2:
                              self.warn("Failed to preserve workspace state for failed build sysroot: {}"
 -                                      .format(e))
 +                                      .format(e2))
                      self.__set_build_result(success=False, description=str(e), detail=e.detail)
                      self._cache_artifact(rootdir, sandbox, e.collect)

doc/source/using_commands.rst

@@ -86,13 +86,6 @@ project's main directory.
  ----
 -.. _invoking_source_bundle:
+-
 -.. click:: buildstream._frontend.cli:source_bundle
 -   :prog: bst source bundle
+-
 -----
+-
  .. _invoking_workspace:
  .. click:: buildstream._frontend.cli:workspace

tests/artifactcache/expiry.py

@@ -18,6 +18,7 @@
+ #
  import os
 +from unittest import mock
  import pytest
@@ -311,6 +312,8 @@ def test_never_delete_required_track(cli, datafiles, tmpdir):
      ("0", True),
      ("-1", False),
      ("pony", False),
 +    ("7K", False),
 +    ("70%", False),
      ("200%", False)
  ])
  @pytest.mark.datafiles(DATA_DIR)
@@ -324,7 +327,35 @@ def test_invalid_cache_quota(cli, datafiles, tmpdir, quota, success):
+         }
      })
 -    res = cli.run(project=project, args=['workspace', 'list'])
 +    # We patch how we get space information
 +    # Ideally we would instead create a FUSE device on which we control
 +    # everything.
 +    # If the value is a percentage, we fix the current values to take into
 +    # account the block size, since this is important in how we compute the size
++
 +    if quota.endswith("%"):  # We set the used space at 60% of total space
 +        stats = os.statvfs(".")
 +        free_space = 0.6 * stats.f_bsize * stats.f_blocks
 +        total_space = stats.f_bsize * stats.f_blocks
 +    else:
 +        free_space = 6000
 +        total_space = 10000
++
 +    volume_space_patch = mock.patch(
 +        "buildstream._artifactcache.artifactcache.ArtifactCache._get_volume_space_info_for",
 +        autospec=True,
 +        return_value=(free_space, total_space),
 +    )
++
 +    cache_size_patch = mock.patch(
 +        "buildstream._artifactcache.artifactcache.ArtifactCache.get_cache_size",
 +        autospec=True,
 +        return_value=0,
 +    )
++
 +    with volume_space_patch, cache_size_patch:
 +        res = cli.run(project=project, args=['workspace', 'list'])
++
      if success:
          res.assert_success()
      else:

tests/completions/completions.py

@@ -6,6 +6,7 @@ from tests.testutils import cli
  DATA_DIR = os.path.dirname(os.path.realpath(__file__))
  MAIN_COMMANDS = [
 +    'artifact ',
      'build ',
      'checkout ',
      'fetch ',
@@ -16,7 +17,6 @@ MAIN_COMMANDS = [
      'shell ',
      'show ',
      'source-checkout ',
 -    'source-bundle ',
      'track ',
      'workspace '
+ ]

tests/frontend/help.py

@@ -25,7 +25,6 @@ def test_help_main(cli):
      ('push'),
      ('shell'),
      ('show'),
 -    ('source-bundle'),
      ('track'),
      ('workspace')
  ])

tests/frontend/source_checkout.py

  import os
  import pytest
 +import tarfile
 +from pathlib import Path
  from tests.testutils import cli
@@ -55,6 +57,39 @@ def test_source_checkout(datafiles, cli, tmpdir_factory, with_workspace, guess_e
      assert os.path.exists(os.path.join(checkout, 'checkout-deps', 'etc', 'buildstream', 'config'))
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize('force_flag', ['--force', '-f'])
 +def test_source_checkout_force(datafiles, cli, force_flag):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    checkout = os.path.join(cli.directory, 'source-checkout')
 +    target = 'checkout-deps.bst'
++
 +    os.makedirs(os.path.join(checkout, 'some-thing'))
 +    # Path(os.path.join(checkout, 'some-file')).touch()
++
 +    result = cli.run(project=project, args=['source-checkout', force_flag, target, '--deps', 'none', checkout])
 +    result.assert_success()
++
 +    assert os.path.exists(os.path.join(checkout, 'checkout-deps', 'etc', 'buildstream', 'config'))
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_source_checkout_tar(datafiles, cli):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    checkout = os.path.join(cli.directory, 'source-checkout.tar')
 +    target = 'checkout-deps.bst'
++
 +    result = cli.run(project=project, args=['source-checkout', '--tar', target, '--deps', 'none', checkout])
 +    result.assert_success()
++
 +    assert os.path.exists(checkout)
 +    with tarfile.open(checkout) as tf:
 +        expected_content = os.path.join(checkout, 'checkout-deps', 'etc', 'buildstream', 'config')
 +        tar_members = [f.name for f in tf]
 +        for member in tar_members:
 +            assert member in expected_content
++
++
  @pytest.mark.datafiles(DATA_DIR)
  @pytest.mark.parametrize('deps', [('build'), ('none'), ('run'), ('all')])
  def test_source_checkout_deps(datafiles, cli, deps):
@@ -135,3 +170,38 @@ def test_source_checkout_fetch(datafiles, cli, fetch):
          assert os.path.exists(os.path.join(checkout, 'remote-import-dev', 'pony.h'))
      else:
          result.assert_main_error(ErrorDomain.PIPELINE, 'uncached-sources')
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_source_checkout_build_scripts(cli, tmpdir, datafiles):
 +    project_path = os.path.join(datafiles.dirname, datafiles.basename)
 +    element_name = 'source-bundle/source-bundle-hello.bst'
 +    normal_name = 'source-bundle-source-bundle-hello'
 +    checkout = os.path.join(str(tmpdir), 'source-checkout')
++
 +    args = ['source-checkout', '--include-build-scripts', element_name, checkout]
 +    result = cli.run(project=project_path, args=args)
 +    result.assert_success()
++
 +    # There sould be a script for each element (just one in this case) and a top level build script
 +    expected_scripts = ['build.sh', 'build-' + normal_name]
 +    for script in expected_scripts:
 +        assert script in os.listdir(checkout)
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_source_checkout_tar_buildscripts(cli, tmpdir, datafiles):
 +    project_path = os.path.join(datafiles.dirname, datafiles.basename)
 +    element_name = 'source-bundle/source-bundle-hello.bst'
 +    normal_name = 'source-bundle-source-bundle-hello'
 +    tar_file = os.path.join(str(tmpdir), 'source-checkout.tar')
++
 +    args = ['source-checkout', '--include-build-scripts', '--tar', element_name, tar_file]
 +    result = cli.run(project=project_path, args=args)
 +    result.assert_success()
++
 +    expected_scripts = ['build.sh', 'build-' + normal_name]
++
 +    with tarfile.open(tar_file, 'r') as tf:
 +        for script in expected_scripts:
 +            assert script in tf.getnames()

tests/frontend/source_bundle.py → tests/integration/artifact.py

+ #
 +#  Copyright (C) 2018 Codethink Limited
  #  Copyright (C) 2018 Bloomberg Finance LP
+ #
  #  This program is free software; you can redistribute it and/or
@@ -14,15 +15,17 @@
  #  You should have received a copy of the GNU Lesser General Public
  #  License along with this library. If not, see <http://www.gnu.org/licenses/>.
+ #
 -#  Authors: Chandan Singh <csingh43 bloomberg net>
 +#  Authors: Richard Maw <richard maw codethink co uk>
+ #
  import os
 -import tarfile
+-
  import pytest
 -from tests.testutils import cli
 +from tests.testutils import cli_integration as cli
++
++
 +pytestmark = pytest.mark.integration
++
  # Project directory
  DATA_DIR = os.path.join(
@@ -31,18 +34,35 @@ DATA_DIR = os.path.join(
+ )
 +@pytest.mark.integration
  @pytest.mark.datafiles(DATA_DIR)
 -def test_source_bundle(cli, tmpdir, datafiles):
 -    project_path = os.path.join(datafiles.dirname, datafiles.basename)
 -    element_name = 'source-bundle/source-bundle-hello.bst'
 -    normal_name = 'source-bundle-source-bundle-hello'
+-
 -    # Verify that we can correctly produce a source-bundle
 -    args = ['source-bundle', element_name, '--directory', str(tmpdir)]
 -    result = cli.run(project=project_path, args=args)
 -    result.assert_success()
+-
 -    # Verify that the source-bundle contains our sources and a build script
 -    with tarfile.open(os.path.join(str(tmpdir), '{}.tar.gz'.format(normal_name))) as bundle:
 -        assert os.path.join(normal_name, 'source', normal_name, 'llamas.txt') in bundle.getnames()
 -        assert os.path.join(normal_name, 'build.sh') in bundle.getnames()
 +def test_artifact_log(cli, tmpdir, datafiles):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
++
 +    # Get the cache key of our test element
 +    result = cli.run(project=project, silent=True, args=[
 +        '--no-colors',
 +        'show', '--deps', 'none', '--format', '%{full-key}',
 +        'base.bst'
 +    ])
 +    key = result.output.strip()
++
 +    # Ensure we have an artifact to read
 +    result = cli.run(project=project, args=['build', 'base.bst'])
 +    assert result.exit_code == 0
++
 +    # Read the log via the element name
 +    result = cli.run(project=project, args=['artifact', 'log', 'base.bst'])
 +    assert result.exit_code == 0
 +    log = result.output
++
 +    # Read the log via the key
 +    result = cli.run(project=project, args=['artifact', 'log', 'test/base/' + key])
 +    assert result.exit_code == 0
 +    assert log == result.output
++
 +    # Read the log via glob
 +    result = cli.run(project=project, args=['artifact', 'log', 'test/base/*'])
 +    assert result.exit_code == 0
 +    # The artifact is cached under both a strong key and a weak key
 +    assert (log + log) == result.output

tests/testutils/mock_os.py deleted

 -from contextlib import contextmanager
 -import os
+-
+-
 -# MockAttributeResult
 -#
 -# A class to take a dictionary of kwargs and make them accessible via
 -# attributes of the object.
 -#
 -class MockAttributeResult(dict):
 -    __getattr__ = dict.get
+-
+-
 -# mock_statvfs():
 -#
 -# Gets a function which mocks statvfs and returns a statvfs result with the kwargs accessible.
 -#
 -# Returns:
 -#    func(path) -> object: object will have all the kwargs accessible via object.kwarg
 -#
 -# Example:
 -#    statvfs = mock_statvfs(f_blocks=10)
 -#    result = statvfs("regardless/of/path")
 -#    assert result.f_blocks == 10 # True
 -def mock_statvfs(**kwargs):
 -    def statvfs(path):
 -        return MockAttributeResult(kwargs)
 -    return statvfs
+-
+-
 -# monkey_patch()
 -#
 -# with monkey_patch("statvfs", custom_statvfs):
 -#    assert os.statvfs == custom_statvfs # True
 -# assert os.statvfs == custom_statvfs # False
 -#
 -@contextmanager
 -def monkey_patch(to_patch, patched_func):
 -    orig = getattr(os, to_patch)
 -    setattr(os, to_patch, patched_func)
 -    try:
 -        yield
 -    finally:
 -        setattr(os, to_patch, orig)

tests/utils/misc.py

 +import os
 +from unittest import mock
++
  from buildstream import _yaml
 -from ..testutils import mock_os
 -from ..testutils.runcli import cli
 -import os
 -import pytest
 +from ..testutils.runcli import cli
  KiB = 1024
@@ -13,7 +13,6 @@ TiB = (GiB * 1024)
  def test_parse_size_over_1024T(cli, tmpdir):
 -    BLOCK_SIZE = 4096
      cli.configure({
          'cache': {
              'quota': 2048 * TiB
@@ -23,9 +22,13 @@ def test_parse_size_over_1024T(cli, tmpdir):
      os.makedirs(str(project))
      _yaml.dump({'name': 'main'}, str(project.join("project.conf")))
 -    bavail = (1025 * TiB) / BLOCK_SIZE
 -    patched_statvfs = mock_os.mock_statvfs(f_bavail=bavail, f_bsize=BLOCK_SIZE)
 -    with mock_os.monkey_patch("statvfs", patched_statvfs):
 +    volume_space_patch = mock.patch(
 +        "buildstream._artifactcache.artifactcache.ArtifactCache._get_volume_space_info_for",
 +        autospec=True,
 +        return_value=(1025 * TiB, 1025 * TiB)
 +    )
++
 +    with volume_space_patch:
          result = cli.run(project, args=["build", "file.bst"])
          failure_msg = 'Your system does not have enough available space to support the cache quota specified.'
          assert failure_msg in result.stderr

[Notes] [Git][BuildStream/buildstream][bschubert/fix-unbound-variable-exception] 15 commits: Add --tar option to source-checkout command

Benjamin Schubert pushed to branch bschubert/fix-unbound-variable-exception at BuildStream / buildstream

Commits:

13 changed files:

Changes: