[Notes] [Git][BuildStream/buildstream][jonathan/workspace-fragment-creat

Jonathan Maw pushed to branch jonathan/workspace-fragment-create at BuildStream / buildstream

Commits:

fe33e328

by James Ennis at 2018-11-08T17:54:18Z

using_config.rst: Add documentation to showing how to impose quotas on the local cache

This patch partially resolves #700

09faf002

by James Ennis at 2018-11-08T17:54:18Z

artifactcache.py: Fix misleading error message when using % cache quota

Due to the changed Exception message, this patch also changes
the test_parse_size_over_1024T test in misc.py

d153453c

by Javier Jardón at 2018-11-08T18:22:54Z

Merge branch 'jennis/quota_declaration_fix' into 'master'

Add local cache expiry documentation and fix misleading error message when specifying a percentage cache quota

Closes #700

See merge request BuildStream/buildstream!939

642a1a3b

by Jonathan Maw at 2018-11-09T11:00:27Z

_workspaces.py: Add a WorkspaceLocal object

WorkspaceLocal contains all the information needed to generate data
that's stored within the workspace. Currently this is just a list of
project directories.

96675873

by Jonathan Maw at 2018-11-09T11:00:27Z

_stream.py: Opened workspaces store the project that opened them

f818c118

by Jonathan Maw at 2018-11-09T11:00:27Z

_stream.py: workspace_close deletes the stored project reference

299fe0e2

by Jonathan Maw at 2018-11-09T11:00:27Z

tests: Workspace tests check existence of .bstproject.yaml

3b6ecbfe

by Jonathan Maw at 2018-11-09T11:00:27Z

_project.py: Rename _ensure_project_dir to _find_project_dir

50e4f7e0

by Jonathan Maw at 2018-11-09T11:00:27Z

utils.py: Add a helper for searching upwards for files

i.e. with a given directory and filename, check parent directories until
either a directory with the filename is found, or you reach the root of
the filesystem.

77bc3d85

by Jonathan Maw at 2018-11-09T11:00:27Z

cli.py: Use utils' search upwards helper when searching for project.conf

0793dae9

by Jonathan Maw at 2018-11-09T11:00:27Z

_project.py: Find project from workspace if outside of a project

14405073

by Jonathan Maw at 2018-11-09T11:00:27Z

_workspaces.py: Do not include .bstproject.yaml in the cache key

59433d7a

by Jonathan Maw at 2018-11-09T11:00:27Z

cli: Interactively warn if the user is trying to close the workspace they're using to load the project

This involves changes in:
* _stream.py:
  * Add the helper Stream.workspace_is_required()
* cli.py:
  * If buildstream is invoked interactively, prompt the user to confirm
    that they want to close the workspace they're using to load this
    project.

ca696da8

by Jonathan Maw at 2018-11-09T11:00:27Z

cli.py: Add '--force' arg to workspace_close

--force will skip confirmation prompts, and removing the workspace
you're currently using to load the project will fail without "--force".

91231d75

by Jonathan Maw at 2018-11-09T11:00:27Z

tests: Test bst commands from an external workspace

10 changed files:

buildstream/_artifactcache/artifactcache.py
buildstream/_frontend/cli.py
buildstream/_project.py
buildstream/_stream.py
buildstream/_workspaces.py
buildstream/utils.py
doc/source/using_config.rst
tests/frontend/workspace.py
tests/integration/shell.py
tests/utils/misc.py

Changes:

buildstream/_artifactcache/artifactcache.py

@@ -937,15 +937,22 @@ class ArtifactCache():
                              "Invalid cache quota ({}): ".format(utils._pretty_size(cache_quota)) +
                              "BuildStream requires a minimum cache quota of 2G.")
          elif cache_quota > cache_size + available_space:  # Check maximum
 +            if '%' in self.context.config_cache_quota:
 +                available = (available_space / (stat.f_blocks * stat.f_bsize)) * 100
 +                available = '{}% of total disk space'.format(round(available, 1))
 +            else:
 +                available = utils._pretty_size(available_space)
++
              raise LoadError(LoadErrorReason.INVALID_DATA,
                              ("Your system does not have enough available " +
                               "space to support the cache quota specified.\n" +
 -                             "You currently have:\n" +
 -                             "- {used} of cache in use at {local_cache_path}\n" +
 -                             "- {available} of available system storage").format(
 -                                 used=utils._pretty_size(cache_size),
 -                                 local_cache_path=self.context.artifactdir,
 -                                 available=utils._pretty_size(available_space)))
 +                             "\nYou have specified a quota of {quota} total disk space.\n" +
 +                             "- The filesystem containing {local_cache_path} only " +
 +                             "has: {available_size} available.")
 +                            .format(
 +                                quota=self.context.config_cache_quota,
 +                                local_cache_path=self.context.artifactdir,
 +                                available_size=available))
          # Place a slight headroom (2e9 (2GB) on the cache_quota) into
          # cache_quota to try and avoid exceptions.

buildstream/_frontend/cli.py

@@ -59,18 +59,9 @@ def complete_target(args, incomplete):
      :return: all the possible user-specified completions for the param
      """
 +    from .. import utils
      project_conf = 'project.conf'
 -    def ensure_project_dir(directory):
 -        directory = os.path.abspath(directory)
 -        while not os.path.isfile(os.path.join(directory, project_conf)):
 -            parent_dir = os.path.dirname(directory)
 -            if directory == parent_dir:
 -                break
 -            directory = parent_dir
+-
 -        return directory
+-
      # First resolve the directory, in case there is an
      # active --directory/-C option
+     #
@@ -89,7 +80,7 @@ def complete_target(args, incomplete):
      else:
          # Check if this directory or any of its parent directories
          # contain a project config file
 -        base_directory = ensure_project_dir(base_directory)
 +        base_directory = utils._search_upward_for_file(base_directory, project_conf)
      # Now parse the project.conf just to find the element path,
      # this is unfortunately a bit heavy.
@@ -713,10 +704,12 @@ def workspace_open(app, no_checkout, force, track_, element, directory):
                help="Remove the path that contains the closed workspace")
  @click.option('--all', '-a', 'all_', default=False, is_flag=True,
                help="Close all open workspaces")
 +@click.option('--force', '-f', default=False, is_flag=True,
 +              help="Always close the workspace and/or delete your changes")
  @click.argument('elements', nargs=-1,
                  type=click.Path(readable=False))
  @click.pass_obj
 -def workspace_close(app, remove_dir, all_, elements):
 +def workspace_close(app, remove_dir, all_, force, elements):
      """Close a workspace"""
      if not (all_ or elements):
@@ -735,15 +728,25 @@ def workspace_close(app, remove_dir, all_, elements):
          elements = app.stream.redirect_element_names(elements)
 -        # Check that the workspaces in question exist
 +        # Check that the workspaces in question exist, and that it's safe to
 +        # remove them.
          nonexisting = []
          for element_name in elements:
              if not app.stream.workspace_exists(element_name):
                  nonexisting.append(element_name)
 +            if app.stream.workspace_is_required(element_name):
 +                if app.interactive:
 +                    click.echo("Removing '{}' will prevent you from running buildstream commands".format(element_name))
 +                    if not click.confirm('Are you sure you want to close this workspace?'):
 +                        click.echo('Aborting', err=True)
 +                        sys.exit(-1)
 +                elif not force:
 +                    raise AppError("Cannot close workspaces. Workspace {} is being used to load the project"
 +                                   .format(element_name), reason='closing-required-workspace')
          if nonexisting:
              raise AppError("Workspace does not exist", detail="\n".join(nonexisting))
 -        if app.interactive and remove_dir:
 +        if app.interactive and remove_dir and not force:
              if not click.confirm('This will remove all your changes, are you sure?'):
                  click.echo('Aborting', err=True)
                  sys.exit(-1)

buildstream/_project.py

@@ -40,6 +40,7 @@ from .element import Element
  from ._message import Message, MessageType
  from ._includes import Includes
  from ._platform import Platform
 +from ._workspaces import WorkspaceLocal
  # Project Configuration file
@@ -94,8 +95,8 @@ class Project():
          # The project name
          self.name = None
 -        # The project directory
 -        self.directory = self._ensure_project_dir(directory)
 +        # The project directory, and whether the project was found from an external workspace
 +        self.directory, self._required_workspace_element = self._find_project_dir(directory)
          # Absolute path to where elements are loaded from within the project
          self.element_path = None
@@ -357,6 +358,14 @@ class Project():
          self._load_second_pass()
 +    # required_workspace_element()
 +    #
 +    # Returns the element whose workspace is required to load this project,
 +    # if any.
 +    #
 +    def required_workspace_element(self):
 +        return self._required_workspace_element
++
      # cleanup()
+     #
      # Cleans up resources used loading elements
@@ -645,7 +654,7 @@ class Project():
          # Source url aliases
          output._aliases = _yaml.node_get(config, Mapping, 'aliases', default_value={})
 -    # _ensure_project_dir()
 +    # _find_project_dir()
+     #
      # Returns path of the project directory, if a configuration file is found
      # in given directory or any of its parent directories.
@@ -656,18 +665,24 @@ class Project():
      # Raises:
      #    LoadError if project.conf is not found
+     #
 -    def _ensure_project_dir(self, directory):
 -        directory = os.path.abspath(directory)
 -        while not os.path.isfile(os.path.join(directory, _PROJECT_CONF_FILE)):
 -            parent_dir = os.path.dirname(directory)
 -            if directory == parent_dir:
 +    # Returns:
 +    #    (str) - the directory that contains the project, and
 +    #    (str) - the name of the element required to find the project, or an empty string
 +    def _find_project_dir(self, directory):
 +        workspace_element = ""
 +        project_directory = utils._search_upward_for_file(directory, _PROJECT_CONF_FILE)
 +        if not project_directory:
 +            workspace_local = WorkspaceLocal.load(directory)
 +            if workspace_local:
 +                project_directory = workspace_local.get_default_path()
 +                workspace_element = workspace_local.get_default_element()
 +            else:
                  raise LoadError(
                      LoadErrorReason.MISSING_PROJECT_CONF,
                      '{} not found in current directory or any of its parent directories'
                      .format(_PROJECT_CONF_FILE))
 -            directory = parent_dir
 -        return directory
 +        return project_directory, workspace_element
      def _load_plugin_factories(self, config, output):
          plugin_source_origins = []   # Origins of custom sources

buildstream/_stream.py

@@ -32,6 +32,7 @@ from ._exceptions import StreamError, ImplError, BstError, set_last_task_error
  from ._message import Message, MessageType
  from ._scheduler import Scheduler, SchedStatus, TrackQueue, FetchQueue, BuildQueue, PullQueue, PushQueue
  from ._pipeline import Pipeline, PipelineSelection
 +from ._workspaces import WorkspaceLocal
  from . import utils, _yaml, _site
  from . import Scope, Consistency
@@ -516,6 +517,10 @@ class Stream():
              with target.timed_activity("Staging sources to {}".format(directory)):
                  target._open_workspace()
 +        project = self._context.get_toplevel_project()
 +        workspace_local = WorkspaceLocal.create(directory, project.directory, target._get_full_name())
 +        workspace_local.write()
++
          workspaces.save_config()
          self._message(MessageType.INFO, "Saved workspace configuration")
@@ -540,6 +545,11 @@ class Stream():
                  except OSError as e:
                      raise StreamError("Could not remove  '{}': {}"
                                        .format(workspace.get_absolute_path(), e)) from e
 +        else:
 +            # TODO: At some point, closing a workspace only deletes the file if no projects are using it.
 +            workspace_local = WorkspaceLocal.load(workspace.get_absolute_path())
 +            if workspace_local:
 +                workspace_local.delete()
          # Delete the workspace and save the configuration
          workspaces.delete_workspace(element_name)
@@ -633,6 +643,20 @@ class Stream():
          return False
 +    # workspace_is_required()
 +    #
 +    # Checks whether the workspace belonging to element_name is required to
 +    # load the project
 +    #
 +    # Args:
 +    #    element_name (str): The element whose workspace may be required
 +    #
 +    # Returns:
 +    #    (bool): True if the workspace is required
 +    def workspace_is_required(self, element_name):
 +        required_elm = self._project.required_workspace_element()
 +        return required_elm == element_name
++
      # workspace_list
+     #
      # Serializes the workspaces and dumps them in YAML to stdout.

buildstream/_workspaces.py

@@ -25,6 +25,149 @@ from ._exceptions import LoadError, LoadErrorReason
  BST_WORKSPACE_FORMAT_VERSION = 3
 +BST_WORKSPACE_LOCAL_FORMAT_VERSION = 1
 +WORKSPACE_LOCAL_FILE = ".bstproject.yaml"
++
++
 +# WorkspaceLocal()
 +#
 +# An object to contain various helper functions and data required for
 +# referring from a workspace back to buildstream.
 +#
 +# Args:
 +#    directory (str): The directory that the workspace exists in
 +#    project_path (str): The project path used to refer back
 +#                        to buildstream projects.
 +#    element_name (str): The name of the element used to create this workspace.
 +class WorkspaceLocal():
 +    def __init__(self, directory, project_path="", element_name=""):
 +        self._projects = []
 +        self._directory = directory
++
 +        assert (project_path and element_name) or (not project_path and not element_name)
 +        if project_path:
 +            self._add_project(project_path, element_name)
++
 +    # get_default_path()
 +    #
 +    # Retrieves the default path to a project.
 +    #
 +    # Returns:
 +    #    (str): The path to a project
 +    def get_default_path(self):
 +        return self._projects[0]['project-path']
++
 +    # get_default_element()
 +    #
 +    # Retrieves the name of the element that owns this workspace.
 +    #
 +    # Returns:
 +    #    (str): The name of an element
 +    def get_default_element(self):
 +        return self._projects[0]['element-name']
++
 +    # to_dict()
 +    #
 +    # Turn the members data into a dict for serialization purposes
 +    #
 +    # Returns:
 +    #    (dict): A dict representation of the WorkspaceLocal
 +    #
 +    def to_dict(self):
 +        ret = {
 +            'projects': self._projects,
 +            'format-version': BST_WORKSPACE_LOCAL_FORMAT_VERSION,
 +        }
 +        return ret
++
 +    # from_dict()
 +    #
 +    # Loads a new WorkspaceLocal from a simple dictionary
 +    #
 +    # Args:
 +    #    directory (str): The directory that the workspace exists in
 +    #    dictionary (dict): The dict to generate a WorkspaceLocal from
 +    #
 +    # Returns:
 +    #   (WorkspaceLocal): A newly instantiated WorkspaceLocal
 +    @classmethod
 +    def from_dict(cls, directory, dictionary):
 +        # Only know how to handle one format-version at the moment.
 +        format_version = int(dictionary['format-version'])
 +        assert format_version == BST_WORKSPACE_LOCAL_FORMAT_VERSION, \
 +            "Format version {} not found in {}".format(BST_WORKSPACE_LOCAL_FORMAT_VERSION, dictionary)
++
 +        workspace_local = cls(directory)
 +        for item in dictionary['projects']:
 +            workspace_local._add_project(item['project-path'], item['element-name'])
++
 +        return workspace_local
++
 +    # create()
 +    #
 +    # Creates a new WorkspaceLocal
 +    #
 +    # Args:
 +    #    directory (str): The directory that the workspace exists in
 +    #    project_path (str): The path to the project to store
 +    #    element_name (str): The name of the element within the project
 +    #
 +    # Returns:
 +    #    (WorkspaceLocal): The created WorkspaceLocal
 +    @classmethod
 +    def create(cls, directory, project_path, element_name):
 +        # TODO: Load WorkspaceLocal if it exists, and maybe add project_path to it
 +        return cls(directory, project_path, element_name)
++
 +    # load()
 +    #
 +    # Loads the WorkspaceLocal for a given directory. This directory may be a
 +    # subdirectory of the workspace's directory.
 +    #
 +    # Args:
 +    #    directory (str): The directory
 +    # Returns:
 +    #    (WorkspaceLocal): The created WorkspaceLocal, if in a workspace, or
 +    #    (NoneType): None, if the directory is not inside a workspace.
 +    @classmethod
 +    def load(cls, directory):
 +        local_dir = cls.search_for_dir(directory)
 +        if local_dir:
 +            workspace_file = os.path.join(local_dir, WORKSPACE_LOCAL_FILE)
 +            data_dict = _yaml.load(workspace_file)
 +            return cls.from_dict(local_dir, data_dict)
 +        else:
 +            return None
++
 +    # write()
 +    #
 +    # Writes the WorkspaceLocal to disk
 +    def write(self):
 +        os.makedirs(self._directory, exist_ok=True)
 +        _yaml.dump(self.to_dict(), self._get_filename())
++
 +    # delete()
 +    #
 +    # Deletes the WorkspaceLocal from disk, if it exists.
 +    def delete(self):
 +        try:
 +            os.unlink(self._get_filename())
 +        except FileNotFoundError:
 +            pass
++
 +    # search_for_dir()
 +    #
 +    # Returns the directory that contains the workspace local file,
 +    # searching upwards from search_dir.
 +    @staticmethod
 +    def search_for_dir(search_dir):
 +        return utils._search_upward_for_file(search_dir, WORKSPACE_LOCAL_FILE)
++
 +    def _get_filename(self):
 +        return os.path.join(self._directory, WORKSPACE_LOCAL_FILE)
++
 +    def _add_project(self, project_path, element_name):
 +        self._projects.append({'project-path': project_path, 'element-name': element_name})
  # Workspace()
@@ -174,10 +317,15 @@ class Workspace():
          if recalculate or self._key is None:
              fullpath = self.get_absolute_path()
 +            excluded_files = [WORKSPACE_LOCAL_FILE]
++
              # Get a list of tuples of the the project relative paths and fullpaths
              if os.path.isdir(fullpath):
                  filelist = utils.list_relative_paths(fullpath)
 -                filelist = [(relpath, os.path.join(fullpath, relpath)) for relpath in filelist]
 +                filelist = [
 +                    (relpath, os.path.join(fullpath, relpath)) for relpath in filelist
 +                    if relpath not in excluded_files
 +                ]
              else:
                  filelist = [(self.get_absolute_path(), fullpath)]

buildstream/utils.py

@@ -1199,3 +1199,17 @@ def _deduplicate(iterable, key=None):
  def _get_link_mtime(path):
      path_stat = os.lstat(path)
      return path_stat.st_mtime
++
++
 +# Returns the first directory to contain filename, or an empty string if
 +# none found
 +#
 +def _search_upward_for_file(directory, filename):
 +    directory = os.path.abspath(directory)
 +    while not os.path.isfile(os.path.join(directory, filename)):
 +        parent_dir = os.path.dirname(directory)
 +        if directory == parent_dir:
 +            return ""
 +        directory = parent_dir
++
 +    return directory

doc/source/using_config.rst

@@ -147,6 +147,44 @@ The default mirror is defined by its name, e.g.
     ``--default-mirror`` command-line option.
 +Local cache expiry
 +~~~~~~~~~~~~~~~~~~
 +BuildStream locally caches artifacts, build trees, log files and sources within a
 +cache located at ``~/.cache/buildstream`` (unless a $XDG_CACHE_HOME environment
 +variable exists). When building large projects, this cache can get very large,
 +thus BuildStream will attempt to clean up the cache automatically by expiring the least
 +recently *used* artifacts.
++
 +By default, cache expiry will begin once the file system which contains the cache
 +approaches maximum usage. However, it is also possible to impose a quota on the local
 +cache in the user configuration. This can be done in two ways:
++
 +1. By restricting the maximum size of the cache directory itself.
++
 +For example, to ensure that BuildStream's cache does not grow beyond 100 GB,
 +simply declare the following in your user configuration (``~/.config/buildstream.conf``):
++
 +.. code:: yaml
++
 +  cache:
 +    quota: 100G
++
 +This quota defines the maximum size of the artifact cache in bytes.
 +Other accepted values are: K, M, G or T (or you can simply declare the value in bytes, without the suffix).
 +This uses the same format as systemd's
 +`resource-control <https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html>`_.
++
 +2. By expiring artifacts once the file system which contains the cache exceeds a specified usage.
++
 +To ensure that we start cleaning the cache once we've used 80% of local disk space (on the file system
 +which mounts the cache):
++
 +.. code:: yaml
++
 +  cache:
 +    quota: 80%
++
++
  Default configuration
  ---------------------
  The default BuildStream configuration is specified here for reference:

tests/frontend/workspace.py

@@ -29,6 +29,7 @@ import shutil
  import subprocess
  from ruamel.yaml.comments import CommentedSet
  from tests.testutils import cli, create_repo, ALL_REPO_KINDS, wait_for_cache_granularity
 +from tests.testutils import create_artifact_share
  from buildstream import _yaml
  from buildstream._exceptions import ErrorDomain, LoadError, LoadErrorReason
@@ -93,6 +94,13 @@ def open_workspace(cli, tmpdir, datafiles, kind, track, suffix='', workspace_dir
      result.assert_success()
 +    # Assert that a .bstproject.yaml file has been created
 +    # and contains the path to the project
 +    bstproject_path = os.path.join(workspace_dir, '.bstproject.yaml')
 +    assert os.path.exists(bstproject_path)
 +    with open(bstproject_path) as f:
 +        assert project_path in f.read()
++
      # Assert that we are now buildable because the source is
      # now cached.
      assert cli.get_element_state(project_path, element_name) == 'buildable'
@@ -148,6 +156,10 @@ def test_open_force(cli, tmpdir, datafiles, kind):
      # Assert the workspace dir still exists
      assert os.path.exists(workspace)
 +    # Assert the bstproject doesn't exist
 +    bstproject_path = os.path.join(workspace, '.bstproject.yaml')
 +    assert not os.path.exists(bstproject_path)
++
      # Now open the workspace again with --force, this should happily succeed
      result = cli.run(project=project, args=[
          'workspace', 'open', '--force', element_name, workspace
@@ -436,7 +448,9 @@ def test_list(cli, tmpdir, datafiles):
  @pytest.mark.datafiles(DATA_DIR)
  @pytest.mark.parametrize("kind", repo_kinds)
  @pytest.mark.parametrize("strict", [("strict"), ("non-strict")])
 -def test_build(cli, tmpdir, datafiles, kind, strict):
 +@pytest.mark.parametrize("call_from", [("project"), ("workspace")])
 +def test_build(cli, tmpdir_factory, datafiles, kind, strict, call_from):
 +    tmpdir = tmpdir_factory.mktemp('')
      element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, kind, False)
      checkout = os.path.join(str(tmpdir), 'checkout')
@@ -458,16 +472,18 @@ def test_build(cli, tmpdir, datafiles, kind, strict):
+         }
      })
 +    from_dir = project if call_from == "project" else workspace
++
      # Build modified workspace
 -    assert cli.get_element_state(project, element_name) == 'buildable'
 -    assert cli.get_element_key(project, element_name) == "{:?<64}".format('')
 -    result = cli.run(project=project, args=['build', element_name])
 +    assert cli.get_element_state(from_dir, element_name) == 'buildable'
 +    assert cli.get_element_key(from_dir, element_name) == "{:?<64}".format('')
 +    result = cli.run(project=from_dir, args=['build', element_name])
      result.assert_success()
 -    assert cli.get_element_state(project, element_name) == 'cached'
 -    assert cli.get_element_key(project, element_name) != "{:?<64}".format('')
 +    assert cli.get_element_state(from_dir, element_name) == 'cached'
 +    assert cli.get_element_key(from_dir, element_name) != "{:?<64}".format('')
      # Checkout the result
 -    result = cli.run(project=project, args=[
 +    result = cli.run(project=from_dir, args=[
          'checkout', element_name, checkout
      ])
      result.assert_success()
@@ -876,3 +892,127 @@ def test_multiple_failed_builds(cli, tmpdir, datafiles):
          result = cli.run(project=project, args=["build", element_name])
          assert "BUG" not in result.stderr
          assert cli.get_element_state(project, element_name) != "cached"
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_external_fetch(cli, datafiles, tmpdir_factory):
 +    # Fetching from a workspace outside a project doesn't fail horribly
 +    tmpdir = tmpdir_factory.mktemp('')
 +    element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", False)
++
 +    result = cli.run(project=workspace, args=['fetch', element_name])
 +    result.assert_success()
++
 +    # We already fetched it by opening the workspace, but we're also checking
 +    # `bst show` works here
 +    assert cli.get_element_state(workspace, element_name) == 'buildable'
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_external_push_pull(cli, datafiles, tmpdir_factory):
 +    # Pushing and pulling to/from an artifact cache works from an external workspace
 +    tmpdir = tmpdir_factory.mktemp('')
 +    element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", False)
++
 +    with create_artifact_share(os.path.join(str(tmpdir), 'artifactshare')) as share:
 +        result = cli.run(project=workspace, args=['build', element_name])
 +        result.assert_success()
++
 +        cli.configure({
 +            'artifacts': {'url': share.repo, 'push': True}
 +        })
++
 +        result = cli.run(project=workspace, args=['push', element_name])
 +        result.assert_success()
++
 +        result = cli.run(project=workspace, args=['pull', '--deps', 'all', 'target.bst'])
 +        result.assert_success()
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_external_track(cli, datafiles, tmpdir_factory):
 +    # Tracking does not get horribly confused
 +    tmpdir = tmpdir_factory.mktemp('')
 +    element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", True)
++
 +    # The workspace is necessarily already tracked, so we only care that
 +    # there's no weird errors.
 +    result = cli.run(project=workspace, args=['track', element_name])
 +    result.assert_success()
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_external_open_other(cli, datafiles, tmpdir_factory):
 +    # From inside an external workspace, open another workspace
 +    tmpdir1 = tmpdir_factory.mktemp('')
 +    tmpdir2 = tmpdir_factory.mktemp('')
 +    # Making use of the assumption that it's the same project in both invocations of open_workspace
 +    alpha_element, project, alpha_workspace = open_workspace(cli, tmpdir1, datafiles, "git", False, suffix="-alpha")
 +    beta_element, _, beta_workspace = open_workspace(cli, tmpdir2, datafiles, "git", False, suffix="-beta")
++
 +    # Closing the other element first, because I'm too lazy to create an
 +    # element without opening it
 +    result = cli.run(project=project, args=['workspace', 'close', beta_element])
 +    result.assert_success()
++
 +    result = cli.run(project=alpha_workspace, args=['workspace', 'open', '--force', beta_element, beta_workspace])
 +    result.assert_success()
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_external_close_other(cli, datafiles, tmpdir_factory):
 +    # From inside an external workspace, close the other workspace
 +    tmpdir1 = tmpdir_factory.mktemp('')
 +    tmpdir2 = tmpdir_factory.mktemp('')
 +    # Making use of the assumption that it's the same project in both invocations of open_workspace
 +    alpha_element, project, alpha_workspace = open_workspace(cli, tmpdir1, datafiles, "git", False, suffix="-alpha")
 +    beta_element, _, beta_workspace = open_workspace(cli, tmpdir2, datafiles, "git", False, suffix="-beta")
++
 +    result = cli.run(project=alpha_workspace, args=['workspace', 'close', beta_element])
 +    result.assert_success()
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize("force", [("force"), ("no-force")])
 +def test_external_close_self(cli, datafiles, tmpdir_factory, force):
 +    # From inside an external workspace, close it
 +    # This is unwise, so is only allowed if --force
 +    tmpdir1 = tmpdir_factory.mktemp('')
 +    tmpdir2 = tmpdir_factory.mktemp('')
 +    # Making use of the assumption that it's the same project in both invocations of open_workspace
 +    alpha_element, project, alpha_workspace = open_workspace(cli, tmpdir1, datafiles, "git", False, suffix="-alpha")
 +    beta_element, _, beta_workspace = open_workspace(cli, tmpdir2, datafiles, "git", False, suffix="-beta")
++
 +    args = ['workspace', 'close']
 +    if force == "force":
 +        args.append('--force')
 +    args.append(alpha_element)
++
 +    result = cli.run(project=alpha_workspace, args=args)
 +    if force == "force":
 +        result.assert_success()
 +    else:
 +        # TODO: Proper error domain and reason
 +        result.assert_main_error(ErrorDomain.APP, 'closing-required-workspace')
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_external_reset_other(cli, datafiles, tmpdir_factory):
 +    tmpdir1 = tmpdir_factory.mktemp('')
 +    tmpdir2 = tmpdir_factory.mktemp('')
 +    # Making use of the assumption that it's the same project in both invocations of open_workspace
 +    alpha_element, project, alpha_workspace = open_workspace(cli, tmpdir1, datafiles, "git", False, suffix="-alpha")
 +    beta_element, _, beta_workspace = open_workspace(cli, tmpdir2, datafiles, "git", False, suffix="-beta")
++
 +    result = cli.run(project=alpha_workspace, args=['workspace', 'reset', beta_element])
 +    result.assert_success()
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +def test_external_list(cli, datafiles, tmpdir_factory):
 +    tmpdir = tmpdir_factory.mktemp('')
 +    # Making use of the assumption that it's the same project in both invocations of open_workspace
 +    element, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", False)
++
 +    result = cli.run(project=workspace, args=['workspace', 'list'])
 +    result.assert_success()

tests/integration/shell.py

@@ -339,3 +339,28 @@ def test_integration_devices(cli, tmpdir, datafiles):
      result = execute_shell(cli, project, ["true"], element=element_name)
      assert result.exit_code == 0
++
++
 +# Test that a shell can be opened from an external workspace
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize("build_shell", [("build"), ("nobuild")])
 +def test_integration_external_workspace(cli, tmpdir_factory, datafiles, build_shell):
 +    tmpdir = tmpdir_factory.mktemp("")
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    element_name = 'autotools/amhello.bst'
 +    workspace_dir = os.path.join(str(tmpdir), 'workspace')
++
 +    result = cli.run(project=project, args=[
 +        'workspace', 'open', element_name, workspace_dir
 +    ])
 +    result.assert_success()
++
 +    result = cli.run(project=project, cwd=workspace_dir, args=['build', element_name])
 +    result.assert_success()
++
 +    command = ['shell']
 +    if build_shell == 'build':
 +        command.append('--build')
 +    command.extend([element_name, '--', 'true'])
 +    result = cli.run(project=project, cwd=workspace_dir, args=command)
 +    result.assert_success()

tests/utils/misc.py

@@ -27,4 +27,5 @@ def test_parse_size_over_1024T(cli, tmpdir):
      patched_statvfs = mock_os.mock_statvfs(f_bavail=bavail, f_bsize=BLOCK_SIZE)
      with mock_os.monkey_patch("statvfs", patched_statvfs):
          result = cli.run(project, args=["build", "file.bst"])
 -        assert "1025T of available system storage" in result.stderr
 +        failure_msg = 'Your system does not have enough available space to support the cache quota specified.'
 +        assert failure_msg in result.stderr

[Notes] [Git][BuildStream/buildstream][jonathan/workspace-fragment-create] 15 commits: using_config.rst: Add documentation to showing how to impose quotas on the local cache

Jonathan Maw pushed to branch jonathan/workspace-fragment-create at BuildStream / buildstream

Commits:

10 changed files:

Changes: