[Notes] [Git][BuildStream/buildstream][jennis/filter-docs] 18 commits:

James Ennis pushed to branch jennis/filter-docs at BuildStream / buildstream

Commits:

a3e2cdd2

by Tom Pollard at 2019-01-28T12:14:40Z

_stream.py: Add use_artifact_config opt arg for load_selection()

use_artifact_config added as an optional default arg, allowing
for loading of given elements artifact remote config.

805baf7d

by Tom Pollard at 2019-01-28T12:14:40Z

Download buildtrees on demand for bst shell --use-buildtree

Provide bst shell --use-buildtree the ability to attempt to
acquire missing buildtrees, given respective option, user
pull-buildtree context and remote availability.

_frontend/cli.py: Refactor logic for determining --use-buildtree
option with given opportunity to attempt pulling a non-local
buildtree. Element loaded with artifact_config to allow remote
querying.

_stream.py: With given user option and element state, construct
PullQueue to fetch remote buildtree. Continue or Error without
buildtree if cannot be attained.

tests/integration/build-tree.py: Update to support new usecases

38356932

by Tom Pollard at 2019-01-28T13:44:37Z

Merge branch 'tpollard/829' into 'master'

Download buildtrees on demand for bst shell --use-buildtree

Closes #829

See merge request BuildStream/buildstream!1050

3590ca8c

by Abderrahim Kitouni at 2019-01-28T14:59:19Z

requirements/requirements.in: require protobuf >= 3.6

This is needed since 0f2bc3754

39b952dc

by Abderrahim Kitouni at 2019-01-28T15:19:04Z

requirements/requirements.in: require Click >= 7.0

This is needed since 629a6e524, and was lost in the conversion to requirements.in

80b36d0c

by Javier Jardón at 2019-01-28T17:54:52Z

Merge branch 'abderrahim/protobuf-version' into 'master'

requirements/requirements.in: update minimum versions

Closes #884

See merge request BuildStream/buildstream!1114

a1ab48da

by Valentin David at 2019-01-28T21:30:26Z

Fix crash when spawned job completes very fast

Job can complete before we return from `Job.span()` to
`Scheduler._spawn_job()`, so that `_active_jobs` would not yet contain
the job.

This would print a stack on the console and try to run a second time
the job which can have unexpected effects.

In order to reproduce the issue, in
`buildstream/_scheduler/jobs/job.py`, in `Job.spawn`,
add a call to `time.sleep()` right before call to
`asyncio.get_child_watcher()`.

This fixes issue #857.

2fcb4491

by Jürg Billeter at 2019-01-28T22:36:22Z

Merge branch 'valentindavid/crash_in_scheduler_857' into 'master'

Fix crash when spawned job completes very fast

Closes #857

See merge request BuildStream/buildstream!1095

1c05a092

by Valentin David at 2019-01-29T05:58:17Z

Fix type of error codes in CAS server

Fixes #882.

785da59c

by Jürg Billeter at 2019-01-29T06:49:10Z

Merge branch 'valentindavid/wrong_type_in_status_code' into 'master'

Fix type of error codes in CAS server

Closes #882

See merge request BuildStream/buildstream!1099

ddef91ea

by Valentin David at 2019-01-29T07:23:35Z

Make sure testing cache directory exists

Fixes #873

6a4c8611

by Jürg Billeter at 2019-01-29T08:18:45Z

Merge branch 'valentindavid/make_cache_dir' into 'master'

Make sure testing cache directory exists

Closes #873

See merge request BuildStream/buildstream!1092

86c8e414

by Angelos Evripiotis at 2019-01-29T10:39:29Z

BREAK:remove unconditional 'are you sure?' prompts

This is a breaking change, as it affects behaviour that people might be
relying on. An entry has been added to NEWS.

As proposed on the mailing list, this change removes the unconditional
prompts on:

    o: bst workspace reset
    o: bst workspace close --remove-dir

If interactive, these commands would always interrupt you with a prompt
like this:

    This will remove all your changes, are you sure?

This seems like it may just save someone's work some time. It may also
condition folks to hit 'y' quickly without thinking.

This change also makes the non-interactive behaviour consistent with the
interactive behaviour in the default case. There is also the case of the
prompt configured by 'really-workspace-close-project-inaccessible',
which may be tackled in later work.

This change also removes the new config options to suppress those
prompts, and their associated news entry.

The relevant bit of the mailing list conversation is here:
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00106.html

The issue to make interactive and non-interactive behaviour consistent
is here:
https://gitlab.com/BuildStream/buildstream/issues/744

aae35e13

by Angelos Evripiotis at 2019-01-29T12:08:07Z

Merge branch 'aevri/are_you_sure' into 'master'

BREAK:remove unconditional 'are you sure?' prompts

See merge request BuildStream/buildstream!1061

03111d39

by Dor Askayo at 2019-01-30T10:35:06Z

filter.py: don't recurse when staging dependencies

Also bump the element's version so cached artifacts would be
invalidated.

Fixes #883

7256bb0c

by James Ennis at 2019-01-30T11:34:04Z

Merge branch 'doraskayo/filter-indirect-deps-fix' into 'master'

filter.py: don't recurse when staging dependencies

Closes #883

See merge request BuildStream/buildstream!1110

4ae91c15

by James Ennis at 2019-01-31T11:13:41Z

filter.py/filter.yaml: Documentation improvements

64d8af6a

by James Ennis at 2019-01-31T11:13:41Z

filter.py: Add an example to the documentation

15 changed files:

NEWS
buildstream/_cas/casserver.py
buildstream/_context.py
buildstream/_frontend/cli.py
buildstream/_scheduler/scheduler.py
buildstream/_stream.py
buildstream/data/userconfig.yaml
buildstream/plugins/elements/filter.py
buildstream/plugins/elements/filter.yaml
conftest.py
requirements/requirements.in
tests/elements/filter.py
+ tests/elements/filter/basic/elements/input-with-deps.bst
+ tests/elements/filter/basic/elements/output-include-with-indirect-deps.bst
tests/integration/build-tree.py

Changes:

NEWS

@@ -50,6 +50,11 @@ buildstream 1.3.1
      an error message and a hint instead, to avoid bothering folks that just
      made a mistake.
 +  o BREAKING CHANGE: The unconditional 'Are you sure?' prompts have been
 +    removed. These would always ask you if you were sure when running
 +    'bst workspace close --remove-dir' or 'bst workspace reset'. They got in
 +    the way too often.
++
    o Failed builds are included in the cache as well.
      `bst checkout` will provide anything in `%{install-root}`.
      A build including cached fails will cause any dependant elements
@@ -87,12 +92,6 @@ buildstream 1.3.1
      instead of just a specially-formatted build-root with a `root` and `scratch`
      subdirectory.
 -  o The buildstream.conf file learned new
 -    'prompt.really-workspace-close-remove-dir' and
 -    'prompt.really-workspace-reset-hard' options. These allow users to suppress
 -    certain confirmation prompts, e.g. double-checking that the user meant to
 -    run the command as typed.
+-
    o Due to the element `build tree` being cached in the respective artifact their
      size in some cases has significantly increased. In *most* cases the build trees
      are not utilised when building targets, as such by default bst 'pull' & 'build'

buildstream/_cas/casserver.py

@@ -324,7 +324,7 @@ class _ContentAddressableStorageServicer(remote_execution_pb2_grpc.ContentAddres
              blob_response.digest.size_bytes = digest.size_bytes
              if len(blob_request.data) != digest.size_bytes:
 -                blob_response.status.code = grpc.StatusCode.FAILED_PRECONDITION
 +                blob_response.status.code = code_pb2.FAILED_PRECONDITION
                  continue
              try:
@@ -335,10 +335,10 @@ class _ContentAddressableStorageServicer(remote_execution_pb2_grpc.ContentAddres
                      out.flush()
                      server_digest = self.cas.add_object(path=out.name)
                      if server_digest.hash != digest.hash:
 -                        blob_response.status.code = grpc.StatusCode.FAILED_PRECONDITION
 +                        blob_response.status.code = code_pb2.FAILED_PRECONDITION
              except ArtifactTooLargeException:
 -                blob_response.status.code = grpc.StatusCode.RESOURCE_EXHAUSTED
 +                blob_response.status.code = code_pb2.RESOURCE_EXHAUSTED
          return response

buildstream/_context.py

@@ -121,18 +121,10 @@ class Context():
          # Whether or not to attempt to pull build trees globally
          self.pull_buildtrees = None
 -        # Boolean, whether we double-check with the user that they meant to
 -        # remove a workspace directory.
 -        self.prompt_workspace_close_remove_dir = None
+-
          # Boolean, whether we double-check with the user that they meant to
          # close the workspace when they're using it to access the project.
          self.prompt_workspace_close_project_inaccessible = None
 -        # Boolean, whether we double-check with the user that they meant to do
 -        # a hard reset of a workspace, potentially losing changes.
 -        self.prompt_workspace_reset_hard = None
+-
          # Whether elements must be rebuilt when their dependencies have changed
          self._strict_build_plan = None
@@ -260,16 +252,10 @@ class Context():
          prompt = _yaml.node_get(
              defaults, Mapping, 'prompt')
          _yaml.node_validate(prompt, [
 -            'really-workspace-close-remove-dir',
              'really-workspace-close-project-inaccessible',
 -            'really-workspace-reset-hard',
          ])
 -        self.prompt_workspace_close_remove_dir = _node_get_option_str(
 -            prompt, 'really-workspace-close-remove-dir', ['ask', 'yes']) == 'ask'
          self.prompt_workspace_close_project_inaccessible = _node_get_option_str(
              prompt, 'really-workspace-close-project-inaccessible', ['ask', 'yes']) == 'ask'
 -        self.prompt_workspace_reset_hard = _node_get_option_str(
 -            prompt, 'really-workspace-reset-hard', ['ask', 'yes']) == 'ask'
          # Load per-projects overrides
          self._project_overrides = _yaml.node_get(defaults, Mapping, 'projects', default_value={})

buildstream/_frontend/cli.py

@@ -526,7 +526,7 @@ def shell(app, element, sysroot, mount, isolate, build_, cli_buildtree, command)
      else:
          scope = Scope.RUN
 -    use_buildtree = False
 +    use_buildtree = None
      with app.initialized():
          if not element:
@@ -534,7 +534,8 @@ def shell(app, element, sysroot, mount, isolate, build_, cli_buildtree, command)
              if not element:
                  raise AppError('Missing argument "ELEMENT".')
 -        dependencies = app.stream.load_selection((element,), selection=PipelineSelection.NONE)
 +        dependencies = app.stream.load_selection((element,), selection=PipelineSelection.NONE,
 +                                                 use_artifact_config=True)
          element = dependencies[0]
          prompt = app.shell_prompt(element)
          mounts = [
@@ -543,20 +544,31 @@ def shell(app, element, sysroot, mount, isolate, build_, cli_buildtree, command)
+         ]
          cached = element._cached_buildtree()
 -        if cli_buildtree == "always":
 -            if cached:
 -                use_buildtree = True
 -            else:
 -                raise AppError("No buildtree is cached but the use buildtree option was specified")
 -        elif cli_buildtree == "never":
 -            pass
 -        elif cli_buildtree == "try":
 -            use_buildtree = cached
 +        if cli_buildtree in ("always", "try"):
 +            use_buildtree = cli_buildtree
 +            if not cached and use_buildtree == "always":
 +                click.echo("WARNING: buildtree is not cached locally, will attempt to pull from available remotes",
 +                           err=True)
          else:
 -            if app.interactive and cached:
 -                use_buildtree = bool(click.confirm('Do you want to use the cached buildtree?'))
 +            # If the value has defaulted to ask and in non interactive mode, don't consider the buildtree, this
 +            # being the default behaviour of the command
 +            if app.interactive and cli_buildtree == "ask":
 +                if cached and bool(click.confirm('Do you want to use the cached buildtree?')):
 +                    use_buildtree = "always"
 +                elif not cached:
 +                    try:
 +                        choice = click.prompt("Do you want to pull & use a cached buildtree?",
 +                                              type=click.Choice(['try', 'always', 'never']),
 +                                              err=True, show_choices=True)
 +                    except click.Abort:
 +                        click.echo('Aborting', err=True)
 +                        sys.exit(-1)
++
 +                    if choice != "never":
 +                        use_buildtree = choice
++
          if use_buildtree and not element._cached_success():
 -            click.echo("Warning: using a buildtree from a failed build.")
 +            click.echo("WARNING: using a buildtree from a failed build.", err=True)
          try:
              exitcode = app.stream.shell(element, scope, prompt,
@@ -829,11 +841,6 @@ def workspace_close(app, remove_dir, all_, elements):
          if nonexisting:
              raise AppError("Workspace does not exist", detail="\n".join(nonexisting))
 -        if app.interactive and remove_dir and app.context.prompt_workspace_close_remove_dir:
 -            if not click.confirm('This will remove all your changes, are you sure?'):
 -                click.echo('Aborting', err=True)
 -                sys.exit(-1)
+-
          for element_name in elements:
              app.stream.workspace_close(element_name, remove_dir=remove_dir)
@@ -867,11 +874,6 @@ def workspace_reset(app, soft, track_, all_, elements):
          if all_ and not app.stream.workspace_exists():
              raise AppError("No open workspaces to reset")
 -        if app.interactive and not soft and app.context.prompt_workspace_reset_hard:
 -            if not click.confirm('This will remove all your changes, are you sure?'):
 -                click.echo('Aborting', err=True)
 -                sys.exit(-1)
+-
          if all_:
              elements = tuple(element_name for element_name, _ in app.context.get_workspaces().list())

buildstream/_scheduler/scheduler.py

@@ -314,10 +314,10 @@ class Scheduler():
      #    job (Job): The job to spawn
+     #
      def _spawn_job(self, job):
 -        job.spawn()
          self._active_jobs.append(job)
          if self._job_start_callback:
              self._job_start_callback(job)
 +        job.spawn()
      # Callback for the cache size job
      def _cache_size_job_complete(self, status, cache_size):

buildstream/_stream.py

@@ -101,19 +101,22 @@ class Stream():
      #    targets (list of str): Targets to pull
      #    selection (PipelineSelection): The selection mode for the specified targets
      #    except_targets (list of str): Specified targets to except from fetching
 +    #    use_artifact_config (bool): If artifact remote config should be loaded
+     #
      # Returns:
      #    (list of Element): The selected elements
      def load_selection(self, targets, *,
                         selection=PipelineSelection.NONE,
 -                       except_targets=()):
 +                       except_targets=(),
 +                       use_artifact_config=False):
          profile_start(Topics.LOAD_SELECTION, "_".join(t.replace(os.sep, '-') for t in targets))
          elements, _ = self._load(targets, (),
                                   selection=selection,
                                   except_targets=except_targets,
 -                                 fetch_subprojects=False)
 +                                 fetch_subprojects=False,
 +                                 use_artifact_config=use_artifact_config)
          profile_end(Topics.LOAD_SELECTION, "_".join(t.replace(os.sep, '-') for t in targets))
@@ -131,7 +134,7 @@ class Stream():
      #    mounts (list of HostMount): Additional directories to mount into the sandbox
      #    isolate (bool): Whether to isolate the environment like we do in builds
      #    command (list): An argv to launch in the sandbox, or None
 -    #    usebuildtree (bool): Wheather to use a buildtree as the source.
 +    #    usebuildtree (str): Whether to use a buildtree as the source, given cli option
+     #
      # Returns:
      #    (int): The exit code of the launched shell
@@ -141,7 +144,7 @@ class Stream():
                mounts=None,
                isolate=False,
                command=None,
 -              usebuildtree=False):
 +              usebuildtree=None):
          # Assert we have everything we need built, unless the directory is specified
          # in which case we just blindly trust the directory, using the element
@@ -156,8 +159,31 @@ class Stream():
                  raise StreamError("Elements need to be built or downloaded before staging a shell environment",
                                    detail="\n".join(missing_deps))
 +        buildtree = False
 +        # Check if we require a pull queue attempt, with given artifact state and context
 +        if usebuildtree:
 +            if not element._cached_buildtree():
 +                require_buildtree = self._buildtree_pull_required([element])
 +                # Attempt a pull queue for the given element if remote and context allow it
 +                if require_buildtree:
 +                    self._message(MessageType.INFO, "Attempting to fetch missing artifact buildtree")
 +                    self._add_queue(PullQueue(self._scheduler))
 +                    self._enqueue_plan(require_buildtree)
 +                    self._run()
 +                    # Now check if the buildtree was successfully fetched
 +                    if element._cached_buildtree():
 +                        buildtree = True
 +                if not buildtree:
 +                    if usebuildtree == "always":
 +                        raise StreamError("Buildtree is not cached locally or in available remotes")
 +                    else:
 +                        self._message(MessageType.INFO, """Buildtree is not cached locally or in available remotes,
 +                                                        shell will be loaded without it""")
 +            else:
 +                buildtree = True
++
          return element._shell(scope, directory, mounts=mounts, isolate=isolate, prompt=prompt, command=command,
 -                              usebuildtree=usebuildtree)
 +                              usebuildtree=buildtree)
      # build()
+     #

buildstream/data/userconfig.yaml

@@ -112,14 +112,6 @@ logging:
+ #
  prompt:
 -  # Whether to really proceed with 'bst workspace close --remove-dir' removing
 -  # a workspace directory, potentially losing changes.
 -  #
 -  #  ask - Ask the user if they are sure.
 -  #  yes - Always remove, without asking.
 -  #
 -  really-workspace-close-remove-dir: ask
+-
    # Whether to really proceed with 'bst workspace close' when doing so would
    # stop them from running bst commands in this workspace.
+   #
@@ -127,11 +119,3 @@ prompt:
    #  yes - Always close, without asking.
+   #
    really-workspace-close-project-inaccessible: ask
+-
 -  # Whether to really proceed with 'bst workspace reset' doing a hard reset of
 -  # a workspace, potentially losing changes.
 -  #
 -  #  ask - Ask the user if they are sure.
 -  #  yes - Always hard reset, without asking.
 -  #
 -  really-workspace-reset-hard: ask

buildstream/plugins/elements/filter.py

@@ -20,25 +20,127 @@
  """
  filter - Extract a subset of files from another element
  =======================================================
 -This filters another element by producing an output that is a subset of
 -the filtered element.
 +Filter another element by producing an output that is a subset of
 +the parent element's output. Subsets are defined by the parent element's
 +:ref:`split rules <public_split_rules>`.
 -To specify the element to filter, specify it as the one and only build
 -dependency to filter. See :ref:`Dependencies <format_dependencies>`
 -for what dependencies are and how to specify them.
 +Overview
 +--------
 +A filter element must have exactly one *build* dependency, where said
 +dependency is the 'parent' element which we would like to filter.
 +Runtime dependencies may also be specified, which can be useful to propagate
 +forward from this filter element onto its reverse dependencies.
 +See :ref:`Dependencies <format_dependencies>` to see how we specify dependencies.
 -Dependencies aside from the filtered element may be specified, but
 -they must be runtime dependencies only. This can be useful to propagate
 -runtime dependencies forward from this filter element onto its reverse
 -dependencies.
 +When workspaces are opened, closed or reset on a filter element, or this
 +element is tracked, the filter element will transparently pass on the command
 +to its parent element (the sole build-dependency).
 -When workspaces are opened, closed or reset on this element, or this
 -element is tracked, instead of erroring due to a lack of sources, this
 -element will transparently pass on the command to its sole build-dependency.
 +Example
 +-------
 +Consider a simple import element, ``import.bst`` which imports the local files
 +'foo', 'bar' and 'baz' (each stored in ``files/``, relative to the project's root):
 -The default configuration and possible options are as such:
 -  .. literalinclude:: ../../../buildstream/plugins/elements/filter.yaml
 -     :language: yaml
 +.. code:: yaml
++
 +   kind: import
++
 +   # Specify sources to import
 +   sources:
 +   - kind: local
 +     path: files
++
 +   # Specify public domain data, visible to other elements
 +   public:
 +     bst:
 +       split-rules:
 +         foo:
 +         - /foo
 +         bar:
 +         - /bar
++
 +.. note::
++
 +   We can make an element's metadata visible to all reverse dependencies by making use
 +   of the ``public:`` field. See the :ref:`public data documentation <format_public>`
 +   for more information.
++
 +In this example, ``import.bst`` will serve as the 'parent' of the filter element, thus
 +its output will be filtered. It is important to understand that the artifact of the
 +above element will contain the files: 'foo', 'bar' and 'baz'.
++
 +Now, to produce an element whose artifact contains the file 'foo', and exlusively 'foo',
 +we can define the following filter, ``filter-foo.bst``:
++
 +.. code:: yaml
++
 +   kind: filter
++
 +   # Declare the sole build-dependency of the filter element
 +   depends:
 +   - filename: import.bst
 +     type: build
++
 +   # Declare a list of domains to include in the filter's artifact
 +   config:
 +     include:
 +     - foo
++
 +.. note::
++
 +   We can also specify build-dependencies with a 'build-depends' field which has been
 +   available since :ref:`format version 14 <project_format_version>`. See the
 +   :ref:`Build-Depends documentation <format_build_depends>` for more detail.
++
 +It should be noted that an 'empty' ``include:`` list would, by default, include all
 +split-rules specified in the parent element, which, in this example, would be the
 +files 'foo' and 'bar' (the file 'baz' was not covered by any split rules).
++
 +Equally, we can use the ``exclude:`` statement to create the same artifact (which
 +only contains the file 'foo') by declaring the following element, ``exclude-bar.bst``:
++
 +.. code:: yaml
++
 +   kind: filter
++
 +   # Declare the sole build-dependency of the filter element
 +   depends:
 +   - filename: import.bst
 +     type: build
++
 +   # Declare a list of domains to exclude in the filter's artifact
 +   config:
 +     exclude:
 +     - bar
++
 +In addition to the ``include:`` and ``exclude:`` fields, there exists an ``include-orphans:``
 +(Boolean) field, which defaults to ``False``. This will determine whether to include files
 +which are not present in the 'split-rules'. For example, if we wanted to filter out all files
 +which are not included as split rules we can define the following element, ``filter-misc.bst``:
++
 +.. code:: yaml
++
 +   kind: filter
++
 +   # Declare the sole build-dependency of the filter element
 +   depends:
 +   - filename: import.bst
 +     type: build
++
 +   # Filter out all files which are not declared as split rules
 +   config:
 +     exclude:
 +     - foo
 +     - bar
 +     include-orphans: True
++
 +The artifact of ``filter-misc.bst`` will only contain the file 'baz'.
++
 +Below is more information regarding the the default configurations and possible options
 +of the filter element:
++
 +.. literalinclude:: ../../../buildstream/plugins/elements/filter.yaml
 +   :language: yaml
  """
  from buildstream import Element, ElementError, Scope
@@ -47,6 +149,8 @@ from buildstream import Element, ElementError, Scope
  class FilterElement(Element):
      # pylint: disable=attribute-defined-outside-init
 +    BST_ARTIFACT_VERSION = 1
++
      # The filter element's output is its dependencies, so
      # we must rebuild if the dependencies change even when
      # not in strict build plans.
@@ -102,7 +206,7 @@ class FilterElement(Element):
      def assemble(self, sandbox):
          with self.timed_activity("Staging artifact", silent_nested=True):
 -            for dep in self.dependencies(Scope.BUILD):
 +            for dep in self.dependencies(Scope.BUILD, recurse=False):
                  dep.stage_artifact(sandbox, include=self.include,
                                     exclude=self.exclude, orphans=self.include_orphans)
          return ""

buildstream/plugins/elements/filter.yaml

@@ -2,20 +2,20 @@
  # Filter element configuration
  config:
 -  # A list of domains to include from each artifact, as
 -  # they were defined in the element's 'split-rules'.
 +  # A list of domains to include in each artifact, as
 +  # they were defined as public data in the parent
 +  # element's 'split-rules'.
+   #
 -  # Since domains can be added, it is not an error to
 -  # specify domains which may not exist for all of the
 -  # elements in this composition.
 +  # The filter element will fail when trying to include
 +  # domains that do not exist.
+   #
    # The default empty list indicates that all domains
 -  # from each dependency should be included.
 +  # of the parent's artifact should be included.
+   #
    include: []
    # A list of domains to exclude from each artifact, as
 -  # they were defined in the element's 'split-rules'.
 +  # they were defined in the parent element's 'split-rules'.
+   #
    # In the case that a file is spoken for by a domain
    # in the 'include' list and another in the 'exclude'
@@ -23,7 +23,7 @@ config:
    exclude: []
    # Whether to include orphan files which are not
 -  # included by any of the 'split-rules' present on
 -  # a given element.
 +  # included by any of the 'split-rules' present in
 +  # the parent element.
+   #
    include-orphans: False

conftest.py

@@ -54,6 +54,7 @@ class IntegrationCache():
      def __init__(self, cache):
          cache = os.path.abspath(cache)
 +        os.makedirs(cache, exist_ok=True)
          # Use the same sources every time
          self.sources = os.path.join(cache, 'sources')

requirements/requirements.in

 -Click
 +Click >= 7.0
  grpcio >= 1.10
  Jinja2 >= 2.10
  pluginbase
 -protobuf >= 3.5
 +protobuf >= 3.6
  psutil
  # According to ruamel.yaml's PyPI page, we are suppose to use
  # "<=0.15" in production until 0.15 becomes API stable.

tests/elements/filter.py

@@ -464,3 +464,23 @@ def test_filter_track_multi_exclude(datafiles, cli, tmpdir):
      assert "ref" not in new_input["sources"][0]
      new_input2 = _yaml.load(input2_file)
      assert new_input2["sources"][0]["ref"] == ref
++
++
 +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'basic'))
 +def test_filter_include_with_indirect_deps(datafiles, cli, tmpdir):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    result = cli.run(project=project, args=[
 +        'build', 'output-include-with-indirect-deps.bst'])
 +    result.assert_success()
++
 +    checkout = os.path.join(tmpdir.dirname, tmpdir.basename, 'checkout')
 +    result = cli.run(project=project, args=[
 +        'artifact', 'checkout', 'output-include-with-indirect-deps.bst', '--directory', checkout])
 +    result.assert_success()
++
 +    # direct dependencies should be staged and filtered
 +    assert os.path.exists(os.path.join(checkout, "baz"))
++
 +    # indirect dependencies shouldn't be staged and filtered
 +    assert not os.path.exists(os.path.join(checkout, "foo"))
 +    assert not os.path.exists(os.path.join(checkout, "bar"))

tests/elements/filter/basic/elements/input-with-deps.bst

 +kind: import
++
 +depends:
 +- filename: input.bst
++
 +sources:
 +- kind: local
 +  path: files
++
 +public:
 +  bst:
 +    split-rules:
 +      baz:
 +      - /baz

tests/elements/filter/basic/elements/output-include-with-indirect-deps.bst

 +kind: filter
++
 +depends:
 +- filename: input-with-deps.bst
 +  type: build

tests/integration/build-tree.py

@@ -101,7 +101,7 @@ def test_buildtree_from_failure(cli_integration, tmpdir, datafiles):
          'shell', '--build', element_name, '--use-buildtree', 'always', '--', 'cat', 'test'
      ])
      res.assert_success()
 -    assert "Warning: using a buildtree from a failed build" in res.output
 +    assert "WARNING: using a buildtree from a failed build" in res.stderr
      assert 'Hi' in res.output
@@ -141,7 +141,7 @@ def test_buildtree_pulled(cli, tmpdir, datafiles):
          res.assert_success()
 -# This test checks for correct behaviour if a buildtree is not present.
 +# This test checks for correct behaviour if a buildtree is not present in the local cache.
  @pytest.mark.datafiles(DATA_DIR)
  @pytest.mark.skipif(IS_LINUX and not HAVE_BWRAP, reason='Only available with bubblewrap on Linux')
  def test_buildtree_options(cli, tmpdir, datafiles):
@@ -156,6 +156,7 @@ def test_buildtree_options(cli, tmpdir, datafiles):
          result = cli.run(project=project, args=['build', element_name])
          result.assert_success()
          assert cli.get_element_state(project, element_name) == 'cached'
 +        assert share.has_artifact('test', element_name, cli.get_element_key(project, element_name))
          # Discard the cache
          cli.configure({
@@ -168,8 +169,6 @@ def test_buildtree_options(cli, tmpdir, datafiles):
          result = cli.run(project=project, args=['artifact', 'pull', '--deps', 'all', element_name])
          result.assert_success()
 -        # The above is the simplest way I know to create a local cache without any buildtrees.
+-
          # Check it's not using the cached build tree
          res = cli.run(project=project, args=[
              'shell', '--build', element_name, '--use-buildtree', 'never', '--', 'cat', 'test'
@@ -177,24 +176,51 @@ def test_buildtree_options(cli, tmpdir, datafiles):
          res.assert_shell_error()
          assert 'Hi' not in res.output
 -        # Check it's not correctly handling the lack of buildtree
 +        # Check it's not using the cached build tree, default is to ask, and fall back to not
 +        # for non interactive behavior
          res = cli.run(project=project, args=[
 -            'shell', '--build', element_name, '--use-buildtree', 'try', '--', 'cat', 'test'
 +            'shell', '--build', element_name, '--', 'cat', 'test'
          ])
          res.assert_shell_error()
          assert 'Hi' not in res.output
 -        # Check it's not using the cached build tree, default is to ask, and fall back to not
 -        # for non interactive behavior
 +        # Check correctly handling the lack of buildtree, with 'try' not attempting to
 +        # pull the buildtree as the user context is by default set to not pull them
          res = cli.run(project=project, args=[
 -            'shell', '--build', element_name, '--', 'cat', 'test'
 +            'shell', '--build', element_name, '--use-buildtree', 'try', '--', 'cat', 'test'
          ])
 -        res.assert_shell_error()
          assert 'Hi' not in res.output
 +        assert 'Attempting to fetch missing artifact buildtrees' not in res.stderr
 +        assert """Buildtree is not cached locally or in available remotes,
 +                shell will be loaded without it"""
 -        # Check it's using the cached build tree
 +        # Check correctly handling the lack of buildtree, with 'try' attempting and succeeding
 +        # to pull the buildtree as the user context allow the pulling of buildtrees and it is
 +        # available in the remote
 +        res = cli.run(project=project, args=[
 +            '--pull-buildtrees', 'shell', '--build', element_name, '--use-buildtree', 'try', '--', 'cat', 'test'
 +        ])
 +        assert 'Attempting to fetch missing artifact buildtree' in res.stderr
 +        assert 'Hi' in res.output
 +        shutil.rmtree(os.path.join(os.path.join(cli.directory, 'artifacts2')))
 +        assert cli.get_element_state(project, element_name) != 'cached'
++
 +        # Check it's not loading the shell at all with always set for the buildtree, when the
 +        # user context does not allow for buildtree pulling
 +        result = cli.run(project=project, args=['artifact', 'pull', '--deps', 'all', element_name])
 +        result.assert_success()
          res = cli.run(project=project, args=[
              'shell', '--build', element_name, '--use-buildtree', 'always', '--', 'cat', 'test'
          ])
          res.assert_main_error(ErrorDomain.PROG_NOT_FOUND, None)
 +        assert 'Buildtree is not cached locally or in available remotes' in res.stderr
          assert 'Hi' not in res.output
 +        assert 'Attempting to fetch missing artifact buildtree' not in res.stderr
++
 +        # Check that when user context is set to pull buildtrees and a remote has the buildtree,
 +        # 'always' will attempt and succeed at pulling the missing buildtree.
 +        res = cli.run(project=project, args=[
 +            '--pull-buildtrees', 'shell', '--build', element_name, '--use-buildtree', 'always', '--', 'cat', 'test'
 +        ])
 +        assert 'Hi' in res.output
 +        assert 'Attempting to fetch missing artifact buildtree' in res.stderr

[Notes] [Git][BuildStream/buildstream][jennis/filter-docs] 18 commits: _stream.py: Add use_artifact_config opt arg for load_selection()

James Ennis pushed to branch jennis/filter-docs at BuildStream / buildstream

Commits:

15 changed files:

Changes: