[Notes] [Git][BuildStream/buildstream][chandan/bst-checkout-build] 4 com

Chandan Singh pushed to branch chandan/bst-checkout-build at BuildStream / buildstream

Commits:

da7e038b

by Jürg Billeter at 2018-09-24T13:58:55Z

element.py: Fix cache check in non-strict mode

The behavior of __assert_cached() was changed when the keystrength
parameter was introduced. This restores the previous behavior.

Fixes #607.

Fixes: 3d308894 ("element.py: Cache the result of checking whether...")

86ea1173

by Jürg Billeter at 2018-09-24T13:59:59Z

tests/frontend/rebuild.py: Add strict/non-strict rebuild test

Test rebuild that requires staging of weakly cached dependency.

This is a regression test for #607.

a76339de

by Jürg Billeter at 2018-09-24T16:02:38Z

Merge branch 'juerg/rebuild' into 'master'

element.py: Fix cache check in non-strict mode

Closes #607

See merge request BuildStream/buildstream!822

c8dfe7f7

by Chandan Singh at 2018-09-25T00:08:15Z

Add `--deps build` option to `bst checkout`

As discussed in
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
add `--deps build` option to `bst checkout`. This will allow users to
checkout the all build dependencies of a given element using a single
command.

This change also fixes an existing issue where `--deps none` always
resulted in an empty output. (#670)

Fixes #670.

8 changed files:

buildstream/_frontend/cli.py
buildstream/element.py
buildstream/element_enums.py
tests/frontend/buildcheckout.py
+ tests/frontend/project/elements/checkout-deps.bst
+ tests/frontend/project/elements/rebuild-target.bst
+ tests/frontend/project/files/etc-files/etc/buildstream/config
+ tests/frontend/rebuild.py

Changes:

buildstream/_frontend/cli.py

@@ -630,7 +630,7 @@ def shell(app, element, sysroot, mount, isolate, build_, command):
  @click.option('--force', '-f', default=False, is_flag=True,
                help="Allow files to be overwritten")
  @click.option('--deps', '-d', default='run',
 -              type=click.Choice(['run', 'none']),
 +              type=click.Choice(['run', 'build', 'none']),
                help='The dependencies to checkout (default: run)')
  @click.option('--integrate/--no-integrate', default=True, is_flag=True,
                help="Whether to run integration commands")

buildstream/element.py

@@ -423,7 +423,7 @@ class Element(Plugin):
                                                  visited=visited, recursed=True)
          # Yeild self only at the end, after anything needed has been traversed
 -        if should_yield and (recurse or recursed) and (scope == Scope.ALL or scope == Scope.RUN):
 +        if should_yield and (recurse or recursed) and scope != Scope.BUILD:
              yield self
      def search(self, scope, name):
@@ -671,37 +671,37 @@ class Element(Plugin):
          if self.__can_build_incrementally() and workspace.last_successful:
              old_dep_keys = self.__get_artifact_metadata_dependencies(workspace.last_successful)
 -        for dep in self.dependencies(scope):
 +        for element in self.dependencies(scope):
              # If we are workspaced, and we therefore perform an
              # incremental build, we must ensure that we update the mtimes
              # of any files created by our dependencies since the last
              # successful build.
              to_update = None
              if workspace and old_dep_keys:
 -                dep.__assert_cached()
 +                element.__assert_cached()
 -                if dep.name in old_dep_keys:
 -                    key_new = dep._get_cache_key()
 -                    key_old = old_dep_keys[dep.name]
 +                if element.name in old_dep_keys:
 +                    key_new = element._get_cache_key()
 +                    key_old = old_dep_keys[element.name]
                      # We only need to worry about modified and added
                      # files, since removed files will be picked up by
                      # build systems anyway.
 -                    to_update, _, added = self.__artifacts.diff(dep, key_old, key_new, subdir='files')
 -                    workspace.add_running_files(dep.name, to_update + added)
 -                    to_update.extend(workspace.running_files[dep.name])
 +                    to_update, _, added = self.__artifacts.diff(element, key_old, key_new, subdir='files')
 +                    workspace.add_running_files(element.name, to_update + added)
 +                    to_update.extend(workspace.running_files[element.name])
                      # In case we are running `bst shell`, this happens in the
                      # main process and we need to update the workspace config
                      if utils._is_main_process():
                          self._get_context().get_workspaces().save_config()
 -            result = dep.stage_artifact(sandbox,
 -                                        path=path,
 -                                        include=include,
 -                                        exclude=exclude,
 -                                        orphans=orphans,
 -                                        update_mtimes=to_update)
 +            result = element.stage_artifact(sandbox,
 +                                            path=path,
 +                                            include=include,
 +                                            exclude=exclude,
 +                                            orphans=orphans,
 +                                            update_mtimes=to_update)
              if result.overwritten:
                  for overwrite in result.overwritten:
                      # Completely new overwrite
@@ -710,13 +710,13 @@ class Element(Plugin):
                          # written the element before
                          for elm, contents in files_written.items():
                              if overwrite in contents:
 -                                overlaps[overwrite] = [elm, dep.name]
 +                                overlaps[overwrite] = [elm, element.name]
                      else:
 -                        overlaps[overwrite].append(dep.name)
 -            files_written[dep.name] = result.files_written
 +                        overlaps[overwrite].append(element.name)
 +            files_written[element.name] = result.files_written
              if result.ignored:
 -                ignored[dep.name] = result.ignored
 +                ignored[element.name] = result.ignored
          if overlaps:
              overlap_warning = False
@@ -1318,17 +1318,24 @@ class Element(Plugin):
                  if scope == Scope.BUILD:
                      self.stage(sandbox)
                  elif scope == Scope.RUN:
++
 +                    if deps == 'build':
 +                        dependency_scope = Scope.BUILD
 +                    elif deps == 'run':
 +                        dependency_scope = Scope.RUN
 +                    else:
 +                        dependency_scope = Scope.NONE
++
                      # Stage deps in the sandbox root
 -                    if deps == 'run':
 -                        with self.timed_activity("Staging dependencies", silent_nested=True):
 -                            self.stage_dependency_artifacts(sandbox, scope)
 +                    with self.timed_activity("Staging dependencies", silent_nested=True):
 +                        self.stage_dependency_artifacts(sandbox, dependency_scope)
 -                        # Run any integration commands provided by the dependencies
 -                        # once they are all staged and ready
 -                        if integrate:
 -                            with self.timed_activity("Integrating sandbox"):
 -                                for dep in self.dependencies(scope):
 -                                    dep.integrate(sandbox)
 +                    # Run any integration commands provided by the dependencies
 +                    # once they are all staged and ready
 +                    if integrate:
 +                        with self.timed_activity("Integrating sandbox"):
 +                            for dep in self.dependencies(dependency_scope):
 +                                dep.integrate(sandbox)
              yield sandbox
@@ -2084,7 +2091,7 @@ class Element(Plugin):
+     #
      # Raises an error if the artifact is not cached.
+     #
 -    def __assert_cached(self, keystrength=_KeyStrength.STRONG):
 +    def __assert_cached(self, keystrength=None):
          assert self.__is_cached(keystrength=keystrength), "{}: Missing artifact {}".format(
              self, self._get_brief_display_key())

buildstream/element_enums.py

@@ -59,3 +59,7 @@ class Scope(Enum):
      """All elements required for running the element. Including the element
      itself.
      """
++
 +    NONE = 4
 +    """Just the element itself, no dependencies.
 +    """

tests/frontend/buildcheckout.py

@@ -61,13 +61,14 @@ def test_build_checkout(datafiles, cli, strict, hardlinks):
  @pytest.mark.datafiles(DATA_DIR)
 -@pytest.mark.parametrize("deps", [("run"), ("none")])
 +@pytest.mark.parametrize("deps", [("run"), ("none"), ("build")])
  def test_build_checkout_deps(datafiles, cli, deps):
      project = os.path.join(datafiles.dirname, datafiles.basename)
      checkout = os.path.join(cli.directory, 'checkout')
 +    element_name = 'checkout-deps.bst'
      # First build it
 -    result = cli.run(project=project, args=['build', 'target.bst'])
 +    result = cli.run(project=project, args=['build', element_name])
      result.assert_success()
      # Assert that after a successful build, the builddir is empty
@@ -76,20 +77,25 @@ def test_build_checkout_deps(datafiles, cli, deps):
      assert not os.listdir(builddir)
      # Now check it out
 -    result = cli.run(project=project, args=['checkout', 'target.bst', '--deps', deps, checkout])
 +    result = cli.run(project=project, args=['checkout', element_name, '--deps', deps, checkout])
      result.assert_success()
 -    # Check that the executable hello file is found in the checkout
 -    filename = os.path.join(checkout, 'usr', 'bin', 'hello')
 +    # Verify output of this element
 +    filename = os.path.join(checkout, 'etc', 'buildstream', 'config')
 +    if deps == "build":
 +        assert not os.path.exists(filename)
 +    else:
 +        assert os.path.exists(filename)
 -    if deps == "run":
 +    # Verify output of this element's build dependencies
 +    filename = os.path.join(checkout, 'usr', 'include', 'pony.h')
 +    if deps == "build":
          assert os.path.exists(filename)
      else:
          assert not os.path.exists(filename)
 -    # Check that the executable hello file is found in the checkout
 -    filename = os.path.join(checkout, 'usr', 'include', 'pony.h')
+-
 +    # Verify output of this element's runtime dependencies
 +    filename = os.path.join(checkout, 'usr', 'bin', 'hello')
      if deps == "run":
          assert os.path.exists(filename)
      else:

tests/frontend/project/elements/checkout-deps.bst

 +kind: import
 +description: It is important for this element to have both build and runtime dependencies
 +sources:
 +- kind: local
 +  path: files/etc-files
 +depends:
 +- filename: import-dev.bst
 +  type: build
 +- filename: import-bin.bst
 +  type: runtime

tests/frontend/project/elements/rebuild-target.bst

 +kind: compose
++
 +build-depends:
 +- target.bst

tests/frontend/project/files/etc-files/etc/buildstream/config

+config

tests/frontend/rebuild.py

 +import os
 +import pytest
 +from tests.testutils import cli
++
 +# Project directory
 +DATA_DIR = os.path.join(
 +    os.path.dirname(os.path.realpath(__file__)),
 +    "project",
 +)
++
++
 +def strict_args(args, strict):
 +    if strict != "strict":
 +        return ['--no-strict'] + args
 +    return args
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize("strict", ["strict", "non-strict"])
 +def test_rebuild(datafiles, cli, strict):
 +    project = os.path.join(datafiles.dirname, datafiles.basename)
 +    checkout = os.path.join(cli.directory, 'checkout')
++
 +    # First build intermediate target.bst
 +    result = cli.run(project=project, args=strict_args(['build', 'target.bst'], strict))
 +    result.assert_success()
++
 +    # Modify base import
 +    with open(os.path.join(project, 'files', 'dev-files', 'usr', 'include', 'new.h'), "w") as f:
 +        f.write("#define NEW")
++
 +    # Rebuild base import and build top-level rebuild-target.bst
 +    # In non-strict mode, this does not rebuild intermediate target.bst,
 +    # which means that a weakly cached target.bst will be staged as dependency.
 +    result = cli.run(project=project, args=strict_args(['build', 'rebuild-target.bst'], strict))
 +    result.assert_success()

[Notes] [Git][BuildStream/buildstream][chandan/bst-checkout-build] 4 commits: element.py: Fix cache check in non-strict mode

Chandan Singh pushed to branch chandan/bst-checkout-build at BuildStream / buildstream

Commits:

8 changed files:

Changes: