[Notes] [Git][BuildStream/buildstream][jjardon/doc_git

Tristan Van Berkom pushed to branch jjardon/doc_git_submodules at BuildStream / buildstream

Commits:

cc5ef68c

by Tristan Van Berkom at 2018-12-03T08:41:42Z

MANIFEST.in: Include the sources for rebuilding the documentation session HTML

This was previously missing, causing our CI to forget to rebuild the sessions
when updating the documentation.

This fixes issue #778

87ba19d7

by Tristan Van Berkom at 2018-12-03T09:19:54Z

Merge branch 'tristan/fix-doc-session-rebuilds' into 'master'

MANIFEST.in: Include the sources for rebuilding the documentation session HTML

Closes #778

See merge request BuildStream/buildstream!984

1b24148f

by Chandan Singh at 2018-12-03T09:46:13Z

Ensure `--deps=none` option works for `bst checkout`

Currently, `bst checkout --deps none` command always produces empty
output. Fix this issue and add regression test for the same.

- element_enums.py: Add Scope.NONE.

- element.py: Ensure Scope.NONE works correctly in addition to
  Scope.RUN/Scope.ALL in Element.dependencies() and Element.search().

- tests/frontend/buildcheckout.py: Fix tests for `--deps none`.

Fixes #670.

8ac1e757

by Chandan Singh at 2018-12-03T09:46:14Z

Add `--deps build` option to `bst checkout`

As discussed in
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
add `--deps build` option to `bst checkout`. This will allow users to
checkout the all build dependencies of a given element using a single
command.

- _frontend/cli.py: Add `--deps build` option for `bst checkout`.

- element.py: Support `deps='build'` in Element._prepare_sandbox().

- tests/frontend/buildcheckout.py: Ensure `--deps build` works as
  expected.

ac60f47a

by Benjamin Schubert at 2018-12-03T09:46:14Z

Refactor and simplify _prepare_sandbox for elements

Before we would have a intricate logics with multiple arguments
that might get ignored.

This simplifies the design and introduces a bool `shell` instead
of having two different variables concerned about scope

b1430fb5

by Tristan Van Berkom at 2018-12-03T11:01:54Z

Merge branch 'chandan/bst-checkout-build' into 'master'

Add `--deps build` option to `bst checkout`

Closes #670

See merge request BuildStream/buildstream!819

9022a778

by Javier Jardón at 2018-12-03T11:14:55Z

buildstream/plugins/sources/git.py: warn about checkout by default

See #783

Changes:

MANIFEST.in

@@ -19,6 +19,7 @@ recursive-include doc/source *.html
  recursive-include doc/source *.odg
  recursive-include doc/source *.svg
  recursive-include doc/examples *
 +recursive-include doc/sessions *.run
  # Tests
  recursive-include tests *

buildstream/_frontend/cli.py

@@ -640,7 +640,7 @@ def shell(app, element, sysroot, mount, isolate, build_, command):
  @click.option('--force', '-f', default=False, is_flag=True,
                help="Allow files to be overwritten")
  @click.option('--deps', '-d', default='run',
 -              type=click.Choice(['run', 'none']),
 +              type=click.Choice(['run', 'build', 'none']),
                help='The dependencies to checkout (default: run)')
  @click.option('--integrate/--no-integrate', default=True, is_flag=True,
                help="Whether to run integration commands")
@@ -657,16 +657,24 @@ def shell(app, element, sysroot, mount, isolate, build_, command):
  def checkout(app, element, location, force, deps, integrate, hardlinks, tar):
      """Checkout a built artifact to the specified location
      """
 +    from ..element import Scope
      if hardlinks and tar:
          click.echo("ERROR: options --hardlinks and --tar conflict", err=True)
          sys.exit(-1)
 +    if deps == "run":
 +        scope = Scope.RUN
 +    elif deps == "build":
 +        scope = Scope.BUILD
 +    elif deps == "none":
 +        scope = Scope.NONE
++
      with app.initialized():
          app.stream.checkout(element,
                              location=location,
                              force=force,
 -                            deps=deps,
 +                            scope=scope,
                              integrate=integrate,
                              hardlinks=hardlinks,
                              tar=tar)

buildstream/_stream.py

@@ -370,7 +370,7 @@ class Stream():
      #    target (str): Target to checkout
      #    location (str): Location to checkout the artifact to
      #    force (bool): Whether files can be overwritten if necessary
 -    #    deps (str): The dependencies to checkout
 +    #    scope (str): The scope of dependencies to checkout
      #    integrate (bool): Whether to run integration commands
      #    hardlinks (bool): Whether checking out files hardlinked to
      #                      their artifacts is acceptable
@@ -383,7 +383,7 @@ class Stream():
      def checkout(self, target, *,
                   location=None,
                   force=False,
 -                 deps='run',
 +                 scope=Scope.RUN,
                   integrate=True,
                   hardlinks=False,
                   tar=False):
@@ -396,7 +396,7 @@ class Stream():
          # Stage deps into a temporary sandbox first
          try:
 -            with target._prepare_sandbox(Scope.RUN, None, deps=deps,
 +            with target._prepare_sandbox(scope=scope, directory=None,
                                           integrate=integrate) as sandbox:
                  # Copy or move the sandbox to the target directory

buildstream/element.py

@@ -438,7 +438,7 @@ class Element(Plugin):
                                                  visited=visited, recursed=True)
          # Yeild self only at the end, after anything needed has been traversed
 -        if should_yield and (recurse or recursed) and (scope in (Scope.ALL, Scope.RUN)):
 +        if should_yield and (recurse or recursed) and scope != Scope.BUILD:
              yield self
      def search(self, scope, name):
@@ -1339,7 +1339,7 @@ class Element(Plugin):
      # is used to stage things by the `bst checkout` codepath
+     #
      @contextmanager
 -    def _prepare_sandbox(self, scope, directory, deps='run', integrate=True):
 +    def _prepare_sandbox(self, scope, directory, shell=False, integrate=True):
          # bst shell and bst checkout require a local sandbox.
          bare_directory = True if directory else False
          with self.__sandbox(directory, config=self.__sandbox_config, allow_remote=False,
@@ -1350,20 +1350,19 @@ class Element(Plugin):
              # Stage something if we need it
              if not directory:
 -                if scope == Scope.BUILD:
 +                if shell and scope == Scope.BUILD:
                      self.stage(sandbox)
 -                elif scope == Scope.RUN:
 +                else:
                      # Stage deps in the sandbox root
 -                    if deps == 'run':
 -                        with self.timed_activity("Staging dependencies", silent_nested=True):
 -                            self.stage_dependency_artifacts(sandbox, scope)
 +                    with self.timed_activity("Staging dependencies", silent_nested=True):
 +                        self.stage_dependency_artifacts(sandbox, scope)
 -                        # Run any integration commands provided by the dependencies
 -                        # once they are all staged and ready
 -                        if integrate:
 -                            with self.timed_activity("Integrating sandbox"):
 -                                for dep in self.dependencies(scope):
 -                                    dep.integrate(sandbox)
 +                    # Run any integration commands provided by the dependencies
 +                    # once they are all staged and ready
 +                    if integrate:
 +                        with self.timed_activity("Integrating sandbox"):
 +                            for dep in self.dependencies(scope):
 +                                dep.integrate(sandbox)
              yield sandbox
@@ -1858,7 +1857,7 @@ class Element(Plugin):
      # If directory is not specified, one will be staged using scope
      def _shell(self, scope=None, directory=None, *, mounts=None, isolate=False, prompt=None, command=None):
 -        with self._prepare_sandbox(scope, directory) as sandbox:
 +        with self._prepare_sandbox(scope, directory, shell=True) as sandbox:
              environment = self.get_environment()
              environment = copy.copy(environment)
              flags = SandboxFlags.INTERACTIVE | SandboxFlags.ROOT_READ_ONLY

buildstream/plugins/sources/git.py

@@ -25,6 +25,11 @@ git - stage files from a git repository
    * git
 +.. attention::
++
 +    Note that this plugin **will checkout git submodules by default**; even if
 +    they are not specified in the `.bst` file.
++
  **Usage:**
  .. code:: yaml

buildstream/types.py

@@ -48,6 +48,12 @@ class Scope(Enum):
      itself.
      """
 +    NONE = 4
 +    """Just the element itself, no dependencies.
++
 +    *Since: 1.4*
 +    """
++
  class Consistency():
      """Defines the various consistency states of a :class:`.Source`.

tests/frontend/buildcheckout.py

@@ -86,13 +86,14 @@ def test_build_invalid_suffix_dep(datafiles, cli, strict, hardlinks):
  @pytest.mark.datafiles(DATA_DIR)
 -@pytest.mark.parametrize("deps", [("run"), ("none")])
 +@pytest.mark.parametrize("deps", [("run"), ("none"), ("build")])
  def test_build_checkout_deps(datafiles, cli, deps):
      project = os.path.join(datafiles.dirname, datafiles.basename)
      checkout = os.path.join(cli.directory, 'checkout')
 +    element_name = "checkout-deps.bst"
      # First build it
 -    result = cli.run(project=project, args=['build', 'target.bst'])
 +    result = cli.run(project=project, args=['build', element_name])
      result.assert_success()
      # Assert that after a successful build, the builddir is empty
@@ -101,20 +102,25 @@ def test_build_checkout_deps(datafiles, cli, deps):
      assert not os.listdir(builddir)
      # Now check it out
 -    result = cli.run(project=project, args=['checkout', 'target.bst', '--deps', deps, checkout])
 +    result = cli.run(project=project, args=['checkout', element_name, '--deps', deps, checkout])
      result.assert_success()
 -    # Check that the executable hello file is found in the checkout
 -    filename = os.path.join(checkout, 'usr', 'bin', 'hello')
 +    # Verify output of this element
 +    filename = os.path.join(checkout, 'etc', 'buildstream', 'config')
 +    if deps == "build":
 +        assert not os.path.exists(filename)
 +    else:
 +        assert os.path.exists(filename)
 -    if deps == "run":
 +    # Verify output of this element's build dependencies
 +    filename = os.path.join(checkout, 'usr', 'include', 'pony.h')
 +    if deps == "build":
          assert os.path.exists(filename)
      else:
          assert not os.path.exists(filename)
 -    # Check that the executable hello file is found in the checkout
 -    filename = os.path.join(checkout, 'usr', 'include', 'pony.h')
+-
 +    # Verify output of this element's runtime dependencies
 +    filename = os.path.join(checkout, 'usr', 'bin', 'hello')
      if deps == "run":
          assert os.path.exists(filename)
      else:

[Notes] [Git][BuildStream/buildstream][jjardon/doc_git_submodules] 7 commits: MANIFEST.in: Include the sources for rebuilding the documentation session HTML

Tristan Van Berkom pushed to branch jjardon/doc_git_submodules at BuildStream / buildstream

Commits:

7 changed files:

Changes: