[Notes] [Git][BuildStream/buildstream][chandan/bst-checkout-build] 6 com

Chandan Singh pushed to branch chandan/bst-checkout-build at BuildStream / buildstream

Commits:

fdbf9be1

by Tiago Gomes at 2018-09-26T08:17:46Z

ci: update freedesktop-sdk ref

The current one doesn't build due server being down.

9280b0cf

by Tiago Gomes at 2018-09-26T15:54:53Z

ci: add ACME challenge

To finish the creation of the Let's Encrypt certificate, which will
allows us to use https://docs.buildstream.build/ for accessing the
documentation.

af131503

by Tiago Gomes at 2018-09-26T16:35:58Z

Merge branch 'tiagogomes/acme-challenge' into 'master'

ci: add ACME challenge

See merge request BuildStream/buildstream!831

f7f14f25
by Tiago Gomes at 2018-09-26T17:42:12Z
```
ci: add missing `-p` argument
```

6d13a58b

by Chandan Singh at 2018-09-26T20:14:09Z

Ensure `--deps=none` option works for `bst checkout`

Currently, `bst checkout --deps none` command always produces empty
output. Fix this issue and add regression test for the same.

As part of this, also rename the `dep` loop variable to `element` in
order to avoid confusion in future.

Fixes #670.

41c4c983

by Chandan Singh at 2018-09-26T20:14:09Z

Add `--deps build` option to `bst checkout`

As discussed in
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
add `--deps build` option to `bst checkout`. This will allow users to
checkout the all build dependencies of a given element using a single
command.

7 changed files:

.gitlab-ci.yml
buildstream/_frontend/cli.py
buildstream/element.py
buildstream/element_enums.py
tests/frontend/buildcheckout.py
+ tests/frontend/project/elements/checkout-deps.bst
+ tests/frontend/project/files/etc-files/etc/buildstream/config

Changes:

.gitlab-ci.yml

@@ -163,7 +163,7 @@ docs:
    variables:
      bst_ext_url: git+https://gitlab.com/BuildStream/bst-external.git
      bst_ext_ref: 1d6ab71151b93c8cbc0a91a36ffe9270f3b835f1 # 0.5.1
 -    fd_sdk_ref: 718ea88089644a1ea5b488de0b90c2c565cb75f8 # 18.08.12
 +    fd_sdk_ref: 88d7c22c2281b987faa02edd57df80d430eecf1f # 18.08.12
    before_script:
    - (cd dist && ./unpack.sh && cd buildstream && pip3 install .)
    - pip3 install --user -e ${bst_ext_url}@${bst_ext_ref}#egg=bst_ext
@@ -273,8 +273,14 @@ pages:
    dependencies:
    - source_dist
    - docs
 +  variables:
 +    ACME_DIR: public/.well-known/acme-challenge
    script:
 -  - find public/
 +  - mkdir -p ${ACME_DIR}
 +    # Required to finish the creation of the Let's Encrypt certificate,
 +    # which allows using https://docs.buildstream.build/ for accessing
 +    # the documentation.
 +  - echo ${ACME_CHALLENGE} > ${ACME_DIR}/$(echo ${ACME_CHALLENGE} | cut -c1-43)
    artifacts:
      paths:
      - public/

buildstream/_frontend/cli.py

@@ -630,7 +630,7 @@ def shell(app, element, sysroot, mount, isolate, build_, command):
  @click.option('--force', '-f', default=False, is_flag=True,
                help="Allow files to be overwritten")
  @click.option('--deps', '-d', default='run',
 -              type=click.Choice(['run', 'none']),
 +              type=click.Choice(['run', 'build', 'none']),
                help='The dependencies to checkout (default: run)')
  @click.option('--integrate/--no-integrate', default=True, is_flag=True,
                help="Whether to run integration commands")

buildstream/element.py

@@ -423,7 +423,7 @@ class Element(Plugin):
                                                  visited=visited, recursed=True)
          # Yeild self only at the end, after anything needed has been traversed
 -        if should_yield and (recurse or recursed) and (scope == Scope.ALL or scope == Scope.RUN):
 +        if should_yield and (recurse or recursed) and scope != Scope.BUILD:
              yield self
      def search(self, scope, name):
@@ -671,37 +671,37 @@ class Element(Plugin):
          if self.__can_build_incrementally() and workspace.last_successful:
              old_dep_keys = self.__get_artifact_metadata_dependencies(workspace.last_successful)
 -        for dep in self.dependencies(scope):
 +        for element in self.dependencies(scope):
              # If we are workspaced, and we therefore perform an
              # incremental build, we must ensure that we update the mtimes
              # of any files created by our dependencies since the last
              # successful build.
              to_update = None
              if workspace and old_dep_keys:
 -                dep.__assert_cached()
 +                element.__assert_cached()
 -                if dep.name in old_dep_keys:
 -                    key_new = dep._get_cache_key()
 -                    key_old = old_dep_keys[dep.name]
 +                if element.name in old_dep_keys:
 +                    key_new = element._get_cache_key()
 +                    key_old = old_dep_keys[element.name]
                      # We only need to worry about modified and added
                      # files, since removed files will be picked up by
                      # build systems anyway.
 -                    to_update, _, added = self.__artifacts.diff(dep, key_old, key_new, subdir='files')
 -                    workspace.add_running_files(dep.name, to_update + added)
 -                    to_update.extend(workspace.running_files[dep.name])
 +                    to_update, _, added = self.__artifacts.diff(element, key_old, key_new, subdir='files')
 +                    workspace.add_running_files(element.name, to_update + added)
 +                    to_update.extend(workspace.running_files[element.name])
                      # In case we are running `bst shell`, this happens in the
                      # main process and we need to update the workspace config
                      if utils._is_main_process():
                          self._get_context().get_workspaces().save_config()
 -            result = dep.stage_artifact(sandbox,
 -                                        path=path,
 -                                        include=include,
 -                                        exclude=exclude,
 -                                        orphans=orphans,
 -                                        update_mtimes=to_update)
 +            result = element.stage_artifact(sandbox,
 +                                            path=path,
 +                                            include=include,
 +                                            exclude=exclude,
 +                                            orphans=orphans,
 +                                            update_mtimes=to_update)
              if result.overwritten:
                  for overwrite in result.overwritten:
                      # Completely new overwrite
@@ -710,13 +710,13 @@ class Element(Plugin):
                          # written the element before
                          for elm, contents in files_written.items():
                              if overwrite in contents:
 -                                overlaps[overwrite] = [elm, dep.name]
 +                                overlaps[overwrite] = [elm, element.name]
                      else:
 -                        overlaps[overwrite].append(dep.name)
 -            files_written[dep.name] = result.files_written
 +                        overlaps[overwrite].append(element.name)
 +            files_written[element.name] = result.files_written
              if result.ignored:
 -                ignored[dep.name] = result.ignored
 +                ignored[element.name] = result.ignored
          if overlaps:
              overlap_warning = False
@@ -1318,17 +1318,24 @@ class Element(Plugin):
                  if scope == Scope.BUILD:
                      self.stage(sandbox)
                  elif scope == Scope.RUN:
++
 +                    if deps == 'build':
 +                        dependency_scope = Scope.BUILD
 +                    elif deps == 'run':
 +                        dependency_scope = Scope.RUN
 +                    else:
 +                        dependency_scope = Scope.NONE
++
                      # Stage deps in the sandbox root
 -                    if deps == 'run':
 -                        with self.timed_activity("Staging dependencies", silent_nested=True):
 -                            self.stage_dependency_artifacts(sandbox, scope)
+-
 -                        # Run any integration commands provided by the dependencies
 -                        # once they are all staged and ready
 -                        if integrate:
 -                            with self.timed_activity("Integrating sandbox"):
 -                                for dep in self.dependencies(scope):
 -                                    dep.integrate(sandbox)
 +                    with self.timed_activity("Staging dependencies", silent_nested=True):
 +                        self.stage_dependency_artifacts(sandbox, dependency_scope)
++
 +                    # Run any integration commands provided by the dependencies
 +                    # once they are all staged and ready
 +                    if integrate:
 +                        with self.timed_activity("Integrating sandbox"):
 +                            for dep in self.dependencies(dependency_scope):
 +                                dep.integrate(sandbox)
              yield sandbox

buildstream/element_enums.py

@@ -59,3 +59,7 @@ class Scope(Enum):
      """All elements required for running the element. Including the element
      itself.
      """
++
 +    NONE = 4
 +    """Just the element itself, no dependencies.
 +    """

tests/frontend/buildcheckout.py

@@ -61,13 +61,14 @@ def test_build_checkout(datafiles, cli, strict, hardlinks):
  @pytest.mark.datafiles(DATA_DIR)
 -@pytest.mark.parametrize("deps", [("run"), ("none")])
 +@pytest.mark.parametrize("deps", [("run"), ("none"), ("build")])
  def test_build_checkout_deps(datafiles, cli, deps):
      project = os.path.join(datafiles.dirname, datafiles.basename)
      checkout = os.path.join(cli.directory, 'checkout')
 +    element_name = "checkout-deps.bst"
      # First build it
 -    result = cli.run(project=project, args=['build', 'target.bst'])
 +    result = cli.run(project=project, args=['build', element_name])
      result.assert_success()
      # Assert that after a successful build, the builddir is empty
@@ -76,20 +77,25 @@ def test_build_checkout_deps(datafiles, cli, deps):
      assert not os.listdir(builddir)
      # Now check it out
 -    result = cli.run(project=project, args=['checkout', 'target.bst', '--deps', deps, checkout])
 +    result = cli.run(project=project, args=['checkout', element_name, '--deps', deps, checkout])
      result.assert_success()
 -    # Check that the executable hello file is found in the checkout
 -    filename = os.path.join(checkout, 'usr', 'bin', 'hello')
 +    # Verify output of this element
 +    filename = os.path.join(checkout, 'etc', 'buildstream', 'config')
 +    if deps == "build":
 +        assert not os.path.exists(filename)
 +    else:
 +        assert os.path.exists(filename)
 -    if deps == "run":
 +    # Verify output of this element's build dependencies
 +    filename = os.path.join(checkout, 'usr', 'include', 'pony.h')
 +    if deps == "build":
          assert os.path.exists(filename)
      else:
          assert not os.path.exists(filename)
 -    # Check that the executable hello file is found in the checkout
 -    filename = os.path.join(checkout, 'usr', 'include', 'pony.h')
+-
 +    # Verify output of this element's runtime dependencies
 +    filename = os.path.join(checkout, 'usr', 'bin', 'hello')
      if deps == "run":
          assert os.path.exists(filename)
      else:

tests/frontend/project/elements/checkout-deps.bst

 +kind: import
 +description: It is important for this element to have both build and runtime dependencies
 +sources:
 +- kind: local
 +  path: files/etc-files
 +depends:
 +- filename: import-dev.bst
 +  type: build
 +- filename: import-bin.bst
 +  type: runtime

tests/frontend/project/files/etc-files/etc/buildstream/config

+config

[Notes] [Git][BuildStream/buildstream][chandan/bst-checkout-build] 6 commits: ci: update freedesktop-sdk ref

Chandan Singh pushed to branch chandan/bst-checkout-build at BuildStream / buildstream

Commits:

7 changed files:

Changes: