[Notes] [Git][BuildStream/buildstream][jonathan/test-missing-workspace-g

Jonathan Maw pushed to branch jonathan/test-missing-workspace-guessing at BuildStream / buildstream

Commits:

cf0e7d26

by Jürg Billeter at 2019-01-04T07:07:59Z

_frontend/cli.py: Reinstate support for guessing targets

Moving fetch and track to the source command group accidentally dropped
the support for guessing targets for these commands when invoked from a
workspace directory. This brings it back.

0751fc8f

by Jürg Billeter at 2019-01-04T11:27:58Z

Merge branch 'juerg/source-guess-target' into 'master'

_frontend/cli.py: Reinstate support for guessing targets

See merge request BuildStream/buildstream!1036

ebd965fe

by Chandan Singh at 2019-01-04T15:21:35Z

Rename "tools" directory to "requirements"

This seems like a better name for the directory, as it more closely
describes the purpose of its contents.

bb712886
by Chandan Singh at 2019-01-04T15:21:35Z
```
Add Makefile to update requirements files
```

b9792837

by Chandan Singh at 2019-01-04T15:21:35Z

CONTRIBUTING.rst: Add instructions to update requirements files

Split the "The MANIFEST.in and setup.py" section in two:
"Managing data files" and "Updating BuildStream's Python dependencies".

Briefly explain the layout of `requirements` directory and add
instructions to use the Makefile added in the last commit.

45692335

by Tristan Van Berkom at 2019-01-04T15:50:14Z

Merge branch 'chandan/update-requirements-one-liner' into 'master'

Add Makefile to update requirements files

See merge request BuildStream/buildstream!1035

f42dcb54

by Chandan Singh at 2019-01-04T17:16:43Z

.gitlab-ci.yml: Remove prepare stage

As we now run tests using `tox`, we don't need to worry about manually
packing and unpacking BuildStream. So, we can remove the preapre stage
entirely.

Update `coverage` and nightly jobs to appropriately cope with this
change. Both these jobs now install all runtime dependencies from
requirements files.

ee29f207

by Chandan Singh at 2019-01-04T17:41:02Z

Merge branch 'chandan/no-sdist' into 'master'

.gitlab-ci.yml: Remove prepare stage

See merge request BuildStream/buildstream!1037

f122cfe8

by Jürg Billeter at 2019-01-06T17:57:33Z

_frontend/app.py: Use buildtree for interactive shell on build failure

Fixes: e29aea36 ("Basic options for shell --build to use buildtrees")

7dfb85b3

by Tristan Van Berkom at 2019-01-06T18:31:25Z

Merge branch 'juerg/shell-buildtree' into 'master'

_frontend/app.py: Use buildtree for interactive shell on build failure

See merge request BuildStream/buildstream!1039

6baec9f5

by Jonathan Maw at 2019-01-07T13:27:15Z

Test that tracking in workspaces actually works

Previously, it merely tested that buildstream did not fall other,
rather than whether it did anything useful.

854c7ea2

by Jonathan Maw at 2019-01-07T13:27:15Z

tests: Test that fetching an open workspace will fetch its dependencies

Previously, there was no way of detecting whether fetching happened, as
an element with an open workspace will not be fetched.

15 changed files:

.gitlab-ci.yml
CONTRIBUTING.rst
MANIFEST.in
buildstream/_frontend/app.py
buildstream/_frontend/cli.py
+ requirements/Makefile
tools/dev-requirements.in → requirements/dev-requirements.in
tools/dev-requirements.txt → requirements/dev-requirements.txt
tools/plugin-requirements.in → requirements/plugin-requirements.in
tools/plugin-requirements.txt → requirements/plugin-requirements.txt
tools/requirements.in → requirements/requirements.in
tools/requirements.txt → requirements/requirements.txt
setup.py
tests/frontend/workspace.py
tox.ini

Changes:

.gitlab-ci.yml

@@ -6,7 +6,6 @@ cache:
      - cache/
  stages:
 -  - prepare
    - test
    - post
@@ -15,41 +14,6 @@ variables:
    INTEGRATION_CACHE: "${CI_PROJECT_DIR}/cache/integration-cache"
    TEST_COMMAND: "tox -- --color=yes --integration"
 -#####################################################
 -#                  Prepare stage                    #
 -#####################################################
+-
 -# Create a source distribution
 -#
 -source_dist:
 -  stage: prepare
 -  script:
+-
 -  # Generate the source distribution tarball
 -  #
 -  - python3 setup.py sdist
 -  - tar -ztf dist/*
 -  - tarball=$(cd dist && echo $(ls *))
+-
 -  # Verify that the source distribution tarball can be installed correctly
 -  #
 -  - pip3 install dist/*.tar.gz
 -  - bst --version
+-
 -  # unpack tarball as `dist/buildstream` directory
 -  - |
 -    cat > dist/unpack.sh << EOF
 -    #!/bin/sh
 -    tar -zxf ${tarball}
 -    mv ${tarball%.tar.gz} buildstream
 -    EOF
+-
 -  # Make our helpers executable
 -  - chmod +x dist/unpack.sh
 -  artifacts:
 -    paths:
 -    - dist/
+-
  #####################################################
  #                    Test stage                     #
@@ -72,8 +36,7 @@ source_dist:
    - useradd -Um buildstream
    - chown -R buildstream:buildstream .
 -  # Run the tests from the source distribution, We run as a simple
 -  # user to test for permission issues
 +  # Run the tests as a simple user to test for permission issues
    - su buildstream -c "${TEST_COMMAND}"
    after_script:
@@ -192,7 +155,8 @@ docs:
      scheduler:
        fetchers: 2
      EOF
 -  - (cd dist && ./unpack.sh && cd buildstream && pip3 install .)
 +  - pip3 install -r requirements/requirements.txt -r requirements/plugin-requirements.txt
 +  - pip3 install --no-index .
    - pip3 install --user -e ${BST_EXT_URL}@${BST_EXT_REF}#egg=bst_ext
    - git clone https://gitlab.com/freedesktop-sdk/freedesktop-sdk.git
    - git -C freedesktop-sdk checkout ${FD_SDK_REF}
@@ -275,13 +239,12 @@ coverage:
    stage: post
    coverage: '/TOTAL +\d+ +\d+ +(\d+\.\d+)%/'
    script:
 -    - cd dist && ./unpack.sh && cd buildstream
 -    - pip3 install -r tools/requirements.txt -r tools/dev-requirements.txt
 +    - pip3 install -r requirements/requirements.txt -r requirements/dev-requirements.txt
      - pip3 install --no-index .
      - mkdir report
      - cd report
 -    - cp ../../../coverage-unix/coverage.* .
 -    - cp ../../../coverage-linux/coverage.* .
 +    - cp ../coverage-unix/coverage.* .
 +    - cp ../coverage-linux/coverage.* .
      - ls coverage.*
      - coverage combine --rcfile=../.coveragerc -a coverage.*
      - coverage report --rcfile=../.coveragerc -m
@@ -290,7 +253,6 @@ coverage:
    - tests-fedora-27
    - tests-fedora-28
    - tests-unix
 -  - source_dist
    except:
    - schedules
@@ -299,7 +261,6 @@ coverage:
  pages:
    stage: post
    dependencies:
 -  - source_dist
    - docs
    variables:
      ACME_DIR: public/.well-known/acme-challenge

CONTRIBUTING.rst

@@ -1736,10 +1736,8 @@ obtain profiles::
        ForceCommand BST_PROFILE=artifact-receive cd /tmp && bst-artifact-receive --pull-url https://example.com/ /home/artifacts/artifacts
 -The MANIFEST.in and setup.py
 -----------------------------
 -When adding a dependency to BuildStream, it's important to update the setup.py accordingly.
+-
 +Managing data files
 +-------------------
  When adding data files which need to be discovered at runtime by BuildStream, update setup.py accordingly.
  When adding data files for the purpose of docs or tests, or anything that is not covered by
@@ -1749,3 +1747,23 @@ At any time, running the following command to create a source distribution shoul
  creating a tarball which contains everything we want it to include::
    ./setup.py sdist
++
++
 +Updating BuildStream's Python dependencies
 +------------------------------------------
 +BuildStream's Python dependencies are listed in multiple
 +`requirements files <https://pip.readthedocs.io/en/latest/reference/pip_install/#requirements-file-format>`
 +present in the ``requirements`` directory.
++
 +All ``.txt`` files in this directory are generated from the corresponding
 +``.in`` file, and each ``.in`` file represents a set of dependencies. For
 +example, ``requirements.in`` contains all runtime dependencies of BuildStream.
 +``requirements.txt`` is generated from it, and contains pinned versions of all
 +runtime dependencies (including transitive dependencies) of BuildStream.
++
 +When adding a new dependency to BuildStream, or updating existing dependencies,
 +it is important to update the appropriate requirements file accordingly. After
 +changing the ``.in`` file, run the following to update the matching ``.txt``
 +file::
++
 +   make -C requirements

MANIFEST.in

@@ -32,12 +32,12 @@ include .pylintrc
  recursive-include buildstream/_protos *.proto
  # Requirements files
 -include tools/requirements.in
 -include tools/requirements.txt
 -include tools/dev-requirements.in
 -include tools/dev-requirements.txt
 -include tools/plugin-requirements.in
 -include tools/plugin-requirements.txt
 +include requirements/requirements.in
 +include requirements/requirements.txt
 +include requirements/dev-requirements.in
 +include requirements/dev-requirements.txt
 +include requirements/plugin-requirements.in
 +include requirements/plugin-requirements.txt
  # Versioneer
  include versioneer.py

buildstream/_frontend/app.py

@@ -599,7 +599,7 @@ class App():
                      click.echo("\nDropping into an interactive shell in the failed build sandbox\n", err=True)
                      try:
                          prompt = self.shell_prompt(element)
 -                        self.stream.shell(element, Scope.BUILD, prompt, isolate=True)
 +                        self.stream.shell(element, Scope.BUILD, prompt, isolate=True, usebuildtree=True)
                      except BstError as e:
                          click.echo("Error while attempting to create interactive shell: {}".format(e), err=True)
                  elif choice == 'log':

buildstream/_frontend/cli.py

@@ -719,6 +719,11 @@ def source_fetch(app, elements, deps, track_, except_, track_cross_junctions):
          deps = PipelineSelection.ALL
      with app.initialized(session_name="Fetch"):
 +        if not elements:
 +            guessed_target = app.context.guess_element()
 +            if guessed_target:
 +                elements = (guessed_target,)
++
          app.stream.fetch(elements,
                           selection=deps,
                           except_targets=except_,
@@ -755,6 +760,11 @@ def source_track(app, elements, deps, except_, cross_junctions):
          all:   All dependencies of all specified elements
      """
      with app.initialized(session_name="Track"):
 +        if not elements:
 +            guessed_target = app.context.guess_element()
 +            if guessed_target:
 +                elements = (guessed_target,)
++
          # Substitute 'none' for 'redirect' so that element redirections
          # will be done
          if deps == 'none':

requirements/Makefile

 +# Makefile for updating BuildStream's requirements files.
 +#
++
 +REQUIREMENTS_IN := $(wildcard *.in)
 +REQUIREMENTS_TXT := $(REQUIREMENTS_IN:.in=.txt)
 +PYTHON := python3
 +VENV := $(PYTHON) -m venv
++
 +VENV_PIP = $(VENVDIR)/bin/pip
++
++
 +.PHONY: all
++
 +all: $(REQUIREMENTS_TXT)
++
 +%.txt: %.in
 +	$(eval VENVDIR := $(shell mktemp -d $(CURDIR)/.bst-venv.XXXXXX))
 +	$(VENV) $(VENVDIR)
 +	$(VENV_PIP) install -r $^
 +	$(VENV_PIP) freeze -r $^ > $@
 +	rm -rf $(VENVDIR)

tools/dev-requirements.in → requirements/dev-requirements.in

tools/dev-requirements.txt → requirements/dev-requirements.txt

tools/plugin-requirements.in → requirements/plugin-requirements.in

tools/plugin-requirements.txt → requirements/plugin-requirements.txt

tools/requirements.in → requirements/requirements.in

tools/requirements.txt → requirements/requirements.txt

setup.py

@@ -270,10 +270,10 @@ def get_cmdclass():
  #####################################################
  #               Gather requirements                 #
  #####################################################
 -with open('tools/dev-requirements.in') as dev_reqs:
 +with open('requirements/dev-requirements.in') as dev_reqs:
      dev_requires = dev_reqs.read().splitlines()
 -with open('tools/requirements.in') as install_reqs:
 +with open('requirements/requirements.in') as install_reqs:
      install_requires = install_reqs.read().splitlines()
  #####################################################

tests/frontend/workspace.py

@@ -31,7 +31,7 @@ import shutil
  import subprocess
  from ruamel.yaml.comments import CommentedSet
  from tests.testutils import cli, create_repo, ALL_REPO_KINDS, wait_for_cache_granularity
 -from tests.testutils import create_artifact_share
 +from tests.testutils import create_artifact_share, create_element_size
  from buildstream import _yaml
  from buildstream._exceptions import ErrorDomain, LoadError, LoadErrorReason
@@ -119,7 +119,7 @@ class WorkspaceCreater():
          return element_tuples
      def open_workspaces(self, kinds, track, suffixs=None, workspace_dir=None,
 -                        element_attrs=None):
 +                        element_attrs=None, no_checkout=False):
          element_tuples = self.create_workspace_elements(kinds, track, suffixs, workspace_dir,
                                                          element_attrs)
@@ -130,6 +130,8 @@ class WorkspaceCreater():
          args = ['workspace', 'open']
          if track:
              args.append('--track')
 +        if no_checkout:
 +            args.append('--no-checkout')
          if workspace_dir is not None:
              assert len(element_tuples) == 1, "test logic error"
              _, workspace_dir = element_tuples[0]
@@ -140,23 +142,24 @@ class WorkspaceCreater():
          result.assert_success()
 -        for element_name, workspace_dir in element_tuples:
 -            # Assert that we are now buildable because the source is
 -            # now cached.
 -            assert self.cli.get_element_state(self.project_path, element_name) == 'buildable'
 +        if not no_checkout:
 +            for element_name, workspace_dir in element_tuples:
 +                # Assert that we are now buildable because the source is
 +                # now cached.
 +                assert self.cli.get_element_state(self.project_path, element_name) == 'buildable'
 -            # Check that the executable hello file is found in the workspace
 -            filename = os.path.join(workspace_dir, 'usr', 'bin', 'hello')
 -            assert os.path.exists(filename)
 +                # Check that the executable hello file is found in the workspace
 +                filename = os.path.join(workspace_dir, 'usr', 'bin', 'hello')
 +                assert os.path.exists(filename)
          return element_tuples
  def open_workspace(cli, tmpdir, datafiles, kind, track, suffix='', workspace_dir=None,
 -                   project_path=None, element_attrs=None):
 +                   project_path=None, element_attrs=None, no_checkout=False):
      workspace_object = WorkspaceCreater(cli, tmpdir, datafiles, project_path)
      workspaces = workspace_object.open_workspaces((kind, ), track, (suffix, ), workspace_dir,
 -                                                  element_attrs)
 +                                                  element_attrs, no_checkout)
      assert len(workspaces) == 1
      element_name, workspace = workspaces[0]
      return element_name, workspace_object.project_path, workspace
@@ -1068,25 +1071,35 @@ def test_multiple_failed_builds(cli, tmpdir, datafiles):
  @pytest.mark.parametrize('subdir', [True, False], ids=["subdir", "no-subdir"])
  @pytest.mark.parametrize("guess_element", [True, False], ids=["guess", "no-guess"])
  def test_external_fetch(cli, datafiles, tmpdir_factory, subdir, guess_element):
 -    # Fetching from a workspace outside a project doesn't fail horribly
 +    # An element with an open workspace can't be fetched, but we still expect fetches
 +    # to fetch any dependencies
      tmpdir = tmpdir_factory.mktemp('')
 -    element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", False)
 +    depend_element = 'fetchable.bst'
++
 +    # Create an element to fetch (local sources do not need to fetch)
 +    create_element_size(depend_element, str(datafiles), 'elements', [], 1024)
++
 +    element_name, project, workspace = open_workspace(
 +        cli, tmpdir, datafiles, "git", False, no_checkout=True,
 +        element_attrs={'depends': [depend_element]}
 +    )
      arg_elm = [element_name] if not guess_element else []
      if subdir:
          call_dir = os.path.join(workspace, 'usr')
 +        os.makedirs(call_dir, exist_ok=True)
      else:
          call_dir = workspace
 +    # Assert that the depended element is not fetched yet
 +    assert cli.get_element_state(str(datafiles), depend_element) == 'fetch needed'
++
 +    # Fetch the workspaced element
      result = cli.run(project=project, args=['-C', call_dir, 'source', 'fetch'] + arg_elm)
      result.assert_success()
 -    # We already fetched it by opening the workspace, but we're also checking
 -    # `bst show` works here
 -    result = cli.run(project=project,
 -                     args=['-C', call_dir, 'show', '--deps', 'none', '--format', '%{state}'] + arg_elm)
 -    result.assert_success()
 -    assert result.output.strip() == 'buildable'
 +    # Assert that the depended element has now been fetched
 +    assert cli.get_element_state(str(datafiles), depend_element) == 'buildable'
  @pytest.mark.datafiles(DATA_DIR)
@@ -1115,16 +1128,24 @@ def test_external_push_pull(cli, datafiles, tmpdir_factory, guess_element):
  @pytest.mark.datafiles(DATA_DIR)
  @pytest.mark.parametrize("guess_element", [True, False], ids=["guess", "no-guess"])
  def test_external_track(cli, datafiles, tmpdir_factory, guess_element):
 -    # Tracking does not get horribly confused
      tmpdir = tmpdir_factory.mktemp('')
 -    element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", True)
 +    element_name, project, workspace = open_workspace(cli, tmpdir, datafiles, "git", False)
 +    element_file = os.path.join(str(datafiles), 'elements', element_name)
      arg_elm = [element_name] if not guess_element else []
 -    # The workspace is necessarily already tracked, so we only care that
 -    # there's no weird errors.
 +    # Delete the ref from the source so that we can detect if the
 +    # element has been tracked
 +    element_contents = _yaml.load(element_file)
 +    del element_contents['sources'][0]['ref']
 +    _yaml.dump(_yaml.node_sanitize(element_contents), element_file)
++
      result = cli.run(project=project, args=['-C', workspace, 'source', 'track'] + arg_elm)
      result.assert_success()
 +    # Element is tracked now
 +    element_contents = _yaml.load(element_file)
 +    assert 'ref' in element_contents['sources'][0]
++
  @pytest.mark.datafiles(DATA_DIR)
  def test_external_open_other(cli, datafiles, tmpdir_factory):

tox.ini

@@ -5,9 +5,9 @@ skip_missing_interpreters = true
  [testenv]
  commands = pytest {posargs}
  deps =
 -    -rtools/requirements.txt
 -    -rtools/dev-requirements.txt
 -    -rtools/plugin-requirements.txt
 +    -rrequirements/requirements.txt
 +    -rrequirements/dev-requirements.txt
 +    -rrequirements/plugin-requirements.txt
  passenv =
      BST_FORCE_BACKEND
      GI_TYPELIB_PATH
@@ -18,9 +18,9 @@ commands =
      pycodestyle
      pylint buildstream
  deps =
 -    -rtools/requirements.txt
 -    -rtools/dev-requirements.txt
 -    -rtools/plugin-requirements.txt
 +    -rrequirements/requirements.txt
 +    -rrequirements/dev-requirements.txt
 +    -rrequirements/plugin-requirements.txt
  [testenv:docs]
  commands =
@@ -30,8 +30,8 @@ deps =
      sphinx==1.7.9
      sphinx-click
      sphinx_rtd_theme
 -    -rtools/requirements.txt
 -    -rtools/plugin-requirements.txt
 +    -rrequirements/requirements.txt
 +    -rrequirements/plugin-requirements.txt
  passenv =
      BST_FORCE_SESSION_REBUILD
      BST_SOURCE_CACHE

[Notes] [Git][BuildStream/buildstream][jonathan/test-missing-workspace-guessing] 12 commits: _frontend/cli.py: Reinstate support for guessing targets

Jonathan Maw pushed to branch jonathan/test-missing-workspace-guessing at BuildStream / buildstream

Commits:

15 changed files:

Changes: