[Notes] [Git][BuildStream/buildstream][chandan/no-sdist] 7 commits:

Chandan Singh pushed to branch chandan/no-sdist at BuildStream / buildstream

Commits:

cf0e7d26

by Jürg Billeter at 2019-01-04T07:07:59Z

_frontend/cli.py: Reinstate support for guessing targets

Moving fetch and track to the source command group accidentally dropped
the support for guessing targets for these commands when invoked from a
workspace directory. This brings it back.

0751fc8f

by Jürg Billeter at 2019-01-04T11:27:58Z

Merge branch 'juerg/source-guess-target' into 'master'

_frontend/cli.py: Reinstate support for guessing targets

See merge request BuildStream/buildstream!1036

ebd965fe

by Chandan Singh at 2019-01-04T15:21:35Z

Rename "tools" directory to "requirements"

This seems like a better name for the directory, as it more closely
describes the purpose of its contents.

bb712886
by Chandan Singh at 2019-01-04T15:21:35Z
```
Add Makefile to update requirements files
```

b9792837

by Chandan Singh at 2019-01-04T15:21:35Z

CONTRIBUTING.rst: Add instructions to update requirements files

Split the "The MANIFEST.in and setup.py" section in two:
"Managing data files" and "Updating BuildStream's Python dependencies".

Briefly explain the layout of `requirements` directory and add
instructions to use the Makefile added in the last commit.

45692335

by Tristan Van Berkom at 2019-01-04T15:50:14Z

Merge branch 'chandan/update-requirements-one-liner' into 'master'

Add Makefile to update requirements files

See merge request BuildStream/buildstream!1035

6a446638

by Chandan Singh at 2019-01-04T15:54:49Z

.gitlab-ci.yml: Remove prepare stage

As we now run tests using `tox`, we don't need to worry about manually
packing and unpacking BuildStream. So, we can remove the preapre stage
entirely.

Update `coverage` and nightly jobs to appropriately cope with this
change.

13 changed files:

.gitlab-ci.yml
CONTRIBUTING.rst
MANIFEST.in
buildstream/_frontend/cli.py
+ requirements/Makefile
tools/dev-requirements.in → requirements/dev-requirements.in
tools/dev-requirements.txt → requirements/dev-requirements.txt
tools/plugin-requirements.in → requirements/plugin-requirements.in
tools/plugin-requirements.txt → requirements/plugin-requirements.txt
tools/requirements.in → requirements/requirements.in
tools/requirements.txt → requirements/requirements.txt
setup.py
tox.ini

Changes:

.gitlab-ci.yml

@@ -6,7 +6,6 @@ cache:
      - cache/
  stages:
 -  - prepare
    - test
    - post
@@ -15,41 +14,6 @@ variables:
    INTEGRATION_CACHE: "${CI_PROJECT_DIR}/cache/integration-cache"
    TEST_COMMAND: "tox -- --color=yes --integration"
 -#####################################################
 -#                  Prepare stage                    #
 -#####################################################
+-
 -# Create a source distribution
 -#
 -source_dist:
 -  stage: prepare
 -  script:
+-
 -  # Generate the source distribution tarball
 -  #
 -  - python3 setup.py sdist
 -  - tar -ztf dist/*
 -  - tarball=$(cd dist && echo $(ls *))
+-
 -  # Verify that the source distribution tarball can be installed correctly
 -  #
 -  - pip3 install dist/*.tar.gz
 -  - bst --version
+-
 -  # unpack tarball as `dist/buildstream` directory
 -  - |
 -    cat > dist/unpack.sh << EOF
 -    #!/bin/sh
 -    tar -zxf ${tarball}
 -    mv ${tarball%.tar.gz} buildstream
 -    EOF
+-
 -  # Make our helpers executable
 -  - chmod +x dist/unpack.sh
 -  artifacts:
 -    paths:
 -    - dist/
+-
  #####################################################
  #                    Test stage                     #
@@ -72,8 +36,7 @@ source_dist:
    - useradd -Um buildstream
    - chown -R buildstream:buildstream .
 -  # Run the tests from the source distribution, We run as a simple
 -  # user to test for permission issues
 +  # Run the tests as a simple user to test for permission issues
    - su buildstream -c "${TEST_COMMAND}"
    after_script:
@@ -192,7 +155,8 @@ docs:
      scheduler:
        fetchers: 2
      EOF
 -  - (cd dist && ./unpack.sh && cd buildstream && pip3 install .)
 +  - pip3 install -r requirements/requirements.txt -r requirements/dev-requirements.txt
 +  - pip3 install --no-index .
    - pip3 install --user -e ${BST_EXT_URL}@${BST_EXT_REF}#egg=bst_ext
    - git clone https://gitlab.com/freedesktop-sdk/freedesktop-sdk.git
    - git -C freedesktop-sdk checkout ${FD_SDK_REF}
@@ -275,22 +239,19 @@ coverage:
    stage: post
    coverage: '/TOTAL +\d+ +\d+ +(\d+\.\d+)%/'
    script:
 -    - cd dist && ./unpack.sh && cd buildstream
 -    - pip3 install -r tools/requirements.txt -r tools/dev-requirements.txt
 -    - pip3 install --no-index .
 +    - pip3 install -r requirements/dev-requirements.txt
      - mkdir report
      - cd report
 -    - cp ../../../coverage-unix/coverage.* .
 -    - cp ../../../coverage-linux/coverage.* .
 +    - cp ../coverage-unix/coverage.* .
 +    - cp ../coverage-linux/coverage.* .
      - ls coverage.*
 -    - coverage combine --rcfile=../.coveragerc -a coverage.*
 -    - coverage report --rcfile=../.coveragerc -m
 +    - coverage combine -a coverage.*
 +    - coverage report -m
    dependencies:
    - tests-debian-9
    - tests-fedora-27
    - tests-fedora-28
    - tests-unix
 -  - source_dist
    except:
    - schedules
@@ -299,7 +260,6 @@ coverage:
  pages:
    stage: post
    dependencies:
 -  - source_dist
    - docs
    variables:
      ACME_DIR: public/.well-known/acme-challenge

CONTRIBUTING.rst

@@ -1736,10 +1736,8 @@ obtain profiles::
        ForceCommand BST_PROFILE=artifact-receive cd /tmp && bst-artifact-receive --pull-url https://example.com/ /home/artifacts/artifacts
 -The MANIFEST.in and setup.py
 -----------------------------
 -When adding a dependency to BuildStream, it's important to update the setup.py accordingly.
+-
 +Managing data files
 +-------------------
  When adding data files which need to be discovered at runtime by BuildStream, update setup.py accordingly.
  When adding data files for the purpose of docs or tests, or anything that is not covered by
@@ -1749,3 +1747,23 @@ At any time, running the following command to create a source distribution shoul
  creating a tarball which contains everything we want it to include::
    ./setup.py sdist
++
++
 +Updating BuildStream's Python dependencies
 +------------------------------------------
 +BuildStream's Python dependencies are listed in multiple
 +`requirements files <https://pip.readthedocs.io/en/latest/reference/pip_install/#requirements-file-format>`
 +present in the ``requirements`` directory.
++
 +All ``.txt`` files in this directory are generated from the corresponding
 +``.in`` file, and each ``.in`` file represents a set of dependencies. For
 +example, ``requirements.in`` contains all runtime dependencies of BuildStream.
 +``requirements.txt`` is generated from it, and contains pinned versions of all
 +runtime dependencies (including transitive dependencies) of BuildStream.
++
 +When adding a new dependency to BuildStream, or updating existing dependencies,
 +it is important to update the appropriate requirements file accordingly. After
 +changing the ``.in`` file, run the following to update the matching ``.txt``
 +file::
++
 +   make -C requirements

MANIFEST.in

@@ -32,12 +32,12 @@ include .pylintrc
  recursive-include buildstream/_protos *.proto
  # Requirements files
 -include tools/requirements.in
 -include tools/requirements.txt
 -include tools/dev-requirements.in
 -include tools/dev-requirements.txt
 -include tools/plugin-requirements.in
 -include tools/plugin-requirements.txt
 +include requirements/requirements.in
 +include requirements/requirements.txt
 +include requirements/dev-requirements.in
 +include requirements/dev-requirements.txt
 +include requirements/plugin-requirements.in
 +include requirements/plugin-requirements.txt
  # Versioneer
  include versioneer.py

buildstream/_frontend/cli.py

@@ -719,6 +719,11 @@ def source_fetch(app, elements, deps, track_, except_, track_cross_junctions):
          deps = PipelineSelection.ALL
      with app.initialized(session_name="Fetch"):
 +        if not elements:
 +            guessed_target = app.context.guess_element()
 +            if guessed_target:
 +                elements = (guessed_target,)
++
          app.stream.fetch(elements,
                           selection=deps,
                           except_targets=except_,
@@ -755,6 +760,11 @@ def source_track(app, elements, deps, except_, cross_junctions):
          all:   All dependencies of all specified elements
      """
      with app.initialized(session_name="Track"):
 +        if not elements:
 +            guessed_target = app.context.guess_element()
 +            if guessed_target:
 +                elements = (guessed_target,)
++
          # Substitute 'none' for 'redirect' so that element redirections
          # will be done
          if deps == 'none':

requirements/Makefile

 +# Makefile for updating BuildStream's requirements files.
 +#
++
 +REQUIREMENTS_IN := $(wildcard *.in)
 +REQUIREMENTS_TXT := $(REQUIREMENTS_IN:.in=.txt)
 +PYTHON := python3
 +VENV := $(PYTHON) -m venv
++
 +VENV_PIP = $(VENVDIR)/bin/pip
++
++
 +.PHONY: all
++
 +all: $(REQUIREMENTS_TXT)
++
 +%.txt: %.in
 +	$(eval VENVDIR := $(shell mktemp -d $(CURDIR)/.bst-venv.XXXXXX))
 +	$(VENV) $(VENVDIR)
 +	$(VENV_PIP) install -r $^
 +	$(VENV_PIP) freeze -r $^ > $@
 +	rm -rf $(VENVDIR)

tools/dev-requirements.in → requirements/dev-requirements.in

tools/dev-requirements.txt → requirements/dev-requirements.txt

tools/plugin-requirements.in → requirements/plugin-requirements.in

tools/plugin-requirements.txt → requirements/plugin-requirements.txt

tools/requirements.in → requirements/requirements.in

tools/requirements.txt → requirements/requirements.txt

setup.py

@@ -270,10 +270,10 @@ def get_cmdclass():
  #####################################################
  #               Gather requirements                 #
  #####################################################
 -with open('tools/dev-requirements.in') as dev_reqs:
 +with open('requirements/dev-requirements.in') as dev_reqs:
      dev_requires = dev_reqs.read().splitlines()
 -with open('tools/requirements.in') as install_reqs:
 +with open('requirements/requirements.in') as install_reqs:
      install_requires = install_reqs.read().splitlines()
  #####################################################

tox.ini

@@ -5,9 +5,9 @@ skip_missing_interpreters = true
  [testenv]
  commands = pytest {posargs}
  deps =
 -    -rtools/requirements.txt
 -    -rtools/dev-requirements.txt
 -    -rtools/plugin-requirements.txt
 +    -rrequirements/requirements.txt
 +    -rrequirements/dev-requirements.txt
 +    -rrequirements/plugin-requirements.txt
  passenv =
      BST_FORCE_BACKEND
      GI_TYPELIB_PATH
@@ -18,9 +18,9 @@ commands =
      pycodestyle
      pylint buildstream
  deps =
 -    -rtools/requirements.txt
 -    -rtools/dev-requirements.txt
 -    -rtools/plugin-requirements.txt
 +    -rrequirements/requirements.txt
 +    -rrequirements/dev-requirements.txt
 +    -rrequirements/plugin-requirements.txt
  [testenv:docs]
  commands =
@@ -30,8 +30,8 @@ deps =
      sphinx==1.7.9
      sphinx-click
      sphinx_rtd_theme
 -    -rtools/requirements.txt
 -    -rtools/plugin-requirements.txt
 +    -rrequirements/requirements.txt
 +    -rrequirements/plugin-requirements.txt
  passenv =
      BST_FORCE_SESSION_REBUILD
      BST_SOURCE_CACHE

[Notes] [Git][BuildStream/buildstream][chandan/no-sdist] 7 commits: _frontend/cli.py: Reinstate support for guessing targets

Chandan Singh pushed to branch chandan/no-sdist at BuildStream / buildstream

Commits:

13 changed files:

Changes: