[Notes] [Git][BuildStream/buildstream][tristan/docs-version-badge] 10 co

Tristan Van Berkom pushed to branch tristan/docs-version-badge at BuildStream / buildstream

Commits:

ddbf7bc2

by Jürg Billeter at 2018-08-28T08:56:06Z

element.py: Schedule assemble for key of workspaced elements

For uncached workspaced elements, assemble is required even just to
calculate the cache key. As dynamic scheduling relies on cache keys,
schedule assemble for uncached workspace elements even if they have not
been marked as required yet.

Fixes #461.

ffc556ff

by Valentin David at 2018-08-28T09:01:05Z

tests/frontend/workspace.py: Add test for workspaced dependencies

This adds a regression test for #461.

dd90d53d

by Jürg Billeter at 2018-08-28T09:01:05Z

element.py: Do not attempt to pull workspaced elements

6a5a8e7a

by Tristan Van Berkom at 2018-08-28T11:29:22Z

Merge branch 'juerg/workspaced-dependencies' into 'master'

Fix key invalidation for workspaced dependencies

Closes #461

See merge request BuildStream/buildstream!740

76d545e6

by Tristan Van Berkom at 2018-08-28T13:02:18Z

tests/frontend/workspaces.py: Removing some redundant tests

Removed redundant tests from recently merged !740, this new
test does not need to run for every different source kind.

d9af99a7

by Tristan Van Berkom at 2018-08-28T13:07:53Z

doc: Adding badges.py release badge generator

This adds a step to the docs generation Makefile to generate
release.svg and snapshot.svg badges, modelled after the gitlab
badges.

df909fdd

by Tristan Van Berkom at 2018-08-28T13:07:53Z

.gitignore: Adding the generated badges directory in docs to be ignored

001f391d

by Tristan Van Berkom at 2018-08-28T13:07:53Z

README.rst: Adding the release and snapshot badges to the readme

bfb1bee0

by Tristan Van Berkom at 2018-08-28T13:07:53Z

doc: Adding the release badges to the install page and the semantic versioning page

30efef00
by Tristan Van Berkom at 2018-08-28T13:15:27Z
```
Testing
```

Changes:

.gitignore

@@ -26,6 +26,7 @@ __pycache__/
  buildstream/__version__.py
  # Autogenerated doc
 +doc/source/badges/
  doc/source/sessions/
  doc/source/elements/
  doc/source/sources/

.gitlab-ci.yml

@@ -50,89 +50,6 @@ source_dist:
  #                    Test stage                     #
  #####################################################
 -# Run premerge commits
 -#
 -.linux-tests-template: &linux-tests
 -  stage: test
 -  variables:
 -    PYTEST_ADDOPTS: "--color=yes"
 -  script:
 -  # Diagnostics
 -  - mount
 -  - df -h
+-
 -  - useradd -Um buildstream
 -  - chown -R buildstream:buildstream .
+-
 -  - export INTEGRATION_CACHE="$(pwd)/cache/integration-cache"
+-
 -  # Unpack and get into dist/buildstream
 -  - cd dist && ./unpack.sh
 -  - chown -R buildstream:buildstream buildstream
 -  - cd buildstream
+-
 -  # Run the tests from the source distribution, We run as a simple
 -  # user to test for permission issues
 -  - su buildstream -c 'python3 setup.py test --index-url invalid://uri --addopts --integration'
+-
 -  # Go back to the toplevel and collect our reports
 -  - cd ../..
 -  - mkdir -p coverage-linux/
 -  - cp dist/buildstream/.coverage.* coverage-linux/coverage."${CI_JOB_NAME}"
 -  artifacts:
 -    paths:
 -    - coverage-linux/
+-
 -tests-debian-9:
 -  image: buildstream/testsuite-debian:9-master-114-4cab18e3
 -  <<: *linux-tests
+-
 -tests-fedora-27:
 -  image: buildstream/testsuite-fedora:27-master-114-4cab18e3
 -  <<: *linux-tests
+-
 -tests-fedora-28:
 -  image: buildstream/testsuite-fedora:28-master-114-4cab18e3
 -  <<: *linux-tests
+-
 -tests-ubuntu-18.04:
 -  image: buildstream/testsuite-ubuntu:18.04-master-114-4cab18e3
 -  <<: *linux-tests
+-
 -tests-unix:
 -  # Use fedora here, to a) run a test on fedora and b) ensure that we
 -  # can get rid of ostree - this is not possible with debian-8
 -  image: buildstream/testsuite-fedora:27-master-114-4cab18e3
 -  stage: test
 -  variables:
 -    BST_FORCE_BACKEND: "unix"
 -    PYTEST_ADDOPTS: "--color=yes"
 -  script:
+-
 -    - export INTEGRATION_CACHE="$(pwd)/cache/integration-cache"
+-
 -    # We remove the Bubblewrap and OSTree packages here so that we catch any
 -    # codepaths that try to use them. Removing OSTree causes fuse-libs to
 -    # disappear unless we mark it as user-installed.
 -    - dnf mark install fuse-libs
 -    - dnf erase -y bubblewrap ostree
+-
 -    # Unpack and get into dist/buildstream
 -    - cd dist && ./unpack.sh && cd buildstream
+-
 -    # Since the unix platform is required to run as root, no user change required
 -    - python3 setup.py test --index-url invalid://uri --addopts --integration
+-
 -    # Go back to the toplevel and collect our reports
 -    - cd ../..
 -    - mkdir -p coverage-unix/
 -    - cp dist/buildstream/.coverage.* coverage-unix/coverage.unix
 -  artifacts:
 -    paths:
 -    - coverage-unix/
 -    - logs-unix/
+-
+-
  # Automatically build documentation for every commit, we want to know
  # if building documentation fails even if we're not deploying it.
  # Note: We still do not enforce a consistent installation of python3-sphinx,
@@ -153,78 +70,10 @@ docs:
      - public/
 -# Check code quality with gitlab's built-in feature.
 -#
 -code_quality:
 -  image: docker:stable
 -  stage: test
 -  variables:
 -    DOCKER_DRIVER: overlay2
 -  allow_failure: true
 -  services:
 -    - docker:stable-dind
 -  script:
 -    - export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')
 -    - docker run
 -        --env SOURCE_CODE="$PWD"
 -        --volume "$PWD":/code
 -        --volume /var/run/docker.sock:/var/run/docker.sock
 -        "registry.gitlab.com/gitlab-org/security-products/codequality:$SP_VERSION" /code
 -  artifacts:
 -    paths: [gl-code-quality-report.json]
+-
  #####################################################
  #                    Post stage                     #
  #####################################################
 -analysis:
 -  stage: post
 -  script:
 -  - |
 -    pip3 install radon
 -    mkdir analysis
+-
 -  - |
 -    echo "Calculating Maintainability Index"
 -    radon mi -s -j buildstream > analysis/mi.json
 -    radon mi -s buildstream
+-
 -  - |
 -    echo "Calculating Cyclomatic Complexity"
 -    radon cc -a -s -j buildstream > analysis/cc.json
 -    radon cc -a -s buildstream
+-
 -  - |
 -    echo "Calculating Raw Metrics"
 -    radon raw -s -j buildstream > analysis/raw.json
 -    radon raw -s buildstream
+-
 -  artifacts:
 -    paths:
 -    - analysis/
+-
 -# Collate coverage reports
 -#
 -coverage:
 -  stage: post
 -  coverage: '/TOTAL +\d+ +\d+ +(\d+\.\d+)%/'
 -  script:
 -    - cd dist && ./unpack.sh && cd buildstream
 -    - pip3 install --no-index .
 -    - mkdir report
 -    - cd report
 -    - cp ../../../coverage-unix/coverage.unix .
 -    - cp ../../../coverage-linux/coverage.* .
 -    - ls coverage.*
 -    - coverage combine --rcfile=../.coveragerc -a coverage.*
 -    - coverage report --rcfile=../.coveragerc -m
 -  dependencies:
 -  - tests-debian-9
 -  - tests-fedora-27
 -  - tests-fedora-28
 -  - tests-unix
 -  - source_dist
+-
  # Deploy, only for merges which land on master branch.
+ #
  pages:
@@ -248,3 +97,4 @@ pages:
    # See https://gitlab.com/gitlab-org/gitlab-ce/issues/35141
+   #
    - master
 +  - tristan/docs-version-badge

README.rst

  About
  -----
++
 +.. image:: https://buildstream.gitlab.io/buildstream/_static/release.svg
++
 +.. image:: https://buildstream.gitlab.io/buildstream/_static/snapshot.svg
++
  .. image:: https://gitlab.com/BuildStream/buildstream/badges/master/pipeline.svg
     :target: https://gitlab.com/BuildStream/buildstream/commits/master

buildstream/element.py

@@ -1101,9 +1101,12 @@ class Element(Plugin):
              # until the full cache query below.
              if (not self.__assemble_scheduled and not self.__assemble_done and
                      not self.__cached_success(keystrength=_KeyStrength.WEAK) and
 -                    not self._pull_pending() and self._is_required()):
 -                self._schedule_assemble()
 -                return
 +                    not self._pull_pending()):
 +                # For uncached workspaced elements, assemble is required
 +                # even if we only need the cache key
 +                if self._is_required() or self._get_workspace():
 +                    self._schedule_assemble()
 +                    return
          if self.__strict_cache_key is None:
              dependencies = [
@@ -1126,13 +1129,17 @@ class Element(Plugin):
                  self.__weak_cached = self.__artifacts.contains(self, self.__weak_cache_key)
          if (not self.__assemble_scheduled and not self.__assemble_done and
 -                not self._cached_success() and not self._pull_pending() and self._is_required()):
 +                not self._cached_success() and not self._pull_pending()):
              # Workspaced sources are considered unstable if a build is pending
              # as the build will modify the contents of the workspace.
              # Determine as early as possible if a build is pending to discard
              # unstable cache keys.
 -            self._schedule_assemble()
 -            return
++
 +            # For uncached workspaced elements, assemble is required
 +            # even if we only need the cache key
 +            if self._is_required() or self._get_workspace():
 +                self._schedule_assemble()
 +                return
          if self.__cache_key is None:
              # Calculate strong cache key
@@ -1430,7 +1437,6 @@ class Element(Plugin):
      # in a subprocess.
+     #
      def _schedule_assemble(self):
 -        assert self._is_required()
          assert not self.__assemble_scheduled
          self.__assemble_scheduled = True
@@ -1438,6 +1444,8 @@ class Element(Plugin):
          for dep in self.dependencies(Scope.BUILD, recurse=False):
              dep._set_required()
 +        self._set_required()
++
          # Invalidate workspace key as the build modifies the workspace directory
          workspace = self._get_workspace()
          if workspace:
@@ -1661,6 +1669,10 @@ class Element(Plugin):
      #   (bool): Whether a pull operation is pending
+     #
      def _pull_pending(self):
 +        if self._get_workspace():
 +            # Workspace builds are never pushed to artifact servers
 +            return False
++
          if self.__strong_cached:
              # Artifact already in local cache
              return False

doc/Makefile

@@ -35,7 +35,7 @@ endif
  PYTHONPATH=$(CURDIR)/..:$(CURDIR)/../buildstream/plugins
 -.PHONY: all clean templates templates-clean sessions sessions-prep sessions-clean html devhelp
 +.PHONY: all clean templates templates-clean sessions sessions-prep sessions-clean badges badges-clean html devhelp
  # Canned recipe for generating plugin api skeletons
  #   $1 = the plugin directory
@@ -70,9 +70,13 @@ endef
  all: html devhelp
 -clean: templates-clean sessions-clean
 +clean: templates-clean sessions-clean badges-clean
  	rm -rf build
 +############################################################
 +#                 Plugin doc templates                     #
 +############################################################
++
  # Generate rst templates for the docs using a mix of sphinx-apidoc and
  # our 'plugin-doc-skeleton' routine for plugin pages.
  templates:
@@ -86,6 +90,10 @@ templates-clean:
  	rm -rf source/elements
  	rm -rf source/sources
 +############################################################
 +#                   Session captures                       #
 +############################################################
++
  # Stage the stored sessions into the place where they will
  # be used in the build.
+ #
@@ -111,10 +119,27 @@ sessions: sessions-prep
  sessions-clean:
  	rm -rf source/sessions
++
 +############################################################
 +#                  Generate release badges                 #
 +############################################################
 +badges-clean:
 +	rm -rf source/badges
++
 +badges:
 +	mkdir -p source/badges
 +	$(CURDIR)/badges.py > source/badges/snapshot.svg
 +	$(CURDIR)/badges.py --release > source/badges/release.svg
++
++
 +############################################################
 +#                    Main sphinx build                     #
 +############################################################
++
  # Targets which generate docs with sphinx build
+ #
+ #
 -html devhelp: templates sessions
 +html devhelp: templates sessions badges
  	@echo "Building $@..."
  	PYTHONPATH=$(PYTHONPATH) \
  	    $(SPHINXBUILD) -b $@ $(ALLSPHINXOPTS) "$(BUILDDIR)/$@" \

doc/badges.py

 +#!/usr/bin/env python3
 +#
 +#  Copyright (C) 2018 Codethink Limited
 +#
 +#  This program is free software; you can redistribute it and/or
 +#  modify it under the terms of the GNU Lesser General Public
 +#  License as published by the Free Software Foundation; either
 +#  version 2 of the License, or (at your option) any later version.
 +#
 +#  This library is distributed in the hope that it will be useful,
 +#  but WITHOUT ANY WARRANTY; without even the implied warranty of
 +#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.	 See the GNU
 +#  Lesser General Public License for more details.
 +#
 +#  You should have received a copy of the GNU Lesser General Public
 +#  License along with this library. If not, see <http://www.gnu.org/licenses/>.
 +#
 +#  Authors:
 +#        Tristan Van Berkom <tristan vanberkom codethink co uk>
 +#
 +import click
 +import subprocess
 +import re
++
 +# The badge template is modeled after the gitlab badge svgs
 +#
 +BADGE_TEMPLATE = """<svg xmlns="http://www.w3.org/2000/svg"
 +                         xmlns:xlink="http://www.w3.org/1999/xlink"
 +                         width="116" height="20">
 +  <a xlink:href="">
 +    <linearGradient id="b" x2="0" y2="100%">
 +      <stop offset="0" stop-color="#bbb" stop-opacity=".1"/>
 +      <stop offset="1" stop-opacity=".1"/>
 +    </linearGradient>
++
 +    <mask id="a">
 +      <rect width="116" height="20" rx="3" fill="#fff"/>
 +    </mask>
++
 +    <g mask="url(#a)">
 +      <path fill="#555"
 +            d="M0 0 h62 v20 H0 z"/>
 +      <path fill="{color}"
 +            d="M62 0 h54 v20 H62 z"/>
 +      <path fill="url(#b)"
 +            d="M0 0 h116 v20 H0 z"/>
 +    </g>
++
 +    <g fill="#fff" text-anchor="middle">
 +      <g font-family="DejaVu Sans,Verdana,Geneva,sans-serif" font-size="11">
 +        <text x="31" y="15" fill="#010101" fill-opacity=".3">
 +          {badge_name}
 +        </text>
 +        <text x="31" y="14">
 +          {badge_name}
 +        </text>
 +        <text x="89" y="15" fill="#010101" fill-opacity=".3">
 +          {version}
 +        </text>
 +        <text x="89" y="14">
 +          {version}
 +        </text>
 +      </g>
 +    </g>
 +  </a>
 +</svg>
 +"""
++
 +URL_FORMAT = 'https://download.gnome.org/sources/BuildStream/{brief_version}/BuildStream-{full_version}.tar.xz'
 +RELEASE_COLOR = '#0040FF'
 +SNAPSHOT_COLOR = '#FF8000'
 +VERSION_TAG_MATCH = r'([0-9]*)\.([0-9]*)\.([0-9]*)'
++
++
 +# Parse a release tag and return a three tuple
 +# of the major, minor and micro version.
 +#
 +# Tags which do not follow the release tag format
 +# will just be returned as (0, 0, 0)
 +#
 +def parse_tag(tag):
 +    match = re.search(VERSION_TAG_MATCH, tag)
 +    if match:
 +        major = match.group(1)
 +        minor = match.group(2)
 +        micro = match.group(3)
 +        return (int(major), int(minor), int(micro))
++
 +    return (0, 0, 0)
++
++
 +# Call out to git and guess the latest version,
 +# this will just return (0, 0, 0) in case of any error.
 +#
 +def guess_version(release):
 +    try:
 +        tags_output = subprocess.check_output(['git', 'tag'])
 +    except CalledProcessError:
 +        return (0, 0, 0)
++
 +    # Parse the `git tag` output into a list of integer tuples
 +    tags_output = tags_output.decode('UTF-8')
 +    all_tags = tags_output.splitlines()
 +    all_versions = [parse_tag(tag) for tag in all_tags]
++
 +    # Filter the list by the minor point version, if
 +    # we are checking for the latest "release" version, then
 +    # only pickup even number minor points.
 +    #
 +    filtered_versions = [
 +        version for version in all_versions
 +        if (version[1] % 2) == (not release)
 +    ]
++
 +    # Make sure they are sorted, and take the last one
 +    sorted_versions = sorted(filtered_versions)
 +    latest_version = sorted_versions[-1]
++
 +    return latest_version
++
++
 +@click.command(short_help="Generate the version badges")
 +@click.option('--release', is_flag=True, default=False,
 +              help="Whether to generate the badge for the release version")
 +def generate_badges(release):
 +    """Generate the version badge svg files
 +    """
 +    major, minor, micro = guess_version(release)
++
 +    if release:
 +        badge_name = 'release'
 +        color = RELEASE_COLOR
 +    else:
 +        badge_name = 'snapshot'
 +        color = SNAPSHOT_COLOR
++
 +    brief_version = '{major}.{minor}'.format(major=major, minor=minor)
 +    full_version = '{major}.{minor}.{micro}'.format(major=major, minor=minor, micro=micro)
 +    url_target = URL_FORMAT.format(brief_version=brief_version, full_version=full_version)
 +    badge = BADGE_TEMPLATE.format(badge_name=badge_name,
 +                                  version=full_version,
 +                                  color=color,
 +                                  url_target=url_target)
 +    click.echo(badge, nl=False)
 +    return 0
++
++
 +if __name__ == '__main__':
 +    generate_badges()

doc/source/conf.py

@@ -160,7 +160,7 @@ html_theme = 'sphinx_rtd_theme'
  # Add any paths that contain custom static files (such as style sheets) here,
  # relative to this directory. They are copied after the builtin static files,
  # so a file named "default.css" will overwrite the builtin "default.css".
 -html_static_path = []
 +html_static_path = ['badges']
  # Add any extra paths that contain custom files (such as robots.txt or
  # .htaccess) here, relative to this directory. These files are copied

doc/source/install_versions.rst

@@ -13,8 +13,12 @@ For example, for a given version number ``X.Y.Z``
   * The ``X.<odd number>.*`` versions are development spanshots intended for testing.
  If you are :ref:`installing from git <install_git_checkout>`, please look for the latest
 -tag in the latest release branch to ensure you're getting the latest release.
 +tag to ensure you're getting the latest release.
 -Current release branches:
 - * `bst-1.2 (latest) <https://gitlab.com/BuildStream/buildstream/commits/bst-1.2>`_
 - * `bst-1.0 (deprecated) <https://gitlab.com/BuildStream/buildstream/commits/bst-1.0>`_
 +* Latest release:
++
 +  .. image:: https://buildstream.gitlab.io/buildstream/_static/release.svg
++
 +* Latest development snapshot:
++
 +  .. image:: https://buildstream.gitlab.io/buildstream/_static/snapshot.svg

doc/source/main_install.rst

@@ -4,6 +4,11 @@
  Install
  =======
++
 +.. image:: https://buildstream.gitlab.io/buildstream/_static/release.svg
++
 +.. image:: https://buildstream.gitlab.io/buildstream/_static/snapshot.svg
++
  This section provides instructions for installing BuildStream and its
  companion artifact server on various platforms, along with any installation
  related materials.

tests/frontend/workspace.py

@@ -780,3 +780,72 @@ def test_inconsitent_pipeline_message(cli, tmpdir, datafiles, kind):
          'build', element_name
      ])
      result.assert_main_error(ErrorDomain.PIPELINE, "inconsistent-pipeline-workspaced")
++
++
 +@pytest.mark.datafiles(DATA_DIR)
 +@pytest.mark.parametrize("strict", [("strict"), ("non-strict")])
 +def test_cache_key_workspace_in_dependencies(cli, tmpdir, datafiles, strict):
 +    checkout = os.path.join(str(tmpdir), 'checkout')
 +    element_name, project, workspace = open_workspace(cli, os.path.join(str(tmpdir), 'repo-a'), datafiles, 'git', False)
++
 +    element_path = os.path.join(project, 'elements')
 +    back_dep_element_name = 'workspace-test-back-dep.bst'
++
 +    # Write out our test target
 +    element = {
 +        'kind': 'compose',
 +        'depends': [
 +            {
 +                'filename': element_name,
 +                'type': 'build'
 +            }
 +        ]
 +    }
 +    _yaml.dump(element,
 +               os.path.join(element_path,
 +                            back_dep_element_name))
++
 +    # Modify workspace
 +    shutil.rmtree(os.path.join(workspace, 'usr', 'bin'))
 +    os.makedirs(os.path.join(workspace, 'etc'))
 +    with open(os.path.join(workspace, 'etc', 'pony.conf'), 'w') as f:
 +        f.write("PONY='pink'")
++
 +    # Configure strict mode
 +    strict_mode = True
 +    if strict != 'strict':
 +        strict_mode = False
 +    cli.configure({
 +        'projects': {
 +            'test': {
 +                'strict': strict_mode
 +            }
 +        }
 +    })
++
 +    # Build artifact with dependency's modified workspace
 +    assert cli.get_element_state(project, element_name) == 'buildable'
 +    assert cli.get_element_key(project, element_name) == "{:?<64}".format('')
 +    assert cli.get_element_state(project, back_dep_element_name) == 'waiting'
 +    assert cli.get_element_key(project, back_dep_element_name) == "{:?<64}".format('')
 +    result = cli.run(project=project, args=['build', back_dep_element_name])
 +    result.assert_success()
 +    assert cli.get_element_state(project, element_name) == 'cached'
 +    assert cli.get_element_key(project, element_name) != "{:?<64}".format('')
 +    assert cli.get_element_state(project, back_dep_element_name) == 'cached'
 +    assert cli.get_element_key(project, back_dep_element_name) != "{:?<64}".format('')
 +    result = cli.run(project=project, args=['build', back_dep_element_name])
 +    result.assert_success()
++
 +    # Checkout the result
 +    result = cli.run(project=project, args=[
 +        'checkout', back_dep_element_name, checkout
 +    ])
 +    result.assert_success()
++
 +    # Check that the pony.conf from the modified workspace exists
 +    filename = os.path.join(checkout, 'etc', 'pony.conf')
 +    assert os.path.exists(filename)
++
 +    # Check that the original /usr/bin/hello is not in the checkout
 +    assert not os.path.exists(os.path.join(checkout, 'usr', 'bin', 'hello'))

[Notes] [Git][BuildStream/buildstream][tristan/docs-version-badge] 10 commits: element.py: Schedule assemble for key of workspaced elements

Tristan Van Berkom pushed to branch tristan/docs-version-badge at BuildStream / buildstream

Commits:

10 changed files:

Changes: