[Notes] [Git][BuildStream/buildstream][chandan/setup.py-metadata] 13 com

Chandan Singh pushed to branch chandan/setup.py-metadata at BuildStream / buildstream

Commits:

56096b60

by Chandan Singh at 2018-08-24T02:12:03Z

.gitlab-ci.yml: Verify that source tarball installs correctly

Add a check to verify that the source distribution tarball that we
generate installs corrctly. It is useful to do this check as this is the
same tarball will be used when installing BuildStream via PyPI.

This check also happens indirectly in the coverage job where we install
BuildStream but this check makes it more explicit and will also cause
the CI to fail early if the tarball is not correct.

86e4729d

by Tristan Van Berkom at 2018-08-24T06:42:03Z

Merge branch 'chandan/ci-sdist-test' into 'master'

.gitlab-ci.yml: Verify that source tarball installs correctly

See merge request BuildStream/buildstream!720

d88568ef

by Valentin David at 2018-08-24T06:48:08Z

tests/integration/source-determinism.py: Use cli_integration.

f7ef151b

by Tristan Van Berkom at 2018-08-24T07:36:19Z

Merge branch 'valentindavid/cli_integration_source_determinism' into 'master'

tests/integration/source-determinism.py: Use cli_integration.

See merge request BuildStream/buildstream!715

fa476d79

by Tristan Van Berkom at 2018-08-24T07:55:43Z

tests/integration/project: Use the same alias for the alpine tarball

This makes the integration tests use the same 'alpine' alias for the
tests as we use in the examples, this avoids a redundant download
of an extra alpine tarball in integration tests.

This is a part of #603

255f9ee3

by Tristan Van Berkom at 2018-08-24T08:39:19Z

Merge branch 'tristan/avoid-redundant-alpine' into 'master'

tests/integration/project: Use the same alias for the alpine tarball

See merge request BuildStream/buildstream!722

e1ed09aa

by Jim MacArthur at 2018-08-24T09:51:27Z

element.py: chmod 777 directories if delete fails.

Staging may produce directories with less than 'rwx' permissions which
will cause tempfile to throw an exception on deleting the temporary
directory if not changed.

01fe504b

by James Ennis at 2018-08-24T09:51:27Z

tar tests: Add tarball which contains a read-only dir

a3755c9a

by James Ennis at 2018-08-24T09:51:27Z

tar.py: Add test for a tarball containing a read-only dir

1bfcca1a

by James Ennis at 2018-08-24T10:31:41Z

Merge branch 'jmac/tempfile-extraction-bug' into 'master'

Correct crash after staging tars with read-only directories

See merge request BuildStream/buildstream!713

744925f3

by James Ennis at 2018-08-24T10:34:57Z

HACKING.rst: Remove -k flag for individual tests as this is not required

dfee6d75

by James Ennis at 2018-08-24T11:15:45Z

Merge branch 'jennis/correct_HACKING' into 'master'

'-k' flag not needed to run individual tests

See merge request BuildStream/buildstream!714

1fab4541

by Chandan Singh at 2018-08-24T11:25:22Z

setup.py: Add more metadata useful for PyPI

- setup.py: List useful links using `project_urls`

- setup.py: Specify minumum python vresion using `python_requires`

  `python_requires` is the new way of specifying requirements for python
  version, as documented @
  https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires.

  This will also make this requirement appear nicely on the PyPI project
  page.

- setup.py: Add project classifiers

  PyPI will use these classifiers to categorize projects while searching
  or browsing. Full list of classifiers can be found at
  https://pypi.org/classifiers.

11 changed files:

.gitlab-ci.yml
HACKING.rst
buildstream/element.py
setup.py
tests/integration/project/elements/base/base-alpine.bst
tests/integration/project/project.conf
tests/integration/source-determinism.py
tests/sources/tar.py
+ tests/sources/tar/read-only/content/a.tar.gz
+ tests/sources/tar/read-only/target.bst
tests/testutils/runcli.py

Changes:

.gitlab-ci.yml

@@ -36,6 +36,11 @@ source_dist:
    - tar -ztf dist/*
    - tarball=$(cd dist && echo $(ls *))
 +  # Verify that the source distribution tarball can be installed correctly
 +  #
 +  - pip3 install dist/*.tar.gz
 +  - bst --version
++
    # unpack tarball as `dist/buildstream` directory
    - |
      cat > dist/unpack.sh << EOF

HACKING.rst

@@ -465,12 +465,12 @@ If you want to run a specific test or a group of tests, you
  can specify a prefix to match. E.g. if you want to run all of
  the frontend tests you can do::
 -  ./setup.py test --addopts '-k tests/frontend/'
 +  ./setup.py test --addopts 'tests/frontend/'
  Specific tests can be chosen by using the :: delimeter after the test module.
  If you wanted to run the test_build_track test within frontend/buildtrack.py you could do::
 -  ./setup.py test --adopts '-k tests/frontend/buildtrack.py::test_build_track'
 +  ./setup.py test --addopts 'tests/frontend/buildtrack.py::test_build_track'
  We also have a set of slow integration tests that are disabled by
  default - you will notice most of them marked with SKIP in the pytest

buildstream/element.py

@@ -1361,8 +1361,12 @@ class Element(Plugin):
              if not vdirectory.is_empty():
                  raise ElementError("Staging directory '{}' is not empty".format(vdirectory))
 -            with tempfile.TemporaryDirectory() as temp_staging_directory:
 +            # While mkdtemp is advertised as using the TMP environment variable, it
 +            # doesn't, so this explicit extraction is necesasry.
 +            tmp_prefix = os.environ.get("TMP", None)
 +            temp_staging_directory = tempfile.mkdtemp(prefix=tmp_prefix)
 +            try:
                  workspace = self._get_workspace()
                  if workspace:
                      # If mount_workspaces is set and we're doing incremental builds,
@@ -1377,6 +1381,19 @@ class Element(Plugin):
                          source._stage(temp_staging_directory)
                  vdirectory.import_files(temp_staging_directory)
++
 +            finally:
 +                # Staging may produce directories with less than 'rwx' permissions
 +                # for the owner, which will break tempfile, so we need to use chmod
 +                # occasionally.
 +                def make_dir_writable(fn, path, excinfo):
 +                    os.chmod(os.path.dirname(path), 0o777)
 +                    if os.path.isdir(path):
 +                        os.rmdir(path)
 +                    else:
 +                        os.remove(path)
 +                shutil.rmtree(temp_staging_directory, onerror=make_dir_writable)
++
          # Ensure deterministic mtime of sources at build time
          vdirectory.set_deterministic_mtime()
          # Ensure deterministic owners of sources at build time

setup.py

@@ -242,11 +242,28 @@ setup(name='BuildStream',
        author='BuildStream Developers',
        author_email='buildstream-list gnome org',
 +      classifiers=[
 +          'Environment :: Console',
 +          'Intended Audience :: Developers',
 +          'License :: OSI Approved :: GNU Lesser General Public License v2 or later (LGPLv2+)',
 +          'Operating System :: POSIX',
 +          'Programming Language :: Python :: 3',
 +          'Programming Language :: Python :: 3.5',
 +          'Programming Language :: Python :: 3.6',
 +          'Programming Language :: Python :: 3.7',
 +          'Topic :: Software Development :: Build Tools'
 +      ],
        description='A framework for modelling build pipelines in YAML',
        license='LGPL',
        long_description=long_description,
        long_description_content_type='text/x-rst; charset=UTF-8',
        url='https://gitlab.com/BuildStream/buildstream',
 +      project_urls={
 +          'Documentation': 'https://buildstream.gitlab.io/buildstream/',
 +          'Tracker': 'https://gitlab.com/BuildStream/buildstream/issues',
 +          'Mailing List': 'https://mail.gnome.org/mailman/listinfo/buildstream-list'
 +      },
 +      python_requires='>=3.5',
        packages=find_packages(exclude=('tests', 'tests.*')),
        package_data={'buildstream': ['plugins/*/*.py', 'plugins/*/*.yaml',
                                      'data/*.yaml', 'data/*.sh.in']},

tests/integration/project/elements/base/base-alpine.bst

@@ -7,6 +7,6 @@ description: |
  sources:
    - kind: tar
 -    url: sysroot:tarballs/integration-tests-base.v1.x86_64.tar.xz
 +    url: alpine:integration-tests-base.v1.x86_64.tar.xz
      base-dir: ''
      ref: 3eb559250ba82b64a68d86d0636a6b127aa5f6d25d3601a79f79214dc9703639

tests/integration/project/project.conf

@@ -2,7 +2,7 @@
  name: test
  element-path: elements
  aliases:
 -  sysroot: https://gnome7.codethink.co.uk/
 +  alpine: https://gnome7.codethink.co.uk/tarballs/
    project_dir: file://{project_dir}
  options:
    linux:

tests/integration/source-determinism.py

@@ -2,7 +2,8 @@ import os
  import pytest
  from buildstream import _yaml, utils
 -from tests.testutils import cli, create_repo, ALL_REPO_KINDS
 +from tests.testutils import create_repo, ALL_REPO_KINDS
 +from tests.testutils import cli_integration as cli
  DATA_DIR = os.path.join(
@@ -28,7 +29,7 @@ def create_test_directory(*path, mode=0o644):
  @pytest.mark.integration
  @pytest.mark.datafiles(DATA_DIR)
  @pytest.mark.parametrize("kind", [(kind) for kind in ALL_REPO_KINDS] + ['local'])
 -def test_deterministic_source_umask(cli, tmpdir, datafiles, kind):
 +def test_deterministic_source_umask(cli, tmpdir, datafiles, kind, integration_cache):
      project = str(datafiles)
      element_name = 'list'
      element_path = os.path.join(project, 'elements', element_name)
@@ -91,14 +92,16 @@ def test_deterministic_source_umask(cli, tmpdir, datafiles, kind):
                  return f.read()
          finally:
              os.umask(old_umask)
 -            cli.remove_artifact_from_cache(project, element_name)
 +            cache_dir = os.path.join(integration_cache, 'artifacts')
 +            cli.remove_artifact_from_cache(project, element_name,
 +                                           cache_dir=cache_dir)
      assert get_value_for_umask(0o022) == get_value_for_umask(0o077)
  @pytest.mark.integration
  @pytest.mark.datafiles(DATA_DIR)
 -def test_deterministic_source_local(cli, tmpdir, datafiles):
 +def test_deterministic_source_local(cli, tmpdir, datafiles, integration_cache):
      """Only user rights should be considered for local source.
      """
      project = str(datafiles)
@@ -150,6 +153,8 @@ def test_deterministic_source_local(cli, tmpdir, datafiles):
              with open(os.path.join(checkoutdir, 'ls-l'), 'r') as f:
                  return f.read()
          finally:
 -            cli.remove_artifact_from_cache(project, element_name)
 +            cache_dir = os.path.join(integration_cache, 'artifacts')
 +            cli.remove_artifact_from_cache(project, element_name,
 +                                           cache_dir=cache_dir)
      assert get_value_for_mask(0o7777) == get_value_for_mask(0o0700)

tests/sources/tar.py

@@ -3,6 +3,7 @@ import pytest
  import tarfile
  import tempfile
  import subprocess
 +from shutil import copyfile, rmtree
  from buildstream._exceptions import ErrorDomain
  from buildstream import _yaml
@@ -257,3 +258,47 @@ def test_stage_default_basedir_lzip(cli, tmpdir, datafiles, srcdir):
      original_contents = list_dir_contents(original_dir)
      checkout_contents = list_dir_contents(checkoutdir)
      assert(checkout_contents == original_contents)
++
++
 +# Test that a tarball that contains a read only dir works
 +@pytest.mark.datafiles(os.path.join(DATA_DIR, 'read-only'))
 +def test_read_only_dir(cli, tmpdir, datafiles):
 +    try:
 +        project = os.path.join(datafiles.dirname, datafiles.basename)
 +        generate_project(project, tmpdir)
++
 +        # Get the tarball in tests/sources/tar/read-only/content
 +        #
 +        # NOTE that we need to do this because tarfile.open and tar.add()
 +        # are packing the tar up with writeable files and dirs
 +        tarball = os.path.join(str(datafiles), 'content', 'a.tar.gz')
 +        if not os.path.exists(tarball):
 +            raise FileNotFoundError('{} does not exist'.format(tarball))
 +        copyfile(tarball, os.path.join(str(tmpdir), 'a.tar.gz'))
++
 +        # Because this test can potentially leave directories behind
 +        # which are difficult to remove, ask buildstream to use
 +        # our temp directory, so we can clean up.
 +        tmpdir_str = str(tmpdir)
 +        if not tmpdir_str.endswith(os.path.sep):
 +            tmpdir_str += os.path.sep
 +        env = {"TMP": tmpdir_str}
++
 +        # Track, fetch, build, checkout
 +        result = cli.run(project=project, args=['track', 'target.bst'], env=env)
 +        result.assert_success()
 +        result = cli.run(project=project, args=['fetch', 'target.bst'], env=env)
 +        result.assert_success()
 +        result = cli.run(project=project, args=['build', 'target.bst'], env=env)
 +        result.assert_success()
++
 +    finally:
++
 +        # Make tmpdir deletable no matter what happens
 +        def make_dir_writable(fn, path, excinfo):
 +            os.chmod(os.path.dirname(path), 0o777)
 +            if os.path.isdir(path):
 +                os.rmdir(path)
 +            else:
 +                os.remove(path)
 +        rmtree(str(tmpdir), onerror=make_dir_writable)

tests/sources/tar/read-only/content/a.tar.gz

No preview for this file type

tests/sources/tar/read-only/target.bst

 +kind: import
 +description: The kind of this element is irrelevant.
 +sources:
 +- kind: tar
 +  url: tmpdir:/a.tar.gz
 +  ref: foo

tests/testutils/runcli.py

@@ -198,8 +198,10 @@ class Cli():
          for key, val in config.items():
              self.config[key] = val
 -    def remove_artifact_from_cache(self, project, element_name):
 -        cache_dir = os.path.join(project, 'cache', 'artifacts')
 +    def remove_artifact_from_cache(self, project, element_name,
 +                                   *, cache_dir=None):
 +        if not cache_dir:
 +            cache_dir = os.path.join(project, 'cache', 'artifacts')
          cache_dir = os.path.join(cache_dir, 'cas', 'refs', 'heads')

[Notes] [Git][BuildStream/buildstream][chandan/setup.py-metadata] 13 commits: .gitlab-ci.yml: Verify that source tarball installs correctly

Chandan Singh pushed to branch chandan/setup.py-metadata at BuildStream / buildstream

Commits:

11 changed files:

Changes: