[Notes] [Git][BuildStream/buildstream][phil/tox-vev-environment] 70 commits: Cache Keys: Update to use JSON rather than pickle

[Phil Dawson <gitlab mg gitlab com>]

Title: GitLab

Phil Dawson pushed to branch phil/tox-vev-environment at BuildStream / buildstream

Commits:

• c2c10d68
  by Daniel Silverstone at 2019-02-14T13:02:07Z

  Cache Keys: Update to use JSON rather than pickle
  
  This affects the cache key version (updated to 7) and introduces
  a dependency on `ujson` which is BSD licenced as of the version
  locked in `requirements.txt`
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 868c057e
  by Tristan Van Berkom at 2019-02-14T13:58:39Z

  Merge branch 'danielsilverstone-ct/json-cache-key' into 'master'
  
  Update cache keys to use JSON
  
  See merge request BuildStream/buildstream!1151

• ace15fa6
  by Benjamin Schubert at 2019-02-14T14:04:45Z

  Don't keep MetaSource around in Source
  
  We can easily rebiuld MetaSource from any source so there is no
  reason to keep them around.
  
  This will slightly improve memory usage.
  

• 174f5394
  by Tristan Van Berkom at 2019-02-14T15:04:27Z

  Merge branch 'bschubert/dont-keep-metasource' into 'master'
  
  Don't keep MetaSource around in Source
  
  See merge request BuildStream/buildstream!1150

• e7e4086e
  by Jürg Billeter at 2019-02-14T15:05:38Z

  utils.py: Change _ensure_real_directory() to not resolve symlinks
  
  Resolving symlinks during staging causes various issues:
  
  * Split rules may not work properly as the resolved paths will differ
    depending on whether another artifact with a directory symlink has
    been staged in the same root directory or not, e.g., as part of
    compose.
  
  * The order of symlinks in file lists is difficult to get right to
    guarantee consistent and predictable behavior as paths in a file list
    might rely on symlinks in the same file list. See #647 and #817.
  
  * Staging order differences can lead to surprising results. See #390.
  
  * Difficult to properly support absolute symlinks. Absolute symlinks are
    currently converted to relative symlinks, however, this doesn't always
    work. See #606 and #830.
  
  This will require changes in projects that rely on the current behavior.
  However, the changes are expected to be small and are often a sign of
  buggy element files. E.g., elements that don't fully obey `bindir` or
  `sbindir` variables.
  

• c0570871
  by Jürg Billeter at 2019-02-14T15:05:38Z

  _casbaseddirectory.py: Do not resolve symlinks
  
  This matches the change in utils._process_list().
  
  This also removes the _Resolver class as it is now unused. We may want
  to support controlled symlink resolution in the future, in which case
  the _Resolver class can be resurrected from this commit.
  

• 55bccf63
  by Jürg Billeter at 2019-02-14T15:05:38Z

  utils.py: Do not mangle absolute symlinks
  
  Copy symlinks as they are, absolute or relative. We no longer resolve
  symlinks when copying files, which makes this safe.
  

• 94c6d59b
  by Jürg Billeter at 2019-02-14T15:05:38Z

  Bump artifact version for changes in symlink handling
  

• 949962f8
  by Jürg Billeter at 2019-02-14T15:05:38Z

  NEWS: Add entry for change in symlink handling
  

• 56c07baa
  by Jürg Billeter at 2019-02-14T16:08:42Z

  Merge branch 'juerg/symlinks2' into 'master'
  
  Do not resolve or mangle symlinks during staging
  
  See merge request BuildStream/buildstream!1140

• 69520011
  by Chandan Singh at 2019-02-15T06:58:27Z

  contrib/bst-graph: Add script to print graph in DOT format
  
  This script leverages the recently added format strings
  (`%{build-deps}`, `%{runtime-deps}`) to `bst show` to print a graph in
  DOT format. This requires users to have the `graphviz` python package
  installed.
  
  Additionally, users can also render the graph using the `--format`
  option if they have the `graphviz` command line tool installed.
  

• 51dae747
  by Chandan Singh at 2019-02-15T08:50:21Z

  Merge branch 'chandan/dot-graph' into 'master'
  
  contrib/bst-graph: Add script to print graph in DOT format
  
  Closes #705
  
  See merge request BuildStream/buildstream!1148

• 0f78a47d
  by Daniel Silverstone at 2019-02-15T10:03:46Z

  Variables: Rework how expansion strings work
  
  Rather than constantly using regular expressions and retrieval from YAML
  nodes, pre-parse expansion strings into a list representation cached
  for reuse, and then expand them as simple string concatenation.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 14176e51
  by Benjamin Schubert at 2019-02-15T11:04:10Z

  Merge branch 'danielsilverstone-ct/variables-rework' into 'master'
  
  Variables: Rework how expansion strings work
  
  See merge request BuildStream/buildstream!1152

• 39febfda
  by Angelos Evripiotis at 2019-02-15T11:10:51Z

  storage.Directory.export_to_tar: mtime=_magic...
  
  Change the default value of mtime when doing export_to_tar() from `0` to
  `_utils._magic_timestamp`. This avoids problems in other software, which
  assume that an mtime of `0` means the file does not exist.
  
  There is more than one example where files with an mtime of zero are
  treated as non-existant by other software. e.g. [ninja][1] and [Template
  Toolkit][2].
  
  The OSTree project also [express an intention][3] to move from an mtime
  of 0 to an mtime of 1:
  
  > For this reason, OSTree acts as though all timestamps are set to
  > time_t 0, so that comparisons will be considered up-to-date. Note that
  > for a few releases, OSTree used 1 to fix warnings such as GNU Tar
  > emitting "implausibly old time stamp" with 0; however, until we have a
  > mechanism to transition cleanly to 1, for compatibilty OSTree is
  > reverted to use zero again.
  
  >From the comments on export_to_tar(), the motivation for having an mtime
  of 0 was to have reproducible results, rather than it specifically being
  the value 0.
  
  Additionally, the reproducible builds project has a [page on
  archives][4]; it mentions the benefits of setting all the files to have
  the same mtime, or clamping the mtime. It makes no mention of a
  motivation for the mtime to be specifically 0.
  
  Fixes #914
  
  [1]: https://github.com/ninja-build/ninja/issues/1120
  [2]: https://github.com/abw/Template2/blob/8d7d37200af436f1ad43628278d3caad257c8e27/lib/Template/Provider.pm#L635
  [3]: https://ostree.readthedocs.io/en/latest/manual/repo/
  [4]: https://reproducible-builds.org/docs/archives/
  

• 8b34e356
  by Angelos Evripiotis at 2019-02-15T12:25:21Z

  Merge branch 'aevri/mtime1' into 'master'
  
  storage.Directory.export_to_tar: default mtime=utils._magic_timestamp
  
  Closes #914
  
  See merge request BuildStream/buildstream!1149

• 3bbb90e7
  by Benjamin Schubert at 2019-02-15T12:34:36Z

  Use sets when checking for existence of an element
  

• 0921ccf4
  by Benjamin Schubert at 2019-02-15T13:39:52Z

  Merge branch 'bschubert/set-as-set' into 'master'
  
  Use sets when checking for existence of an element
  
  See merge request BuildStream/buildstream!1154

• c24f2971
  by Angelos Evripiotis at 2019-02-15T13:56:48Z

  userconfig: rm really-workspace-close-project-inaccessible
  
  Remove the need for the 'really-workspace-close-project-inaccessible'
  config option, as well as the option itself.
  
  As agreed on the mailing list [1], all the 'are you sure?' prompts on
  workspace reset and close were removed. While that discussion was going
  on, this new prompt and option was added. At the 2019 BuildStream
  Gathering, it was verbally agreed between myself and Tristan VB that we
  would also remove this instance.
  
  It was also agreed that we should have a notice to let the user know
  what they'd done, this was already in place if interactive. Moved it to
  be unconditional so that there's no difference in non-interactive
  behaviour. Made it output to stderr, as it's diagnostic meant for the
  user. Made it the last thing echo'd so it's next to the prompt - it's
  very relevant to what they type next. Added a test to make sure the text
  makes it to stderr in the appropriate case, and not in an inappropriate
  one.
  
  This is the last instance of any prompt configuration, so BuildStream
  can also forget all of that machinery.
  
  [1] https://mail.gnome.org/archives/buildstream-list/2018-December/msg00111.html
  

• 91eb29a6
  by Angelos Evripiotis at 2019-02-15T14:55:06Z

  Merge branch 'are_you_sure2' into 'master'
  
  userconfig: rm really-workspace-close-project-inaccessible
  
  Closes #726 and #744
  
  See merge request BuildStream/buildstream!1130

• b8561fb1
  by Chandan Singh at 2019-02-15T14:58:22Z

  contrib/bst-here: Change default image to buildstream/buildstream
  
  See https://gitlab.com/BuildStream/buildstream-docker-images/issues/26
  for detailed discussion around this.
  
  `buildstream/buildstream-fedora` is now considered deprecated. Switch to
  `buildstream/buildstream` image. This image also offers more tags that
  will provide users more flexibility.
  

• 2699c77c
  by Chandan Singh at 2019-02-15T14:58:31Z

  contrib/bst-here: Allow users to specify image variant
  
  Now that the `buildstream/buildstream` image has 9 variants, let's make
  it easier to choose the desired tag, using a command-line option.
  
  This is otherwise possible by specifying the full image name `image:tag`
  using the `-i` option. But, this will make it easier to specify just the
  tag using `-j`.
  
  The following two invocations of `bst-here` are now equivalent:
  
      bst-here -i buildstream/buildstream:dev
      bst-here -j dev
  

• 677fc6c5
  by Chandan Singh at 2019-02-15T16:09:21Z

  Merge branch 'chandan/bst-here-tags' into 'master'
  
  contrib/bst-here: Allow users to specify image variant
  
  See merge request BuildStream/buildstream!1153

• 3b889eb8
  by Javier Jardón at 2019-02-15T20:22:08Z

  README.rst: Add table with distros with packaged buildstream
  
  [ci skip]
  

• 232906b1
  by Javier Jardón at 2019-02-15T23:08:50Z

  Merge branch 'jjardon/distros' into 'master'
  
  README.rst: Add table with distros with packaged buildstream
  
  See merge request BuildStream/buildstream!1143

• ee2296ab
  by Tristan Van Berkom at 2019-02-18T08:41:57Z

  CONTRIBUTING.rst: Documenting the release process
  
  So that other people can also make releases.
  

• c79696fe
  by Tristan Van Berkom at 2019-02-18T09:43:22Z

  Merge branch 'tristan/document-release-process' into 'master'
  
  CONTRIBUTING.rst: Documenting the release process
  
  See merge request BuildStream/buildstream!1155

• fb65af6c
  by Jürg Billeter at 2019-02-18T10:29:25Z

  _casbaseddirectory.py: Fix order in _recalculate_recursing_up()
  
  The directory needs to be serialized after the Digest for the
  subdirectory `caller` has been updated.
  

• f5660fa0
  by Jim MacArthur at 2019-02-18T10:29:25Z

  _casbaseddirectory.py: Correct typo for _get_underlying_directory
  

• cdcf0dc5
  by Jürg Billeter at 2019-02-18T10:29:25Z

  _casbaseddirectory.py: Remove unused find_root() method
  

• f9dd6ea2
  by Jim MacArthur at 2019-02-18T10:29:25Z

  storage: Add Directory.get_size()
  

• 050249bb
  by Jürg Billeter at 2019-02-18T10:29:25Z

  _casbaseddirectory.py: Add _get_digest() method
  

• 3b881efc
  by Jürg Billeter at 2019-02-18T10:29:25Z

  element.py: Expand scope of timed_activity in _cache_artifact()
  
  ArtifactCache.commit() is no longer the only expensive part and even
  more so with the upcoming change to use CASBasedDirectory.
  

• 3832c0d1
  by Jim MacArthur at 2019-02-18T10:29:25Z

  element.py: Alter _cache_artifact to use CasBasedDirectory
  

• ef85e3b2
  by Jürg Billeter at 2019-02-18T11:18:10Z

  Merge branch 'jmac/cache_artifacts_with_vdir' into 'master'
  
  Cache artifacts with virtual directories instead of filesystem.
  
  Closes #787
  
  See merge request BuildStream/buildstream!991

• e0f0a01d
  by Chandan Singh at 2019-02-18T12:41:31Z

  requirements/Makefile: exclude pkg-resources
  
  Having `pkg-resources` in requirements files is never the right thing as
  it is not really a package that one can install. This is basically an
  artifact of Debian-based systems providing incorrect metadata to pip.
  See [this pip issue](https://github.com/pypa/pip/issues/4022) and
  [this Ubuntu issue](https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1635463)
  for more background.
  

• f149fed2
  by Tristan Van Berkom at 2019-02-18T13:43:28Z

  Merge branch 'chandan/remove-pkg-resources' into 'master'
  
  requirements/Makefile: exclude pkg-resources
  
  Closes #918
  
  See merge request BuildStream/buildstream!1156

• 4022234e
  by Tristan Van Berkom at 2019-02-18T13:44:29Z

  .gitlab-ci.yml: Some new test environments were added, but not collecting coverage.
  
  The test jobs which create coverage reports need to be listed
  in the dependencies of the coverage job, as highlighted in
  issue #356.
  

• 86466e7e
  by Tristan Van Berkom at 2019-02-18T14:27:08Z

  Merge branch 'tristan/collect-missing-coverage' into 'master'
  
  .gitlab-ci.yml: Some new test environments were added, but not collecting coverage.
  
  See merge request BuildStream/buildstream!1157

• c8cd24b9
  by Jonathan Maw at 2019-02-18T17:37:00Z

  .gitlab-ci: make wsl-test automatic, allow failure
  

• de70f8c0
  by Jonathan Maw at 2019-02-18T18:57:45Z

  Merge branch 'jonathan/wsl-tests-allow-fail' into 'master'
  
  Allow WSL tests to run, but they don't block the CI
  
  See merge request BuildStream/buildstream!1159

• 00b86b19
  by Chandan Singh at 2019-02-18T18:59:31Z

  doc/badges.py: Fix catching of CalledProcessError
  
  Since `CalledProcessError` is defined by the `subprocess` module, it
  must be caught as `subprocess.CalledProcessError`, and not as simply
  `CalledProcessError`.
  

• 067a0c6b
  by Chandan Singh at 2019-02-18T20:31:10Z

  Merge branch 'chandan/fix-badge-subprocess-error' into 'master'
  
  doc/badges.py: Fix catching of CalledProcessError
  
  See merge request BuildStream/buildstream!1160

• fca37d0d
  by Chandan Singh at 2019-02-19T06:37:04Z

  buildstream/_yaml.py: Import Mapping from collections.abc
  
  Instead of importing `Mapping` from `collections`, it must be imported
  from `collections.abc`, as the old method will stop working from Python
  3.8.
  

• 5f0571d1
  by Jürg Billeter at 2019-02-19T07:37:21Z

  Merge branch 'chandan/fix-mapping-import' into 'master'
  
  buildstream/_yaml.py: Import Mapping from collections.abc
  
  See merge request BuildStream/buildstream!1161

• e2074693
  by Gökçen Nurlu at 2019-02-19T07:44:54Z

  Prevent caching when Element.stage() fails
  
  That step is not actually part of the build phase should not be cached and
  causes hard to find bugs.
  
  Fixes #791
  

• cfbe409d
  by Jürg Billeter at 2019-02-19T09:00:31Z

  Merge branch 'gokcennurlu/dont_cache_if_stage_fails' into 'master'
  
  Prevent caching when Element.stage() fails
  
  Closes #791
  
  See merge request BuildStream/buildstream!1012

• 1ae17968
  by Valentin David at 2019-02-19T09:01:48Z

  element.py: Expect absolute paths in overlap whitelists
  
  This allows use of variables such as %{prefix} and matches the
  documentation. This is a breaking change.
  
  Fixes #721
  

• afe823e8
  by Jürg Billeter at 2019-02-19T10:33:59Z

  Merge branch 'valentindavid/absolute-whitelist' into 'master'
  
  Allow absolute paths in whitelist
  
  Closes #721
  
  See merge request BuildStream/buildstream!968

• d70bfc38
  by Adam Jones at 2019-02-19T10:34:37Z

  Add a link to the label descriptions in the issue templates
  

• 857e7414
  by Tom Pollard at 2019-02-19T11:37:15Z

  Merge branch 'adamjones/labels' into 'master'
  
  Add a link to the label descriptions in the issue templates
  
  See merge request BuildStream/buildstream!913

• 9bc389a8
  by Jürg Billeter at 2019-02-19T11:39:44Z

  tests: Don't use integration test for artifact log
  
  A non-integration test works just as well for this and is a lot quicker,
  especially on WSL.
  

• 4a002bee
  by Jürg Billeter at 2019-02-19T13:17:00Z

  Merge branch 'juerg/artifact-test' into 'master'
  
  tests: Don't use integration test for artifact log
  
  See merge request BuildStream/buildstream!1162

• 1cd27edf
  by Jürg Billeter at 2019-02-19T13:19:43Z

  tests/integration/shell.py: Skip test_isolated_no_mount without sandbox
  
  This test passes without a sandbox but only because the exit code is
  also non-zero in that case. It didn't actually perform the desired test.
  
  This test is very slow on WSL, so let's skip it if there is no sandbox.
  

• 69675d22
  by Jürg Billeter at 2019-02-19T13:27:31Z

  tests/integration/shell.py: More asserts in test_isolated_no_mount
  
  This verifies that the command fails with the right error.
  

• 99764715
  by Jürg Billeter at 2019-02-19T15:03:58Z

  Merge branch 'juerg/shell-test' into 'master'
  
  tests/integration/shell.py: Improve test_isolated_no_mount
  
  See merge request BuildStream/buildstream!1163

• 3df140b2
  by Dor Askayo at 2019-02-19T16:01:16Z

  setup.cfg: exclude .eggs/**, build/** and tmp/** from pycodestyle linting
  

• fec626e2
  by Dor Askayo at 2019-02-19T16:01:16Z

  .gitignore: ignore the build output directory
  

• 52c0c185
  by Phil Dawson at 2019-02-19T17:04:58Z

  Merge branch 'phil/additional-linting-excludes' into 'master'
  
  setup.cfg: exclude tmp/**, .eggs/** and build/** from pycodestyle linting
  
  See merge request BuildStream/buildstream!1165

• c7f76268
  by Raoul Hidalgo Charman at 2019-02-19T17:05:17Z

  cachedir: add new dir option that's default root to other dirs
  
  Makes artifactdir and builddir obsolete.
  
  Fixes #870
  

• 4b62b936
  by Raoul Hidalgo Charman at 2019-02-19T17:05:17Z

  context: remove artifactdir
  
  Will check and move old artifact directory if it exists, and create
  symlink linking old directory to new.
  
  Part of #870
  

• 5e10e2e8
  by Raoul Hidalgo Charman at 2019-02-19T17:05:17Z

  CASQuota: Move cache check methods into new Class
  
  This sits in Context allowing artifact cache to check the cas quota
  while not being used for CASServer.
  
  A lot of code that checks cache quota has been touched.
  
  Part of #870
  

• 36a60abe
  by Jürg Billeter at 2019-02-19T18:42:56Z

  Merge branch 'raoul/870-root-cache-dir' into 'master'
  
  root cache directory
  
  Closes #870
  
  See merge request BuildStream/buildstream!1100

• 6e86d4db
  by Angelos Evripiotis at 2019-02-20T06:06:53Z

  doc/./arch_data_model: fixup affect/effect
  

• d7596d1c
  by Angelos Evripiotis at 2019-02-20T06:06:53Z

  doc/./arch_data_model: add some missing full-stops
  

• f6b8ac57
  by Angelos Evripiotis at 2019-02-20T06:06:53Z

  doc/./arch_data_model:edit comp stages for clarity
  

• 803268fe
  by Jürg Billeter at 2019-02-20T06:54:35Z

  Merge branch 'aevri/doc_arch_tweaks' into 'master'
  
  doc/../arch_data_model: minor tweaks
  
  See merge request BuildStream/buildstream!1166

• 670ac479
  by Jürg Billeter at 2019-02-20T06:55:14Z

  tests/integration/cachedfail.py: Fix test_push_cached_fail
  
  Ensure that a unique cache key is used for different test variants. This
  revealed that the test actually fails with --on-error=quit. Marking that
  test variant as expected failure until this is fixed.
  
  https://gitlab.com/BuildStream/buildstream/issues/534
  

• cf01c3bf
  by Jürg Billeter at 2019-02-20T08:00:16Z

  Merge branch 'juerg/cachedfail' into 'master'
  
  tests/integration/cachedfail.py: Fix test_push_cached_fail
  
  See merge request BuildStream/buildstream!1167

• 25e3e01f
  by Phil Dawson at 2019-02-20T09:55:00Z

  tox.ini: Add 'venv' environment to run arbitrary commands in a venv
  
  This is a handy way for developers to make use of the venvs we're
  constructing for our test suite to run in.
  

30 changed files:

• .gitignore
• .gitlab-ci.yml
• .gitlab/issue_templates/bst_bug.md
• .gitlab/issue_templates/bst_task.md
• CONTRIBUTING.rst
• NEWS
• README.rst
• buildstream/_artifactcache.py
• buildstream/_cachekey.py
• buildstream/_cas/__init__.py
• buildstream/_cas/cascache.py
• buildstream/_context.py
• buildstream/_frontend/cli.py
• buildstream/_frontend/status.py
• buildstream/_frontend/widget.py
• buildstream/_loader/loader.py
• buildstream/_profile.py
• buildstream/_scheduler/jobs/cachesizejob.py
• buildstream/_scheduler/jobs/cleanupjob.py
• buildstream/_variables.py
• buildstream/_versions.py
• buildstream/_yaml.py
• buildstream/data/userconfig.yaml
• buildstream/element.py
• buildstream/plugins/sources/tar.py
• buildstream/plugintestutils/runcli.py
• buildstream/source.py
• buildstream/storage/_casbaseddirectory.py
• buildstream/storage/_filebaseddirectory.py
• buildstream/storage/directory.py

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.