[Notes] [Git][BuildStream/buildstream][raoul/870-root-cache-dir] 47 commits: Variables: Rework how expansion strings work

[Jürg Billeter <gitlab mg gitlab com>]

Title: GitLab

Jürg Billeter pushed to branch raoul/870-root-cache-dir at BuildStream / buildstream

Commits:

• 0f78a47d
  by Daniel Silverstone at 2019-02-15T10:03:46Z

  Variables: Rework how expansion strings work
  
  Rather than constantly using regular expressions and retrieval from YAML
  nodes, pre-parse expansion strings into a list representation cached
  for reuse, and then expand them as simple string concatenation.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 14176e51
  by Benjamin Schubert at 2019-02-15T11:04:10Z

  Merge branch 'danielsilverstone-ct/variables-rework' into 'master'
  
  Variables: Rework how expansion strings work
  
  See merge request BuildStream/buildstream!1152

• 39febfda
  by Angelos Evripiotis at 2019-02-15T11:10:51Z

  storage.Directory.export_to_tar: mtime=_magic...
  
  Change the default value of mtime when doing export_to_tar() from `0` to
  `_utils._magic_timestamp`. This avoids problems in other software, which
  assume that an mtime of `0` means the file does not exist.
  
  There is more than one example where files with an mtime of zero are
  treated as non-existant by other software. e.g. [ninja][1] and [Template
  Toolkit][2].
  
  The OSTree project also [express an intention][3] to move from an mtime
  of 0 to an mtime of 1:
  
  > For this reason, OSTree acts as though all timestamps are set to
  > time_t 0, so that comparisons will be considered up-to-date. Note that
  > for a few releases, OSTree used 1 to fix warnings such as GNU Tar
  > emitting "implausibly old time stamp" with 0; however, until we have a
  > mechanism to transition cleanly to 1, for compatibilty OSTree is
  > reverted to use zero again.
  
  >From the comments on export_to_tar(), the motivation for having an mtime
  of 0 was to have reproducible results, rather than it specifically being
  the value 0.
  
  Additionally, the reproducible builds project has a [page on
  archives][4]; it mentions the benefits of setting all the files to have
  the same mtime, or clamping the mtime. It makes no mention of a
  motivation for the mtime to be specifically 0.
  
  Fixes #914
  
  [1]: https://github.com/ninja-build/ninja/issues/1120
  [2]: https://github.com/abw/Template2/blob/8d7d37200af436f1ad43628278d3caad257c8e27/lib/Template/Provider.pm#L635
  [3]: https://ostree.readthedocs.io/en/latest/manual/repo/
  [4]: https://reproducible-builds.org/docs/archives/
  

• 8b34e356
  by Angelos Evripiotis at 2019-02-15T12:25:21Z

  Merge branch 'aevri/mtime1' into 'master'
  
  storage.Directory.export_to_tar: default mtime=utils._magic_timestamp
  
  Closes #914
  
  See merge request BuildStream/buildstream!1149

• 3bbb90e7
  by Benjamin Schubert at 2019-02-15T12:34:36Z

  Use sets when checking for existence of an element
  

• 0921ccf4
  by Benjamin Schubert at 2019-02-15T13:39:52Z

  Merge branch 'bschubert/set-as-set' into 'master'
  
  Use sets when checking for existence of an element
  
  See merge request BuildStream/buildstream!1154

• c24f2971
  by Angelos Evripiotis at 2019-02-15T13:56:48Z

  userconfig: rm really-workspace-close-project-inaccessible
  
  Remove the need for the 'really-workspace-close-project-inaccessible'
  config option, as well as the option itself.
  
  As agreed on the mailing list [1], all the 'are you sure?' prompts on
  workspace reset and close were removed. While that discussion was going
  on, this new prompt and option was added. At the 2019 BuildStream
  Gathering, it was verbally agreed between myself and Tristan VB that we
  would also remove this instance.
  
  It was also agreed that we should have a notice to let the user know
  what they'd done, this was already in place if interactive. Moved it to
  be unconditional so that there's no difference in non-interactive
  behaviour. Made it output to stderr, as it's diagnostic meant for the
  user. Made it the last thing echo'd so it's next to the prompt - it's
  very relevant to what they type next. Added a test to make sure the text
  makes it to stderr in the appropriate case, and not in an inappropriate
  one.
  
  This is the last instance of any prompt configuration, so BuildStream
  can also forget all of that machinery.
  
  [1] https://mail.gnome.org/archives/buildstream-list/2018-December/msg00111.html
  

• 91eb29a6
  by Angelos Evripiotis at 2019-02-15T14:55:06Z

  Merge branch 'are_you_sure2' into 'master'
  
  userconfig: rm really-workspace-close-project-inaccessible
  
  Closes #726 and #744
  
  See merge request BuildStream/buildstream!1130

• b8561fb1
  by Chandan Singh at 2019-02-15T14:58:22Z

  contrib/bst-here: Change default image to buildstream/buildstream
  
  See https://gitlab.com/BuildStream/buildstream-docker-images/issues/26
  for detailed discussion around this.
  
  `buildstream/buildstream-fedora` is now considered deprecated. Switch to
  `buildstream/buildstream` image. This image also offers more tags that
  will provide users more flexibility.
  

• 2699c77c
  by Chandan Singh at 2019-02-15T14:58:31Z

  contrib/bst-here: Allow users to specify image variant
  
  Now that the `buildstream/buildstream` image has 9 variants, let's make
  it easier to choose the desired tag, using a command-line option.
  
  This is otherwise possible by specifying the full image name `image:tag`
  using the `-i` option. But, this will make it easier to specify just the
  tag using `-j`.
  
  The following two invocations of `bst-here` are now equivalent:
  
      bst-here -i buildstream/buildstream:dev
      bst-here -j dev
  

• 677fc6c5
  by Chandan Singh at 2019-02-15T16:09:21Z

  Merge branch 'chandan/bst-here-tags' into 'master'
  
  contrib/bst-here: Allow users to specify image variant
  
  See merge request BuildStream/buildstream!1153

• 3b889eb8
  by Javier Jardón at 2019-02-15T20:22:08Z

  README.rst: Add table with distros with packaged buildstream
  
  [ci skip]
  

• 232906b1
  by Javier Jardón at 2019-02-15T23:08:50Z

  Merge branch 'jjardon/distros' into 'master'
  
  README.rst: Add table with distros with packaged buildstream
  
  See merge request BuildStream/buildstream!1143

• ee2296ab
  by Tristan Van Berkom at 2019-02-18T08:41:57Z

  CONTRIBUTING.rst: Documenting the release process
  
  So that other people can also make releases.
  

• c79696fe
  by Tristan Van Berkom at 2019-02-18T09:43:22Z

  Merge branch 'tristan/document-release-process' into 'master'
  
  CONTRIBUTING.rst: Documenting the release process
  
  See merge request BuildStream/buildstream!1155

• fb65af6c
  by Jürg Billeter at 2019-02-18T10:29:25Z

  _casbaseddirectory.py: Fix order in _recalculate_recursing_up()
  
  The directory needs to be serialized after the Digest for the
  subdirectory `caller` has been updated.
  

• f5660fa0
  by Jim MacArthur at 2019-02-18T10:29:25Z

  _casbaseddirectory.py: Correct typo for _get_underlying_directory
  

• cdcf0dc5
  by Jürg Billeter at 2019-02-18T10:29:25Z

  _casbaseddirectory.py: Remove unused find_root() method
  

• f9dd6ea2
  by Jim MacArthur at 2019-02-18T10:29:25Z

  storage: Add Directory.get_size()
  

• 050249bb
  by Jürg Billeter at 2019-02-18T10:29:25Z

  _casbaseddirectory.py: Add _get_digest() method
  

• 3b881efc
  by Jürg Billeter at 2019-02-18T10:29:25Z

  element.py: Expand scope of timed_activity in _cache_artifact()
  
  ArtifactCache.commit() is no longer the only expensive part and even
  more so with the upcoming change to use CASBasedDirectory.
  

• 3832c0d1
  by Jim MacArthur at 2019-02-18T10:29:25Z

  element.py: Alter _cache_artifact to use CasBasedDirectory
  

• ef85e3b2
  by Jürg Billeter at 2019-02-18T11:18:10Z

  Merge branch 'jmac/cache_artifacts_with_vdir' into 'master'
  
  Cache artifacts with virtual directories instead of filesystem.
  
  Closes #787
  
  See merge request BuildStream/buildstream!991

• e0f0a01d
  by Chandan Singh at 2019-02-18T12:41:31Z

  requirements/Makefile: exclude pkg-resources
  
  Having `pkg-resources` in requirements files is never the right thing as
  it is not really a package that one can install. This is basically an
  artifact of Debian-based systems providing incorrect metadata to pip.
  See [this pip issue](https://github.com/pypa/pip/issues/4022) and
  [this Ubuntu issue](https://bugs.launchpad.net/ubuntu/+source/python-pip/+bug/1635463)
  for more background.
  

• f149fed2
  by Tristan Van Berkom at 2019-02-18T13:43:28Z

  Merge branch 'chandan/remove-pkg-resources' into 'master'
  
  requirements/Makefile: exclude pkg-resources
  
  Closes #918
  
  See merge request BuildStream/buildstream!1156

• 4022234e
  by Tristan Van Berkom at 2019-02-18T13:44:29Z

  .gitlab-ci.yml: Some new test environments were added, but not collecting coverage.
  
  The test jobs which create coverage reports need to be listed
  in the dependencies of the coverage job, as highlighted in
  issue #356.
  

• 86466e7e
  by Tristan Van Berkom at 2019-02-18T14:27:08Z

  Merge branch 'tristan/collect-missing-coverage' into 'master'
  
  .gitlab-ci.yml: Some new test environments were added, but not collecting coverage.
  
  See merge request BuildStream/buildstream!1157

• c8cd24b9
  by Jonathan Maw at 2019-02-18T17:37:00Z

  .gitlab-ci: make wsl-test automatic, allow failure
  

• de70f8c0
  by Jonathan Maw at 2019-02-18T18:57:45Z

  Merge branch 'jonathan/wsl-tests-allow-fail' into 'master'
  
  Allow WSL tests to run, but they don't block the CI
  
  See merge request BuildStream/buildstream!1159

• 00b86b19
  by Chandan Singh at 2019-02-18T18:59:31Z

  doc/badges.py: Fix catching of CalledProcessError
  
  Since `CalledProcessError` is defined by the `subprocess` module, it
  must be caught as `subprocess.CalledProcessError`, and not as simply
  `CalledProcessError`.
  

• 067a0c6b
  by Chandan Singh at 2019-02-18T20:31:10Z

  Merge branch 'chandan/fix-badge-subprocess-error' into 'master'
  
  doc/badges.py: Fix catching of CalledProcessError
  
  See merge request BuildStream/buildstream!1160

• fca37d0d
  by Chandan Singh at 2019-02-19T06:37:04Z

  buildstream/_yaml.py: Import Mapping from collections.abc
  
  Instead of importing `Mapping` from `collections`, it must be imported
  from `collections.abc`, as the old method will stop working from Python
  3.8.
  

• 5f0571d1
  by Jürg Billeter at 2019-02-19T07:37:21Z

  Merge branch 'chandan/fix-mapping-import' into 'master'
  
  buildstream/_yaml.py: Import Mapping from collections.abc
  
  See merge request BuildStream/buildstream!1161

• e2074693
  by Gökçen Nurlu at 2019-02-19T07:44:54Z

  Prevent caching when Element.stage() fails
  
  That step is not actually part of the build phase should not be cached and
  causes hard to find bugs.
  
  Fixes #791
  

• cfbe409d
  by Jürg Billeter at 2019-02-19T09:00:31Z

  Merge branch 'gokcennurlu/dont_cache_if_stage_fails' into 'master'
  
  Prevent caching when Element.stage() fails
  
  Closes #791
  
  See merge request BuildStream/buildstream!1012

• 1ae17968
  by Valentin David at 2019-02-19T09:01:48Z

  element.py: Expect absolute paths in overlap whitelists
  
  This allows use of variables such as %{prefix} and matches the
  documentation. This is a breaking change.
  
  Fixes #721
  

• afe823e8
  by Jürg Billeter at 2019-02-19T10:33:59Z

  Merge branch 'valentindavid/absolute-whitelist' into 'master'
  
  Allow absolute paths in whitelist
  
  Closes #721
  
  See merge request BuildStream/buildstream!968

• d70bfc38
  by Adam Jones at 2019-02-19T10:34:37Z

  Add a link to the label descriptions in the issue templates
  

• 857e7414
  by Tom Pollard at 2019-02-19T11:37:15Z

  Merge branch 'adamjones/labels' into 'master'
  
  Add a link to the label descriptions in the issue templates
  
  See merge request BuildStream/buildstream!913

• 9bc389a8
  by Jürg Billeter at 2019-02-19T11:39:44Z

  tests: Don't use integration test for artifact log
  
  A non-integration test works just as well for this and is a lot quicker,
  especially on WSL.
  

• 4a002bee
  by Jürg Billeter at 2019-02-19T13:17:00Z

  Merge branch 'juerg/artifact-test' into 'master'
  
  tests: Don't use integration test for artifact log
  
  See merge request BuildStream/buildstream!1162

• 1cd27edf
  by Jürg Billeter at 2019-02-19T13:19:43Z

  tests/integration/shell.py: Skip test_isolated_no_mount without sandbox
  
  This test passes without a sandbox but only because the exit code is
  also non-zero in that case. It didn't actually perform the desired test.
  
  This test is very slow on WSL, so let's skip it if there is no sandbox.
  

• 69675d22
  by Jürg Billeter at 2019-02-19T13:27:31Z

  tests/integration/shell.py: More asserts in test_isolated_no_mount
  
  This verifies that the command fails with the right error.
  

• 99764715
  by Jürg Billeter at 2019-02-19T15:03:58Z

  Merge branch 'juerg/shell-test' into 'master'
  
  tests/integration/shell.py: Improve test_isolated_no_mount
  
  See merge request BuildStream/buildstream!1163

• 7288fd94
  by Raoul Hidalgo Charman at 2019-02-19T16:50:52Z

  cachedir: add new dir option that's default root to other dirs
  
  Makes artifactdir and builddir obsolete.
  
  Fixes #870
  

• b07c8882
  by Raoul Hidalgo Charman at 2019-02-19T16:50:52Z

  context: remove artifactdir
  
  Will check and move old artifact directory if it exists, and create
  symlink linking old directory to new.
  
  Part of #870
  

• 5f3bcd65
  by Raoul Hidalgo Charman at 2019-02-19T16:50:52Z

  CASQuota: Move cache check methods into new Class
  
  This sits in Context allowing artifact cache to check the cas quota
  while not being used for CASServer.
  
  A lot of code that checks cache quota has been touched.
  
  Part of #870
  

30 changed files:

• .gitlab-ci.yml
• .gitlab/issue_templates/bst_bug.md
• .gitlab/issue_templates/bst_task.md
• CONTRIBUTING.rst
• NEWS
• README.rst
• buildstream/_artifactcache.py
• buildstream/_cas/__init__.py
• buildstream/_cas/cascache.py
• buildstream/_context.py
• buildstream/_frontend/cli.py
• buildstream/_frontend/status.py
• buildstream/_frontend/widget.py
• buildstream/_loader/loader.py
• buildstream/_profile.py
• buildstream/_scheduler/jobs/cachesizejob.py
• buildstream/_scheduler/jobs/cleanupjob.py
• buildstream/_variables.py
• buildstream/_yaml.py
• buildstream/data/userconfig.yaml
• buildstream/element.py
• buildstream/plugins/sources/tar.py
• buildstream/plugintestutils/runcli.py
• buildstream/storage/_casbaseddirectory.py
• buildstream/storage/_filebaseddirectory.py
• buildstream/storage/directory.py
• conftest.py
• contrib/bst-here
• doc/badges.py
• doc/bst2html.py

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.