[Notes] [Git][BuildStream/buildstream][raoul/870-root-cache-dir] 34 commits: Test that tracking in workspaces actually works

[Raoul Hidalgo Charman <gitlab mg gitlab com>]

Title: GitLab

Raoul Hidalgo Charman pushed to branch raoul/870-root-cache-dir at BuildStream / buildstream

Commits:

• 5b0bba85
  by Jonathan Maw at 2019-01-24T10:15:06Z

  Test that tracking in workspaces actually works
  
  Previously, it merely tested that buildstream did not fall other,
  rather than whether it did anything useful.
  

• 58d7d722
  by Jonathan Maw at 2019-01-24T10:18:01Z

  tests: Test that fetching an open workspace will fetch its dependencies
  
  Previously, there was no way of detecting whether fetching happened, as
  an element with an open workspace will not be fetched.
  

• 24bd8994
  by Jürg Billeter at 2019-01-24T13:36:33Z

  Merge branch 'jonathan/test-missing-workspace-guessing' into 'master'
  
  Add tests to cover reinstated support for guessing targets
  
  See merge request BuildStream/buildstream!1042

• e03dd5fc
  by Jürg Billeter at 2019-01-24T13:37:28Z

  _frontend/cli.py: Guess element also for bst build --all
  
  There is no reason to disallow guess_element() for bst build --all.
  

• 3850274b
  by Phillip Smyth at 2019-01-24T13:37:28Z

  projectconfig.yaml: Add key for default targets
  
  _versions.py: Bump format version
  

• ab72d384
  by Jürg Billeter at 2019-01-24T13:37:28Z

  _project.py: Add get_default_target() and get_default_targets() methods
  
  _frontend/cli.py: Use new methods.
  
  Based on patches by Phillip Smyth.
  

• ee2d8434
  by Jürg Billeter at 2019-01-24T13:37:28Z

  _stream.py: Add ignore_junction_targets parameter
  
  This filters out junctions from the list of targets.
  

• 2f175f0a
  by Jürg Billeter at 2019-01-24T13:37:28Z

  _frontend/cli.py: Ignore junctions in default targets where appropriate
  
  Junctions cannot be built, pulled, or pushed. Specifying a junction on
  the command line for these commands will result in an error. However,
  junctions may be in the list of default targets, so they need to be
  ignored for build, pull, and push commands.
  

• 5a351dee
  by Jürg Billeter at 2019-01-24T13:38:50Z

  format_project.rst: Add documentation for default targets
  

• 77414518
  by Phillip Smyth at 2019-01-24T13:38:50Z

  tests/frontend: Add default target tests for bst show and build
  

• ce1c10ec
  by Jürg Billeter at 2019-01-24T13:38:50Z

  tests/frontend/fetch.py: Add default target test for bst source fetch
  

• 99b5c0af
  by Jürg Billeter at 2019-01-24T13:38:50Z

  tests/frontend/pull.py: Add default target test for bst push/pull
  

• 3642c8e7
  by Jürg Billeter at 2019-01-24T13:38:50Z

  tests/frontend/buildcheckout.py: Add default target test with junction
  
  Test that `bst build` does not fail in a project where the list of
  default targets includes a junction (junctions cannot be built).
  

• 50c5159f
  by Phillip Smyth at 2019-01-24T13:38:50Z

  NEWS: Add entry for default target feature
  

• 05587f22
  by Jürg Billeter at 2019-01-24T15:10:08Z

  Merge branch 'issue-638-validate-all-files' into 'master'
  
  Add support for default targets
  
  See merge request BuildStream/buildstream!925

• 3e36e363
  by Tristan Van Berkom at 2019-01-24T16:55:24Z

  _scheduler/resources.py: Dont error out in unregister_exclusive_interest()
  
  Don't require the interest to be registered, just discard any interest,
  this function just sets a bit in a mask, and is not intended to maintain
  a balance like the reserve() function is.
  

• ce01f87e
  by Tristan Van Berkom at 2019-01-24T16:55:24Z

  _scheduler/scheduler.py: Run cache size exclusively at startup
  
  When running any session that has Queues which require Resource.CACHE,
  check if our loaded estimated size exceeds the quota, and if so;
  lock the Resource.CACHE resource exclusively right away and run
  an exclusive initial cache size job.
  
  This ensures we cleanup first before doing anything which might
  add to the cache at startup time, if deemed needed.
  
  This is a partial fix for issue #737
  

• 2479e8df
  by Tristan Van Berkom at 2019-01-24T16:55:24Z

  _frontend/widget.py: Render core messages more like other messages
  
  In order to test when core activities occur by parsing the stderr
  in tests, we should make the messages conform more.
  
  At the same time, this restores alignment of columns in core
  messages with the element processing related messages.
  
  Also, _scheduler/scheduler.py is updated to make it's activity names
  conform to the (current) 5 character limit for the sake of alignment.
  
  The tests/frontend/logging.py test gets it's regexes updated for
  the log lines it checks for in stderr.
  

• fdb8ff65
  by Tristan Van Berkom at 2019-01-24T16:55:24Z

  tests/artifactcache/expiry.py: Test that expiry happens first
  

• acd0bf22
  by Tristan Van Berkom at 2019-01-24T18:01:41Z

  Merge branch 'tristan/cache-management' into 'master'
  
  Cache management fixes
  
  See merge request BuildStream/buildstream!1091

• 41f03296
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _frontend/app.py: Initialize logging before preflighting the artifact cache
  
  The artifact cache emits messages, and we want to allow that in preflight.
  

• 24ca2f46
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  utils.py: Add _get_volume_size()
  
  We can streamline this call to os.statvfs() in a few places.
  

• 9fd9fbb9
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Refactored to use utils._get_volume_size()
  
  This will benefit from a better UtilError being raised, and
  and turns the artifact cache's local function into a one liner.
  
  The loop which finds the first existing directory in the
  given path has been removed, being meaningless due to the
  call to os.makedirs() in ArtifactCache.__init__().
  
  The local function was renamed to _get_cache_volume_size() and
  no longer takes any arguments, which is more suitable for the
  function as it serves as a testing override surface for
  unittest.mock().
  
  The following test cases which use the function to override
  the ArtifactCache behavior have been updated to use the new
  overridable function name:
  
     tests/artifactcache/cache_size.py
     tests/artifactcache/expiry.py
  

• 7ee0c579
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Added ArtifactCacheUsage()
  
  A simple object which creates a snapshot of current
  usage statistics for easy reporting in the frontend.
  

• 353293b6
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _context.py: Added get_artifact_cache_usage()
  
  A frontend facing API for obtaining usage statistics.
  
  I would have put this on Stream instead, but the Context
  seems to be the de facto place for looking up the artifact cache
  in general so let's put it here.
  

• 51ed36de
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _frontend/widget.py: Added cache usage entry in the startup heading
  

• 5797238b
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _frontend/status.py: Added Cache size usage indicator to status bar
  
  This also adds some comments around the main status bar heading
  rendering function.
  

• 8074ebf4
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Add status messages in cache management operations
  
  Added some useful status messages when:
  
    * Calculating a new artifact cache usage size
  
    * Starting a cleanup
  
    * Finishing a cleanup
  
  Also enhanced messaging about what was cleaned up so far when
  aborting a cleanup.
  

• 49c11bc8
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Added client progress callback to ArtifactCache.clean()
  

• 3616e939
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _scheduler/jobs/job.py: Allow subclasses to message the frontend
  

• bcd19266
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _scheduler/jobs/cleanupjob.py: Update cache size while processing
  
  Updates the known cache size in the main process while the cleanup
  process is ongoing, so that the status indicators update live
  while the cleanup happens.
  

• 9c33107f
  by Tristan Van Berkom at 2019-01-24T18:02:08Z

  _artifactcache.py: Correcting API documenting comment for remove()
  
  This seems to have been copy/pasted from cascache, and
  documents the function to possibly return None if defer_prune
  was specified, but this function does not expose defer_prune.
  

• a2140d74
  by Tristan Van Berkom at 2019-01-24T19:44:52Z

  Merge branch 'tristan/cache-management-logging' into 'master'
  
  Cache management logging enhancements
  
  See merge request BuildStream/buildstream!1105

• bf7ede29
  by Raoul Hidalgo Charman at 2019-01-25T15:20:15Z

  rootcachedir: add new dir option that's default root to other dirs
  
  Deprecates artifactdir and builddir.
  
  Fixes #870
  

30 changed files:

• NEWS
• buildstream/_artifactcache.py
• buildstream/_context.py
• buildstream/_frontend/app.py
• buildstream/_frontend/cli.py
• buildstream/_frontend/status.py
• buildstream/_frontend/widget.py
• buildstream/_project.py
• buildstream/_scheduler/jobs/cleanupjob.py
• buildstream/_scheduler/jobs/job.py
• buildstream/_scheduler/resources.py
• buildstream/_scheduler/scheduler.py
• buildstream/_stream.py
• buildstream/_versions.py
• buildstream/data/projectconfig.yaml
• buildstream/data/userconfig.yaml
• buildstream/utils.py
• conftest.py
• doc/bst2html.py
• doc/sessions/running-commands.run
• doc/source/format_project.rst
• tests/artifactcache/cache_size.py
• tests/artifactcache/expiry.py
• tests/artifactcache/junctions.py
• tests/artifactcache/pull.py
• tests/artifactcache/push.py
• tests/frontend/buildcheckout.py
• tests/frontend/fetch.py
• tests/frontend/logging.py
• + tests/frontend/project_default/elements/target.bst

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.