[Notes] [Git][BuildStream/buildstream][raoul/870-root-cache-dir] 35 commits: _yaml.py: Only retrieve provenance in node_get() when needed

[Raoul Hidalgo Charman <gitlab mg gitlab com>]

Title: GitLab

Raoul Hidalgo Charman pushed to branch raoul/870-root-cache-dir at BuildStream / buildstream

Commits:

• a39c4767
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Only retrieve provenance in node_get() when needed
  
  We were indiscriminately retrieving the node's provenance data in the
  `node_get()` function which was accounting for approximately a third of
  the total runtime of `node_get()` which dominates pre-scheduler time in
  `bst build`.  This change ameliorates that situation by only retrieving
  the provenance data when it's actually needed.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• cda03313
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Reduce use of `isinstance()` in `node_sanitize()`
  
  We know that nodes are typically one of:
  str, list, dict, bool, tuple, NoneType or our ChainMap
  Of these, dict and ChainMap are Mapping, only list is list
  and the rest are returned unchanged.  We can reduce/defer our use
  of isinstance here, dramatically, improving performance.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• b6f08e1b
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Remove use of isinstance() in `{node,list}_{chain_,}copy`
  
  A non-trivial proportion of time pre-scheduler in `bst build` is spent
  copying (or chain-copying) nodes.  Approximately a quarter of the time
  spent in that effort is in `isinstance()`.  This removes that CPU load.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 69ee11c6
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Reduce cost of node_final_assertions
  
  By re-using the isinstance replacements from earlier commits and
  using a tuple of the string constants for checking for composition
  markers, we reduce the cost of node_final_assertions by two thirds
  in basic testing.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 0928e570
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _context.py: Cache result of get_strict()
  
  While get_strict() doesn't look expensive per-se, it is called so many
  times that it is valuable to cache the result once computed.  Since I
  don't think it can change once it is computable, cache it immediately
  that becomes possible and we save 20s in my test case.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 022a59f0
  by Benjamin Schubert at 2019-02-12T16:14:05Z

  Merge branch 'danielsilverstone-ct/further-optimisations' into 'master'
  
  Further optimisations
  
  See merge request BuildStream/buildstream!1131

• 352f4ad9
  by Jonathan Maw at 2019-02-12T16:44:35Z

  loader.py: Stage junctions into .bst instead of a tmpdir
  

• 62396af9
  by Jonathan Maw at 2019-02-12T16:44:35Z

  loader.py: Load junction from workspace if one's open
  

• f1e9cb66
  by Jonathan Maw at 2019-02-12T16:44:35Z

  Remove cleanup functionality from loader
  
  Since junction check-outs are now persistent, there is no need to clean
  up temporary directories once finished.
  Now, junction checkouts are stored within the top-level project dir and
  will be cleaned-up when that directory is removed.
  
  This involves changes in:
  * loader.py: Remove cleanup logic and passing-around of tmpdir.
  * _project.py: Remove passing around tmpdir and calling loader's
                 cleanup.
  

• e51116d5
  by Jonathan Maw at 2019-02-12T17:59:25Z

  Merge branch 'jonathan/junction-no-tmpdir' into 'master'
  
  Stage junctions into .bst instead of a tmpdir
  
  Closes #895
  
  See merge request BuildStream/buildstream!1134

• 478e5c47
  by Valentin David at 2019-02-12T18:06:50Z

  buildstream/_cas/cascache.py: Set 0644 rights to pulled files
  
  This was broken by 5ef19a0b31df84caed1e41719ef7ea5c6bd8a8bc.
  

• 6de65306
  by Valentin David at 2019-02-12T19:13:43Z

  Merge branch 'valentindavid/pull-chmod-bug' into 'master'
  
  buildstream/_cas/cascache.py: Set 0644 rights to pulled files
  
  See merge request BuildStream/buildstream!1144

• 9b140fa0
  by James Ennis at 2019-02-13T09:35:45Z

  element.py: Lift ArtifactCache.get_artifact_fullname() to here
  
  This commit removes the method ArtifactCache.get_artifact_fullname()
  and replaces it with Element.get_artifact_name()
  
  Given a key, we are now able to construct the full name of any of an
  element's artifacts.
  

• 039d7c1d
  by James Ennis at 2019-02-13T09:35:45Z

  element.py: Make calculate_cache_key() API private
  

• 7cb6be74
  by James Ennis at 2019-02-13T09:35:45Z

  element.py: Return early in __init_defaults if no plugin config
  

• ddf642e1
  by James Ennis at 2019-02-13T09:35:45Z

  metaelement.py: Provide constructor with default keyword arguments
  

• 55c15f8f
  by James Ennis at 2019-02-13T09:35:45Z

  _artifactelement.py: New ArtifactElement object (derived from Element)
  
  This object should be used when we want to handle artifact refs directly
  from the command line.
  An ArtifactElementError has also been added to _exceptions.py
  

• 6951cfc8
  by James Ennis at 2019-02-13T09:35:45Z

  _project.py: Add create_artifact_element() method
  

• bf1933b6
  by James Ennis at 2019-02-13T09:35:45Z

  _stream.py: Add the _classify_artifacts() helper
  

• ea6ff6da
  by James Ennis at 2019-02-13T09:35:45Z

  _stream.py: Modify behaviour of _classify_artifacts()
  
  * There is no need for this method to use a cas object.
  * Search for artifact globs in the project's element path
  * An artifact key is always 64 chars long
  

• fcc79917
  by James Ennis at 2019-02-13T09:35:45Z

  _stream.py: Allow loading to handle artifact refs
  

• 77345317
  by James Ennis at 2019-02-13T09:35:45Z

  _artifactcache.py: Add get_artifacts_log() method
  
  A CasBasedDirectory object of an artifacts logs can be obtained
  with ArtifactCache.get_artifacts_log(). This ultimately calls
  CASCache.get_top_level_dir() to obtain a CasBasedDirectory
  of an artifact's subdirectory (or subdirectories).
  

• 775ac472
  by James Ennis at 2019-02-13T09:35:45Z

  cli.py: Move artifact ref handling logic to stream
  
  The loading of elements and the handling of artifacts does not belong
  in this module. Such logic should be invoked using the Stream API
  

• f95d6ee8
  by James Ennis at 2019-02-13T09:35:45Z

  cli.py: Remove _classify_artifacts() method
  
  This method has been moved to Stream, where is it used there exclusively.
  

• 491937d7
  by James Ennis at 2019-02-13T09:35:45Z

  cascache.py: Allow CASCache.list_refs() to handle globs
  
  This commit ensures that CASCache.list_refs(), and
  ArtifactCache.list_artifacts(), can both handle glob expressions.
  

• 6ae41474
  by James Ennis at 2019-02-13T09:35:45Z

  _stream.py: _classify_artifacts() should be able to handle globs
  
  _classify_artifacts() no longer filters glob expressions by first
  obtaining a list of ALL refs locally cached. We now only obtain refs
  specified by the glob _expression_.
  
  Furthermore, the Project.element_path is used to start searching for
  globbed elements as opposed to the Project.directory.
  

• 8c6baf23
  by James Ennis at 2019-02-13T10:48:21Z

  Merge branch 'jennis/refactor_artifact_log' into 'master'
  
  Refactor artifact log command
  
  See merge request BuildStream/buildstream!1101

• 14db3287
  by Jürg Billeter at 2019-02-13T10:48:45Z

  projectconfig.yaml: Consistently include directories in split rules
  
  Most split rules already included the relevant directories themselves in
  addition to the directory contents.  Add the missing bin, sbin, and
  libexec directories.
  
  This is required to fix tests with the following commit that changes
  list_relative_paths() to return all directories.
  

• c7625cdf
  by Jürg Billeter at 2019-02-13T10:48:45Z

  tests/integration/project: Add tests directory to split rule
  
  This is required to fix tests with the following commit that changes
  list_relative_paths() to return all directories.
  

• 925429ea
  by Jürg Billeter at 2019-02-13T10:48:45Z

  utils.py: Return all directories in list_relative_paths()
  
  Returning only empty directories leads to inconsistencies when computing
  a manifest by combining results from multiple list_relative_paths()
  calls as done by the compose plugin.
  
  I.e., the same directory may be empty in one dependency and non-empty in
  another dependency.  The merged file list will still contain that
  directory even though it's no longer empty.
  
  This inconsistency causes problems when calculating differences between
  manifests. Returning all directories fixes these inconsistencies.
  
  This is a change in API behavior.
  

• 353745a4
  by Jürg Billeter at 2019-02-13T10:48:45Z

  _casbaseddirectory.py: Return all directories in list_relative_paths()
  
  This matches the change in utils.list_relative_paths().
  

• ae0ee361
  by Jürg Billeter at 2019-02-13T12:42:42Z

  Merge branch 'juerg/list-all-directories' into 'master'
  
  Return all directories in list_relative_paths()
  
  See merge request BuildStream/buildstream!1139

• 8fb9f8e5
  by Raoul Hidalgo Charman at 2019-02-13T14:03:26Z

  cachedir: add new dir option that's default root to other dirs
  
  Makes artifactdir and builddir obsolete.
  
  Fixes #870
  

• c44257a5
  by Raoul Hidalgo Charman at 2019-02-13T14:03:26Z

  context: remove artifactdir
  
  Will check and move old artifact directory if it exists, and create
  symlink linking old directory to new.
  

• 0cf0633d
  by Raoul Hidalgo Charman at 2019-02-13T14:13:04Z

  CASQuota: Move cache check methods into new Class
  
  This sits in Context allowing artifact cache to check the cas quota
  while not being used for CASServer.
  
  A lot of code that checks cache quota has been touched.
  
  Part of #870
  

30 changed files:

• buildstream/_artifactcache.py
• + buildstream/_artifactelement.py
• buildstream/_cas/__init__.py
• buildstream/_cas/cascache.py
• buildstream/_context.py
• buildstream/_exceptions.py
• buildstream/_frontend/cli.py
• buildstream/_frontend/status.py
• buildstream/_frontend/widget.py
• buildstream/_loader/loader.py
• buildstream/_loader/metaelement.py
• buildstream/_project.py
• buildstream/_scheduler/jobs/cachesizejob.py
• buildstream/_scheduler/jobs/cleanupjob.py
• buildstream/_stream.py
• buildstream/_yaml.py
• buildstream/data/projectconfig.yaml
• buildstream/data/userconfig.yaml
• buildstream/element.py
• buildstream/plugintestutils/runcli.py
• buildstream/storage/_casbaseddirectory.py
• buildstream/utils.py
• conftest.py
• doc/bst2html.py
• doc/sessions/running-commands.run
• tests/artifactcache/cache_size.py
• tests/artifactcache/expiry.py
• tests/artifactcache/junctions.py
• tests/artifactcache/pull.py
• tests/artifactcache/push.py

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.