[Notes] [Git][BuildStream/buildstream][jennis/refactor_artifact_log] 40 commits: filter.py/filter.yaml: Documentation improvements

[James Ennis <gitlab mg gitlab com>]

Title: GitLab

James Ennis pushed to branch jennis/refactor_artifact_log at BuildStream / buildstream

Commits:

• dbb3d232
  by James Ennis at 2019-02-01T15:51:32Z

  filter.py/filter.yaml: Documentation improvements
  

• 7e4205cb
  by James Ennis at 2019-02-01T15:51:32Z

  filter.py: Add an example to the documentation
  

• 4109a34a
  by James Ennis at 2019-02-01T17:11:29Z

  Merge branch 'jennis/filter-docs' into 'master'
  
  Improve our filter documentation
  
  Closes #278
  
  See merge request BuildStream/buildstream!1112

• c9345014
  by James Ennis at 2019-02-04T13:53:42Z

  filter.py: Fail if declared domains do not exist in the parent element
  
  This patch also uncovered the fact that our test_filter_deps_ok() test
  has been inaccurate. Thus the element built in this test
  (deps-permitted.bst) has been modified so that it build depends on the
  input.bst element, as it should.
  
  tests/filter.py: Ensure deps_ok test passes
  

• 3ab09651
  by James Ennis at 2019-02-04T14:47:43Z

  Merge branch 'jennis/warn_for_nonexistent_domains' into 'master'
  
  Fail when we explictly try to include/exclude non-existent domains in a filter element
  
  See merge request BuildStream/buildstream!1117

• cd8e5e27
  by Tom Pollard at 2019-02-05T11:22:01Z

  Add --remote, -r option to bst build, inline with pull & push
  
  Providing a remote will limit build's pull/push remote actions to
  the given remote specifically, ignoring those defined via user or
  project configuration.
  

• a46650d0
  by Tom Pollard at 2019-02-05T12:41:30Z

  Merge branch 'tpollard/buildremote' into 'master'
  
  Add --remote, -r option to bst build, inline with pull & push
  
  See merge request BuildStream/buildstream!1119

• 9c94e8e5
  by Daniel Silverstone at 2019-02-05T13:15:24Z

  requirements: Add roaringbitmap to the requirements
  
  In order to support use of roaring bitmaps in the loader, we need to
  depend on it here.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 7682ef49
  by Daniel Silverstone at 2019-02-05T13:15:24Z

  loadelement.py: Use roaring bitmaps in dep cache
  
  The dependency cache in LoadElement causes the peak RAM consumption of
  the loader to be exceedingly large, upwards of 25GB for a test of a
  Debian stack.  By switching from the old dict cache to a roaring bitmap
  based cache, we reduce the cost of the loader cache in RAM terms to
  around 5.5GB for the same stack, which makes it plausible for the short
  term.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 0e471144
  by Benjamin Schubert at 2019-02-05T15:45:37Z

  Merge branch 'danielsilverstone-ct/roaring-bitmaps' into 'master'
  
  Switch to roaring bitmaps for the loader dependency caches
  
  See merge request BuildStream/buildstream!1128

• f17ea6a6
  by Jonathan Maw at 2019-02-06T10:32:05Z

  gitlab-ci: Add commands to be run by WSL runners
  

• d4d2f897
  by Jonathan Maw at 2019-02-06T10:45:34Z

  tests: Skip tests that use sandboxes on WSL
  

• f4c8a0ec
  by Jonathan Maw at 2019-02-06T10:45:34Z

  tests: Fix test failures caused by unix sockets' path length limit
  
  The unix specification for unix sockets only allocates ~100 characters
  for the length of the path. This may be longer than the path for a CI
  runner, or the path to a user's buildstream directory.
  

• 256bcde5
  by Jonathan Maw at 2019-02-06T15:29:18Z

  Merge branch 'jonathan/wsl-tests' into 'master'
  
  Add pre-merge tests that use a WSL runner
  
  Closes #852
  
  See merge request BuildStream/buildstream!1108

• 335ef2b8
  by Daniel Silverstone at 2019-02-06T17:30:13Z

  loadelement.py: Switch from roaringbitmap to pyroaring
  
  The licence for roaringbitmap was not compatible.  pyroaring is
  a similar project which is MIT licensed and is slightly better on
  memory too.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 8f90be91
  by Benjamin Schubert at 2019-02-06T18:36:17Z

  Update testing images to include c++ compiler
  

• 1a75b252
  by Benjamin Schubert at 2019-02-06T19:34:09Z

  Merge branch 'danielsilverstone-ct/other-roaring' into 'master'
  
  loadelement.py: Switch from roaringbitmap to pyroaring
  
  Closes #899
  
  See merge request BuildStream/buildstream!1133

• 7285e87e
  by William Salmon at 2019-02-07T09:42:36Z

  Add more log formatting options
  
  The 'wallclock-us' is a option that we thought we already had.
  

• e61f4713
  by Will Salmon at 2019-02-07T10:58:54Z

  Merge branch 'willsalmon/log_formating' into 'master'
  
  Add more log formatting options
  
  See merge request BuildStream/buildstream!1125

• 40457a10
  by James Ennis at 2019-02-08T10:03:57Z

  element.py: Lift ArtifactCache.get_artifact_fullname() to here
  
  This commit removes the method ArtifactCache.get_artifact_fullname()
  and replaces it with Element.get_artifact_name()
  
  Given a key, we are now able to construct the full name of any of an
  element's artifacts.
  

• 87d652a5
  by James Ennis at 2019-02-08T10:03:57Z

  element.py: Make calculate_cache_key() API private
  

• b0575210
  by James Ennis at 2019-02-08T10:03:57Z

  element.py: Return early in __init_defaults if no plugin config
  

• 9b6d2a35
  by James Ennis at 2019-02-08T10:03:57Z

  metaelement.py: Provide constructor with default keyword arguments
  

• 322a36a3
  by James Ennis at 2019-02-08T10:03:57Z

  _artifactelement.py: New ArtifactElement object (derived from Element)
  
  This object should be used when we want to handle artifact refs directly
  from the command line.
  An ArtifactElementError has also been added to _exceptions.py
  

• fca39b7e
  by James Ennis at 2019-02-08T10:03:57Z

  _project.py: Add create_artifact_element() method
  

• adab6942
  by James Ennis at 2019-02-08T10:03:57Z

  _stream.py: Add the _classify_artifacts() helper
  

• 1f94dcf9
  by James Ennis at 2019-02-08T10:03:57Z

  _stream.py: Modify behaviour of _classify_artifacts()
  
  * There is no need for this method to use a cas object.
  * Search for artifact globs in the project's element path
  * An artifact key is always 64 chars long
  

• 5bb16a62
  by James Ennis at 2019-02-08T10:03:57Z

  _stream.py: Allow loading to handle artifact refs
  

• 6a6e2105
  by James Ennis at 2019-02-08T10:03:57Z

  _artifactcache.py: Add get_artifacts_log() method
  
  A CasBasedDirectory object of an artifacts logs can be obtained
  with ArtifactCache.get_artifacts_log(). This ultimately calls
  CASCache.get_top_level_dir() to obtain a CasBasedDirectory
  of an artifact's subdirectory (or subdirectories).
  

• b7212bf0
  by James Ennis at 2019-02-08T10:03:57Z

  cli.py: Move artifact ref handling logic to stream
  
  The loading of elements and the handling of artifacts does not belong
  in this module. Such logic should be invoked using the Stream API
  

• 59a09a47
  by James Ennis at 2019-02-08T10:03:57Z

  cli.py: Remove _classify_artifacts() method
  
  This method has been moved to Stream, where is it used there exclusively.
  

• ef7459d3
  by James Ennis at 2019-02-08T10:03:57Z

  cascache.py: Allow CASCache.list_refs() to handle globs
  
  This commit ensures that CASCache.list_refs(), and
  ArtifactCache.list_artifacts(), can both handle glob expressions.
  

• a7ae7ce9
  by James Ennis at 2019-02-08T10:03:57Z

  _stream.py: _classify_artifacts() should be able to handle globs
  
  _classify_artifacts() no longer filters glob expressions by first
  obtaining a list of ALL refs locally cached. We now only obtain refs
  specified by the glob _expression_.
  
  Furthermore, the Project.element_path is used to start searching for
  globbed elements as opposed to the Project.directory.
  

• 4b6c6afc
  by James Ennis at 2019-02-08T10:20:34Z

  _artifactelement.py: Import _cachekey and use its method to determine length
  

• a4bf528a
  by James Ennis at 2019-02-08T11:02:33Z

  _exceptions.py: ArtifactElementError cleanup
  

• 2283cafd
  by James Ennis at 2019-02-08T16:54:20Z

  _artifactelement.py: Add _verify_artifact_ref() helper
  
  If the artifact ref cannot be split by exactly two forward slashes,
  and if the length of the key is not as expected, we should raise
  an ArtifactError.
  

• 51977412
  by James Ennis at 2019-02-08T16:54:20Z

  _stream.py: Use the verify_artifact_ref() helper method
  

• 281d1a1f
  by James Ennis at 2019-02-08T16:54:20Z

  _stream.py: Add newline delimiter when unable to load multiple artifact refs
  

• 2cb971f0
  by James Ennis at 2019-02-08T16:54:20Z

  cascache.py: Move get_top_level_dir logic to ArtifactCache
  

• 3660ff18
  by James Ennis at 2019-02-08T16:54:20Z

  _stream.py: Remove guess_element()
  

30 changed files:

• .gitlab-ci.yml
• NEWS
• buildstream/_artifactcache.py
• + buildstream/_artifactelement.py
• buildstream/_cas/cascache.py
• buildstream/_exceptions.py
• buildstream/_frontend/cli.py
• buildstream/_frontend/widget.py
• buildstream/_loader/loadelement.py
• buildstream/_loader/metaelement.py
• buildstream/_project.py
• buildstream/_stream.py
• buildstream/element.py
• buildstream/plugins/elements/filter.py
• buildstream/plugins/elements/filter.yaml
• requirements/requirements.in
• requirements/requirements.txt
• tests/artifactcache/pull.py
• tests/artifactcache/push.py
• tests/elements/filter.py
• tests/elements/filter/basic/elements/deps-permitted.bst
• + tests/elements/filter/basic/elements/output-include-nonexistent-domain.bst
• tests/frontend/completions.py
• tests/frontend/logging.py
• tests/frontend/pull.py
• tests/frontend/push.py
• tests/integration/autotools.py
• tests/integration/build-tree.py
• tests/integration/build-uid.py
• tests/integration/cachedfail.py

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.