[Notes] [Git][BuildStream/buildstream][jonathan/junction-no-tmpdir] 45 commits: Expose basic api for testing external plugins.

[Jonathan Maw <gitlab mg gitlab com>]

Title: GitLab

Jonathan Maw pushed to branch jonathan/junction-no-tmpdir at BuildStream / buildstream

Commits:

• b4d4c4f5
  by Phil Dawson at 2019-02-08T14:27:54Z

  Expose basic api for testing external plugins.
  
  We want external plugins to be able to make use of the core testing utils.
  This commit exposes the basic utilities which are currently in use in
  bst-external plugins. If necessary, more utilities could be exposed in the
  future.
  
  Moves the following files from tests/testutils/ to
  buildstream/plugintestingutils/:
  o runcli.py
  o integration.py
  
  As part of this, this commit makes the following changes to runcli.py
  and integration.py:
  o runcli.py: Fix linting errors
  o runcli.py: Add user facing documentation
  o Integration.py: Add user facing documentation
  

• 77ce0c16
  by Phil Dawson at 2019-02-08T16:28:05Z

  Merge branch 'phil/plugin-testing-api' into 'master'
  
  Expose basic api for testing external plugins.
  
  Closes #847
  
  See merge request BuildStream/buildstream!1075

• 1ee4a4ba
  by Abderrahim Kitouni at 2019-02-08T16:42:23Z

  _artifactcache.py: don't leak the project specific remote caches
  
  the code for initializing remotes added the project specific remote
  caches to the global list instead of making a copy.
  
  Fixes #618
  

• 24c0de16
  by Abderrahim Kitouni at 2019-02-08T16:42:23Z

  _project.py: use artifact caches from the parent project for junctions
  
  This makes a junction use the artifact cache of the parent project
  before the ones defined for the junction
  
  Fixes #401
  

• a937f99a
  by Javier Jardón at 2019-02-08T20:29:05Z

  Merge branch 'abderrahim/artifact-cache-junction' into 'master'
  
  Use artifact cache specs from the parent project before those defined in junctions
  
  Closes #618 and #401
  
  See merge request BuildStream/buildstream!1113

• da1560e4
  by Javier Jardón at 2019-02-09T09:10:01Z

  .gitlab-ci.yml: Remove testing on Fedora 27
  
  Fedora 27 is EOL since 2018-11-30
  
  See https://fedoraproject.org/wiki/End_of_life
  

• 79fbab61
  by Jürg Billeter at 2019-02-09T10:30:20Z

  Merge branch 'jjardon/fedora_27' into 'master'
  
  .gitlab-ci.yml: Remove testing on Fedora 27
  
  See merge request BuildStream/buildstream!1136

• 354c563d
  by Jürg Billeter at 2019-02-10T20:03:46Z

  import.py: Validate config node
  
  Fixes #662.
  

• 5e1be71f
  by Jürg Billeter at 2019-02-11T05:10:56Z

  Merge branch 'juerg/import' into 'master'
  
  import.py: Validate config node
  
  Closes #662
  
  See merge request BuildStream/buildstream!1141

• 99e1be45
  by Jürg Billeter at 2019-02-11T05:12:25Z

  local.py: Do not follow symlinks in local directories
  
  isdir() follows symlinks on the host, resulting in potential host
  contamination. This change reorders the file checks to avoid this issue.
  

• f95e222e
  by Jürg Billeter at 2019-02-11T05:12:25Z

  sandbox/sandbox.py: Do not follow symlinks in _has_command()
  
  This is required to ensure symlinks are not resolved on the host.
  

• 89973fb3
  by Jürg Billeter at 2019-02-11T05:12:25Z

  utils.py: Remove list_dirs parameter from list_relative_paths()
  
  list_dirs was always True in the BuildStream code base. There was also a
  bug in the list_dirs=False code path as it did not return symlinks in
  `dirnames`.
  
  This is an API break, however, there are no known external callers.
  

• d1da3fb0
  by Jürg Billeter at 2019-02-11T05:12:25Z

  utils.py: Fix sorting of symlinks to directories
  
  os.walk() resolves symlinks to check whether they point to a directory
  even when followlinks is set to False. We already work around that
  broken behavior by extracting symlinks from `dirnames`. However, the
  sort order was still incorrect as we returned symlinks in dirnames
  before files and other symlinks. This change fixes this, sorting all
  files and symlinks in a single list.
  

• 7cf67ed3
  by Jürg Billeter at 2019-02-11T05:12:25Z

  _casbaseddirectory.py: Do not mimic os.walk() in list_relative_paths()
  
  This matches the change in utils.list_relative_paths() that now sorts
  all symlinks as files, instead of following the broken behavior of
  os.walk().
  

• 7ec0bb5e
  by Jürg Billeter at 2019-02-11T05:44:20Z

  tests/sources/local.py: Add directory symlink test
  

• c07cc967
  by Jürg Billeter at 2019-02-11T07:13:28Z

  Merge branch 'juerg/symlinks' into 'master'
  
  Symlink fixes
  
  See merge request BuildStream/buildstream!1138

• 9e3d3e05
  by Angelos Evripiotis at 2019-02-11T07:14:50Z

  contributing: snakeviz replaces pyflame+flamegraph
  
  Replace the instructions for pyflame+flamegraph with simpler ones for
  snakeviz. For our general use-case this seems to be easier and better.
  Usage of this tool was demonstrated at the 2019 BuildStream Gathering in
  January by Daniel Silverstone, when presenting the aggregate results of
  profiling on many target environments.
  
  Here is the relevant mailing list thread:
  
  "Profiling before the gathering"
  https://mail.gnome.org/archives/buildstream-list/2019-January/msg00057.html
  

• bb6a692d
  by Jürg Billeter at 2019-02-11T09:20:34Z

  Merge branch 'snakeviz' into 'master'
  
  contributing: snakeviz replaces pyflame+flamegraph
  
  See merge request BuildStream/buildstream!1129

• 1ed63e54
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: better provenance on recursive include
  
  Use the provenance of the include block, instead of the whole node.
  

• 02e48209
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: better error on missing include
  
  Previously, a missing include would result in an error like this:
  
      Could not find file at not-a-file.include
  
  Note that the file containing the include was not mentioned.
  
  Now we get an error like this instead:
  
      element.bst [line 7 column 5]: Include block references a file that
      could not be found: 'not-a-file.include'.
  

• 4336e3bf
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: better error on including directory
  
  Previously, include a directory result in an error like this:
  
      mydir is a directory. bst command expects a .bst file.
  
  Note that the file containing the include was not mentioned.
  
  Now we get an error like this instead:
  
      element.bst [line 12 column 0]: Include block references a
      directory instead of a file: 'mydir'.
  

• 3f6c5000
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: re-use file_path variable
  
  Avoid an unnecessary call to os.path.join().
  

• adde0c94
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  tests/format/include: remove unused tmpdir's
  
  Don't create and remove temp dirs unnecessarily when they are not used,
  looks like these were just copy-pastes without intended side-effects.
  

• a66f8379
  by Jürg Billeter at 2019-02-11T13:52:54Z

  Merge branch 'aevri/include-error' into 'master'
  
  More user-friendly reporting on include errors
  
  See merge request BuildStream/buildstream!891

• a482b008
  by Javier Jardón at 2019-02-11T19:23:02Z

  .gitlab-ci.yml: Do not automatically run "test-wsl" until runner problems are sorted out
  

• 0816b8b1
  by Jürg Billeter at 2019-02-11T21:29:29Z

  Merge branch 'jjardon/allow_fail_WSL' into 'master'
  
  .gitlab-ci.yml: allow "test-wsl" to fail until runner problems are sorted out
  
  See merge request BuildStream/buildstream!1145

• a7aed65a
  by Jürg Billeter at 2019-02-12T05:52:07Z

  utils.py: Increase buffer size in sha256sum()
  
  Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
  factor 4, according to a very simple benchmark.
  

• 8b9e1d24
  by Jürg Billeter at 2019-02-12T05:52:07Z

  _cas/cascache.py: Increase buffer size in add_object()
  
  Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
  factor 4, according to a very simple benchmark.
  

• 86a9048a
  by Jürg Billeter at 2019-02-12T07:08:18Z

  Merge branch 'juerg/buffer-size' into 'master'
  
  Increase read buffer size to improve performance
  
  See merge request BuildStream/buildstream!1142

• 9b6c18e4
  by Chandan Singh at 2019-02-12T10:20:52Z

  _frontend: Allow printing dependencies using `bst show`
  
  At present, there isn't an easy way to print anything from `bst show`
  that would give the users an idea of what the dependency graph looks
  like. One could use things like `--deps build`, but that will just print
  a list, without any information about the dependency edges.
  
  Add `%{deps}`, `%{build-deps}` and `%{runtime-deps}` format strings to
  `bst show` that would simply print the list of all dependencies, build
  dependencies and runtime dependencies respectively.
  
  Summary of changes:
  
    * buildstream/_frontend/cli.py: Add help for new format symbols.
    * buildstream/_frontend/widget.py: Add support for new format symbols
                                       for dependencies.
    * tests/frontend/show.py: Add tests for new format symbols.
  

• 126bc006
  by Chandan Singh at 2019-02-12T10:23:44Z

  Re-generate man pages
  

• dbce3434
  by Chandan Singh at 2019-02-12T12:19:57Z

  Merge branch 'chandan/deps' into 'master'
  
  _frontend: Allow printing dependencies using `bst show`
  
  Closes #890
  
  See merge request BuildStream/buildstream!1121

• 24d29a17
  by Javier Jardón at 2019-02-12T12:27:42Z

  .gitlab-ci.yml: Add job to run test in current Fedora:29
  

• 80dcf65d
  by Javier Jardón at 2019-02-12T12:27:42Z

  .gitlab-ci.yml: run all generic fedora test in current stable version:29
  

• 58b84905
  by Javier Jardón at 2019-02-12T12:27:42Z

  .gitlab-ci.yml: Remove "tests-python-3.7-stretch" job
  
  Fedora 29 includes python 3.7 so this is not needed anymore
  

• 95d9b9ae
  by Javier Jardón at 2019-02-12T13:31:56Z

  Merge branch 'jjardon/fedora_29' into 'master'
  
  .gitlab-ci.yml: Test with current fedora release: 29
  
  See merge request BuildStream/buildstream!1137

• a39c4767
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Only retrieve provenance in node_get() when needed
  
  We were indiscriminately retrieving the node's provenance data in the
  `node_get()` function which was accounting for approximately a third of
  the total runtime of `node_get()` which dominates pre-scheduler time in
  `bst build`.  This change ameliorates that situation by only retrieving
  the provenance data when it's actually needed.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• cda03313
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Reduce use of `isinstance()` in `node_sanitize()`
  
  We know that nodes are typically one of:
  str, list, dict, bool, tuple, NoneType or our ChainMap
  Of these, dict and ChainMap are Mapping, only list is list
  and the rest are returned unchanged.  We can reduce/defer our use
  of isinstance here, dramatically, improving performance.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• b6f08e1b
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Remove use of isinstance() in `{node,list}_{chain_,}copy`
  
  A non-trivial proportion of time pre-scheduler in `bst build` is spent
  copying (or chain-copying) nodes.  Approximately a quarter of the time
  spent in that effort is in `isinstance()`.  This removes that CPU load.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 69ee11c6
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Reduce cost of node_final_assertions
  
  By re-using the isinstance replacements from earlier commits and
  using a tuple of the string constants for checking for composition
  markers, we reduce the cost of node_final_assertions by two thirds
  in basic testing.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 0928e570
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _context.py: Cache result of get_strict()
  
  While get_strict() doesn't look expensive per-se, it is called so many
  times that it is valuable to cache the result once computed.  Since I
  don't think it can change once it is computable, cache it immediately
  that becomes possible and we save 20s in my test case.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 022a59f0
  by Benjamin Schubert at 2019-02-12T16:14:05Z

  Merge branch 'danielsilverstone-ct/further-optimisations' into 'master'
  
  Further optimisations
  
  See merge request BuildStream/buildstream!1131

• 352f4ad9
  by Jonathan Maw at 2019-02-12T16:44:35Z

  loader.py: Stage junctions into .bst instead of a tmpdir
  

• 62396af9
  by Jonathan Maw at 2019-02-12T16:44:35Z

  loader.py: Load junction from workspace if one's open
  

• f1e9cb66
  by Jonathan Maw at 2019-02-12T16:44:35Z

  Remove cleanup functionality from loader
  
  Since junction check-outs are now persistent, there is no need to clean
  up temporary directories once finished.
  Now, junction checkouts are stored within the top-level project dir and
  will be cleaned-up when that directory is removed.
  
  This involves changes in:
  * loader.py: Remove cleanup logic and passing-around of tmpdir.
  * _project.py: Remove passing around tmpdir and calling loader's
                 cleanup.
  

30 changed files:

• .gitlab-ci.yml
• CONTRIBUTING.rst
• buildstream/_artifactcache.py
• buildstream/_cas/cascache.py
• buildstream/_context.py
• buildstream/_frontend/cli.py
• buildstream/_frontend/widget.py
• buildstream/_includes.py
• buildstream/_loader/loader.py
• buildstream/_project.py
• buildstream/_yaml.py
• buildstream/plugins/elements/import.py
• buildstream/plugins/sources/local.py
• + buildstream/plugintestutils/__init__.py
• tests/testutils/integration.py → buildstream/plugintestutils/integration.py
• tests/testutils/runcli.py → buildstream/plugintestutils/runcli.py
• buildstream/sandbox/sandbox.py
• buildstream/storage/_casbaseddirectory.py
• buildstream/utils.py
• man/bst-artifact-checkout.1
• man/bst-artifact-log.1
• man/bst-artifact-pull.1
• man/bst-artifact-push.1
• man/bst-artifact-server.1
• man/bst-artifact.1
• man/bst-build.1
• man/bst-help.1
• man/bst-init.1
• man/bst-shell.1
• man/bst-show.1

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.