[Notes] [Git][BuildStream/buildstream][juerg/symlinks2] 35 commits: contributing: snakeviz replaces pyflame+flamegraph

[Jürg Billeter <gitlab mg gitlab com>]

Title: GitLab

Jürg Billeter pushed to branch juerg/symlinks2 at BuildStream / buildstream

Commits:

• 9e3d3e05
  by Angelos Evripiotis at 2019-02-11T07:14:50Z

  contributing: snakeviz replaces pyflame+flamegraph
  
  Replace the instructions for pyflame+flamegraph with simpler ones for
  snakeviz. For our general use-case this seems to be easier and better.
  Usage of this tool was demonstrated at the 2019 BuildStream Gathering in
  January by Daniel Silverstone, when presenting the aggregate results of
  profiling on many target environments.
  
  Here is the relevant mailing list thread:
  
  "Profiling before the gathering"
  https://mail.gnome.org/archives/buildstream-list/2019-January/msg00057.html
  

• bb6a692d
  by Jürg Billeter at 2019-02-11T09:20:34Z

  Merge branch 'snakeviz' into 'master'
  
  contributing: snakeviz replaces pyflame+flamegraph
  
  See merge request BuildStream/buildstream!1129

• 1ed63e54
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: better provenance on recursive include
  
  Use the provenance of the include block, instead of the whole node.
  

• 02e48209
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: better error on missing include
  
  Previously, a missing include would result in an error like this:
  
      Could not find file at not-a-file.include
  
  Note that the file containing the include was not mentioned.
  
  Now we get an error like this instead:
  
      element.bst [line 7 column 5]: Include block references a file that
      could not be found: 'not-a-file.include'.
  

• 4336e3bf
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: better error on including directory
  
  Previously, include a directory result in an error like this:
  
      mydir is a directory. bst command expects a .bst file.
  
  Note that the file containing the include was not mentioned.
  
  Now we get an error like this instead:
  
      element.bst [line 12 column 0]: Include block references a
      directory instead of a file: 'mydir'.
  

• 3f6c5000
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  _includes: re-use file_path variable
  
  Avoid an unnecessary call to os.path.join().
  

• adde0c94
  by Angelos Evripiotis at 2019-02-11T09:24:48Z

  tests/format/include: remove unused tmpdir's
  
  Don't create and remove temp dirs unnecessarily when they are not used,
  looks like these were just copy-pastes without intended side-effects.
  

• a66f8379
  by Jürg Billeter at 2019-02-11T13:52:54Z

  Merge branch 'aevri/include-error' into 'master'
  
  More user-friendly reporting on include errors
  
  See merge request BuildStream/buildstream!891

• a482b008
  by Javier Jardón at 2019-02-11T19:23:02Z

  .gitlab-ci.yml: Do not automatically run "test-wsl" until runner problems are sorted out
  

• 0816b8b1
  by Jürg Billeter at 2019-02-11T21:29:29Z

  Merge branch 'jjardon/allow_fail_WSL' into 'master'
  
  .gitlab-ci.yml: allow "test-wsl" to fail until runner problems are sorted out
  
  See merge request BuildStream/buildstream!1145

• a7aed65a
  by Jürg Billeter at 2019-02-12T05:52:07Z

  utils.py: Increase buffer size in sha256sum()
  
  Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
  factor 4, according to a very simple benchmark.
  

• 8b9e1d24
  by Jürg Billeter at 2019-02-12T05:52:07Z

  _cas/cascache.py: Increase buffer size in add_object()
  
  Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
  factor 4, according to a very simple benchmark.
  

• 86a9048a
  by Jürg Billeter at 2019-02-12T07:08:18Z

  Merge branch 'juerg/buffer-size' into 'master'
  
  Increase read buffer size to improve performance
  
  See merge request BuildStream/buildstream!1142

• 9b6c18e4
  by Chandan Singh at 2019-02-12T10:20:52Z

  _frontend: Allow printing dependencies using `bst show`
  
  At present, there isn't an easy way to print anything from `bst show`
  that would give the users an idea of what the dependency graph looks
  like. One could use things like `--deps build`, but that will just print
  a list, without any information about the dependency edges.
  
  Add `%{deps}`, `%{build-deps}` and `%{runtime-deps}` format strings to
  `bst show` that would simply print the list of all dependencies, build
  dependencies and runtime dependencies respectively.
  
  Summary of changes:
  
    * buildstream/_frontend/cli.py: Add help for new format symbols.
    * buildstream/_frontend/widget.py: Add support for new format symbols
                                       for dependencies.
    * tests/frontend/show.py: Add tests for new format symbols.
  

• 126bc006
  by Chandan Singh at 2019-02-12T10:23:44Z

  Re-generate man pages
  

• dbce3434
  by Chandan Singh at 2019-02-12T12:19:57Z

  Merge branch 'chandan/deps' into 'master'
  
  _frontend: Allow printing dependencies using `bst show`
  
  Closes #890
  
  See merge request BuildStream/buildstream!1121

• 24d29a17
  by Javier Jardón at 2019-02-12T12:27:42Z

  .gitlab-ci.yml: Add job to run test in current Fedora:29
  

• 80dcf65d
  by Javier Jardón at 2019-02-12T12:27:42Z

  .gitlab-ci.yml: run all generic fedora test in current stable version:29
  

• 58b84905
  by Javier Jardón at 2019-02-12T12:27:42Z

  .gitlab-ci.yml: Remove "tests-python-3.7-stretch" job
  
  Fedora 29 includes python 3.7 so this is not needed anymore
  

• 95d9b9ae
  by Javier Jardón at 2019-02-12T13:31:56Z

  Merge branch 'jjardon/fedora_29' into 'master'
  
  .gitlab-ci.yml: Test with current fedora release: 29
  
  See merge request BuildStream/buildstream!1137

• a39c4767
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Only retrieve provenance in node_get() when needed
  
  We were indiscriminately retrieving the node's provenance data in the
  `node_get()` function which was accounting for approximately a third of
  the total runtime of `node_get()` which dominates pre-scheduler time in
  `bst build`.  This change ameliorates that situation by only retrieving
  the provenance data when it's actually needed.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• cda03313
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Reduce use of `isinstance()` in `node_sanitize()`
  
  We know that nodes are typically one of:
  str, list, dict, bool, tuple, NoneType or our ChainMap
  Of these, dict and ChainMap are Mapping, only list is list
  and the rest are returned unchanged.  We can reduce/defer our use
  of isinstance here, dramatically, improving performance.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• b6f08e1b
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Remove use of isinstance() in `{node,list}_{chain_,}copy`
  
  A non-trivial proportion of time pre-scheduler in `bst build` is spent
  copying (or chain-copying) nodes.  Approximately a quarter of the time
  spent in that effort is in `isinstance()`.  This removes that CPU load.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 69ee11c6
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _yaml.py: Reduce cost of node_final_assertions
  
  By re-using the isinstance replacements from earlier commits and
  using a tuple of the string constants for checking for composition
  markers, we reduce the cost of node_final_assertions by two thirds
  in basic testing.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 0928e570
  by Daniel Silverstone at 2019-02-12T15:12:30Z

  _context.py: Cache result of get_strict()
  
  While get_strict() doesn't look expensive per-se, it is called so many
  times that it is valuable to cache the result once computed.  Since I
  don't think it can change once it is computable, cache it immediately
  that becomes possible and we save 20s in my test case.
  
  Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
  

• 022a59f0
  by Benjamin Schubert at 2019-02-12T16:14:05Z

  Merge branch 'danielsilverstone-ct/further-optimisations' into 'master'
  
  Further optimisations
  
  See merge request BuildStream/buildstream!1131

• 0f45c296
  by Jürg Billeter at 2019-02-12T16:45:59Z

  projectconfig.yaml: Consistently include directories in split rules
  
  Most split rules already included the relevant directories themselves in
  addition to the directory contents.  Add the missing bin, sbin, and
  libexec directories.
  
  This is required to fix tests with the following commit that changes
  list_relative_paths() to return all directories.
  

• bfa40529
  by Jürg Billeter at 2019-02-12T16:45:59Z

  tests/integration/project: Add tests directory to split rule
  
  This is required to fix tests with the following commit that changes
  list_relative_paths() to return all directories.
  

• 5ed19a4a
  by Jürg Billeter at 2019-02-12T16:45:59Z

  utils.py: Return all directories in list_relative_paths()
  
  Returning only empty directories leads to inconsistencies when computing
  a manifest by combining results from multiple list_relative_paths()
  calls as done by the compose plugin.
  
  I.e., the same directory may be empty in one dependency and non-empty in
  another dependency.  The merged file list will still contain that
  directory even though it's no longer empty.
  
  This inconsistency causes problems when calculating differences between
  manifests. Returning all directories fixes these inconsistencies.
  
  This is a change in API behavior.
  

• f2172e3b
  by Jürg Billeter at 2019-02-12T16:45:59Z

  _casbaseddirectory.py: Return all directories in list_relative_paths()
  
  This matches the change in utils.list_relative_paths().
  

• 5f928aee
  by Jürg Billeter at 2019-02-12T16:45:59Z

  utils.py: Change _ensure_real_directory() to not resolve symlinks
  
  Resolving symlinks during staging causes various issues:
  
  * Split rules may not work properly as the resolved paths will differ
    depending on whether another artifact with a directory symlink has
    been staged in the same root directory or not, e.g., as part of
    compose.
  
  * The order of symlinks in file lists is difficult to get right to
    guarantee consistent and predictable behavior as paths in a file list
    might rely on symlinks in the same file list. See #647 and #817.
  
  * Staging order differences can lead to surprising results.
  
  * Difficult to properly support absolute symlinks. Absolute symlinks are
    currently converted to relative symlinks, however, this doesn't always
    work. See #606 and #830.
  
  This will require changes in projects that rely on the current behavior.
  However, the changes are expected to be small and are often a sign of
  buggy element files. E.g., elements that don't fully obey `bindir` or
  `sbindir` variables.
  

• fd857155
  by Jürg Billeter at 2019-02-12T16:45:59Z

  _casbaseddirectory.py: Do not resolve symlinks
  
  This matches the change in utils._process_list().
  
  This also removes the _Resolver class as it is now unused. We may want
  to support controlled symlink resolution in the future, in which case
  the _Resolver class can be resurrected from this commit.
  

• 6e7fab35
  by Jürg Billeter at 2019-02-12T16:46:11Z

  utils.py: Do not mangle absolute symlinks
  
  Copy symlinks as they are, absolute or relative. We no longer resolve
  symlinks when copying files, which makes this safe.
  

• 578a516b
  by Jürg Billeter at 2019-02-12T16:46:11Z

  Bump artifact version for changes in symlink handling
  

• 773dc4ab
  by Jürg Billeter at 2019-02-12T16:46:11Z

  NEWS: Add entry for change in symlink handling
  

30 changed files:

• .gitlab-ci.yml
• CONTRIBUTING.rst
• NEWS
• buildstream/_cas/cascache.py
• buildstream/_context.py
• buildstream/_frontend/cli.py
• buildstream/_frontend/widget.py
• buildstream/_includes.py
• buildstream/_versions.py
• buildstream/_yaml.py
• buildstream/data/projectconfig.yaml
• buildstream/storage/_casbaseddirectory.py
• buildstream/utils.py
• man/bst-artifact-checkout.1
• man/bst-artifact-log.1
• man/bst-artifact-pull.1
• man/bst-artifact-push.1
• man/bst-artifact-server.1
• man/bst-artifact.1
• man/bst-build.1
• man/bst-help.1
• man/bst-init.1
• man/bst-shell.1
• man/bst-show.1
• man/bst-source-checkout.1
• man/bst-source-fetch.1
• man/bst-source-track.1
• man/bst-source.1
• man/bst-workspace-close.1
• man/bst-workspace-list.1

The diff was not included because it is too large.

—
View it on GitLab.
You're receiving this email because of your account on gitlab.com. If you'd like to receive fewer emails, you can adjust your notification settings.