-
f17ea6a6
by Jonathan Maw
at 2019-02-06T10:32:05Z
gitlab-ci: Add commands to be run by WSL runners
-
d4d2f897
by Jonathan Maw
at 2019-02-06T10:45:34Z
tests: Skip tests that use sandboxes on WSL
-
f4c8a0ec
by Jonathan Maw
at 2019-02-06T10:45:34Z
tests: Fix test failures caused by unix sockets' path length limit
The unix specification for unix sockets only allocates ~100 characters
for the length of the path. This may be longer than the path for a CI
runner, or the path to a user's buildstream directory.
-
256bcde5
by Jonathan Maw
at 2019-02-06T15:29:18Z
Merge branch 'jonathan/wsl-tests' into 'master'
Add pre-merge tests that use a WSL runner
Closes #852
See merge request BuildStream/buildstream!1108
-
335ef2b8
by Daniel Silverstone
at 2019-02-06T17:30:13Z
loadelement.py: Switch from roaringbitmap to pyroaring
The licence for roaringbitmap was not compatible. pyroaring is
a similar project which is MIT licensed and is slightly better on
memory too.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
8f90be91
by Benjamin Schubert
at 2019-02-06T18:36:17Z
Update testing images to include c++ compiler
-
1a75b252
by Benjamin Schubert
at 2019-02-06T19:34:09Z
Merge branch 'danielsilverstone-ct/other-roaring' into 'master'
loadelement.py: Switch from roaringbitmap to pyroaring
Closes #899
See merge request BuildStream/buildstream!1133
-
7285e87e
by William Salmon
at 2019-02-07T09:42:36Z
Add more log formatting options
The 'wallclock-us' is a option that we thought we already had.
-
e61f4713
by Will Salmon
at 2019-02-07T10:58:54Z
Merge branch 'willsalmon/log_formating' into 'master'
Add more log formatting options
See merge request BuildStream/buildstream!1125
-
b4d4c4f5
by Phil Dawson
at 2019-02-08T14:27:54Z
Expose basic api for testing external plugins.
We want external plugins to be able to make use of the core testing utils.
This commit exposes the basic utilities which are currently in use in
bst-external plugins. If necessary, more utilities could be exposed in the
future.
Moves the following files from tests/testutils/ to
buildstream/plugintestingutils/:
o runcli.py
o integration.py
As part of this, this commit makes the following changes to runcli.py
and integration.py:
o runcli.py: Fix linting errors
o runcli.py: Add user facing documentation
o Integration.py: Add user facing documentation
-
77ce0c16
by Phil Dawson
at 2019-02-08T16:28:05Z
Merge branch 'phil/plugin-testing-api' into 'master'
Expose basic api for testing external plugins.
Closes #847
See merge request BuildStream/buildstream!1075
-
1ee4a4ba
by Abderrahim Kitouni
at 2019-02-08T16:42:23Z
_artifactcache.py: don't leak the project specific remote caches
the code for initializing remotes added the project specific remote
caches to the global list instead of making a copy.
Fixes #618
-
24c0de16
by Abderrahim Kitouni
at 2019-02-08T16:42:23Z
_project.py: use artifact caches from the parent project for junctions
This makes a junction use the artifact cache of the parent project
before the ones defined for the junction
Fixes #401
-
a937f99a
by Javier Jardón
at 2019-02-08T20:29:05Z
Merge branch 'abderrahim/artifact-cache-junction' into 'master'
Use artifact cache specs from the parent project before those defined in junctions
Closes #618 and #401
See merge request BuildStream/buildstream!1113
-
da1560e4
by Javier Jardón
at 2019-02-09T09:10:01Z
.gitlab-ci.yml: Remove testing on Fedora 27
Fedora 27 is EOL since 2018-11-30
See https://fedoraproject.org/wiki/End_of_life
-
79fbab61
by Jürg Billeter
at 2019-02-09T10:30:20Z
Merge branch 'jjardon/fedora_27' into 'master'
.gitlab-ci.yml: Remove testing on Fedora 27
See merge request BuildStream/buildstream!1136
-
354c563d
by Jürg Billeter
at 2019-02-10T20:03:46Z
import.py: Validate config node
Fixes #662.
-
5e1be71f
by Jürg Billeter
at 2019-02-11T05:10:56Z
Merge branch 'juerg/import' into 'master'
import.py: Validate config node
Closes #662
See merge request BuildStream/buildstream!1141
-
99e1be45
by Jürg Billeter
at 2019-02-11T05:12:25Z
local.py: Do not follow symlinks in local directories
isdir() follows symlinks on the host, resulting in potential host
contamination. This change reorders the file checks to avoid this issue.
-
f95e222e
by Jürg Billeter
at 2019-02-11T05:12:25Z
sandbox/sandbox.py: Do not follow symlinks in _has_command()
This is required to ensure symlinks are not resolved on the host.
-
89973fb3
by Jürg Billeter
at 2019-02-11T05:12:25Z
utils.py: Remove list_dirs parameter from list_relative_paths()
list_dirs was always True in the BuildStream code base. There was also a
bug in the list_dirs=False code path as it did not return symlinks in
`dirnames`.
This is an API break, however, there are no known external callers.
-
d1da3fb0
by Jürg Billeter
at 2019-02-11T05:12:25Z
utils.py: Fix sorting of symlinks to directories
os.walk() resolves symlinks to check whether they point to a directory
even when followlinks is set to False. We already work around that
broken behavior by extracting symlinks from `dirnames`. However, the
sort order was still incorrect as we returned symlinks in dirnames
before files and other symlinks. This change fixes this, sorting all
files and symlinks in a single list.
-
7cf67ed3
by Jürg Billeter
at 2019-02-11T05:12:25Z
_casbaseddirectory.py: Do not mimic os.walk() in list_relative_paths()
This matches the change in utils.list_relative_paths() that now sorts
all symlinks as files, instead of following the broken behavior of
os.walk().
-
7ec0bb5e
by Jürg Billeter
at 2019-02-11T05:44:20Z
tests/sources/local.py: Add directory symlink test
-
c07cc967
by Jürg Billeter
at 2019-02-11T07:13:28Z
Merge branch 'juerg/symlinks' into 'master'
Symlink fixes
See merge request BuildStream/buildstream!1138
-
9e3d3e05
by Angelos Evripiotis
at 2019-02-11T07:14:50Z
contributing: snakeviz replaces pyflame+flamegraph
Replace the instructions for pyflame+flamegraph with simpler ones for
snakeviz. For our general use-case this seems to be easier and better.
Usage of this tool was demonstrated at the 2019 BuildStream Gathering in
January by Daniel Silverstone, when presenting the aggregate results of
profiling on many target environments.
Here is the relevant mailing list thread:
"Profiling before the gathering"
https://mail.gnome.org/archives/buildstream-list/2019-January/msg00057.html
-
bb6a692d
by Jürg Billeter
at 2019-02-11T09:20:34Z
Merge branch 'snakeviz' into 'master'
contributing: snakeviz replaces pyflame+flamegraph
See merge request BuildStream/buildstream!1129
-
1ed63e54
by Angelos Evripiotis
at 2019-02-11T09:24:48Z
_includes: better provenance on recursive include
Use the provenance of the include block, instead of the whole node.
-
02e48209
by Angelos Evripiotis
at 2019-02-11T09:24:48Z
_includes: better error on missing include
Previously, a missing include would result in an error like this:
Could not find file at not-a-file.include
Note that the file containing the include was not mentioned.
Now we get an error like this instead:
element.bst [line 7 column 5]: Include block references a file that
could not be found: 'not-a-file.include'.
-
4336e3bf
by Angelos Evripiotis
at 2019-02-11T09:24:48Z
_includes: better error on including directory
Previously, include a directory result in an error like this:
mydir is a directory. bst command expects a .bst file.
Note that the file containing the include was not mentioned.
Now we get an error like this instead:
element.bst [line 12 column 0]: Include block references a
directory instead of a file: 'mydir'.
-
3f6c5000
by Angelos Evripiotis
at 2019-02-11T09:24:48Z
_includes: re-use file_path variable
Avoid an unnecessary call to os.path.join().
-
adde0c94
by Angelos Evripiotis
at 2019-02-11T09:24:48Z
tests/format/include: remove unused tmpdir's
Don't create and remove temp dirs unnecessarily when they are not used,
looks like these were just copy-pastes without intended side-effects.
-
a66f8379
by Jürg Billeter
at 2019-02-11T13:52:54Z
Merge branch 'aevri/include-error' into 'master'
More user-friendly reporting on include errors
See merge request BuildStream/buildstream!891
-
a482b008
by Javier Jardón
at 2019-02-11T19:23:02Z
.gitlab-ci.yml: Do not automatically run "test-wsl" until runner problems are sorted out
-
0816b8b1
by Jürg Billeter
at 2019-02-11T21:29:29Z
Merge branch 'jjardon/allow_fail_WSL' into 'master'
.gitlab-ci.yml: allow "test-wsl" to fail until runner problems are sorted out
See merge request BuildStream/buildstream!1145
-
a7aed65a
by Jürg Billeter
at 2019-02-12T05:52:07Z
utils.py: Increase buffer size in sha256sum()
Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
factor 4, according to a very simple benchmark.
-
8b9e1d24
by Jürg Billeter
at 2019-02-12T05:52:07Z
_cas/cascache.py: Increase buffer size in add_object()
Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
factor 4, according to a very simple benchmark.
-
86a9048a
by Jürg Billeter
at 2019-02-12T07:08:18Z
Merge branch 'juerg/buffer-size' into 'master'
Increase read buffer size to improve performance
See merge request BuildStream/buildstream!1142
-
9b6c18e4
by Chandan Singh
at 2019-02-12T10:20:52Z
_frontend: Allow printing dependencies using `bst show`
At present, there isn't an easy way to print anything from `bst show`
that would give the users an idea of what the dependency graph looks
like. One could use things like `--deps build`, but that will just print
a list, without any information about the dependency edges.
Add `%{deps}`, `%{build-deps}` and `%{runtime-deps}` format strings to
`bst show` that would simply print the list of all dependencies, build
dependencies and runtime dependencies respectively.
Summary of changes:
* buildstream/_frontend/cli.py: Add help for new format symbols.
* buildstream/_frontend/widget.py: Add support for new format symbols
for dependencies.
* tests/frontend/show.py: Add tests for new format symbols.
-
126bc006
by Chandan Singh
at 2019-02-12T10:23:44Z
Re-generate man pages
-
dbce3434
by Chandan Singh
at 2019-02-12T12:19:57Z
Merge branch 'chandan/deps' into 'master'
_frontend: Allow printing dependencies using `bst show`
Closes #890
See merge request BuildStream/buildstream!1121
-
24d29a17
by Javier Jardón
at 2019-02-12T12:27:42Z
.gitlab-ci.yml: Add job to run test in current Fedora:29
-
80dcf65d
by Javier Jardón
at 2019-02-12T12:27:42Z
.gitlab-ci.yml: run all generic fedora test in current stable version:29
-
58b84905
by Javier Jardón
at 2019-02-12T12:27:42Z
.gitlab-ci.yml: Remove "tests-python-3.7-stretch" job
Fedora 29 includes python 3.7 so this is not needed anymore
-
95d9b9ae
by Javier Jardón
at 2019-02-12T13:31:56Z
Merge branch 'jjardon/fedora_29' into 'master'
.gitlab-ci.yml: Test with current fedora release: 29
See merge request BuildStream/buildstream!1137
-
a39c4767
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_yaml.py: Only retrieve provenance in node_get() when needed
We were indiscriminately retrieving the node's provenance data in the
`node_get()` function which was accounting for approximately a third of
the total runtime of `node_get()` which dominates pre-scheduler time in
`bst build`. This change ameliorates that situation by only retrieving
the provenance data when it's actually needed.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
cda03313
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_yaml.py: Reduce use of `isinstance()` in `node_sanitize()`
We know that nodes are typically one of:
str, list, dict, bool, tuple, NoneType or our ChainMap
Of these, dict and ChainMap are Mapping, only list is list
and the rest are returned unchanged. We can reduce/defer our use
of isinstance here, dramatically, improving performance.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b6f08e1b
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_yaml.py: Remove use of isinstance() in `{node,list}_{chain_,}copy`
A non-trivial proportion of time pre-scheduler in `bst build` is spent
copying (or chain-copying) nodes. Approximately a quarter of the time
spent in that effort is in `isinstance()`. This removes that CPU load.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
69ee11c6
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_yaml.py: Reduce cost of node_final_assertions
By re-using the isinstance replacements from earlier commits and
using a tuple of the string constants for checking for composition
markers, we reduce the cost of node_final_assertions by two thirds
in basic testing.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
0928e570
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_context.py: Cache result of get_strict()
While get_strict() doesn't look expensive per-se, it is called so many
times that it is valuable to cache the result once computed. Since I
don't think it can change once it is computable, cache it immediately
that becomes possible and we save 20s in my test case.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
022a59f0
by Benjamin Schubert
at 2019-02-12T16:14:05Z
Merge branch 'danielsilverstone-ct/further-optimisations' into 'master'
Further optimisations
See merge request BuildStream/buildstream!1131
-
352f4ad9
by Jonathan Maw
at 2019-02-12T16:44:35Z
loader.py: Stage junctions into .bst instead of a tmpdir
-
62396af9
by Jonathan Maw
at 2019-02-12T16:44:35Z
loader.py: Load junction from workspace if one's open
-
f1e9cb66
by Jonathan Maw
at 2019-02-12T16:44:35Z
Remove cleanup functionality from loader
Since junction check-outs are now persistent, there is no need to clean
up temporary directories once finished.
Now, junction checkouts are stored within the top-level project dir and
will be cleaned-up when that directory is removed.
This involves changes in:
* loader.py: Remove cleanup logic and passing-around of tmpdir.
* _project.py: Remove passing around tmpdir and calling loader's
cleanup.
-
e51116d5
by Jonathan Maw
at 2019-02-12T17:59:25Z
Merge branch 'jonathan/junction-no-tmpdir' into 'master'
Stage junctions into .bst instead of a tmpdir
Closes #895
See merge request BuildStream/buildstream!1134
-
478e5c47
by Valentin David
at 2019-02-12T18:06:50Z
buildstream/_cas/cascache.py: Set 0644 rights to pulled files
This was broken by 5ef19a0b31df84caed1e41719ef7ea5c6bd8a8bc.
-
6de65306
by Valentin David
at 2019-02-12T19:13:43Z
Merge branch 'valentindavid/pull-chmod-bug' into 'master'
buildstream/_cas/cascache.py: Set 0644 rights to pulled files
See merge request BuildStream/buildstream!1144
-
9b140fa0
by James Ennis
at 2019-02-13T09:35:45Z
element.py: Lift ArtifactCache.get_artifact_fullname() to here
This commit removes the method ArtifactCache.get_artifact_fullname()
and replaces it with Element.get_artifact_name()
Given a key, we are now able to construct the full name of any of an
element's artifacts.
-
039d7c1d
by James Ennis
at 2019-02-13T09:35:45Z
element.py: Make calculate_cache_key() API private
-
7cb6be74
by James Ennis
at 2019-02-13T09:35:45Z
element.py: Return early in __init_defaults if no plugin config
-
ddf642e1
by James Ennis
at 2019-02-13T09:35:45Z
metaelement.py: Provide constructor with default keyword arguments
-
55c15f8f
by James Ennis
at 2019-02-13T09:35:45Z
_artifactelement.py: New ArtifactElement object (derived from Element)
This object should be used when we want to handle artifact refs directly
from the command line.
An ArtifactElementError has also been added to _exceptions.py
-
6951cfc8
by James Ennis
at 2019-02-13T09:35:45Z
_project.py: Add create_artifact_element() method
-
bf1933b6
by James Ennis
at 2019-02-13T09:35:45Z
_stream.py: Add the _classify_artifacts() helper
-
ea6ff6da
by James Ennis
at 2019-02-13T09:35:45Z
_stream.py: Modify behaviour of _classify_artifacts()
* There is no need for this method to use a cas object.
* Search for artifact globs in the project's element path
* An artifact key is always 64 chars long
-
fcc79917
by James Ennis
at 2019-02-13T09:35:45Z
_stream.py: Allow loading to handle artifact refs
-
77345317
by James Ennis
at 2019-02-13T09:35:45Z
_artifactcache.py: Add get_artifacts_log() method
A CasBasedDirectory object of an artifacts logs can be obtained
with ArtifactCache.get_artifacts_log(). This ultimately calls
CASCache.get_top_level_dir() to obtain a CasBasedDirectory
of an artifact's subdirectory (or subdirectories).
-
775ac472
by James Ennis
at 2019-02-13T09:35:45Z
cli.py: Move artifact ref handling logic to stream
The loading of elements and the handling of artifacts does not belong
in this module. Such logic should be invoked using the Stream API
-
f95d6ee8
by James Ennis
at 2019-02-13T09:35:45Z
cli.py: Remove _classify_artifacts() method
This method has been moved to Stream, where is it used there exclusively.
-
491937d7
by James Ennis
at 2019-02-13T09:35:45Z
cascache.py: Allow CASCache.list_refs() to handle globs
This commit ensures that CASCache.list_refs(), and
ArtifactCache.list_artifacts(), can both handle glob expressions.
-
6ae41474
by James Ennis
at 2019-02-13T09:35:45Z
_stream.py: _classify_artifacts() should be able to handle globs
_classify_artifacts() no longer filters glob expressions by first
obtaining a list of ALL refs locally cached. We now only obtain refs
specified by the glob _expression_.
Furthermore, the Project.element_path is used to start searching for
globbed elements as opposed to the Project.directory.
-
8c6baf23
by James Ennis
at 2019-02-13T10:48:21Z
Merge branch 'jennis/refactor_artifact_log' into 'master'
Refactor artifact log command
See merge request BuildStream/buildstream!1101
-
52c7567f
by Daniel Silverstone
at 2019-02-13T11:09:41Z
WIP: fixme! reword! Current dodgy expansion string variant