-
9b6c18e4
by Chandan Singh
at 2019-02-12T10:20:52Z
_frontend: Allow printing dependencies using `bst show`
At present, there isn't an easy way to print anything from `bst show`
that would give the users an idea of what the dependency graph looks
like. One could use things like `--deps build`, but that will just print
a list, without any information about the dependency edges.
Add `%{deps}`, `%{build-deps}` and `%{runtime-deps}` format strings to
`bst show` that would simply print the list of all dependencies, build
dependencies and runtime dependencies respectively.
Summary of changes:
* buildstream/_frontend/cli.py: Add help for new format symbols.
* buildstream/_frontend/widget.py: Add support for new format symbols
for dependencies.
* tests/frontend/show.py: Add tests for new format symbols.
-
126bc006
by Chandan Singh
at 2019-02-12T10:23:44Z
Re-generate man pages
-
dbce3434
by Chandan Singh
at 2019-02-12T12:19:57Z
Merge branch 'chandan/deps' into 'master'
_frontend: Allow printing dependencies using `bst show`
Closes #890
See merge request BuildStream/buildstream!1121
-
24d29a17
by Javier Jardón
at 2019-02-12T12:27:42Z
.gitlab-ci.yml: Add job to run test in current Fedora:29
-
80dcf65d
by Javier Jardón
at 2019-02-12T12:27:42Z
.gitlab-ci.yml: run all generic fedora test in current stable version:29
-
58b84905
by Javier Jardón
at 2019-02-12T12:27:42Z
.gitlab-ci.yml: Remove "tests-python-3.7-stretch" job
Fedora 29 includes python 3.7 so this is not needed anymore
-
95d9b9ae
by Javier Jardón
at 2019-02-12T13:31:56Z
Merge branch 'jjardon/fedora_29' into 'master'
.gitlab-ci.yml: Test with current fedora release: 29
See merge request BuildStream/buildstream!1137
-
a39c4767
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_yaml.py: Only retrieve provenance in node_get() when needed
We were indiscriminately retrieving the node's provenance data in the
`node_get()` function which was accounting for approximately a third of
the total runtime of `node_get()` which dominates pre-scheduler time in
`bst build`. This change ameliorates that situation by only retrieving
the provenance data when it's actually needed.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
cda03313
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_yaml.py: Reduce use of `isinstance()` in `node_sanitize()`
We know that nodes are typically one of:
str, list, dict, bool, tuple, NoneType or our ChainMap
Of these, dict and ChainMap are Mapping, only list is list
and the rest are returned unchanged. We can reduce/defer our use
of isinstance here, dramatically, improving performance.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b6f08e1b
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_yaml.py: Remove use of isinstance() in `{node,list}_{chain_,}copy`
A non-trivial proportion of time pre-scheduler in `bst build` is spent
copying (or chain-copying) nodes. Approximately a quarter of the time
spent in that effort is in `isinstance()`. This removes that CPU load.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
69ee11c6
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_yaml.py: Reduce cost of node_final_assertions
By re-using the isinstance replacements from earlier commits and
using a tuple of the string constants for checking for composition
markers, we reduce the cost of node_final_assertions by two thirds
in basic testing.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
0928e570
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_context.py: Cache result of get_strict()
While get_strict() doesn't look expensive per-se, it is called so many
times that it is valuable to cache the result once computed. Since I
don't think it can change once it is computable, cache it immediately
that becomes possible and we save 20s in my test case.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
022a59f0
by Benjamin Schubert
at 2019-02-12T16:14:05Z
Merge branch 'danielsilverstone-ct/further-optimisations' into 'master'
Further optimisations
See merge request BuildStream/buildstream!1131
-
352f4ad9
by Jonathan Maw
at 2019-02-12T16:44:35Z
loader.py: Stage junctions into .bst instead of a tmpdir
-
62396af9
by Jonathan Maw
at 2019-02-12T16:44:35Z
loader.py: Load junction from workspace if one's open
-
f1e9cb66
by Jonathan Maw
at 2019-02-12T16:44:35Z
Remove cleanup functionality from loader
Since junction check-outs are now persistent, there is no need to clean
up temporary directories once finished.
Now, junction checkouts are stored within the top-level project dir and
will be cleaned-up when that directory is removed.
This involves changes in:
* loader.py: Remove cleanup logic and passing-around of tmpdir.
* _project.py: Remove passing around tmpdir and calling loader's
cleanup.
-
e51116d5
by Jonathan Maw
at 2019-02-12T17:59:25Z
Merge branch 'jonathan/junction-no-tmpdir' into 'master'
Stage junctions into .bst instead of a tmpdir
Closes #895
See merge request BuildStream/buildstream!1134
-
478e5c47
by Valentin David
at 2019-02-12T18:06:50Z
buildstream/_cas/cascache.py: Set 0644 rights to pulled files
This was broken by 5ef19a0b31df84caed1e41719ef7ea5c6bd8a8bc.
-
6de65306
by Valentin David
at 2019-02-12T19:13:43Z
Merge branch 'valentindavid/pull-chmod-bug' into 'master'
buildstream/_cas/cascache.py: Set 0644 rights to pulled files
See merge request BuildStream/buildstream!1144
-
9b140fa0
by James Ennis
at 2019-02-13T09:35:45Z
element.py: Lift ArtifactCache.get_artifact_fullname() to here
This commit removes the method ArtifactCache.get_artifact_fullname()
and replaces it with Element.get_artifact_name()
Given a key, we are now able to construct the full name of any of an
element's artifacts.
-
039d7c1d
by James Ennis
at 2019-02-13T09:35:45Z
element.py: Make calculate_cache_key() API private
-
7cb6be74
by James Ennis
at 2019-02-13T09:35:45Z
element.py: Return early in __init_defaults if no plugin config
-
ddf642e1
by James Ennis
at 2019-02-13T09:35:45Z
metaelement.py: Provide constructor with default keyword arguments
-
55c15f8f
by James Ennis
at 2019-02-13T09:35:45Z
_artifactelement.py: New ArtifactElement object (derived from Element)
This object should be used when we want to handle artifact refs directly
from the command line.
An ArtifactElementError has also been added to _exceptions.py
-
6951cfc8
by James Ennis
at 2019-02-13T09:35:45Z
_project.py: Add create_artifact_element() method
-
bf1933b6
by James Ennis
at 2019-02-13T09:35:45Z
_stream.py: Add the _classify_artifacts() helper
-
ea6ff6da
by James Ennis
at 2019-02-13T09:35:45Z
_stream.py: Modify behaviour of _classify_artifacts()
* There is no need for this method to use a cas object.
* Search for artifact globs in the project's element path
* An artifact key is always 64 chars long
-
fcc79917
by James Ennis
at 2019-02-13T09:35:45Z
_stream.py: Allow loading to handle artifact refs
-
77345317
by James Ennis
at 2019-02-13T09:35:45Z
_artifactcache.py: Add get_artifacts_log() method
A CasBasedDirectory object of an artifacts logs can be obtained
with ArtifactCache.get_artifacts_log(). This ultimately calls
CASCache.get_top_level_dir() to obtain a CasBasedDirectory
of an artifact's subdirectory (or subdirectories).
-
775ac472
by James Ennis
at 2019-02-13T09:35:45Z
cli.py: Move artifact ref handling logic to stream
The loading of elements and the handling of artifacts does not belong
in this module. Such logic should be invoked using the Stream API
-
f95d6ee8
by James Ennis
at 2019-02-13T09:35:45Z
cli.py: Remove _classify_artifacts() method
This method has been moved to Stream, where is it used there exclusively.
-
491937d7
by James Ennis
at 2019-02-13T09:35:45Z
cascache.py: Allow CASCache.list_refs() to handle globs
This commit ensures that CASCache.list_refs(), and
ArtifactCache.list_artifacts(), can both handle glob expressions.
-
6ae41474
by James Ennis
at 2019-02-13T09:35:45Z
_stream.py: _classify_artifacts() should be able to handle globs
_classify_artifacts() no longer filters glob expressions by first
obtaining a list of ALL refs locally cached. We now only obtain refs
specified by the glob _expression_.
Furthermore, the Project.element_path is used to start searching for
globbed elements as opposed to the Project.directory.
-
8c6baf23
by James Ennis
at 2019-02-13T10:48:21Z
Merge branch 'jennis/refactor_artifact_log' into 'master'
Refactor artifact log command
See merge request BuildStream/buildstream!1101
-
14db3287
by Jürg Billeter
at 2019-02-13T10:48:45Z
projectconfig.yaml: Consistently include directories in split rules
Most split rules already included the relevant directories themselves in
addition to the directory contents. Add the missing bin, sbin, and
libexec directories.
This is required to fix tests with the following commit that changes
list_relative_paths() to return all directories.
-
c7625cdf
by Jürg Billeter
at 2019-02-13T10:48:45Z
tests/integration/project: Add tests directory to split rule
This is required to fix tests with the following commit that changes
list_relative_paths() to return all directories.
-
925429ea
by Jürg Billeter
at 2019-02-13T10:48:45Z
utils.py: Return all directories in list_relative_paths()
Returning only empty directories leads to inconsistencies when computing
a manifest by combining results from multiple list_relative_paths()
calls as done by the compose plugin.
I.e., the same directory may be empty in one dependency and non-empty in
another dependency. The merged file list will still contain that
directory even though it's no longer empty.
This inconsistency causes problems when calculating differences between
manifests. Returning all directories fixes these inconsistencies.
This is a change in API behavior.
-
353745a4
by Jürg Billeter
at 2019-02-13T10:48:45Z
_casbaseddirectory.py: Return all directories in list_relative_paths()
This matches the change in utils.list_relative_paths().
-
ae0ee361
by Jürg Billeter
at 2019-02-13T12:42:42Z
Merge branch 'juerg/list-all-directories' into 'master'
Return all directories in list_relative_paths()
See merge request BuildStream/buildstream!1139
-
118644b2
by Tom Pollard
at 2019-02-13T13:37:32Z
Add cli main & user conf option for 'cache-buildtrees' context
_context.py: Add cache_buildtrees global user context, the default
of which is set to by default to 'always' via the addition of
cache-buildtrees to userconfig.yaml cache group. 'failure' & 'never'
can be given as valid options.
app.py & cli.py: Add --cache-buildtrees as a bst main option, which
when passed with a valid option can override the default or user
defined context for cache_buildtrees.
tests/completions/completions.py: Update for the added flag.
-
b546bac1
by Tom Pollard
at 2019-02-13T13:37:32Z
_frontend/cli.py: Ensure failed buildtree warning is correct
not _cached_sucess() could resolve to true if the element wasn't
cached at all. switch to _cached_failure() to ensure condition
reflects expected artifact state
-
d2029444
by Tom Pollard
at 2019-02-13T13:37:32Z
Provide configuration for the optional creation of buildtrees
Artifacts can be cached explicitly with an empty `build tree` when
built via the cli main options or user config for all or only
successful build artifacts. Default behaviour is to still create
and cache all expected buildtrees.
element.py: _cache_artifact() Check if context for cache_buildtrees
has been set to always or failure with a corresponding build
result, if not skip attempting to export the build-root. Element
types without a build-root are cached with an empty buildtree
regardless. Update _stage_sources_at() to warn the user that the
buildtree import is empty.
tests/integration: Add test to artifact.py for the optional caching
of buildtree content from bst build. Rename build-tree.py to
shellbuildtrees.py to reflect included test cases, add test for
empty buildtree warning and failure option.
NEWS: Add entry for new option.
-
54ec032a
by Jürg Billeter
at 2019-02-13T14:50:15Z
Merge branch 'tpollard/896' into 'master'
Optional creation of buildtrees
Closes #896
See merge request BuildStream/buildstream!1135
-
ec4bbf35
by Benjamin Schubert
at 2019-02-13T14:59:47Z
Cleanup internal Loader cache after loading elements
-
b41a82d3
by Benjamin Schubert
at 2019-02-13T14:59:47Z
Don't register exceptions when not running the testsuite
This fix a problem with the garbage collector not being able to
clean the MetaElements that are loaded.
On small projects this is not a problem, but in bigger projects, this
can save a few hundred of MBs at runtime
The reason behind this is, whenever we have a "stack" element, which
has no stack.yaml configuration, since it doesn't need it, we would
get an exception thrown when initiating the first one, as loading the
yaml file would fail.
This would capture the frame in which this command was executed, which
references meta_elements. Therefore, as long as another exception is not
thrown, the garbage collector would not be able to clean all the
MetaElements.
-
9db7f489
by Benjamin Schubert
at 2019-02-13T16:13:16Z
Merge branch 'bschubert/cleanup-local-state' into 'master'
Cleanup MetaElement local state
See merge request BuildStream/buildstream!1147
-
c2c10d68
by Daniel Silverstone
at 2019-02-14T13:02:07Z
Cache Keys: Update to use JSON rather than pickle
This affects the cache key version (updated to 7) and introduces
a dependency on `ujson` which is BSD licenced as of the version
locked in `requirements.txt`
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
868c057e
by Tristan Van Berkom
at 2019-02-14T13:58:39Z
Merge branch 'danielsilverstone-ct/json-cache-key' into 'master'
Update cache keys to use JSON
See merge request BuildStream/buildstream!1151
-
ace15fa6
by Benjamin Schubert
at 2019-02-14T14:04:45Z
Don't keep MetaSource around in Source
We can easily rebiuld MetaSource from any source so there is no
reason to keep them around.
This will slightly improve memory usage.
-
174f5394
by Tristan Van Berkom
at 2019-02-14T15:04:27Z
Merge branch 'bschubert/dont-keep-metasource' into 'master'
Don't keep MetaSource around in Source
See merge request BuildStream/buildstream!1150
-
e7e4086e
by Jürg Billeter
at 2019-02-14T15:05:38Z
utils.py: Change _ensure_real_directory() to not resolve symlinks
Resolving symlinks during staging causes various issues:
* Split rules may not work properly as the resolved paths will differ
depending on whether another artifact with a directory symlink has
been staged in the same root directory or not, e.g., as part of
compose.
* The order of symlinks in file lists is difficult to get right to
guarantee consistent and predictable behavior as paths in a file list
might rely on symlinks in the same file list. See #647 and #817.
* Staging order differences can lead to surprising results. See #390.
* Difficult to properly support absolute symlinks. Absolute symlinks are
currently converted to relative symlinks, however, this doesn't always
work. See #606 and #830.
This will require changes in projects that rely on the current behavior.
However, the changes are expected to be small and are often a sign of
buggy element files. E.g., elements that don't fully obey `bindir` or
`sbindir` variables.
-
c0570871
by Jürg Billeter
at 2019-02-14T15:05:38Z
_casbaseddirectory.py: Do not resolve symlinks
This matches the change in utils._process_list().
This also removes the _Resolver class as it is now unused. We may want
to support controlled symlink resolution in the future, in which case
the _Resolver class can be resurrected from this commit.
-
55bccf63
by Jürg Billeter
at 2019-02-14T15:05:38Z
utils.py: Do not mangle absolute symlinks
Copy symlinks as they are, absolute or relative. We no longer resolve
symlinks when copying files, which makes this safe.
-
94c6d59b
by Jürg Billeter
at 2019-02-14T15:05:38Z
Bump artifact version for changes in symlink handling
-
949962f8
by Jürg Billeter
at 2019-02-14T15:05:38Z
NEWS: Add entry for change in symlink handling
-
56c07baa
by Jürg Billeter
at 2019-02-14T16:08:42Z
Merge branch 'juerg/symlinks2' into 'master'
Do not resolve or mangle symlinks during staging
See merge request BuildStream/buildstream!1140
-
69520011
by Chandan Singh
at 2019-02-15T06:58:27Z
contrib/bst-graph: Add script to print graph in DOT format
This script leverages the recently added format strings
(`%{build-deps}`, `%{runtime-deps}`) to `bst show` to print a graph in
DOT format. This requires users to have the `graphviz` python package
installed.
Additionally, users can also render the graph using the `--format`
option if they have the `graphviz` command line tool installed.
-
51dae747
by Chandan Singh
at 2019-02-15T08:50:21Z
Merge branch 'chandan/dot-graph' into 'master'
contrib/bst-graph: Add script to print graph in DOT format
Closes #705
See merge request BuildStream/buildstream!1148
-
0f78a47d
by Daniel Silverstone
at 2019-02-15T10:03:46Z
Variables: Rework how expansion strings work
Rather than constantly using regular expressions and retrieval from YAML
nodes, pre-parse expansion strings into a list representation cached
for reuse, and then expand them as simple string concatenation.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
14176e51
by Benjamin Schubert
at 2019-02-15T11:04:10Z
Merge branch 'danielsilverstone-ct/variables-rework' into 'master'
Variables: Rework how expansion strings work
See merge request BuildStream/buildstream!1152
-
39febfda
by Angelos Evripiotis
at 2019-02-15T11:10:51Z
storage.Directory.export_to_tar: mtime=_magic...
Change the default value of mtime when doing export_to_tar() from `0` to
`_utils._magic_timestamp`. This avoids problems in other software, which
assume that an mtime of `0` means the file does not exist.
There is more than one example where files with an mtime of zero are
treated as non-existant by other software. e.g. [ninja][1] and [Template
Toolkit][2].
The OSTree project also [express an intention][3] to move from an mtime
of 0 to an mtime of 1:
> For this reason, OSTree acts as though all timestamps are set to
> time_t 0, so that comparisons will be considered up-to-date. Note that
> for a few releases, OSTree used 1 to fix warnings such as GNU Tar
> emitting "implausibly old time stamp" with 0; however, until we have a
> mechanism to transition cleanly to 1, for compatibilty OSTree is
> reverted to use zero again.
>From the comments on export_to_tar(), the motivation for having an mtime
of 0 was to have reproducible results, rather than it specifically being
the value 0.
Additionally, the reproducible builds project has a [page on
archives][4]; it mentions the benefits of setting all the files to have
the same mtime, or clamping the mtime. It makes no mention of a
motivation for the mtime to be specifically 0.
Fixes #914
[1]: https://github.com/ninja-build/ninja/issues/1120
[2]: https://github.com/abw/Template2/blob/8d7d37200af436f1ad43628278d3caad257c8e27/lib/Template/Provider.pm#L635
[3]: https://ostree.readthedocs.io/en/latest/manual/repo/
[4]: https://reproducible-builds.org/docs/archives/
-
8b34e356
by Angelos Evripiotis
at 2019-02-15T12:25:21Z
Merge branch 'aevri/mtime1' into 'master'
storage.Directory.export_to_tar: default mtime=utils._magic_timestamp
Closes #914
See merge request BuildStream/buildstream!1149
-
3bbb90e7
by Benjamin Schubert
at 2019-02-15T12:34:36Z
Use sets when checking for existence of an element
-
0921ccf4
by Benjamin Schubert
at 2019-02-15T13:39:52Z
Merge branch 'bschubert/set-as-set' into 'master'
Use sets when checking for existence of an element
See merge request BuildStream/buildstream!1154
-
c24f2971
by Angelos Evripiotis
at 2019-02-15T13:56:48Z
userconfig: rm really-workspace-close-project-inaccessible
Remove the need for the 'really-workspace-close-project-inaccessible'
config option, as well as the option itself.
As agreed on the mailing list [1], all the 'are you sure?' prompts on
workspace reset and close were removed. While that discussion was going
on, this new prompt and option was added. At the 2019 BuildStream
Gathering, it was verbally agreed between myself and Tristan VB that we
would also remove this instance.
It was also agreed that we should have a notice to let the user know
what they'd done, this was already in place if interactive. Moved it to
be unconditional so that there's no difference in non-interactive
behaviour. Made it output to stderr, as it's diagnostic meant for the
user. Made it the last thing echo'd so it's next to the prompt - it's
very relevant to what they type next. Added a test to make sure the text
makes it to stderr in the appropriate case, and not in an inappropriate
one.
This is the last instance of any prompt configuration, so BuildStream
can also forget all of that machinery.
[1] https://mail.gnome.org/archives/buildstream-list/2018-December/msg00111.html
-
91eb29a6
by Angelos Evripiotis
at 2019-02-15T14:55:06Z
Merge branch 'are_you_sure2' into 'master'
userconfig: rm really-workspace-close-project-inaccessible
Closes #726 and #744
See merge request BuildStream/buildstream!1130
-
b8561fb1
by Chandan Singh
at 2019-02-15T14:58:22Z
contrib/bst-here: Change default image to buildstream/buildstream
See https://gitlab.com/BuildStream/buildstream-docker-images/issues/26
for detailed discussion around this.
`buildstream/buildstream-fedora` is now considered deprecated. Switch to
`buildstream/buildstream` image. This image also offers more tags that
will provide users more flexibility.
-
2699c77c
by Chandan Singh
at 2019-02-15T14:58:31Z
contrib/bst-here: Allow users to specify image variant
Now that the `buildstream/buildstream` image has 9 variants, let's make
it easier to choose the desired tag, using a command-line option.
This is otherwise possible by specifying the full image name `image:tag`
using the `-i` option. But, this will make it easier to specify just the
tag using `-j`.
The following two invocations of `bst-here` are now equivalent:
bst-here -i buildstream/buildstream:dev
bst-here -j dev
-
677fc6c5
by Chandan Singh
at 2019-02-15T16:09:21Z
Merge branch 'chandan/bst-here-tags' into 'master'
contrib/bst-here: Allow users to specify image variant
See merge request BuildStream/buildstream!1153
-
3b889eb8
by Javier Jardón
at 2019-02-15T20:22:08Z
README.rst: Add table with distros with packaged buildstream
[ci skip]