-
d55b9e39
by Phil Dawson
at 2018-12-12T13:55:19Z
Add --tar option to source-checkout command
This commit is part of the work towards #672
-
733aab53
by Phil Dawson
at 2018-12-12T13:55:19Z
Add --force / -f option to source-checkout command
-
c2efeba0
by Phil Dawson
at 2018-12-12T14:43:40Z
Add option to source-checkout command to generate build scripts
-
85c61894
by Phil Dawson
at 2018-12-12T14:43:40Z
Remove source bundle command
This is part of the work towards #672
-
a322d5c0
by Phil Dawson
at 2018-12-12T14:43:40Z
NEWS: Add entry about the removal of source-bundle command
-
ec909605
by Phil Dawson
at 2018-12-12T15:45:42Z
Merge branch 'phil/source-checkout-options' into 'master'
Retire bst source bundle command
Closes #672
See merge request BuildStream/buildstream!959
-
3697a611
by Richard Maw
at 2018-12-12T16:31:38Z
cli: Add support for auto-completing artifact ref names
-
b3dceb16
by Richard Maw
at 2018-12-12T16:32:41Z
cli: Add artifact command group
-
ba08a0cd
by Richard Maw
at 2018-12-12T16:32:41Z
cli: Add artifact log command
-
70fb9554
by Richard Maw
at 2018-12-12T16:32:41Z
tests: Add tests for artifact group commands
-
f773e746
by Richard Maw
at 2018-12-12T16:33:02Z
NEWS: Mention `bst artifact log`
-
b6528441
by richardmaw-codethink
at 2018-12-12T18:00:59Z
Merge branch 'richardmaw/artifact-log' into 'master'
Add artifact log command
See merge request BuildStream/buildstream!920
-
d03bf316
by Benjamin Schubert
at 2018-12-13T10:24:11Z
Mock storage space checks for tests.
Fix #530
- Extract free space computation in a function for easier
mocking
- Mock space computation during cache quota tests
- Mock cache size during cachque quota tests
- Add two more tests when the configuration would require
to much storage space
-
a116f576
by Tristan Van Berkom
at 2018-12-13T10:58:46Z
Merge branch 'BenjaminSchubert/fix-quota-tests' into 'master'
Mock storage space checks for tests.
Closes #530
See merge request BuildStream/buildstream!702
-
180fa774
by Benjamin Schubert
at 2018-12-13T12:05:15Z
element.py: fix unbounded variable in nested python exception
As per https://docs.python.org/3.7/reference/compound_stmts.html#except
variables used in the "except" clause are erased at the end of the
block.
We had a conflict with a nested exception using the same variable as
before.
This renames the nested variable to avoid the clash
-
224aa4c2
by Benjamin Schubert
at 2018-12-13T12:34:41Z
Merge branch 'bschubert/fix-unbound-variable-exception' into 'master'
element.py: fix unbounded variable in nested python exception
See merge request BuildStream/buildstream!1006
-
053beb66
by Tristan Van Berkom
at 2018-12-13T14:23:19Z
manual plugin: Support virtual directories
-
29ab271c
by Tristan Van Berkom
at 2018-12-13T14:23:19Z
makemaker plugin: Support virtual directories
-
ba955cf0
by Tristan Van Berkom
at 2018-12-13T14:23:19Z
pip element plugin: Support virtual directories
-
6010b5a4
by Tristan Van Berkom
at 2018-12-13T14:23:19Z
modulebuild plugin: Support virtual directories
-
3a6d27a4
by Tristan Van Berkom
at 2018-12-13T14:23:19Z
distutils plugin: Support virtual directories
-
4c0e602c
by Tristan Van Berkom
at 2018-12-13T14:23:19Z
BuildElement: Don't enable batching of prepare and assemble by default
Some external plugins depend derive from BuildElement and are broken
by BuildElement enabling this batching by default.
Instead, enable it in all of the individual build element plugin
implementations.
This fixes issue #800
-
60ddeeb9
by Tristan Van Berkom
at 2018-12-13T14:58:28Z
Merge branch 'tristan/dont-batch-prepare-assemble-by-default' into 'master'
Dont batch prepare assemble by default
Closes #800
See merge request BuildStream/buildstream!1009
-
bf72cc42
by Angelos Evripiotis
at 2018-12-13T17:31:17Z
contributing: mandate end-to-end tests, allow others
Update the section on adding tests, to fulfill these broad points:
- Mandate end-to-end testing, with rationale.
- Mention internal APIs as endpoints for testing.
- Warn against pitfalls of unit-testing.
It's more text than I would have liked, perhaps in later work we'll be
able to say it with less.
This change integrates feedback from the mailing list
'Guidance on Unit Tests' thread:
https://mail.gnome.org/archives/buildstream-list/2018-November/msg00045.html
-
13eb7ed2
by Angelos Evripiotis
at 2018-12-13T18:01:56Z
Merge branch 'aevri/contributing_e2e_tests' into 'master'
contributing: more clarity on testing
See merge request BuildStream/buildstream!973
-
629a6e52
by Chandan Singh
at 2018-12-14T19:34:20Z
Introduce new "source" command group
Following the message thread
https://mail.gnome.org/archives/buildstream-list/2018-November/msg00106.html,
implement a new command group called `source`. Move existing `track`,
`fetch`, and the recently added `source-checkout` commands under this
group.
For `track` and `fetch`, this is a BREAKING change, as the old commands
have been marked as obsolete. Using them will result in an error message
that refers people to use the new versions, like `bst source fetch`
instead of old `bst fetch`. `source-checkout` will now become
`source checkout` (the dash has turned into a space), and is not a
breaking change as it was added in the current development cycle.
Note that the functionality to hide commands from help output was added
only recently in Click, so the minimum version of Click that we now
require is 7.0.
Summary of changes:
* _frontend/cli.py: Add `source` command group, mark previous versions
as obsolete and hide them from the help output.
* _frontend/complete.py: Fix completion for hidden commands.
* setup.py: Bump Click minimum version to 7.0.
* tests: Update to cope with the new command names.
Fixes #814.
-
f894c0a8
by Chandan Singh
at 2018-12-14T19:34:20Z
NEWS: Add entry for new source command group
Also, change the existing entry about `source-checkout` to be
`source checkout` instead. And, while we are there, move it near the
other announcements about the source command group.
-
b23bec55
by Chandan Singh
at 2018-12-14T20:07:13Z
Merge branch 'chandan/source-subgroup' into 'master'
Introduce new "source" command group
Closes #814
See merge request BuildStream/buildstream!1003
-
d2105909
by Raoul Hidalgo Charman
at 2018-12-18T11:13:56Z
Add remote execution instance option
This is used when sending execution requests, to specify which instance of the
execution server to use.
Partial fix for #627.
-
a3bbec23
by Jim MacArthur
at 2018-12-18T11:13:56Z
sandboxremote: Add server/storage config with defaults
Adds instance name support for the remote execution storage service.
-
89219f61
by Jim MacArthur
at 2018-12-18T11:13:56Z
_cascache.py: Add instance names to GRPC calls
Adds the 'instance_name' parameter, which may be None, to most GRPC
calls in the CASCache object. ByteStream requests already have
instance_name supplied in the resource name, so do not need the
parameter.
Closes #627.
-
3dc20963
by Jim MacArthur
at 2018-12-18T11:13:56Z
Documentation: Include instance-name in remote execution documentation
-
644d8b28
by Jim MacArthur
at 2018-12-18T11:45:30Z
Merge branch 'raoul/627-RE-instance-config' into 'master'
Remote-execution instance configuration support
Closes #627
See merge request BuildStream/buildstream!952
-
e29aea36
by William Salmon
at 2018-12-19T13:23:19Z
Basic options for shell --build to use buildtrees
Fixes issue #740
-
898a23a5
by Will Salmon
at 2018-12-19T14:35:41Z
Merge branch 'willsalmon/shellBuildTrees' into 'master'
Shell --build has optional buildtrees
Closes #740
See merge request BuildStream/buildstream!986
-
a2f1d879
by Javier Jardón
at 2018-12-19T15:36:11Z
README.rst: Add license badge
-
aae5e4b3
by Javier Jardón
at 2018-12-19T16:09:48Z
Merge branch 'jjardon/license_badge' into 'master'
README.rst: Add license badge
See merge request BuildStream/buildstream!1014
-
2b767fe8
by Jürg Billeter
at 2018-12-20T10:06:11Z
Move fetch logic from FetchQueue to Element
The queue shouldn't need to know about individual sources. This is in
line with _track() and _get_consistency().
-
7a102144
by Jürg Billeter
at 2018-12-20T10:06:11Z
element.py: Do not call fetch() for cached sources
-
b325989e
by Jürg Billeter
at 2018-12-20T10:07:20Z
tests/sources: Test that fetch() is not called for cached sources
-
77d8ad45
by Jürg Billeter
at 2018-12-20T10:42:39Z
Merge branch 'juerg/fetch' into 'master'
Do not call fetch() for cached sources
See merge request BuildStream/buildstream!992
-
9835b7f1
by Chandan Singh
at 2018-12-20T12:50:02Z
Expose base class for Git source plugins
Add a `_GitSourceBase` class to act as the base class for deriving
source plugins that work with Git. The first user of this base class is
the `git` source plugin that's in core at the moment.
The contents of this base class is almost identical to the existing `GitSource`
class. The two notable differences are:
- the private methods now have leading underscores, as some were missing
it originally
- the name of the class
Note that we are exposing a private member here as we expect it to move to a
separate package soon. See the following discussion for more details:
https://gitlab.com/BuildStream/buildstream/issues/739#note_124819869
Fixes #739.
-
14da6955
by Chandan Singh
at 2018-12-20T13:34:10Z
Merge branch 'chandan/abstract-git-source' into 'master'
Expose base class for Git source plugins
Closes #739
See merge request BuildStream/buildstream!1019
-
7368f569
by Angelos Evripiotis
at 2018-12-20T13:59:23Z
_project::_find_project_dir: fix error message
Now that we are also looking for WORKSPACE_PROJECT_FILE, we should add
it to the error message. While we're there, also mention the directory
we are looking in - this could help scripters be clear about what went
wrong.
The new error message looks like this:
Error loading project: None of ['project.conf', '.bstproject.yaml']
found in '/src/temp/blah' or any of its parent directories
-
46efc91d
by Angelos Evripiotis
at 2018-12-20T13:59:23Z
BREAK: remove auto-init behaviour
In the event that the project could not be found, stop BuildStream from
asking if the user would like to create a new project. Exit with error
instead, and give a hint to the user in case they're new.
As proposed on the mailing list here:
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00082.html
The new interaction looks like this:
$ bst show nonsuch.bst
No project found. You can create a new project like so:
bst init
Error loading project: None of ['project.conf', '.bstproject.yaml']
found in '/src/temp/blah' or any of its parent directories
Fixes #826
-
e0c575c4
by Angelos Evripiotis
at 2018-12-20T14:37:38Z
Merge branch 'aevri/rm-autoinit' into 'master'
BREAK: remove auto-init behaviour
Closes #826
See merge request BuildStream/buildstream!1015
-
c05d8b4f
by Chandan Singh
at 2018-12-20T15:53:12Z
tests/frontend/logging.py: Fix regex Deprecation Warning
Use raw strings for regex searches, which is the preferred way to do
regular expressions in Python.
Without this patch, currently we get the following warnings:
```
tests/frontend/logging.py:44
/builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:44: DeprecationWarning: invalid escape sequence \[
m = re.search("\[\d\d:\d\d:\d\d\]\[\]\[\] SUCCESS Checking sources", result.stderr)
tests/frontend/logging.py:80
/builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:80: DeprecationWarning: invalid escape sequence \d
m = re.search("\d\d:\d\d:\d\d,\d\d:\d\d:\d\d.\d{6},\d\d:\d\d:\d\d,,,SUCCESS,Checking sources", result.stderr)
```
-
e8055a56
by Chandan Singh
at 2018-12-20T15:53:12Z
buildstream/utils.py: Fix regex Deprecation Warning
Specify flags at the start of the _expression_ as per the recommendation
of the standard library.
Without this patch, we currently get the following warning:
```
tests/examples/junctions.py::test_open_cross_junction_workspace
/builds/BuildStream/buildstream/dist/buildstream/buildstream/utils.py:213: DeprecationWarning: Flags not at the start of the _expression_ '\\/[^/]*\\Z(?ms)'
regexer = re.compile(_expression_)
```
-
cd4889af
by Chandan Singh
at 2018-12-20T16:34:08Z
Merge branch 'chandan/fix-warning' into 'master'
Fix Deprecation warnings from regex module
See merge request BuildStream/buildstream!1010
-
ac995236
by Tom Pollard
at 2018-12-20T17:18:02Z
_stream.py: fix _buildtree_pull_required logic
-
c3153dea
by Tom Pollard
at 2018-12-21T10:10:58Z
Merge branch 'tpollard/fixbuildtreereq' into 'master'
_stream.py: fix _buildtree_pull_required logic
See merge request BuildStream/buildstream!1021
-
caf49669
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
dev-requirements.txt: Require at least pytest 3.9
Recently we have been adding some tests which use the new tmp_path
fixture from pytest, this is new in 3.9.
-
425781db
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
_signals.py: Fixing new (simplifiable-if-_expression_) linter errors
-
1f8cc996
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
element.py: Fixing new (unnecessary-pass) and (simplifiable-if-_expression_) linter errors
-
89632150
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
source.py: Fixing new (unnecessary-pass) linter error
-
26d135ba
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
_artifactcache/casserver.py: Fixed new (wrong-import-order) linter error
-
d0988f80
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
_frontend/cli.py: Fixing new (unnecessary-pass) linter errors
-
4d54ab78
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
_frontend/complete.py: Fixing new (simplifiable-if-_expression_) linter error
-
ee0d5bf6
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
_frontend/widget.py: Fixing new (wrong-import-order) linter error
-
36295282
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
plugins/elements/import.py: Fixing new (duplicate-string-formatting-argument) linter error
-
4ffe03c6
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
buildstream/storage/_casbaseddirectory.py: Fixing new (unnecessary-pass) linter errors
-
80c137d5
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
dev-requirements.txt: Require new version of pytest-datafiles
Since Thomas Nixon fixed the datafiles plugin to be compatible
with pytest >= 3.8.0, we can now depend on it and avoid the
warnings and errors which the older datafiles plugin causes.
This fixes issue #636
-
61906a14
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
dev-requirements.txt: Remove the restriction on pylint.
After having fixed the more recent linting errors, the
tests should pass with recent versions of pylint. Tested
with pylint 2.2.2.
-
94b9948c
by Tristan Van Berkom
at 2018-12-26T22:35:29Z
Merge branch 'tristan/test-fixes' into 'master'
Misc fixes to the tests and newer linter errors
Closes #636
See merge request BuildStream/buildstream!1011
-
fe4fc0b5
by Tristan Van Berkom
at 2018-12-26T23:19:07Z
_downloadablefilesource.py: Avoid crashes when HOME is unset
The python netrc module will raise OSError in the case
that HOME is not set, this was discovered while running
tests under tox.
-
c61e1793
by Tristan Van Berkom
at 2018-12-26T23:19:07Z
tests/testutils/runcli.py: Fixed broken environment handling
Treat None values in the passed dictionary as keys to be removed
from the environment, this was already happening at restoration time.
-
50c1bb7f
by Tristan Van Berkom
at 2018-12-26T23:19:07Z
tests/sources/tar.py: Test that we don't crash when HOME is unset
This happens when the netrc module is searching for a ~/.netrc
file and it doesnt find any HOME set.
-
32a101f6
by Tristan Van Berkom
at 2018-12-26T23:53:29Z
Merge branch 'tristan/fix-netrc-crasher' into 'master'
Fix netrc crasher
See merge request BuildStream/buildstream!1025
-
a080b6c7
by Tristan Van Berkom
at 2018-12-31T19:03:25Z
tests/loader/junctions.py: Use Result checking APIs instead of manually
Better to use the same API everywhere for thecking CLI results.
-
d7c8b54d
by Tristan Van Berkom
at 2018-12-31T19:14:16Z
tests/loader/junctions.py: Test the error we expect when a cross junction element is not found
-
560a6342
by Tristan Van Berkom
at 2018-12-31T19:49:24Z
Merge branch 'tristan/junction-tests' into 'master'
Junction test addition
See merge request BuildStream/buildstream!1029
-
8a020a5a
by Chandan Singh
at 2018-12-31T19:54:25Z
_loader/loader.py: Refactor warnings about element names in one method
Currently we some duplication in the way we check for invalid filenames.
To make it more robust and allow room for adding more warnings, refactor
it into a separate method `_warn_invalid_elements()` that handles just
this.
-
87568074
by Chandan Singh
at 2018-12-31T19:54:25Z
_loader/loader.py: Add warnings about invalid characters in filename
See https://mail.gnome.org/archives/buildstream-list/2018-December/msg00061.html
for some related discussion.
-
4b2992c1
by Chandan Singh
at 2018-12-31T19:54:25Z
tests/buildcheckout.py: Add tests for invalid chars in filename
-
ce9e9c2d
by Chandan Singh
at 2018-12-31T19:54:25Z
doc: Add note about valid element names
-
b848172c
by Tristan Van Berkom
at 2018-12-31T20:29:51Z
Merge branch 'chandan/element-filename-requirements' into 'master'
Add warnings about invalid characters in filename
See merge request BuildStream/buildstream!1028
-
bb9df2a1
by Chandan Singh
at 2019-01-02T11:59:35Z
doc/Makefile: Update comment about sphinx entrypoint
https://github.com/sphinx-doc/sphinx/issues/4375 has been closed as
`wontfix`. Update the comment in doc/Makefile that mentions it, based on
the discussion on the issue.
-
43eaf6ee
by Chandan Singh
at 2019-01-02T13:42:10Z
Merge branch 'chandan/update-doc-makefile-note' into 'master'
doc/Makefile: Update comment about sphinx entrypoint
See merge request BuildStream/buildstream!1026
-
3b5c8a28
by Tristan Maat
at 2019-01-02T15:57:13Z
.gitlab-ci.yml: Use new testsuite images with pycodestyle 1.4.0
We need to bump the testsuite image versions to use the new
pycodestyle update that allows excluding file paths, so that we can
exclude generated python code.
-
bd60e8a9
by Javier Jardón
at 2019-01-02T16:59:41Z
Use pycodestyle instead of pep8
pep8 package was renamed to pycodestyle to reduce confusion.
Disabled warnings:
W504 - Line break after binary operator (seems people like this)
W605 - Invalid escape sequence (some of our regexes use these)
-
32c47d1c
by Tristan Maat
at 2019-01-02T17:32:09Z
Merge branch 'jjardon/pycodestyle' into 'master'
Use pycodestyle instead pep8 python module
See merge request BuildStream/buildstream!638
-
1cbd9a73
by Chandan Singh
at 2019-01-02T19:39:58Z
Add requirements files for install, test and plugin dependencies
Add `.in` and `.txt` requirements files for BuildStream's pure python
dependencies. For each pair, the `.in` file is supposed to capture the
loose version requirements, and the corresponding `.txt` file is
supposed to have frozen requirements. We have 3 such sets:
* `requirements`: BuildStream's runtime dependencies
* `dev-requirements`: Dependencies for running tests
* `plugin-requirements`: Dependencies for core plugins
Note that the frozen requirements files will only be used for testing
purposes, and `setup.py` will continue to read loose requirements.
-
6b0cb5f3
by Chandan Singh
at 2019-01-02T19:41:21Z
Add tox.ini to enable running tests using tox
Add `tox.ini` file that will enable us to use
[tox](https://tox.readthedocs.io/) as a frontend for running tests.
Since we share the config via `setup.cfg` and requirements via
requirements files, commands like `python3 setup.py test` will continue
to work.
-
9d2d1d4f
by Chandan Singh
at 2019-01-03T03:31:52Z
.gitlab-ci.yml: Run tests using tox
Instead of invoking tests throung `setup.py`, use `tox` as a frontend in
the CI pipelines.
-
8ae04283
by Tristan Van Berkom
at 2019-01-03T03:31:52Z
CONTRIBUTING.rst: Updated to reflect running tests using tox.
-
e7e2a5c1
by Chandan Singh
at 2019-01-03T03:31:52Z
CONTRIBUTING.rst: Add steps for installing non-python build dependencies
Since we don't allow use of site packages by default in our `tox`
configuration, people will need to install non-python build dependencies
of some of our dependencies that do not provide pre-built wheels. We
have two such packages at the moment:
* psuitl: requires python C headers, compiler
* pygobject: requires python C headers, pkg-config, compiler, cairo
headers, and gobject libraries
-
afa0a369
by Chandan Singh
at 2019-01-03T03:31:52Z
Move all requirements files into "tools" directory
These new `.in` and `.txt` are making the repository look very
cluttered. Move them to a separate `tools` directory to make it look a
bit cleaner.
-
3fae3446
by Chandan Singh
at 2019-01-03T03:44:17Z
Move sphinx build functionality to tox
Currently the CI and the docs both have to duplicate the same inforation
about how to gather dependencies etc, and have to use hacky ways to run
them.
Add a new `docs` environment to our tox setup so that building docs is
as simple as running `tox -e docs`.
-
1f7bc655
by Chandan Singh
at 2019-01-03T03:44:17Z
CONTRIBUTING.rst: Add instructions to recreate tox environment
When we update our dependencies, developers will eventually need to
recreate their `tox` environments to get correct results. This happens
because `tox` isn't particularly good at recognizing changes in the
requirements files.
-
669b55b0
by Tristan Van Berkom
at 2019-01-03T15:47:27Z
Merge branch 'chandan/toxify' into 'master'
Use tox as a frontend for running tests
See merge request BuildStream/buildstream!1027
-
ca547f19
by Chandan Singh
at 2019-01-03T19:34:49Z
tests/frontend/buildcheckout.py: Fix bad filename issue for Windows
In BuildStream/buildstream!1028, we added a test specifically to test
that BuildStream correctly raises an warning when the name of an element
contains characters that are invalid on Windows. Unfortunately, we
didn't see it coming that it would make it impossible to checkout this
branch on Windows.
Fix it by generating this file, only if we are not running on Windows.
* tests/testutils/site.py: Add `IS_WINDOWS` check
* tests/frontend/buildcheckout.py: Generate file with invalid filename
on the fly
* Remove tests/frontend/project/elements/invalid-chars.
Fixes #842.
Note that this may still cause issues on WSL when running tests on a
shared filesystem, but that seems to be a generic issue on WSL with
`os.rename`.
-
01c3761c
by Chandan Singh
at 2019-01-03T20:03:32Z
Merge branch 'chandan/fix-bad-filename-mini-disaster' into 'master'
tests/frontend/buildcheckout.py: Fix bad filename issue for Windows
Closes #842
See merge request BuildStream/buildstream!1032
-
7850e4e2
by Tristan Van Berkom
at 2019-01-03T20:55:00Z
setup.cfg: Specify the pycodestyle configurations separately from pytest configuration
-
3c9d592d
by Tristan Van Berkom
at 2019-01-03T20:55:00Z
setup.cfg: Don't lint by default through setup.py test invocations
-
feba48e7
by Tristan Van Berkom
at 2019-01-03T20:55:00Z
tox.ini: Added new 'lint' environment to lint separately
-
9caa0ea0
by Tristan Van Berkom
at 2019-01-03T20:55:00Z
.gitlab-ci.yml: Update the CI to run the linter separately through tox
This should save us some cycles in CI, and also allow the
developer to more conveniently lint separately from testing.
-
91c8b634
by Tristan Van Berkom
at 2019-01-03T21:04:49Z
CONTRIBUTING.rst: Adding instructions to run the linter separately
-
892bf99e
by Tristan Van Berkom
at 2019-01-03T21:31:04Z
tools/requirements.{in,txt}: Fix warning about duplicate jinja
Apparently you need to spell Jinja2 with a capital J, otherwise
we find it listed twice in the resulting requirements.txt files.
-
7c681c42
by Tristan Van Berkom
at 2019-01-03T21:32:40Z
tools/dev-requirements.{in,txt}: Removed requirements on pylint/codestyle pytest plugins
We no longer run the linters through pytest, so we don't need these
plugins anymore.
-
d6689ed2
by Tristan Van Berkom
at 2019-01-03T22:09:02Z
Merge branch 'tristan/toxic-linting' into 'master'
Run the linter separately from the tests
See merge request BuildStream/buildstream!1033
-
e15278d7
by Thomas Coldrick
at 2019-01-03T22:14:01Z
Use collections.abc for Mapping, Iterable
In _yaml.py and _frontend/complete.py we were getting pylint warnings
for using collections.Mapping and collections.Iterable, which are
abstract classes now provided from collections.abc. This patch just
uses the classes from the right place.
-
a85da591
by Tristan Van Berkom
at 2019-01-03T22:36:16Z
Merge branch 'coldtom/collections' into 'master'
Use collections.abc for Mapping, Iterable
Closes #831
See merge request BuildStream/buildstream!1020
-
cf0e7d26
by Jürg Billeter
at 2019-01-04T07:07:59Z
_frontend/cli.py: Reinstate support for guessing targets
Moving fetch and track to the source command group accidentally dropped
the support for guessing targets for these commands when invoked from a
workspace directory. This brings it back.
-
0751fc8f
by Jürg Billeter
at 2019-01-04T11:27:58Z
Merge branch 'juerg/source-guess-target' into 'master'
_frontend/cli.py: Reinstate support for guessing targets
See merge request BuildStream/buildstream!1036
-
ebd965fe
by Chandan Singh
at 2019-01-04T15:21:35Z
Rename "tools" directory to "requirements"
This seems like a better name for the directory, as it more closely
describes the purpose of its contents.
-
bb712886
by Chandan Singh
at 2019-01-04T15:21:35Z
Add Makefile to update requirements files
-
b9792837
by Chandan Singh
at 2019-01-04T15:21:35Z
CONTRIBUTING.rst: Add instructions to update requirements files
Split the "The MANIFEST.in and setup.py" section in two:
"Managing data files" and "Updating BuildStream's Python dependencies".
Briefly explain the layout of `requirements` directory and add
instructions to use the Makefile added in the last commit.
-
45692335
by Tristan Van Berkom
at 2019-01-04T15:50:14Z
Merge branch 'chandan/update-requirements-one-liner' into 'master'
Add Makefile to update requirements files
See merge request BuildStream/buildstream!1035
-
f42dcb54
by Chandan Singh
at 2019-01-04T17:16:43Z
.gitlab-ci.yml: Remove prepare stage
As we now run tests using `tox`, we don't need to worry about manually
packing and unpacking BuildStream. So, we can remove the preapre stage
entirely.
Update `coverage` and nightly jobs to appropriately cope with this
change. Both these jobs now install all runtime dependencies from
requirements files.
-
ee29f207
by Chandan Singh
at 2019-01-04T17:41:02Z
Merge branch 'chandan/no-sdist' into 'master'
.gitlab-ci.yml: Remove prepare stage
See merge request BuildStream/buildstream!1037
-
f122cfe8
by Jürg Billeter
at 2019-01-06T17:57:33Z
_frontend/app.py: Use buildtree for interactive shell on build failure
Fixes: e29aea36 ("Basic options for shell --build to use buildtrees")
-
7dfb85b3
by Tristan Van Berkom
at 2019-01-06T18:31:25Z
Merge branch 'juerg/shell-buildtree' into 'master'
_frontend/app.py: Use buildtree for interactive shell on build failure
See merge request BuildStream/buildstream!1039
-
891fcb0e
by Tristan Van Berkom
at 2019-01-07T16:47:01Z
Fix stack traces discovered with ^C forceful termination.
* utils.py:_kill_process_tree(): Ignore NoSuchProcess errors
These are caused because we issue SIGTERM, and if the process
has not exited after a timeout, we kill it.
* _scheduler/jobs/job.py: Stop handling NoSuchProcess errors here
redundantly, they are already ignored.
It seems that we were ignoring it after sleeping when terminating
tasks from the scheduler... but we were not ignoring it when performing
the same pattern in the `Plugin.call()` -> `utils._call()` path, so
we were still getting these exceptions at termination time from host
tool processes launched by source plugins.
-
5de42d43
by Tristan Van Berkom
at 2019-01-07T18:00:37Z
Merge branch 'tristan/keyboard-interrupt-stack-trace' into 'master'
Fix stack traces discovered with ^C forceful termination.
See merge request BuildStream/buildstream!1043
-
059035b9
by Tristan Van Berkom
at 2019-01-07T18:02:00Z
_scheduler/scheduler.py: Make _schedule_jobs() private
This is not used anywhere outside of the Scheduler, currently
only the Scheduler itself is allowed to queue a job at this level.
If the highlevel business logic for automatic queueing of auxiliary
jobs moves to another location, we can make this public again.
-
b83d1b1f
by Tristan Van Berkom
at 2019-01-07T18:02:00Z
_scheduler/scheduler.py: Only run one cache size job at a time
When queuing the special cache management related cleanup and
cache size jobs, we now treat these jobs as special and do the
following:
* Avoid queueing a cleanup/cache_size job if one is already queued
We just drop redundantly queued jobs here.
* Ensure that jobs of this type only run one at a time
This could have been done with the Resources mechanics,
however as these special jobs have the same properties and
are basically owned by the Scheduler, it seemed more straight
forward to handle the behaviors of these special jobs together.
This fixes issue #753
-
16a8816f
by Tristan Van Berkom
at 2019-01-07T18:02:00Z
Scheduler: Introduced JobStatus instead of simple success boolean
This changes the deepest callback from when a Job completes to
propagate a JobStatus value instead of a simple boolean, and updates
all of the effected code paths which used to receive a boolean
to now handle the JobStatus values.
This further improves the situation for issue #753, as now we avoid
queueing cache size jobs for pull jobs which are skipped.
-
c2fc2a5e
by Tristan Van Berkom
at 2019-01-07T18:02:00Z
_scheduler/jobs/job.py: Removed 'skipped' property
This is redundant now that we report it through the JobStatus.
-
3e3984ad
by Tristan Van Berkom
at 2019-01-07T18:50:23Z
Merge branch 'tristan/one-cache-size-job' into 'master'
Only queue one cache size job
Closes #753
See merge request BuildStream/buildstream!1040
-
512c726e
by Tristan Van Berkom
at 2019-01-08T03:38:11Z
sandbox/sandbox.py: Fix regression of command logging
Since we added batch commands, the batch commands print the
text of the commands directly in the message text, but this is wrong.
The detail string is the appropriate place for text of unknown lengths
(the user can actually configure how many max lines of commands they
want to see in their log), the message text itself should be controlled
and brief enough to avoid text wrapping.
-
01171988
by Tristan Van Berkom
at 2019-01-08T04:20:14Z
Merge branch 'tristan/fix-command-status-messages' into 'master'
sandbox/sandbox.py: Fix regression of command logging
See merge request BuildStream/buildstream!1044
-
6c1d06d6
by Phil Dawson
at 2019-01-08T10:24:32Z
element.py: remove reference to source bundle command
This command has been replacved by the bst source checkout command
-
914ecb72
by Jürg Billeter
at 2019-01-08T10:54:02Z
Merge branch 'phil/remove-source-bundle-reference' into 'master'
element.py: remove documentation reference to source bundle command
See merge request BuildStream/buildstream!1041
-
81f7a964
by Tristan Van Berkom
at 2019-01-08T16:55:58Z
sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.
When code is faulty, in a plugin or in the core, we should
get a stack trace and a BUG message.
-
edcc43ed
by Tristan Van Berkom
at 2019-01-08T19:19:10Z
Merge branch 'tristan/sandbox-programming-error' into 'master'
sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.
See merge request BuildStream/buildstream!1046
-
17c5ca2d
by Valentin David
at 2019-01-09T10:45:23Z
Use relative path to project directory for remote execution certificates/keys
-
605f8d11
by Valentin David
at 2019-01-09T10:45:23Z
Add support for user remote execution configuration
Fixes #631.
-
65ea03ab
by Valentin David
at 2019-01-09T10:45:23Z
Add support for https channel to remote execution and actions servers
Fixes #780.
-
ba05c4fa
by Valentin David
at 2019-01-09T12:58:59Z
Merge branch 'valentindavid/remote_execution_configuration' into 'master'
Remote execution configuration: HTTPS and user configuration
Closes #780 and #631
See merge request BuildStream/buildstream!1030
-
eb0dbcfc
by Valentin David
at 2019-01-09T13:56:58Z
buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY
ScriptElement does not use Sandbox.get_directory. It works using it
with remote execution.
Fixes #850
-
f7681925
by Valentin David
at 2019-01-09T14:44:22Z
Merge branch 'valentindavid/script_virtual_directory' into 'master'
buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY
Closes #850
See merge request BuildStream/buildstream!1047
-
f87d1c93
by Jim MacArthur
at 2019-01-09T15:01:18Z
sandboxremote.py: Rename 'instance_name' option to 'instance-name'
It was 'instance-name' in the documentation.
-
ed8bc5bc
by Jim MacArthur
at 2019-01-09T15:30:28Z
Merge branch 'jmac/rename_instance_name' into 'master'
Rename 'instance_name' option to 'instance-name'
See merge request BuildStream/buildstream!1048
-
f29a0995
by Tristan Van Berkom
at 2019-01-09T19:51:14Z
Support running test environments in parallel with `detox`
This patch namespaces the test temp directory and the output
coverage report file with the name of the environment under test,
such that separately run tests do not access the same files.
When running tests without tox, directly through setup.py,
then the tmp directory will still be `./tmp`.
* .gitignore: Added new .coverage-reports/ directory
* .gitlab-ci.yml: Rely on tox to combine and report coverage, only
tell tox about the COVERAGE_PREFIX so that results can be namespaced
by CI job name.
This change also publishes the sources and final combined `.coverage`
file in an output gitlab artifact for inspection, and lists some missing
dependencies to the `coverage` job.
* tox.ini: Add comments and refactor main [testenv] section so that
other environments dont inherit too much unrelated cruft.
Generate the coverate reports in the respective {envtmpdir} so that
all per-process coverage files are prefixed with a full path, ensuring
that concurrent runs don't mix reports and addressing concerns
raised in #844.
Also implemented new `tox -e coverage` environment to combine
any found coverage and print a report.
* .coveragerc: Omit .tox/ directory from coverage stats
Fixes issue #844
-
d364ad02
by Tristan Van Berkom
at 2019-01-09T19:52:22Z
CONTRIBUTING.rst: Updated to mention new `coverage` tox environment.
Also point out that it is possible to run test environments in
parallel using the `detox` tool.
-
1e352434
by Tristan Van Berkom
at 2019-01-09T20:36:06Z
Merge branch 'tristan/detox-tests' into 'master'
Allow using detox, and fix/refactor collection of coverage reports
Closes #844
See merge request BuildStream/buildstream!1051
-
116c1070
by Tristan Van Berkom
at 2019-01-09T20:39:15Z
.coveragerc: Omit some things which are irrelevant to cover.
* Omit versioneer's _version.py
* Omit our __main__.py which is used only internally for generating
documentation
-
c8f3616d
by Tristan Van Berkom
at 2019-01-09T21:12:29Z
Merge branch 'tristan/refine-coverage' into 'master'
.coveragerc: Omit some things which are irrelevant to cover.
See merge request BuildStream/buildstream!1052
-
c87bb592
by Jürg Billeter
at 2019-01-10T09:43:37Z
_platform/platform.py: Add canonicalize_arch() method
-
26e33346
by Jürg Billeter
at 2019-01-10T12:50:15Z
_options/optionarch.py: Accept architecture aliases
Accept common architecture aliases for arch options instead of only
accepting the canonicalized, OS-independent architecture name. This
restores compatibility with existing projects and may simplify option
handling for projects that only target a single OS (and thus, do not
need OS-independent architecture names).
-
06deb4c4
by Jürg Billeter
at 2019-01-10T12:50:15Z
element.py: Accept architecture aliases for sandbox config
Accept common architecture aliases for the sandbox config for
consistency with arch options.
-
0c2a66b3
by Jürg Billeter
at 2019-01-10T13:06:02Z
tests/format/optionarch.py: Add tests for architecture aliases
-
f86bc760
by Jürg Billeter
at 2019-01-10T13:34:56Z
Merge branch 'juerg/arch' into 'master'
Accept architecture aliases
See merge request BuildStream/buildstream!1034
-
347eb34e
by Angelos Evripiotis
at 2019-01-10T14:38:23Z
contributing: fix 'oprtation' and some other typos
-
6bc27bc1
by Angelos Evripiotis
at 2019-01-10T15:07:46Z
Merge branch 'aevri/contrib-typos' into 'master'
contributing: fix 'oprtation' and some other typos
See merge request BuildStream/buildstream!1055
-
0b83d024
by Tristan Van Berkom
at 2019-01-10T20:02:50Z
_pipeline.py: Fix the planner to retain the original order when depth sorting
The algorithm adds elements to a dictionary and then sorts the dictionary
by the discovered depths while recursing - using an OrderedDict is enough
to ensure a stable order.
This fixes `bst show --deps plan ...` reporting different results on
each invocation.
This fixes an aspect of #824
-
630e26f1
by Tristan Van Berkom
at 2019-01-10T20:03:52Z
testutils/runcli.py: Added Result.get_start_order()
Added a function to report the list of elements which appeared in
a given queue in the order of their first appearance.
-
d1d7de57
by Tristan Van Berkom
at 2019-01-11T04:13:16Z
tests/frontend/order.py: Adding tests ensuring stable element processing order
For each sample project and expected result order, this test ensures that:
* bst show --deps plan: Always shows the expected build order.
* bst source fetch: Always executes in the expected build order.
* bst build: Always builds in the expected order (disabled).
The build part of the test is currently disabled as the scheduler
seems to be messing with the order even in a `--builders 1` scenario.
-
276b8d48
by Tristan Van Berkom
at 2019-01-11T04:42:24Z
Merge branch 'tristan/element-processing-order' into 'master'
Make build plan element list stable
See merge request BuildStream/buildstream!1058
-
fc3dcec8
by Angelos Evripiotis
at 2019-01-11T16:45:00Z
dev-requirements: pytest-cov==2.6.1 for '--no-cov'
Bump the version of pytest-cov, so that we pick up the fix for
the '--no-cov' option breaking in pytest 4.0:
https://github.com/pytest-dev/pytest-cov/pull/230
This was the error you would get prior to this version:
_pytest.warning_types.RemovedInPytest4Warning: config.warn has been
deprecated, use warnings.warn instead
Although it says 'deprecated', it did result in
termination with stack trace.
-
f86b7ff3
by Angelos Evripiotis
at 2019-01-11T16:45:00Z
contributing: '--last-failed' and '--no-cov' tips
To help new folks get to grips with pytest, add some tips on other
frequently used features.
-
d983f231
by Angelos Evripiotis
at 2019-01-11T17:19:06Z
Merge branch 'aevri/tox_no_cov' into 'master'
pytest-cov==2.6.1, contributing: '--last-failed' and '--no-cov' tips
See merge request BuildStream/buildstream!1059
-
829a2b93
by Tristan Van Berkom
at 2019-01-12T21:32:13Z
_scheduler/queues/queue.py: Put elements in the skip list, not jobs
Minor correction, looks like we're not observing this queue
otherwise we'd be seeing crashes.
-
993e30ae
by Tristan Van Berkom
at 2019-01-12T22:02:55Z
Merge branch 'tristan/fix-skipped-elements' into 'master'
_scheduler/queues/queue.py: Put elements in the skip list, not jobs
See merge request BuildStream/buildstream!1063
-
32732e01
by Javier Jardón
at 2019-01-14T09:04:01Z
.gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25
-
4f5f1184
by Valentin David
at 2019-01-14T09:40:56Z
Merge branch 'jjardon/freedesktop-sdk_latest' into 'master'
.gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25
Closes #858
See merge request BuildStream/buildstream!1064
-
bb80a2b8
by Chandan Singh
at 2019-01-14T14:30:15Z
.gitlab-ci.yml: Use newer image for overning aarch tests
During recent reorganizing of `.gitlab-ci.yml` to work with `tox`, seems
like we missed to update the image used by the overnight aarch tests,
meaning that they currently fail due to `tox` being missing from them.
While these tests will be skipped on MRs usually, here is an example of
what the tests look like if they are actually run with this change:
https://gitlab.com/BuildStream/buildstream/-/jobs/145449561.
Fixes #859.
-
10b3ee62
by Chandan Singh
at 2019-01-14T14:57:08Z
Merge branch 'chandan/fix-overnight-aarch' into 'master'
Chandan/fix overnight aarch
Closes #859
See merge request BuildStream/buildstream!1065
-
542cdaf0
by Jürg Billeter
at 2019-01-14T18:46:57Z
_frontend/cli.py: Also check original args for --config in bash completion
The path of the config file generated by testutils for completion tests
is passed as regular argument, not via COMP_WORDS. Use that config file
in complete_artifact() to ensure the test uses the right artifact
directory.
-
ff666e76
by James Ennis
at 2019-01-14T18:46:57Z
completions.py: Add a test for our artifact ref autocompletions
Since the artifact subcommand group has been created, we have been able
to tab complete both element names and artifact refs as arguments to
the artifact subcommands (e.g. bst artifact log), this commit adds a
test for this.
-
273b0f55
by Tristan Van Berkom
at 2019-01-14T20:24:57Z
Merge branch 'jennis/add_artifacts_completion' into 'master'
completions.py: Add a test for our artifact ref autocompletions
See merge request BuildStream/buildstream!1054
-
2e3c2ea2
by Javier Jardón
at 2019-01-14T22:26:04Z
.gitlab-ci.yml: Use latest bst-external
which include flatpak_repo plugin needed to build fdsdk
-
d60d2e31
by Javier Jardón
at 2019-01-15T00:39:22Z
Merge branch 'jjardon/bst_external_0_9' into 'master'
.gitlab-ci.yml: Use latest bst-external
Closes #861
See merge request BuildStream/buildstream!1068
-
605836c1
by Chandan Singh
at 2019-01-15T00:57:05Z
.gitlab-ci.yml: Add job to attempt to update dependencies
This will help us check if BuildStream is working with the latest
version of dependencies, as per our constraints. This job is allowed to
fail but its failure should signal that we need to add stricter
constraints in some of our `.in` requirements files.
-
006370af
by Chandan Singh
at 2019-01-15T01:21:33Z
Merge branch 'chandan/reqs-update-test' into 'master'
.gitlab-ci.yml: Add job to attempt to update dependencies
See merge request BuildStream/buildstream!1038
-
4b544555
by Chandan Singh
at 2019-01-15T21:28:40Z
.gitlab-ci.yml: Add tests for python 3.7
We already have tests for python 3.5 and 3.6 but not 3.7.
Fixes https://gitlab.com/BuildStream/buildstream/issues/838.
-
80fe0d9a
by Javier Jardón
at 2019-01-15T22:48:22Z
Merge branch 'chandan/python37-tests' into 'master'
.gitlab-ci.yml: Add tests for python 3.7
Closes #838
See merge request BuildStream/buildstream!1074
-
c91784ab
by Chandan Singh
at 2019-01-15T22:49:01Z
conftest.py: Don't use deprecated get_marker() function
Starting from `pytest` version 4.1.0, `Node.get_marker()` has been
removed, and hence our tests break when running with newer versions of
`pytest`. It was deprecated since a while back but it has recently been
removed completely. Use `get_closest_marker()` as a replacement that is
suggested in the changelog, and seems to work fine for our use case.
See https://github.com/pytest-dev/pytest/pull/4564 for more context on
the upstream issue.
One way of verifying this change is that this should fix the recently
added `tests-fedora-update-deps` job, that was failing before due to
this issue.
-
ecae4d73
by Javier Jardón
at 2019-01-15T23:23:41Z
Merge branch 'chandan/fix-pytest-get-marker' into 'master'
conftest.py: Don't use deprecated get_marker() function
See merge request BuildStream/buildstream!1073
-
0eac4008
by Valentin David
at 2019-01-16T10:04:57Z
buildstream/_gitsourcebase.py: Reduce git history for git describe.
Found during #833.
`git rev-list --boundary tag..HEAD` unfortunately gives boundaries
that are deeper than should when there is a merge commit between `tag`
and `HEAD`. The common ancestory of the two parents of the merge is
a boundary instead of the parent of the branch that is not traversed.
`--ancestry-path` fixes this issue by restricting `git` traversing
those branches.
-
a405e08f
by Valentin David
at 2019-01-16T10:04:57Z
buildstream/_gitsourcebase.py: Fix case where HEAD is tagged
`git rev-list --boundary HEAD..HEAD` does not return any boundary.
So in this case we need to manually tag the HEAD as a boundary.
-
c0631d48
by Valentin David
at 2019-01-16T10:33:52Z
Merge branch 'valentindavid/git-reduced-history' into 'master'
buildstream/_gitsourcebase.py: Reduce git history for git describe.
See merge request BuildStream/buildstream!1069
-
b608ac86
by James Ennis
at 2019-01-16T11:05:44Z
_context.py: Doc fix: get_toplevel_project() returns object not list
-
97a3beb6
by James Ennis
at 2019-01-16T11:05:44Z
_context.py: Add documentation to get_workspaces() command
-
d6587aa0
by James Ennis
at 2019-01-16T11:33:21Z
Merge branch 'jennis/doc_fixes_in_context' into 'master'
Small documentation/comment fixes in context.py
See merge request BuildStream/buildstream!1072
-
2683f98a
by Raoul Hidalgo Charman
at 2019-01-16T11:55:07Z
_cas: Rename artifactcache folder and move that to a root module
Other components will start to reply on cas modules, and not the artifact cache
modules so it should be organized to reflect this.
All relevant imports have been changed.
Part #802
-
d2cc4798
by Raoul Hidalgo Charman
at 2019-01-16T11:55:07Z
casremote.py: Move remote CAS classes into its own file
Part of #802
-
76f67483
by Raoul Hidalgo Charman
at 2019-01-16T11:55:07Z
cas: move remote only functions to CASRemote
List of methods moved
* Initialization check: made it a class method that is run in a subprocess, for
when checking in the main buildstream process.
* fetch_blobs
* send_blobs
* verify_digest_on_remote
* push_method
Part of #802
-
6c428bc9
by Jürg Billeter
at 2019-01-16T12:56:38Z
Merge branch 'raoul/cas-refactor' into 'master'
Cas refactor
See merge request BuildStream/buildstream!1071
-
2648b0bd
by Angelos Evripiotis
at 2019-01-16T14:44:25Z
bst-docker-import: fix Chandan's name
-
780d0f3a
by Angelos Evripiotis
at 2019-01-16T15:13:06Z
Merge branch 'chadnan' into 'master'
bst-docker-import: fix Chandan's name
See merge request BuildStream/buildstream!1076
-
4ca37dcd
by Tristan Van Berkom
at 2019-01-16T15:47:21Z
source-determinism.py integration test: Dont use integration_cache fixture
This was just deadcode, it's not really used here.
-
6286d820
by Tristan Van Berkom
at 2019-01-16T15:47:21Z
conftest.py: Use different artifact directory for integration tests
To ensure we can run integration tests in parallel, use a tempdir
for the artifact cache of each separate integration test run.
This makes it possible to run multiple full test runs including
integration tests in parallel under detox.
-
471af316
by Tristan Van Berkom
at 2019-01-16T16:32:52Z
Merge branch 'tristan/detoxing-integration-tests' into 'master'
Make integration tests parallelizable
See merge request BuildStream/buildstream!1077
-
081e8426
by James Ennis
at 2019-01-16T17:58:41Z
setup.py: Import server main from _cas/casserver.py
Since the CAS refactor, we have not been able to execute
bst-artifact-server. This commit ensures that we can.
This fix closes #867
-
e385660c
by James Ennis
at 2019-01-16T18:33:20Z
Merge branch 'jennis/fix_bst_artifact_server' into 'master'
Ensure that we can execute `bst-artifact-server`
Closes #867
See merge request BuildStream/buildstream!1079
-
6991d6d6
by Tristan Van Berkom
at 2019-01-16T19:28:14Z
_scheduler: Refactor of queues and resources.
This branch makes the following changes:
* jobs/job.py: No longer stores any interested resource list
Jobs are ephemeral again, they only ever exist while they
are running.
* queues/queue.py: Revert to only handling lists of elements
Elements pass through the queues, Queue.harvest_jobs()
replaces Queue.pop_ready_jobs() and now the Queue stops
creating jobs as soon as there are not enough resources
for the job.
Also removed unused `prepare()` abstract method.
* queues/buildqueue.py: Adapt the part where we launch a job
This part needs to be reworked anyway, just touch it up for
now so that it doesnt break with the surrounding changes.
* jobs/{cachesize,cleanup}job.py: Expose uniform complete callback
Allows the scheduler to manage resource deallocation for these
two job completions as a custom thing, at the same phase
that the Queues take care of their own resource deallocation.
* resources.py: No longer has knowledge of the job
Since jobs are ephemeral, they are not a suitable place
to store the resource identifiers, these must be provided
by the callers wherever needed.
Now the main Resources object is owned by the Scheduler
but shared with Queues, each take care of managing the
resources of the jobs they create through the same
resource API.
* scheduler.py: Reverted to only creating jobs on demand
This changes the flow of the scheduler such that whenever
jobs complete, the queues are interrogated for as many
jobs which can run at the moment but not more; and this
completely removes the waiting list.
For the internal cache management jobs, we handle this
with a little state instead of having a waiting list
and only launch when the resources permit it.
By abolishing the scheduler waiting list and creating jobs
on demand, we fix the order of element processing and consequently
fix issue #712.
-
04ba59bc
by Tristan Van Berkom
at 2019-01-16T19:28:14Z
tests/frontend/order.py: Enable the test for build and fix the fetch tests
With the scheduler changes, fetch jobs get automatically skipped
so the output is changed, using a separate repo for each element
fixes the test such that every fetch job gets a job launched.
-
32bdded8
by Tristan Van Berkom
at 2019-01-16T19:28:14Z
_artifactcache/artifactcache.py: Rephrase failure message
It was saying "There is not enough space to build the given element.",
this makes me think the error is associated to a specific element, but
this does not make sense to show up in a cleanup task.
Instead say "There is not enough space to complete the build.", which
should be more clear that even after cleaning up there is not enough
space.
-
17f6e5a8
by Tristan Van Berkom
at 2019-01-16T20:10:30Z
Merge branch 'tristan/element-processing-order' into 'master'
Scheduler refactor, fix processing order
Closes #712
See merge request BuildStream/buildstream!1067
-
19adc9e5
by Tristan Van Berkom
at 2019-01-16T20:48:10Z
element.py: Cleanup temporary staging directories on termination
Use utils._tempdir() which in turn uses _signals.terminator() for
this purpose.
This also changes utils._tempdir() to use utils._force_rmtree()
so that it is safe to use for cleaning up the staging directory.
This patch fixes orphaned temporary directories being left behind
in the artifact cache directory at forceful termination time.
-
5e9598c5
by Tristan Van Berkom
at 2019-01-16T21:17:26Z
Merge branch 'tristan/cleanup-staging-tempdir' into 'master'
element.py: Cleanup temporary staging directories on termination
See merge request BuildStream/buildstream!1080
-
5ea2f59d
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate completions test to tests/frontend
-
1eafc25c
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate `--except` testing to tests/frontend/show.py
The tests/pipeline directory will be removed, and this test
is the better of the two but also redundant with the one in
tests/frontend/show.
Renamed existing test in show.py to `test_show_except_simple`
and added the better test as `test_show_except` below it.
-
b6183123
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate virtual directory storage test to internals directory
Grouping bits of internal testing together here
-
9b6cc972
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate storage test to the internals directory
-
7e2300ce
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate yaml test to the internals directory
-
353a682f
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate utils tests into internals directory
-
3324490e
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate context test into the internals directory
-
99ea157a
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate plugin factory test to internals directory
-
e825bdd3
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate plugin loading test to internals directory
-
c7f69bcc
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Removing tests/plugins/third_party.py
This tests exactly the same thing that is tested in
tests/internals/pluginfactory.py (the new location
of tests/plugins/basics.py).
-
42150422
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Rename `plugins` directory to `elements` directory
Now that the remaining test "filter.py" in the plugins directory
tests a specific element, it makes sense to create a place for
testing elements, just like we do for sources.
-
f712aaa5
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate internal Loader basic test to internals directory
-
8d676a84
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate dependencies test to tests/format
This used to be an internal test, converted this to use the `cli` fixture.
-
bbde1c21
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate dependency order/iteration testing to the format tests
Created new `tests/format/iteration.py` which tests the order in
which elements are iterated over in various scopes in a loaded
data model.
-
8f2b4e9a
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate test for load_ref() unsupporting plugins into format/project.py
The tests/format/project.py test already has some tests about how
we error gracefully for bad plugins and bad plugin configurations,
lets put it there rather than tests/pipeline/load.py which we
will remove.
-
de881c2a
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate preflight error handling check to tests/format/project.py
This is where other load time related plugin error handling is
checked, and is the last thing to remove in the `tests/pipeline`
directory.
-
a7fbd900
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Renaming some tests in tests/format/project.py
Remove some redundancy from the test lines.
-
aa997450
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Removing tests/pipeline/load.py
The remaining test simply loads a project with one element
and asserts a value on it. This is already sufficiently tested
in tests/format/project.py.
-
561eddf1
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate protected variable handling tests to tests/format/variables.py
-
3bf40cf3
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate junctions test to tests/format/junctions.py
This is the directory for all things related to loading.
-
d34a4fd1
by Tristan Van Berkom
at 2019-01-17T00:26:43Z
Merge branch 'tristan/organizing-tests' into 'master'
General refactor in tests directory
See merge request BuildStream/buildstream!1062
-
d212cdfa
by Tristan Van Berkom
at 2019-01-17T00:28:03Z
sandbox/sandbox.py: Display failed commands in the detail string
We should only display commands in detail strings, not in the
message texts.
This also updates tests/integration/sandbox-bwrap.py to expect
the new message string which only contains the command exit status
and not the whole command itself, this does not alter the validity
of the text case which is checking that we can obtain the expected
return value.
-
ce91ce5d
by Tristan Van Berkom
at 2019-01-17T05:05:16Z
Merge branch 'tristan/error-message-regression' into 'master'
sandbox/sandbox.py: Display failed commands in the detail string
See merge request BuildStream/buildstream!1081
-
63b02f7c
by James Ennis
at 2019-01-17T15:13:27Z
_profile.py: Add timestamp to the logs
-
f64ee46f
by James Ennis
at 2019-01-17T17:02:28Z
_profile.py: Write binaries as well as logs
Private class methods which write the logs and write the binaries
have been added to Profile. The binaries are able to be used
by various visualisation tools.
-
2cb37a7e
by James Ennis
at 2019-01-18T09:31:24Z
Merge branch 'jennis/profiling_outputs_binaries' into 'master'
Make the profiler output binaries (which can be used to visualise the data)
See merge request BuildStream/buildstream!1082
-
56ec33cc
by Valentin David
at 2019-01-18T11:14:41Z
.gitlab-ci.yml: Add overnight tests logs to artifacts.
-
ad2df651
by Javier Jardón
at 2019-01-18T12:01:56Z
Merge branch 'valentindavid/overnight-tests-logs-artifacts' into 'master'
.gitlab-ci.yml: Add overnight tests logs to artifacts.
See merge request BuildStream/buildstream!1084
-
f874295f
by Tristan Van Berkom
at 2019-01-18T15:59:28Z
bzr source plugin: Use lock files to avoid corrupting the source cache
This patch by itself fixes #868
-
42a2fe3c
by Tristan Van Berkom
at 2019-01-18T15:59:28Z
bzr source plugins: Remove all of the atomic backup directory business logic
Follow up of last commit which uses exclusive locking to
protect bzr operations instead.
-
a8713ed2
by Tristan Van Berkom
at 2019-01-18T15:59:28Z
testutils/runcli.py: Added cli.get_element_states()
With get_element_state(), you need to invoke BuildStream once
for every element state you want to observe in a pipeline.
The new get_element_states() reports a dictionary with
the element state hashed by element name and is better to use
if you have more than one element to observe the state of.
-
a895cb2a
by Tristan Van Berkom
at 2019-01-18T15:59:28Z
tests/frontend/track.py: Use test_track_recurse() to stress test Sources
This causes multiple source instances to interact with the same
backing data store at the same time, increasing the likelyhood
of triggering issues around concurrent access.
This more reliably triggers issue #868
-
4236bcc7
by Tristan Van Berkom
at 2019-01-18T16:35:23Z
Merge branch 'tristan/fix-bzr-race' into 'master'
Fix bzr race conditions
Closes #868
See merge request BuildStream/buildstream!1083
-
951a8df1
by Angelos Evripiotis
at 2019-01-18T18:02:22Z
cli.py: add a hint about '--' to 'bst shell' help
-
9911023f
by Angelos Evripiotis
at 2019-01-18T18:56:21Z
Merge branch 'aevri/shell_separator_hint' into 'master'
cli.py: add a hint about '--' to 'bst shell' help
See merge request BuildStream/buildstream!1078
-
c9ce89d2
by Tristan Van Berkom
at 2019-01-18T19:36:26Z
_cas/cascache.py: Cleanup directories when removing refs
With out this, empty directories in the refs/heads directory just
grow unconditionally.
-
c536ab6a
by Tristan Van Berkom
at 2019-01-18T19:36:26Z
tests/artifactcache/expiry.py: Test refs directory cleanup
Enhance the test which checks removal of the extract directories
with an additional check that the ref directories are cleaned up
when removing artifacts.
-
99699ffc
by Tristan Van Berkom
at 2019-01-18T19:36:26Z
utils.py: Added _tempnamedfile()
When used in a child process in BuildStream, this should be
used instead of tempfile.NamedTemporaryFile() directly, otherwise
we fail to cleanup the file on SIGTERM.
-
8ce483d4
by Tristan Van Berkom
at 2019-01-18T19:36:26Z
_cas/cascache.py: Use utils._tempdir() and utils._tempnamedfile()
The direct usage of tempfile.TemporaryDirectory() and
tempfile.NamedTemporaryFile() here causes leakage of any temporary
data when the process operating on temporary data is terminated
with SIGTERM.
Using the utilities ensures that trash is not left behind in
~/.cache/buildstream/artifacts/tmp when the user terminates
BuildStream with ^C.
-
73c7252d
by Tristan Van Berkom
at 2019-01-18T20:57:42Z
Merge branch 'tristan/cas-cleanup-improve' into 'master'
CASCache cleanup improvements
See merge request BuildStream/buildstream!1087
-
adfb9291
by Tristan Van Berkom
at 2019-01-18T21:23:05Z
tests/testutils/python_repo.py: Use subprocess to run sdist
The current approach using setuptools.sandbox.run_setup() was
causing a spurious (but highly frequent) failure where setuptools
gets mixed up with it's manipulation of sys.modules and hits a
RuntimeError as a dictionary changes size while being iterated over.
For instance: https://gitlab.com/BuildStream/buildstream/-/jobs/147967307
Since this already happens in an isolated virtual environment created
by tox, we should not need additional sandboxing here from setuptools,
and launching this as a subprocess will be safer.
-
d114a6bd
by Tristan Van Berkom
at 2019-01-18T22:07:22Z
Merge branch 'tristan/fix-pip-source-test' into 'master'
tests/testutils/python_repo.py: Use subprocess to run sdist
See merge request BuildStream/buildstream!1090
-
bf591ade
by Chandan Singh
at 2019-01-18T22:09:04Z
Derive import plugin from Element instead of BuildElement
The `import` element is not really a build element. The main purpose of
the `BuildElement` class is to allow users to run `build-commands`,
`install-commands` etc. But, `import` does not run such commands.
Moreover, we already override all the methods provided by
`BuildElement`. So it only makes it confusing to have it derived from
`BuildElement` class when it is not a build element.
So, derive it from the base `Element` class instead.
-
2233a532
by Chandan Singh
at 2019-01-18T22:46:41Z
Merge branch 'chandan/import-is-not-buildelement' into 'master'
Derive import plugin from Element instead of BuildElement
See merge request BuildStream/buildstream!1089
-
aa3411f9
by Tristan Maat
at 2019-01-18T22:47:46Z
testutils/runcli.py: Allow removing artifacts from arbitrary dirs
`remove_artifact_from_cache` used a hard-coded path to remove
artifacts, which wasn't sufficient for integration tests.
-
fff882fe
by Tristan Maat
at 2019-01-18T22:47:46Z
widget.py: Avoid "showing 0 lines" messages when there are no lines
This happened when bst is invoked with --message-lines 0 or
--error-lines 0, and was arguably a little too verbose (the user
explicitly asked us not to show them any lines, after all).
Fixes #779
-
e230dedb
by Tristan Van Berkom
at 2019-01-18T23:23:35Z
Merge branch 'tlater/message-lines' into 'master'
Avoid "showing 0 lines" messages when we're asked to show no lines
Closes #779
See merge request BuildStream/buildstream!1031
-
43797617
by Richard Maw
at 2019-01-21T10:41:37Z
_gitsourcebase.py: Fetch with explicit refspecs
Old versions of git lack --force and --tags,
but the same effect can be had by specifying refspecs.
-
ce8dab0f
by Richard Maw
at 2019-01-21T10:41:37Z
tests/frontend/workspace.py: Skip test_open_multi_unwritable when root
The test assumes that a directory with write permission removed isn't writable,
this isn't the case if the process running the tests has CAP_DAC_OVERRIDE
which is common when running as root.
-
50165081
by Richard Maw
at 2019-01-21T10:41:37Z
tests/sources/git.py: Skip tests that assume too new a git
test_track_invalid_submodule depends on being able to remove a submodule
by `git rm $submoduledir`, but old versions of git don't update .gitmodules
so BuildStream still thinks there's a submodule present.
For expediency the test is skipped rather than changed to manually remove
the entry from .gitmodules if git hasn't done it,
since in the common case git is new enough to do that itself.
test_git_describe expects --first-parent to find another tag,
but `bst track` will gracefully degrade if the option doesn't work
so a different history will be retained with old versions of git.
It's of marginal benefit to add additional cruft
to test for different output on old versions of git that won't persist forever.
-
8677a256
by Benjamin Schubert
at 2019-01-21T10:41:37Z
Add a Centos runner
Centos is apparently different enough from fedora when running tests.
-
33782865
by Valentin David
at 2019-01-21T11:47:18Z
Merge branch 'richardmaw/centos-oldgit-test-fixes' into 'master'
Fix CentOS
Closes #833
See merge request BuildStream/buildstream!1085
-
c00b3782
by Chandan Singh
at 2019-01-21T19:37:53Z
CONTRIBUTING.rst: Fix formatting of link to pip docs
The link explaining python's requirements files was formatted
incorrectly (missing underscore at the end). This resulted in the text
being rendered literally, as opposed to being converted into a
hyperlink.
-
638ceb8a
by Chandan Singh
at 2019-01-21T20:16:38Z
Merge branch 'chandan/fix-contrib-link' into 'master'
CONTRIBUTING.rst: Fix formatting of link to pip docs
See merge request BuildStream/buildstream!1094
-
383142e3
by James Ennis
at 2019-01-22T12:32:43Z
using_commands.rst: Split out top level commands and subcommand groups
-
ac135333
by James Ennis
at 2019-01-22T12:32:43Z
using_commands.rst: Add the source checkout command
-
a3fc350f
by James Ennis
at 2019-01-22T12:32:43Z
Move push and pull to the new artifact subcommand group
This commit also ensures that if we try to use the 'old' commands,
BuildStream will fail and instruct the user to use the new command.
-
9eefe863
by James Ennis
at 2019-01-22T12:32:43Z
cli: Add artifact checkout subcommand
'artifact checkout' has slightly different behaviour from 'checkout',
that is, either '--directory' or '--tar' are now required options.
This is a step towards allowing checkout to take multiple args.
-
fbd15939
by James Ennis
at 2019-01-22T12:32:43Z
Mark 'old' checkout command as obsolete
This commit marks 'bst checkout' as a 'hidden' command. If used,
the user will be prompted to use the new 'bst artifact checkout'
command.
All tests which used 'bst checkout' have been modified to use
the new artifact sub-command.
This partially solves #822.
-
9e8034e9
by James Ennis
at 2019-01-22T12:32:43Z
Add NEWS entry describing the deprecation of checkout, pull and push
-
1bccf1d2
by James Ennis
at 2019-01-22T12:38:40Z
using_commands.rst: Split out the artifact subcommands
-
d9072371
by James Ennis
at 2019-01-22T12:38:40Z
using_commands.rst: Add artifact log subcommand
-
077d5a96
by James Ennis
at 2019-01-22T12:41:29Z
Change bst checkout/pull/push references to bst artifact checkout/pull/push
-
ea2fbe4d
by James Ennis
at 2019-01-22T13:02:14Z
man: Regenerate all of our man pages
Due to an upstream click_man issue:
https://github.com/click-contrib/click-man/issues/10
generating the man pages with our current setup.py only generates
a man page for bst-artifact-server, our first entry point.
I then had to remove this entry point from setup.py and regenerate to
obtain man pages for the commands in cli.py
-
9c2a6b93
by James Ennis
at 2019-01-22T15:08:17Z
Merge branch 'jennis/migrate_pull_push_commands' into 'master'
Move push/pull/checkout to the artifact subcommand group
See merge request BuildStream/buildstream!1045
-
35377522
by James Ennis
at 2019-01-22T15:28:13Z
cli.py: Add an obsoletion note to our deprecated commands.
Click 7.0 allows us to declare commands as 'hidden'.
However, sphinx-click and click-man still generate these hidden
commands in the documentation and man pages, respectively.
This is a stop gap solution until the upstream issues (tracked in
issues #879 and #881) have been addressed.
-
76148785
by James Ennis
at 2019-01-22T16:15:20Z
Merge branch 'jennis/add_obsoletion_note' into 'master'
Add an obsoletion note to our deprecated commands.
See merge request BuildStream/buildstream!1097
-
1443c542
by Tristan Van Berkom
at 2019-01-22T16:26:56Z
_scheduler: Fix dont display a failure for terminated jobs
This fixes a recent regression introduced in c2fc2a5ea
-
6e5c9987
by Tristan Van Berkom
at 2019-01-22T17:14:50Z
Merge branch 'tristan/fix-terminated-status' into 'master'
_scheduler: Fix dont display a failure for terminated jobs
See merge request BuildStream/buildstream!1096
-
3315b9a1
by Tristan Van Berkom
at 2019-01-22T18:46:24Z
tests/integration/pullbuildtrees.py: Fix the non-integration case.
This test has one test case which is marked as an integration test,
and the other is not an integration test, but was using the integration
cli. The integration cli does not work correctly if not run in
integration mode.
This was causing an error locally in conftest.py when trying to
create a tmpdir inside a nonexisting integration cache directory.
-
56e857f4
by Tristan Van Berkom
at 2019-01-22T19:59:57Z
Merge branch 'tristan/fix-pullbuildtrees-test' into 'master'
tests/integration/pullbuildtrees.py: Fix the non-integration case.
See merge request BuildStream/buildstream!1098
-
3895571a
by Chandan Singh
at 2019-01-23T00:45:47Z
tox.ini: Unpin sphinx dependency
https://github.com/rtfd/sphinx_rtd_theme/pull/672 has been fixed
upstream, and the newer versions of `sphinx_rtd_theme` do not break
search functionality with Sphinx >= 1.8.
-
528e8ed7
by Chandan Singh
at 2019-01-23T01:31:49Z
Merge branch 'chandan/unpin-sphinx' into 'master'
tox.ini: Unpin sphinx dependency
See merge request BuildStream/buildstream!1093
-
0098a900
by Tristan Van Berkom
at 2019-01-23T13:57:53Z
tests: Migrated cache quota test into artifactcache/cache_size.py
Instead of sitting mysteriously alone in internals/utils.py
-
9be31641
by Tristan Van Berkom
at 2019-01-23T13:57:53Z
_artifactcache.py: Raise ArtifactError() when quota size exceeds disk space.
This is not an error related to loading data, like a parse error
in the quota specification is, but a problem raised by the artifact
cache - this allows us to assert more specific machine readable
errors in test cases (instead of checking the string in stderr, which
this patch also fixes).
This also removes a typo from the error message in the said error.
* tests/artifactcache/cache_size.py
Updated test case to expect the artifact error, which consequently
changes the test case to properly assert a machine readable error
instead of asserting text in the stderr (which is the real, secret
motivation behind this patch).
* tests/artifactcache/expiry.py: Reworked test_invalid_cache_quota()
Now expect the artifact error for the tests which check configurations
which create caches too large to fit on the disk.
-
3a6e953f
by Tristan Van Berkom
at 2019-01-24T01:59:13Z
Merge branch 'tristan/insufficient-storage-error' into 'master'
Tristan/insufficient storage error
See merge request BuildStream/buildstream!1102
-
68339b19
by Tristan Van Berkom
at 2019-01-24T05:01:55Z
tests/testutils/runcli.py: Make get_element_states() take a list of targets
Instead of a single target, we can always provide a single target
in a list.
-
6ecc2b0a
by Tristan Van Berkom
at 2019-01-24T05:01:55Z
tests/artifactcache/expiry.py: Refactored to use get_element_states()
-
1140aed5
by Tristan Van Berkom
at 2019-01-24T05:01:55Z
tests/elements/filter.py: Refactored to use get_element_states()
-
b7ea8b74
by Tristan Van Berkom
at 2019-01-24T05:01:55Z
tests/sources/remote.py: Refactored to use get_element_states()
-
c2c004f9
by Tristan Van Berkom
at 2019-01-24T05:01:55Z
tests/frontend/pull.py: Refactored to use get_element_states()
-
21b2958b
by Tristan Van Berkom
at 2019-01-24T05:01:55Z
tests/frontend/push.py: Refactored to use get_element_states()
-
83fcaa9f
by Tristan Van Berkom
at 2019-01-24T05:01:56Z
tests/frontend/track.py: Refactored to use get_element_states()
-
46eb3018
by Tristan Van Berkom
at 2019-01-24T05:01:56Z
tests/frontend/workspace.py: Refactored to use get_element_states()
-
341b131b
by Tristan Van Berkom
at 2019-01-24T06:13:57Z
Merge branch 'tristan/test-element-states' into 'master'
Reduce number of calls to `bst show` in tests
See merge request BuildStream/buildstream!1103
-
ef2b4648
by Tristan Van Berkom
at 2019-01-24T06:14:20Z
tests/frontend/track.py: test_track_error_cannot_write_file() fixup
This tests how BuildStream reacts when it fails to write the tracking
results to the element files or project.refs file, which is an operation
that plugins do not play a part in.
As such, removing the per repo kind parameterization from this test
as multiple runs are redundant here.
-
40c18174
by Tristan Van Berkom
at 2019-01-24T07:00:50Z
Merge branch 'tristan/track-test-reduce' into 'master'
test_track_error_cannot_write_file() fixup
See merge request BuildStream/buildstream!1104
-
5b0bba85
by Jonathan Maw
at 2019-01-24T10:15:06Z
Test that tracking in workspaces actually works
Previously, it merely tested that buildstream did not fall other,
rather than whether it did anything useful.
-
58d7d722
by Jonathan Maw
at 2019-01-24T10:18:01Z
tests: Test that fetching an open workspace will fetch its dependencies
Previously, there was no way of detecting whether fetching happened, as
an element with an open workspace will not be fetched.
-
24bd8994
by Jürg Billeter
at 2019-01-24T13:36:33Z
Merge branch 'jonathan/test-missing-workspace-guessing' into 'master'
Add tests to cover reinstated support for guessing targets
See merge request BuildStream/buildstream!1042
-
e03dd5fc
by Jürg Billeter
at 2019-01-24T13:37:28Z
_frontend/cli.py: Guess element also for bst build --all
There is no reason to disallow guess_element() for bst build --all.
-
3850274b
by Phillip Smyth
at 2019-01-24T13:37:28Z
projectconfig.yaml: Add key for default targets
_versions.py: Bump format version
-
ab72d384
by Jürg Billeter
at 2019-01-24T13:37:28Z
_project.py: Add get_default_target() and get_default_targets() methods
_frontend/cli.py: Use new methods.
Based on patches by Phillip Smyth.
-
ee2d8434
by Jürg Billeter
at 2019-01-24T13:37:28Z
_stream.py: Add ignore_junction_targets parameter
This filters out junctions from the list of targets.
-
2f175f0a
by Jürg Billeter
at 2019-01-24T13:37:28Z
_frontend/cli.py: Ignore junctions in default targets where appropriate
Junctions cannot be built, pulled, or pushed. Specifying a junction on
the command line for these commands will result in an error. However,
junctions may be in the list of default targets, so they need to be
ignored for build, pull, and push commands.
-
5a351dee
by Jürg Billeter
at 2019-01-24T13:38:50Z
format_project.rst: Add documentation for default targets
-
77414518
by Phillip Smyth
at 2019-01-24T13:38:50Z
tests/frontend: Add default target tests for bst show and build
-
ce1c10ec
by Jürg Billeter
at 2019-01-24T13:38:50Z
tests/frontend/fetch.py: Add default target test for bst source fetch
-
99b5c0af
by Jürg Billeter
at 2019-01-24T13:38:50Z
tests/frontend/pull.py: Add default target test for bst push/pull
-
3642c8e7
by Jürg Billeter
at 2019-01-24T13:38:50Z
tests/frontend/buildcheckout.py: Add default target test with junction
Test that `bst build` does not fail in a project where the list of
default targets includes a junction (junctions cannot be built).
-
50c5159f
by Phillip Smyth
at 2019-01-24T13:38:50Z
NEWS: Add entry for default target feature
-
05587f22
by Jürg Billeter
at 2019-01-24T15:10:08Z
Merge branch 'issue-638-validate-all-files' into 'master'
Add support for default targets
See merge request BuildStream/buildstream!925
-
3e36e363
by Tristan Van Berkom
at 2019-01-24T16:55:24Z
_scheduler/resources.py: Dont error out in unregister_exclusive_interest()
Don't require the interest to be registered, just discard any interest,
this function just sets a bit in a mask, and is not intended to maintain
a balance like the reserve() function is.
-
ce01f87e
by Tristan Van Berkom
at 2019-01-24T16:55:24Z
_scheduler/scheduler.py: Run cache size exclusively at startup
When running any session that has Queues which require Resource.CACHE,
check if our loaded estimated size exceeds the quota, and if so;
lock the Resource.CACHE resource exclusively right away and run
an exclusive initial cache size job.
This ensures we cleanup first before doing anything which might
add to the cache at startup time, if deemed needed.
This is a partial fix for issue #737
-
2479e8df
by Tristan Van Berkom
at 2019-01-24T16:55:24Z
_frontend/widget.py: Render core messages more like other messages
In order to test when core activities occur by parsing the stderr
in tests, we should make the messages conform more.
At the same time, this restores alignment of columns in core
messages with the element processing related messages.
Also, _scheduler/scheduler.py is updated to make it's activity names
conform to the (current) 5 character limit for the sake of alignment.
The tests/frontend/logging.py test gets it's regexes updated for
the log lines it checks for in stderr.
-
fdb8ff65
by Tristan Van Berkom
at 2019-01-24T16:55:24Z
tests/artifactcache/expiry.py: Test that expiry happens first
-
acd0bf22
by Tristan Van Berkom
at 2019-01-24T18:01:41Z
Merge branch 'tristan/cache-management' into 'master'
Cache management fixes
See merge request BuildStream/buildstream!1091
-
41f03296
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_frontend/app.py: Initialize logging before preflighting the artifact cache
The artifact cache emits messages, and we want to allow that in preflight.
-
24ca2f46
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
utils.py: Add _get_volume_size()
We can streamline this call to os.statvfs() in a few places.
-
9fd9fbb9
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Refactored to use utils._get_volume_size()
This will benefit from a better UtilError being raised, and
and turns the artifact cache's local function into a one liner.
The loop which finds the first existing directory in the
given path has been removed, being meaningless due to the
call to os.makedirs() in ArtifactCache.__init__().
The local function was renamed to _get_cache_volume_size() and
no longer takes any arguments, which is more suitable for the
function as it serves as a testing override surface for
unittest.mock().
The following test cases which use the function to override
the ArtifactCache behavior have been updated to use the new
overridable function name:
tests/artifactcache/cache_size.py
tests/artifactcache/expiry.py
-
7ee0c579
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Added ArtifactCacheUsage()
A simple object which creates a snapshot of current
usage statistics for easy reporting in the frontend.
-
353293b6
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_context.py: Added get_artifact_cache_usage()
A frontend facing API for obtaining usage statistics.
I would have put this on Stream instead, but the Context
seems to be the de facto place for looking up the artifact cache
in general so let's put it here.
-
51ed36de
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_frontend/widget.py: Added cache usage entry in the startup heading
-
5797238b
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_frontend/status.py: Added Cache size usage indicator to status bar
This also adds some comments around the main status bar heading
rendering function.
-
8074ebf4
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Add status messages in cache management operations
Added some useful status messages when:
* Calculating a new artifact cache usage size
* Starting a cleanup
* Finishing a cleanup
Also enhanced messaging about what was cleaned up so far when
aborting a cleanup.
-
49c11bc8
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Added client progress callback to ArtifactCache.clean()
-
3616e939
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_scheduler/jobs/job.py: Allow subclasses to message the frontend
-
bcd19266
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_scheduler/jobs/cleanupjob.py: Update cache size while processing
Updates the known cache size in the main process while the cleanup
process is ongoing, so that the status indicators update live
while the cleanup happens.
-
9c33107f
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Correcting API documenting comment for remove()
This seems to have been copy/pasted from cascache, and
documents the function to possibly return None if defer_prune
was specified, but this function does not expose defer_prune.
-
a2140d74
by Tristan Van Berkom
at 2019-01-24T19:44:52Z
Merge branch 'tristan/cache-management-logging' into 'master'
Cache management logging enhancements
See merge request BuildStream/buildstream!1105
-
d5847462
by James Ennis
at 2019-01-25T16:35:21Z
_profile.py: Added a new profiling topic, scheduler
profile_start() and profile_end() calls have been incorporated into
Scheduler.run()
-
56a3954c
by James Ennis
at 2019-01-25T16:35:21Z
_profile.py: Added a new profiling topic, load-selection
profile_start() and profile_end() calls have been added to
Stream.load_selection()
-
116da6d7
by James Ennis
at 2019-01-25T16:35:21Z
_profile.py: Update copyright statement
-
137d31cd
by James Ennis
at 2019-01-25T17:59:22Z
Merge branch 'jennis/add_new_profile_topic' into 'master'
Add new 'scheduler' and 'load-selection' profiling topics
See merge request BuildStream/buildstream!1088
-
22b3a0c1
by Tristan Van Berkom
at 2019-01-25T18:35:36Z
_artifactcache.py: Don't require the quota to be available on disk.
Instead only rely on the headroom to be enough to protect against
out of space conditions. The headroom can become configurable as
a separate step is required.
The changes to achieve this are:
* Rename ArtifactCache.has_quota_exceeded() to ArtifactCache.full().
* ArtifactCache.full() now also reports True if the available
space on the artifact cache volume is smaller than the headroom.
This ensures jobs get triggered to cleanup the cache when
reaching the end of the disk.
* When loading the artifact quota, it is now only an error if
the quota exceeds the overall disk space, not if it does not
fit in the available space.
It is still a warning if the quota does not fit in the
available space on the artifact cache volume.
* Updated scheduler.py and buildqueue.py for the API rename
* tests: Updated the artifactcache/expiry.py test for its
expectations in this regard.
Added a new test to test an error when quota was specified to
exceed total disk space, and adjusted the existing tests to
expect a warning when the quota does not fit in the available
space.
This fixes issue #733 and #869.
-
27ca6cc7
by Tristan Van Berkom
at 2019-01-25T21:18:15Z
Merge branch 'tristan/cache-quota-max-only' into 'master'
_artifactcache.py: Don't require the quota to be available on disk.
Closes #869 and #733
See merge request BuildStream/buildstream!1106
-
89ba2abe
by Chandan Singh
at 2019-01-25T23:50:49Z
tox.ini: Add environment to update man pages
Previously, one had to manually install `click-man` package, and
remember to run the correct command. Now, we can simply run `tox -e man`
to update the man pages.
-
d7438688
by Chandan Singh
at 2019-01-25T23:50:49Z
man: Refresh man pages
Notable changes:
* New `source` and `artifact` command groups
* Man pages corresponding to obsolete commands, that were marked as
hidden in Click, have now been removed.
Fixes #881.
-
4edbbd27
by Chandan Singh
at 2019-01-25T23:50:49Z
setup.py, CONTRIBUTING.rst: Recommend using tox to generate man pages
Simplify our docs, by requesting users to run `tox -e man` to update man
pages, instead of manually installing `click-man` and running the
command manually.
-
a3e7aee8
by Chandan Singh
at 2019-01-25T23:50:49Z
setup.py: Do not error out when man directory is empty/missing
If the `man` directory is empty, then it won't be copied in the source
distribution, and `list_man_pages()` will throw an exception when trying
to list files in a non-existent directory. This prevents us from
installing the BuildStream package when the man pages are not there.
The most common use-case for this is when we want to re-generate the man
pages but want to install the package before re-generating them.
-
222753ac
by Tristan Van Berkom
at 2019-01-26T05:10:36Z
Merge branch 'chandan/toxic-man' into 'master'
Generate man pages using tox & update them
Closes #880 and #881
See merge request BuildStream/buildstream!1107
-
5df4105a
by Angelos Evripiotis
at 2019-01-28T10:13:40Z
news: fix 'osbolete' spelling
-
9c981eff
by Angelos Evripiotis
at 2019-01-28T10:17:57Z
Fixup refs to 'bst track'
Now that 'bst track' is obsolete, change guidance to refer to the
replacement 'bst source track' instead.
-
bef80291
by Angelos Evripiotis
at 2019-01-28T10:19:17Z
Fixup refs to 'bst fetch'
Now that 'bst fetch' is obsolete, change guidance to refer to the
replacement 'bst source fetch' instead.
-
564cb245
by Angelos Evripiotis
at 2019-01-28T11:36:16Z
Merge branch 'aevri/bst_track_guidance' into 'master'
Fixup refs to 'bst track' and 'bst fetch'
See merge request BuildStream/buildstream!1086
-
a3e2cdd2
by Tom Pollard
at 2019-01-28T12:14:40Z
_stream.py: Add use_artifact_config opt arg for load_selection()
use_artifact_config added as an optional default arg, allowing
for loading of given elements artifact remote config.
-
805baf7d
by Tom Pollard
at 2019-01-28T12:14:40Z
Download buildtrees on demand for bst shell --use-buildtree
Provide bst shell --use-buildtree the ability to attempt to
acquire missing buildtrees, given respective option, user
pull-buildtree context and remote availability.
_frontend/cli.py: Refactor logic for determining --use-buildtree
option with given opportunity to attempt pulling a non-local
buildtree. Element loaded with artifact_config to allow remote
querying.
_stream.py: With given user option and element state, construct
PullQueue to fetch remote buildtree. Continue or Error without
buildtree if cannot be attained.
tests/integration/build-tree.py: Update to support new usecases
-
38356932
by Tom Pollard
at 2019-01-28T13:44:37Z
Merge branch 'tpollard/829' into 'master'
Download buildtrees on demand for bst shell --use-buildtree
Closes #829
See merge request BuildStream/buildstream!1050
-
3590ca8c
by Abderrahim Kitouni
at 2019-01-28T14:59:19Z
requirements/requirements.in: require protobuf >= 3.6
This is needed since 0f2bc3754
-
39b952dc
by Abderrahim Kitouni
at 2019-01-28T15:19:04Z
requirements/requirements.in: require Click >= 7.0
This is needed since 629a6e524, and was lost in the conversion to requirements.in
-
80b36d0c
by Javier Jardón
at 2019-01-28T17:54:52Z
Merge branch 'abderrahim/protobuf-version' into 'master'
requirements/requirements.in: update minimum versions
Closes #884
See merge request BuildStream/buildstream!1114
-
a1ab48da
by Valentin David
at 2019-01-28T21:30:26Z
Fix crash when spawned job completes very fast
Job can complete before we return from `Job.span()` to
`Scheduler._spawn_job()`, so that `_active_jobs` would not yet contain
the job.
This would print a stack on the console and try to run a second time
the job which can have unexpected effects.
In order to reproduce the issue, in
`buildstream/_scheduler/jobs/job.py`, in `Job.spawn`,
add a call to `time.sleep()` right before call to
`asyncio.get_child_watcher()`.
This fixes issue #857.
-
2fcb4491
by Jürg Billeter
at 2019-01-28T22:36:22Z
Merge branch 'valentindavid/crash_in_scheduler_857' into 'master'
Fix crash when spawned job completes very fast
Closes #857
See merge request BuildStream/buildstream!1095
-
1c05a092
by Valentin David
at 2019-01-29T05:58:17Z
Fix type of error codes in CAS server
Fixes #882.
-
785da59c
by Jürg Billeter
at 2019-01-29T06:49:10Z
Merge branch 'valentindavid/wrong_type_in_status_code' into 'master'
Fix type of error codes in CAS server
Closes #882
See merge request BuildStream/buildstream!1099
-
ddef91ea
by Valentin David
at 2019-01-29T07:23:35Z
Make sure testing cache directory exists
Fixes #873
-
6a4c8611
by Jürg Billeter
at 2019-01-29T08:18:45Z
Merge branch 'valentindavid/make_cache_dir' into 'master'
Make sure testing cache directory exists
Closes #873
See merge request BuildStream/buildstream!1092
-
86c8e414
by Angelos Evripiotis
at 2019-01-29T10:39:29Z
BREAK:remove unconditional 'are you sure?' prompts
This is a breaking change, as it affects behaviour that people might be
relying on. An entry has been added to NEWS.
As proposed on the mailing list, this change removes the unconditional
prompts on:
o: bst workspace reset
o: bst workspace close --remove-dir
If interactive, these commands would always interrupt you with a prompt
like this:
This will remove all your changes, are you sure?
This seems like it may just save someone's work some time. It may also
condition folks to hit 'y' quickly without thinking.
This change also makes the non-interactive behaviour consistent with the
interactive behaviour in the default case. There is also the case of the
prompt configured by 'really-workspace-close-project-inaccessible',
which may be tackled in later work.
This change also removes the new config options to suppress those
prompts, and their associated news entry.
The relevant bit of the mailing list conversation is here:
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00106.html
The issue to make interactive and non-interactive behaviour consistent
is here:
https://gitlab.com/BuildStream/buildstream/issues/744
-
aae35e13
by Angelos Evripiotis
at 2019-01-29T12:08:07Z
Merge branch 'aevri/are_you_sure' into 'master'
BREAK:remove unconditional 'are you sure?' prompts
See merge request BuildStream/buildstream!1061
-
03111d39
by Dor Askayo
at 2019-01-30T10:35:06Z
filter.py: don't recurse when staging dependencies
Also bump the element's version so cached artifacts would be
invalidated.
Fixes #883
-
7256bb0c
by James Ennis
at 2019-01-30T11:34:04Z
Merge branch 'doraskayo/filter-indirect-deps-fix' into 'master'
filter.py: don't recurse when staging dependencies
Closes #883
See merge request BuildStream/buildstream!1110
-
36746730
by Chandan Singh
at 2019-01-31T10:50:05Z
tox.ini: Specify minimum version of click-man
`click-man` versions < 0.3.0 do not properly support multiple
entrypoints. Since this was added to `tox` after `0.3.0` was released,
`tox` should never be pulling older versions. But, let's add it here for
documentation purposes.
See
https://gitlab.com/BuildStream/buildstream/merge_requests/1107#note_135187046
for some background on this.
-
fa4a21ce
by Chandan Singh
at 2019-01-31T12:15:43Z
Merge branch 'chandan/min-version-click-man' into 'master'
tox.ini: Specify minimum version of click-man
See merge request BuildStream/buildstream!1120
-
dd791373
by Chandan Singh
at 2019-01-31T14:32:44Z
testutils/site.py: Support parsing more exotic git versions
We use output of `git --version` to determine if we can run some tests
that rely on features from newer git versions. Usually, we expect the
output to be like:
git version 2.17.2
On some platforms, like MacOS, there could be a suffix after the version
string, so that it looks something like:
git version 2.17.2 (Apple Git-113)
This causes things to fail like so:
ValueError: invalid literal for int() with base 10: '2 (Apple Git-113)\n'
Fix logic around `HAVE_OLD_GIT` such that we split the output of
`git --version` without limit on how many times we split. Previously we
used to split only twice so the suffixes like `(Apple Git-113)` are not
part of the parsed version.
-
96c0fbd6
by Chandan Singh
at 2019-01-31T15:39:19Z
Merge branch 'chandan/fix-git-version-mac' into 'master'
testutils/site.py: Support parsing more exotic git versions
See merge request BuildStream/buildstream!1118
-
d25e2795
by Benjamin Schubert
at 2019-01-31T17:06:23Z
Add LoadElement in dependency list for LoadElement idrectly
This removes the need for the 'Dependency' list to then be matched
with the corresponding LoadElement and will allow iterating the
graph more easily
-
2d0eebbf
by Benjamin Schubert
at 2019-01-31T17:06:23Z
Pass element directly to collect_element
This simplifies the loading
-
583bd97d
by Benjamin Schubert
at 2019-02-01T10:26:37Z
Merge branch 'bschubert/loader' into 'master'
Cleanup loader by linking LoadElements sooner
See merge request BuildStream/buildstream!1122
-
51cec3da
by Phil Dawson
at 2019-02-01T14:25:44Z
tests/cachekey: Test cache keys are independent of target elements
-
2b38aabe
by Phil Dawson
at 2019-02-01T15:33:00Z
Merge branch 'phil/cache-key-stability-test' into 'master'
tests/cachekey: Test cache keys are independent of target elements
See merge request BuildStream/buildstream!1123
-
dbb3d232
by James Ennis
at 2019-02-01T15:51:32Z
filter.py/filter.yaml: Documentation improvements
-
7e4205cb
by James Ennis
at 2019-02-01T15:51:32Z
filter.py: Add an example to the documentation
-
4109a34a
by James Ennis
at 2019-02-01T17:11:29Z
Merge branch 'jennis/filter-docs' into 'master'
Improve our filter documentation
Closes #278
See merge request BuildStream/buildstream!1112
-
c9345014
by James Ennis
at 2019-02-04T13:53:42Z
filter.py: Fail if declared domains do not exist in the parent element
This patch also uncovered the fact that our test_filter_deps_ok() test
has been inaccurate. Thus the element built in this test
(deps-permitted.bst) has been modified so that it build depends on the
input.bst element, as it should.
tests/filter.py: Ensure deps_ok test passes
-
3ab09651
by James Ennis
at 2019-02-04T14:47:43Z
Merge branch 'jennis/warn_for_nonexistent_domains' into 'master'
Fail when we explictly try to include/exclude non-existent domains in a filter element
See merge request BuildStream/buildstream!1117
-
cd8e5e27
by Tom Pollard
at 2019-02-05T11:22:01Z
Add --remote, -r option to bst build, inline with pull & push
Providing a remote will limit build's pull/push remote actions to
the given remote specifically, ignoring those defined via user or
project configuration.
-
a46650d0
by Tom Pollard
at 2019-02-05T12:41:30Z
Merge branch 'tpollard/buildremote' into 'master'
Add --remote, -r option to bst build, inline with pull & push
See merge request BuildStream/buildstream!1119
-
9c94e8e5
by Daniel Silverstone
at 2019-02-05T13:15:24Z
requirements: Add roaringbitmap to the requirements
In order to support use of roaring bitmaps in the loader, we need to
depend on it here.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
7682ef49
by Daniel Silverstone
at 2019-02-05T13:15:24Z
loadelement.py: Use roaring bitmaps in dep cache
The dependency cache in LoadElement causes the peak RAM consumption of
the loader to be exceedingly large, upwards of 25GB for a test of a
Debian stack. By switching from the old dict cache to a roaring bitmap
based cache, we reduce the cost of the loader cache in RAM terms to
around 5.5GB for the same stack, which makes it plausible for the short
term.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
0e471144
by Benjamin Schubert
at 2019-02-05T15:45:37Z
Merge branch 'danielsilverstone-ct/roaring-bitmaps' into 'master'
Switch to roaring bitmaps for the loader dependency caches
See merge request BuildStream/buildstream!1128
-
f17ea6a6
by Jonathan Maw
at 2019-02-06T10:32:05Z
gitlab-ci: Add commands to be run by WSL runners
-
d4d2f897
by Jonathan Maw
at 2019-02-06T10:45:34Z
tests: Skip tests that use sandboxes on WSL
-
f4c8a0ec
by Jonathan Maw
at 2019-02-06T10:45:34Z
tests: Fix test failures caused by unix sockets' path length limit
The unix specification for unix sockets only allocates ~100 characters
for the length of the path. This may be longer than the path for a CI
runner, or the path to a user's buildstream directory.
-
256bcde5
by Jonathan Maw
at 2019-02-06T15:29:18Z
Merge branch 'jonathan/wsl-tests' into 'master'
Add pre-merge tests that use a WSL runner
Closes #852
See merge request BuildStream/buildstream!1108
-
335ef2b8
by Daniel Silverstone
at 2019-02-06T17:30:13Z
loadelement.py: Switch from roaringbitmap to pyroaring
The licence for roaringbitmap was not compatible. pyroaring is
a similar project which is MIT licensed and is slightly better on
memory too.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
8f90be91
by Benjamin Schubert
at 2019-02-06T18:36:17Z
Update testing images to include c++ compiler
-
1a75b252
by Benjamin Schubert
at 2019-02-06T19:34:09Z
Merge branch 'danielsilverstone-ct/other-roaring' into 'master'
loadelement.py: Switch from roaringbitmap to pyroaring
Closes #899
See merge request BuildStream/buildstream!1133
-
7285e87e
by William Salmon
at 2019-02-07T09:42:36Z
Add more log formatting options
The 'wallclock-us' is a option that we thought we already had.
-
e61f4713
by Will Salmon
at 2019-02-07T10:58:54Z
Merge branch 'willsalmon/log_formating' into 'master'
Add more log formatting options
See merge request BuildStream/buildstream!1125
-
b4d4c4f5
by Phil Dawson
at 2019-02-08T14:27:54Z
Expose basic api for testing external plugins.
We want external plugins to be able to make use of the core testing utils.
This commit exposes the basic utilities which are currently in use in
bst-external plugins. If necessary, more utilities could be exposed in the
future.
Moves the following files from tests/testutils/ to
buildstream/plugintestingutils/:
o runcli.py
o integration.py
As part of this, this commit makes the following changes to runcli.py
and integration.py:
o runcli.py: Fix linting errors
o runcli.py: Add user facing documentation
o Integration.py: Add user facing documentation
-
77ce0c16
by Phil Dawson
at 2019-02-08T16:28:05Z
Merge branch 'phil/plugin-testing-api' into 'master'
Expose basic api for testing external plugins.
Closes #847
See merge request BuildStream/buildstream!1075
-
1ee4a4ba
by Abderrahim Kitouni
at 2019-02-08T16:42:23Z
_artifactcache.py: don't leak the project specific remote caches
the code for initializing remotes added the project specific remote
caches to the global list instead of making a copy.
Fixes #618
-
24c0de16
by Abderrahim Kitouni
at 2019-02-08T16:42:23Z
_project.py: use artifact caches from the parent project for junctions
This makes a junction use the artifact cache of the parent project
before the ones defined for the junction
Fixes #401
-
a937f99a
by Javier Jardón
at 2019-02-08T20:29:05Z
Merge branch 'abderrahim/artifact-cache-junction' into 'master'
Use artifact cache specs from the parent project before those defined in junctions
Closes #618 and #401
See merge request BuildStream/buildstream!1113
-
da1560e4
by Javier Jardón
at 2019-02-09T09:10:01Z
.gitlab-ci.yml: Remove testing on Fedora 27
Fedora 27 is EOL since 2018-11-30
See https://fedoraproject.org/wiki/End_of_life
-
79fbab61
by Jürg Billeter
at 2019-02-09T10:30:20Z
Merge branch 'jjardon/fedora_27' into 'master'
.gitlab-ci.yml: Remove testing on Fedora 27
See merge request BuildStream/buildstream!1136
-
354c563d
by Jürg Billeter
at 2019-02-10T20:03:46Z
import.py: Validate config node
Fixes #662.
-
5e1be71f
by Jürg Billeter
at 2019-02-11T05:10:56Z
Merge branch 'juerg/import' into 'master'
import.py: Validate config node
Closes #662
See merge request BuildStream/buildstream!1141
-
99e1be45
by Jürg Billeter
at 2019-02-11T05:12:25Z
local.py: Do not follow symlinks in local directories
isdir() follows symlinks on the host, resulting in potential host
contamination. This change reorders the file checks to avoid this issue.
-
f95e222e
by Jürg Billeter
at 2019-02-11T05:12:25Z
sandbox/sandbox.py: Do not follow symlinks in _has_command()
This is required to ensure symlinks are not resolved on the host.
-
89973fb3
by Jürg Billeter
at 2019-02-11T05:12:25Z
utils.py: Remove list_dirs parameter from list_relative_paths()
list_dirs was always True in the BuildStream code base. There was also a
bug in the list_dirs=False code path as it did not return symlinks in
`dirnames`.
This is an API break, however, there are no known external callers.
-
d1da3fb0
by Jürg Billeter
at 2019-02-11T05:12:25Z
utils.py: Fix sorting of symlinks to directories
os.walk() resolves symlinks to check whether they point to a directory
even when followlinks is set to False. We already work around that
broken behavior by extracting symlinks from `dirnames`. However, the
sort order was still incorrect as we returned symlinks in dirnames
before files and other symlinks. This change fixes this, sorting all
files and symlinks in a single list.
-
7cf67ed3
by Jürg Billeter
at 2019-02-11T05:12:25Z
_casbaseddirectory.py: Do not mimic os.walk() in list_relative_paths()
This matches the change in utils.list_relative_paths() that now sorts
all symlinks as files, instead of following the broken behavior of
os.walk().
-
7ec0bb5e
by Jürg Billeter
at 2019-02-11T05:44:20Z
tests/sources/local.py: Add directory symlink test
-
c07cc967
by Jürg Billeter
at 2019-02-11T07:13:28Z
Merge branch 'juerg/symlinks' into 'master'
Symlink fixes
See merge request BuildStream/buildstream!1138
-
9e3d3e05
by Angelos Evripiotis
at 2019-02-11T07:14:50Z
contributing: snakeviz replaces pyflame+flamegraph
Replace the instructions for pyflame+flamegraph with simpler ones for
snakeviz. For our general use-case this seems to be easier and better.
Usage of this tool was demonstrated at the 2019 BuildStream Gathering in
January by Daniel Silverstone, when presenting the aggregate results of
profiling on many target environments.
Here is the relevant mailing list thread:
"Profiling before the gathering"
https://mail.gnome.org/archives/buildstream-list/2019-January/msg00057.html
-
bb6a692d
by Jürg Billeter
at 2019-02-11T09:20:34Z
Merge branch 'snakeviz' into 'master'
contributing: snakeviz replaces pyflame+flamegraph
See merge request BuildStream/buildstream!1129
-
1ed63e54
by Angelos Evripiotis
at 2019-02-11T09:24:48Z
_includes: better provenance on recursive include
Use the provenance of the include block, instead of the whole node.
-
02e48209
by Angelos Evripiotis
at 2019-02-11T09:24:48Z
_includes: better error on missing include
Previously, a missing include would result in an error like this:
Could not find file at not-a-file.include
Note that the file containing the include was not mentioned.
Now we get an error like this instead:
element.bst [line 7 column 5]: Include block references a file that
could not be found: 'not-a-file.include'.
-
4336e3bf
by Angelos Evripiotis
at 2019-02-11T09:24:48Z
_includes: better error on including directory
Previously, include a directory result in an error like this:
mydir is a directory. bst command expects a .bst file.
Note that the file containing the include was not mentioned.
Now we get an error like this instead:
element.bst [line 12 column 0]: Include block references a
directory instead of a file: 'mydir'.
-
3f6c5000
by Angelos Evripiotis
at 2019-02-11T09:24:48Z
_includes: re-use file_path variable
Avoid an unnecessary call to os.path.join().
-
adde0c94
by Angelos Evripiotis
at 2019-02-11T09:24:48Z
tests/format/include: remove unused tmpdir's
Don't create and remove temp dirs unnecessarily when they are not used,
looks like these were just copy-pastes without intended side-effects.
-
a66f8379
by Jürg Billeter
at 2019-02-11T13:52:54Z
Merge branch 'aevri/include-error' into 'master'
More user-friendly reporting on include errors
See merge request BuildStream/buildstream!891
-
a482b008
by Javier Jardón
at 2019-02-11T19:23:02Z
.gitlab-ci.yml: Do not automatically run "test-wsl" until runner problems are sorted out
-
0816b8b1
by Jürg Billeter
at 2019-02-11T21:29:29Z
Merge branch 'jjardon/allow_fail_WSL' into 'master'
.gitlab-ci.yml: allow "test-wsl" to fail until runner problems are sorted out
See merge request BuildStream/buildstream!1145
-
a7aed65a
by Jürg Billeter
at 2019-02-12T05:52:07Z
utils.py: Increase buffer size in sha256sum()
Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
factor 4, according to a very simple benchmark.
-
8b9e1d24
by Jürg Billeter
at 2019-02-12T05:52:07Z
_cas/cascache.py: Increase buffer size in add_object()
Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
factor 4, according to a very simple benchmark.
-
86a9048a
by Jürg Billeter
at 2019-02-12T07:08:18Z
Merge branch 'juerg/buffer-size' into 'master'
Increase read buffer size to improve performance
See merge request BuildStream/buildstream!1142
-
9b6c18e4
by Chandan Singh
at 2019-02-12T10:20:52Z
_frontend: Allow printing dependencies using `bst show`
At present, there isn't an easy way to print anything from `bst show`
that would give the users an idea of what the dependency graph looks
like. One could use things like `--deps build`, but that will just print
a list, without any information about the dependency edges.
Add `%{deps}`, `%{build-deps}` and `%{runtime-deps}` format strings to
`bst show` that would simply print the list of all dependencies, build
dependencies and runtime dependencies respectively.
Summary of changes:
* buildstream/_frontend/cli.py: Add help for new format symbols.
* buildstream/_frontend/widget.py: Add support for new format symbols
for dependencies.
* tests/frontend/show.py: Add tests for new format symbols.
-
126bc006
by Chandan Singh
at 2019-02-12T10:23:44Z
Re-generate man pages
-
dbce3434
by Chandan Singh
at 2019-02-12T12:19:57Z
Merge branch 'chandan/deps' into 'master'
_frontend: Allow printing dependencies using `bst show`
Closes #890
See merge request BuildStream/buildstream!1121
-
24d29a17
by Javier Jardón
at 2019-02-12T12:27:42Z
.gitlab-ci.yml: Add job to run test in current Fedora:29
-
80dcf65d
by Javier Jardón
at 2019-02-12T12:27:42Z
.gitlab-ci.yml: run all generic fedora test in current stable version:29
-
58b84905
by Javier Jardón
at 2019-02-12T12:27:42Z
.gitlab-ci.yml: Remove "tests-python-3.7-stretch" job
Fedora 29 includes python 3.7 so this is not needed anymore
-
95d9b9ae
by Javier Jardón
at 2019-02-12T13:31:56Z
Merge branch 'jjardon/fedora_29' into 'master'
.gitlab-ci.yml: Test with current fedora release: 29
See merge request BuildStream/buildstream!1137
-
a39c4767
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_yaml.py: Only retrieve provenance in node_get() when needed
We were indiscriminately retrieving the node's provenance data in the
`node_get()` function which was accounting for approximately a third of
the total runtime of `node_get()` which dominates pre-scheduler time in
`bst build`. This change ameliorates that situation by only retrieving
the provenance data when it's actually needed.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
cda03313
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_yaml.py: Reduce use of `isinstance()` in `node_sanitize()`
We know that nodes are typically one of:
str, list, dict, bool, tuple, NoneType or our ChainMap
Of these, dict and ChainMap are Mapping, only list is list
and the rest are returned unchanged. We can reduce/defer our use
of isinstance here, dramatically, improving performance.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b6f08e1b
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_yaml.py: Remove use of isinstance() in `{node,list}_{chain_,}copy`
A non-trivial proportion of time pre-scheduler in `bst build` is spent
copying (or chain-copying) nodes. Approximately a quarter of the time
spent in that effort is in `isinstance()`. This removes that CPU load.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
69ee11c6
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_yaml.py: Reduce cost of node_final_assertions
By re-using the isinstance replacements from earlier commits and
using a tuple of the string constants for checking for composition
markers, we reduce the cost of node_final_assertions by two thirds
in basic testing.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
0928e570
by Daniel Silverstone
at 2019-02-12T15:12:30Z
_context.py: Cache result of get_strict()
While get_strict() doesn't look expensive per-se, it is called so many
times that it is valuable to cache the result once computed. Since I
don't think it can change once it is computable, cache it immediately
that becomes possible and we save 20s in my test case.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
022a59f0
by Benjamin Schubert
at 2019-02-12T16:14:05Z
Merge branch 'danielsilverstone-ct/further-optimisations' into 'master'
Further optimisations
See merge request BuildStream/buildstream!1131
-
352f4ad9
by Jonathan Maw
at 2019-02-12T16:44:35Z
loader.py: Stage junctions into .bst instead of a tmpdir
-
62396af9
by Jonathan Maw
at 2019-02-12T16:44:35Z
loader.py: Load junction from workspace if one's open
-
f1e9cb66
by Jonathan Maw
at 2019-02-12T16:44:35Z
Remove cleanup functionality from loader
Since junction check-outs are now persistent, there is no need to clean
up temporary directories once finished.
Now, junction checkouts are stored within the top-level project dir and
will be cleaned-up when that directory is removed.
This involves changes in:
* loader.py: Remove cleanup logic and passing-around of tmpdir.
* _project.py: Remove passing around tmpdir and calling loader's
cleanup.
-
e51116d5
by Jonathan Maw
at 2019-02-12T17:59:25Z
Merge branch 'jonathan/junction-no-tmpdir' into 'master'
Stage junctions into .bst instead of a tmpdir
Closes #895
See merge request BuildStream/buildstream!1134
-
478e5c47
by Valentin David
at 2019-02-12T18:06:50Z
buildstream/_cas/cascache.py: Set 0644 rights to pulled files
This was broken by 5ef19a0b31df84caed1e41719ef7ea5c6bd8a8bc.
-
6de65306
by Valentin David
at 2019-02-12T19:13:43Z
Merge branch 'valentindavid/pull-chmod-bug' into 'master'
buildstream/_cas/cascache.py: Set 0644 rights to pulled files
See merge request BuildStream/buildstream!1144
-
9b140fa0
by James Ennis
at 2019-02-13T09:35:45Z
element.py: Lift ArtifactCache.get_artifact_fullname() to here
This commit removes the method ArtifactCache.get_artifact_fullname()
and replaces it with Element.get_artifact_name()
Given a key, we are now able to construct the full name of any of an
element's artifacts.
-
039d7c1d
by James Ennis
at 2019-02-13T09:35:45Z
element.py: Make calculate_cache_key() API private
-
7cb6be74
by James Ennis
at 2019-02-13T09:35:45Z
element.py: Return early in __init_defaults if no plugin config
-
ddf642e1
by James Ennis
at 2019-02-13T09:35:45Z
metaelement.py: Provide constructor with default keyword arguments
-
55c15f8f
by James Ennis
at 2019-02-13T09:35:45Z
_artifactelement.py: New ArtifactElement object (derived from Element)
This object should be used when we want to handle artifact refs directly
from the command line.
An ArtifactElementError has also been added to _exceptions.py
-
6951cfc8
by James Ennis
at 2019-02-13T09:35:45Z
_project.py: Add create_artifact_element() method
-
bf1933b6
by James Ennis
at 2019-02-13T09:35:45Z
_stream.py: Add the _classify_artifacts() helper
-
ea6ff6da
by James Ennis
at 2019-02-13T09:35:45Z
_stream.py: Modify behaviour of _classify_artifacts()
* There is no need for this method to use a cas object.
* Search for artifact globs in the project's element path
* An artifact key is always 64 chars long
-
fcc79917
by James Ennis
at 2019-02-13T09:35:45Z
_stream.py: Allow loading to handle artifact refs
-
77345317
by James Ennis
at 2019-02-13T09:35:45Z
_artifactcache.py: Add get_artifacts_log() method
A CasBasedDirectory object of an artifacts logs can be obtained
with ArtifactCache.get_artifacts_log(). This ultimately calls
CASCache.get_top_level_dir() to obtain a CasBasedDirectory
of an artifact's subdirectory (or subdirectories).
-
775ac472
by James Ennis
at 2019-02-13T09:35:45Z
cli.py: Move artifact ref handling logic to stream
The loading of elements and the handling of artifacts does not belong
in this module. Such logic should be invoked using the Stream API
-
f95d6ee8
by James Ennis
at 2019-02-13T09:35:45Z
cli.py: Remove _classify_artifacts() method
This method has been moved to Stream, where is it used there exclusively.
-
491937d7
by James Ennis
at 2019-02-13T09:35:45Z
cascache.py: Allow CASCache.list_refs() to handle globs
This commit ensures that CASCache.list_refs(), and
ArtifactCache.list_artifacts(), can both handle glob expressions.
-
6ae41474
by James Ennis
at 2019-02-13T09:35:45Z
_stream.py: _classify_artifacts() should be able to handle globs
_classify_artifacts() no longer filters glob expressions by first
obtaining a list of ALL refs locally cached. We now only obtain refs
specified by the glob _expression_.
Furthermore, the Project.element_path is used to start searching for
globbed elements as opposed to the Project.directory.
-
8c6baf23
by James Ennis
at 2019-02-13T10:48:21Z
Merge branch 'jennis/refactor_artifact_log' into 'master'
Refactor artifact log command
See merge request BuildStream/buildstream!1101
-
14db3287
by Jürg Billeter
at 2019-02-13T10:48:45Z
projectconfig.yaml: Consistently include directories in split rules
Most split rules already included the relevant directories themselves in
addition to the directory contents. Add the missing bin, sbin, and
libexec directories.
This is required to fix tests with the following commit that changes
list_relative_paths() to return all directories.
-
c7625cdf
by Jürg Billeter
at 2019-02-13T10:48:45Z
tests/integration/project: Add tests directory to split rule
This is required to fix tests with the following commit that changes
list_relative_paths() to return all directories.
-
925429ea
by Jürg Billeter
at 2019-02-13T10:48:45Z
utils.py: Return all directories in list_relative_paths()
Returning only empty directories leads to inconsistencies when computing
a manifest by combining results from multiple list_relative_paths()
calls as done by the compose plugin.
I.e., the same directory may be empty in one dependency and non-empty in
another dependency. The merged file list will still contain that
directory even though it's no longer empty.
This inconsistency causes problems when calculating differences between
manifests. Returning all directories fixes these inconsistencies.
This is a change in API behavior.
-
353745a4
by Jürg Billeter
at 2019-02-13T10:48:45Z
_casbaseddirectory.py: Return all directories in list_relative_paths()
This matches the change in utils.list_relative_paths().
-
ae0ee361
by Jürg Billeter
at 2019-02-13T12:42:42Z
Merge branch 'juerg/list-all-directories' into 'master'
Return all directories in list_relative_paths()
See merge request BuildStream/buildstream!1139
-
118644b2
by Tom Pollard
at 2019-02-13T13:37:32Z
Add cli main & user conf option for 'cache-buildtrees' context
_context.py: Add cache_buildtrees global user context, the default
of which is set to by default to 'always' via the addition of
cache-buildtrees to userconfig.yaml cache group. 'failure' & 'never'
can be given as valid options.
app.py & cli.py: Add --cache-buildtrees as a bst main option, which
when passed with a valid option can override the default or user
defined context for cache_buildtrees.
tests/completions/completions.py: Update for the added flag.
-
b546bac1
by Tom Pollard
at 2019-02-13T13:37:32Z
_frontend/cli.py: Ensure failed buildtree warning is correct
not _cached_sucess() could resolve to true if the element wasn't
cached at all. switch to _cached_failure() to ensure condition
reflects expected artifact state
-
d2029444
by Tom Pollard
at 2019-02-13T13:37:32Z
Provide configuration for the optional creation of buildtrees
Artifacts can be cached explicitly with an empty `build tree` when
built via the cli main options or user config for all or only
successful build artifacts. Default behaviour is to still create
and cache all expected buildtrees.
element.py: _cache_artifact() Check if context for cache_buildtrees
has been set to always or failure with a corresponding build
result, if not skip attempting to export the build-root. Element
types without a build-root are cached with an empty buildtree
regardless. Update _stage_sources_at() to warn the user that the
buildtree import is empty.
tests/integration: Add test to artifact.py for the optional caching
of buildtree content from bst build. Rename build-tree.py to
shellbuildtrees.py to reflect included test cases, add test for
empty buildtree warning and failure option.
NEWS: Add entry for new option.
-
54ec032a
by Jürg Billeter
at 2019-02-13T14:50:15Z
Merge branch 'tpollard/896' into 'master'
Optional creation of buildtrees
Closes #896
See merge request BuildStream/buildstream!1135
-
ec4bbf35
by Benjamin Schubert
at 2019-02-13T14:59:47Z
Cleanup internal Loader cache after loading elements
-
b41a82d3
by Benjamin Schubert
at 2019-02-13T14:59:47Z
Don't register exceptions when not running the testsuite
This fix a problem with the garbage collector not being able to
clean the MetaElements that are loaded.
On small projects this is not a problem, but in bigger projects, this
can save a few hundred of MBs at runtime
The reason behind this is, whenever we have a "stack" element, which
has no stack.yaml configuration, since it doesn't need it, we would
get an exception thrown when initiating the first one, as loading the
yaml file would fail.
This would capture the frame in which this command was executed, which
references meta_elements. Therefore, as long as another exception is not
thrown, the garbage collector would not be able to clean all the
MetaElements.
-
9db7f489
by Benjamin Schubert
at 2019-02-13T16:13:16Z
Merge branch 'bschubert/cleanup-local-state' into 'master'
Cleanup MetaElement local state
See merge request BuildStream/buildstream!1147
-
c2c10d68
by Daniel Silverstone
at 2019-02-14T13:02:07Z
Cache Keys: Update to use JSON rather than pickle
This affects the cache key version (updated to 7) and introduces
a dependency on `ujson` which is BSD licenced as of the version
locked in `requirements.txt`
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
868c057e
by Tristan Van Berkom
at 2019-02-14T13:58:39Z
Merge branch 'danielsilverstone-ct/json-cache-key' into 'master'
Update cache keys to use JSON
See merge request BuildStream/buildstream!1151
-
ace15fa6
by Benjamin Schubert
at 2019-02-14T14:04:45Z
Don't keep MetaSource around in Source
We can easily rebiuld MetaSource from any source so there is no
reason to keep them around.
This will slightly improve memory usage.
-
174f5394
by Tristan Van Berkom
at 2019-02-14T15:04:27Z
Merge branch 'bschubert/dont-keep-metasource' into 'master'
Don't keep MetaSource around in Source
See merge request BuildStream/buildstream!1150
-
e7e4086e
by Jürg Billeter
at 2019-02-14T15:05:38Z
utils.py: Change _ensure_real_directory() to not resolve symlinks
Resolving symlinks during staging causes various issues:
* Split rules may not work properly as the resolved paths will differ
depending on whether another artifact with a directory symlink has
been staged in the same root directory or not, e.g., as part of
compose.
* The order of symlinks in file lists is difficult to get right to
guarantee consistent and predictable behavior as paths in a file list
might rely on symlinks in the same file list. See #647 and #817.
* Staging order differences can lead to surprising results. See #390.
* Difficult to properly support absolute symlinks. Absolute symlinks are
currently converted to relative symlinks, however, this doesn't always
work. See #606 and #830.
This will require changes in projects that rely on the current behavior.
However, the changes are expected to be small and are often a sign of
buggy element files. E.g., elements that don't fully obey `bindir` or
`sbindir` variables.
-
c0570871
by Jürg Billeter
at 2019-02-14T15:05:38Z
_casbaseddirectory.py: Do not resolve symlinks
This matches the change in utils._process_list().
This also removes the _Resolver class as it is now unused. We may want
to support controlled symlink resolution in the future, in which case
the _Resolver class can be resurrected from this commit.
-
55bccf63
by Jürg Billeter
at 2019-02-14T15:05:38Z
utils.py: Do not mangle absolute symlinks
Copy symlinks as they are, absolute or relative. We no longer resolve
symlinks when copying files, which makes this safe.
-
94c6d59b
by Jürg Billeter
at 2019-02-14T15:05:38Z
Bump artifact version for changes in symlink handling
-
949962f8
by Jürg Billeter
at 2019-02-14T15:05:38Z
NEWS: Add entry for change in symlink handling
-
56c07baa
by Jürg Billeter
at 2019-02-14T16:08:42Z
Merge branch 'juerg/symlinks2' into 'master'
Do not resolve or mangle symlinks during staging
See merge request BuildStream/buildstream!1140
-
81bce0b8
by Jim MacArthur
at 2019-02-14T16:30:27Z
_casbaseddirectory.py: Correct typo for _get_underlying_directory
-
a8389b83
by Jim MacArthur
at 2019-02-14T16:30:27Z
cascache.py: Allow _commit_directory with CasBasedDirectory
-
0fc30532
by Jim MacArthur
at 2019-02-14T16:30:27Z
Virtual directory API: Add get_size with implementations
-
4b208f9c
by Jim MacArthur
at 2019-02-14T16:30:27Z
Virtual Directory: Add _fast_directory_import
-
567e5236
by Jim MacArthur
at 2019-02-14T16:35:33Z
element.py: Alter _cache_artifact to use CasBasedDirectory.