-
cf4741ce
by James Ennis
at 2018-10-25T13:05:43Z
docs: Add helpful notes to artifact server configuration instructions
-
5737bb06
by James Ennis
at 2018-10-25T13:05:43Z
using_configuring_artifact_server.rst: Fix incorrect reference to bst push (with link)
-
28317622
by James Ennis
at 2018-10-25T13:35:22Z
Merge branch 'jennis/patch_remote_cache_docs' into 'master'
Improve our documentation for configuring an artifact server
See merge request BuildStream/buildstream!899
-
d2743c52
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_context.py: Remove useless `return` from `Context.message`
This pointless bare `return` was causing modern pylint to raise an
error.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
95adf09e
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_variables.py: Fix lint issue in `find_recursive_variable`
The python `for`/`else` construct only makes sense if the `for` loop body
has a `break` statement. Otherwise the use of `else` causes pylint
to raise an error.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
9359c067
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_workspaces.py: Simplify chained comparison
In `_parse_workspace_config` there was a chained version comparison
which is simplified in this commit to quieten a pylint issue.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
796bd8c9
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_yaml.py: use `in (a,b)` to simply boolean checks
Where we use a construct `val == foo or val == bar` we can instead use
`val in (foo, bar)` which pylint prefers.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
5a3a537f
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_yaml.py: Quieten a lint in Python < 3.6
For Python before 3.6, `path.resolve()` could not take the `strict`
keyword argument. Linting on such Python versions will raise an unnecessary
issue given the check present. As such, quieten that lint.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b279c730
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_yamlcache.py: Correct bug in `YamlCache._get_filepath()`
The correct file path was computed but never returned. This was
never caught because of a test missing consumption of one of its
parameters. This commit resolves the bug mentioned in #722 leaving
the test to be amended by a future commit.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
527f3b32
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_yamlcache.py: Correct braino in variable name
This braino would mean that we'd fail to raise the right exception
if we failed to find the right project object when unpickling the
yaml cache.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
6d0c697d
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_yamlcache.py: Remove unused imports
The imports of _cachekey and utils were unused.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
32323695
by Daniel Silverstone
at 2018-10-25T14:43:52Z
buildelement.py: Quieten a lint about `__commands`
Unfortunately elements don't override `__init__` in general, so
we have to quieten pylint here.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
c5c0654d
by Daniel Silverstone
at 2018-10-25T14:43:52Z
element.py: Simplify some conditions with `in (foo, bar)`
Where we have conditions of the form `var == foo or var == bar` it
can be simplified to `var in (foo, bar)` which pylint prefers.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
4f84e2f6
by Daniel Silverstone
at 2018-10-25T14:43:52Z
element.py: Silence pylint warning about abstract method
Unfortunately pylint can't understand that we'll always use subclasses
of Element and as such complains of a no-return function (`assemble()`)
having its return value used. This quietens that warning.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b4645745
by Daniel Silverstone
at 2018-10-25T14:43:52Z
plugin.py: use dict.get() instead of an if statement
Pylint recommends the use of dict.get() rather than a multi-line if
statement for handling optional keyword arguments.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
68ec9bcf
by Daniel Silverstone
at 2018-10-25T14:43:52Z
source.py: Quiet several pylint issues with abstract methods
A number of times, abstract methods are called which pylint cannot
possibly understand will only ever happen in subclasses which implement
the abstract methods. This silences those specific warnings
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
1bf0a031
by Daniel Silverstone
at 2018-10-25T14:43:52Z
source.py: Remove unused local variable
Remove `source_kind` as it was unused.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
e9c6db1e
by Daniel Silverstone
at 2018-10-25T14:43:52Z
source.py: Remove unused variable
Remove the `context` variable which was unused.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
5a187a1e
by Daniel Silverstone
at 2018-10-25T14:43:52Z
utils.py: Disable lint warning about popen
We use `preexec_fn` in our use of Popen. Since there are threads
in the codebase, this is not necessarily safe. However we go to
great lengths to ensure that the main process doesn't spawn threads
(they're used in gRPC a lot) and so it should be safe for our use.
As such, we disable the lint here.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b368f57f
by Daniel Silverstone
at 2018-10-25T14:43:52Z
utils.py: Group stat imports
The imports of `stat` and `S_ISDIR` from `stat` should be grouped to
satisfy pylint.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b45fea19
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_artifactcache/artifactcache.py: Remove superfluous parens
The pylint report dislikes the superfluous parens which used to
be here in this assert.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
3e4a6507
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_artifactcache/artifactcache.py: Silence lint issues
The pylint tool is unable to understand that the abstract methods
in the artifact cache will only be called when there's concrete
implementations behind them. Silence the lint errors for these
specific calls.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
447780b0
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_frontend/app.py: Remove unused import of `resource`
The `resource` import was unused. Removed.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
247b9fa8
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_frontend/status.py: Reorder imports
The `curses` import is "standard" whereas `click` is not. As such, despite
alphabetical order normally being the rule, `curses` must come before `click`.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
522867c8
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_fuse/{hardlinks,mount}.py: Remove dangerous use of {} in defaults
Since default values in arguments to functions and methods are created once
at the compilation of the code, it is dangerous to include list and dict
literals in them. This changes the use of {} to None.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
31a7bce5
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_fuse/mount.py: Deal with abstract method lint issue
Unfortunately pylint can't tell this is an abstract method which will
only ever end up invoked from a class which concretises it. As such
we need to silence this warning.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
eb805c6c
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_options/optionbool.py: Refactor conditions
The pylint tool prefers that conditions of the form "var == foo or var == bar"
be refactored into "var in (foo, bar)".
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
aca9a612
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_platform/darwin.py: Remove useless __init__()
An otherwise empty __init__ passing no arguments to super.__init__
is worthless. Remove it.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
77b1506f
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_platform/darwin.py: Remove unused imports
The import of `resource` was unused.
The import of `PlatformError` was unused.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
70000b0d
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_platform/linux.py: Remove unused imports
The imports of `Message` and `MessageType` were unused.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
bab617d0
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_platform/linux.py: Simplify return statement
The linter dislikes an if/return/else/return where it can be
simplified to a return of the if condition.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
45a1d15f
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_scheduler/jobs/job.py: Correct lint issue around abstract method
The linter cannot detect abstract methods the way we use them, so
this silences the warning which ensues.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
e9154b4d
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_scheduler/queue/pullqueue.py: Remove unused return value
The `done()` method does not return anything anywhere else.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
7b082bb7
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_scheduler/queue/trackqueue.py: Remove unused import
The `SourceError` import was never used
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
f1cfc0b7
by Daniel Silverstone
at 2018-10-25T14:43:52Z
plugins/source/git.py: Rationalised unused return values
The `assert_ref_in_track()` method's return value was never used so
rationalise it to never return a value.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
7c72d25b
by Daniel Silverstone
at 2018-10-25T14:43:52Z
sandbox/_mount.py: Do not use dict literals in argument defaults
The use of dictionary literals in argument defaults is disrecommended
due to the way that they are static and thus potentially very confusing.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
4e7408ee
by Daniel Silverstone
at 2018-10-25T14:43:52Z
sandbox/_mounter.py: Remove useless inheritance on `object`
In Python 3, inheriting from `object` explicitly is pointless.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
df495d16
by Daniel Silverstone
at 2018-10-25T14:43:52Z
sandbox/_sandboxbwrap.py: Disable lint about dict.get
Sometimes `dict.get()` is preferable, in this instance it's less clear
so we'll disable that lint
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
63c6e6b1
by Daniel Silverstone
at 2018-10-25T14:43:52Z
sandbox/_sandboxchroot.py: Silence warning about unused variable
The variable was unused, remove it.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
3394741a
by Daniel Silverstone
at 2018-10-25T14:43:52Z
sandbox/_sandboxchroot.py: Silence warning about preexec_fn
We are super-careful to not use threads in places where we might use
Popen and as such this warning is save to quash.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
cd20e7a1
by Daniel Silverstone
at 2018-10-25T14:43:52Z
sandbox/_sandboxchroot.py: Silence warning about dict.get
Sometimes `dict.get()` is preferable to an `if` statement, but this time
it's clearer if we keep the structure. As such, silence the warning.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
5087f5c5
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_artifactcache/cascache.py: Refactor slightly for indentation
Correct some multi-line conditional indentation by slight refactor
if necessary.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
d73d965b
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_artifactcache/cascache.py: Silence len(SEQUENCE) as condition warnings
The lint tool dislikes the use of `if len(SEQ) == 0` type statements
because nominally `len(SEQ)` may be more expensive than `bool(SEQ)`.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
fa740402
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_artifactcache/cascache.py: Prepare attributes in __init__
Python linters prefer that attributes are defined in __init__ rather
than left to later routines.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
ad8271f0
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_artifactcache/cascache.py: Remove unused imports
The imports of `Message` and `MessageType` were unused.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
1445a901
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_artifactcache/cascache.py: Cleanup lint warning about indexing
In order to quieten the linter, and to make a more correct comparison,
we switch to using the `Code.OK` enumeration from `google.rpc.Code`
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
19df07ea
by Tristan Maat
at 2018-10-25T14:43:52Z
Bump CI image tags
We need to do this so that the new linting errors pop up
-
64dbe9ae
by Daniel Silverstone
at 2018-10-25T14:43:52Z
MANIFEST: Include a lot more missing stuff in the manifest
Versioneer needs to be in the MANIFEST.in and its .pyc needs to be ignored
in the .gitignore. Also much docs were not being included, nor conftest.py.
Much of the test suite data files needed including, so changed that to a
raw include of everything in the tests/ tree.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
f6bc25dd
by Daniel Silverstone
at 2018-10-25T14:43:52Z
.gitlab-ci.yml: Change how coverage is recovered
We need to alter how coverage data is recovered in order to work with
newer coverage tooling.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
1c1d14e7
by Chandan Singh
at 2018-10-25T15:04:00Z
Merge branch 'danielsilverstone-ct/fix-lint-issues' into 'master'
Fix lint issues
See merge request BuildStream/buildstream!892
-
23140b6b
by Chandan Singh
at 2018-10-25T15:25:22Z
sources: Don't redundantly document directory configuration
Instead of redundantly documenting "directory" configuration option in
each source plugin, refer to the built-in functionality documentation of
the Source class.
* source.py: Add anchor for built-in functionality.
* sources/plugins/*.py: Remove directory config option, and add reference to
Source's built-in functionality.
Fixes #713.
-
db2a676e
by Chandan Singh
at 2018-10-25T15:26:39Z
build elements: Add reference to BuildElement's built-in config
For each element plugin derived from BuildElement, add reference to the
built-in functionality documentation of BuildElement.
* buildelement.py: Add anchor for built-in functionality.
* plugins/elements/*.py: Add refence to BuildElement's built-in
functionality for each build element.
Related to #713.
-
356d84cd
by Chandan Singh
at 2018-10-25T16:59:13Z
Merge branch 'chandan/source-doc-directory' into 'master'
plugins: Don't redundantly document common configurations
Closes #713
See merge request BuildStream/buildstream!896
-
8c41b5b8
by Daniel Silverstone
at 2018-10-26T09:02:30Z
_yaml.py: Do not insert into cache if retrieved from cache
We were unconditionally inserting into the yaml cache anything we had
retrieved from the yaml cache. This had the side-effect of being expensive
on re-runs of commands where yaml hadn't changed.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
494d7018
by Jonathan Maw
at 2018-10-26T09:26:36Z
Merge branch 'danielsilverstone-ct/do-not-reinsert-yaml' into 'master'
_yaml.py: Do not insert into cache if retrieved from cache
See merge request BuildStream/buildstream!902
-
80762ecb
by Benjamin Schubert
at 2018-10-26T09:45:39Z
Check if command is a str and replace by list before checking existence
Currently we would make sure the sandbox had a command before
converting it to a list if it was given as a string. That meant that a string
command would never exist and the check be invalid.
This also adds the same logic in the dummy sandbox for consistency.
-
f131c407
by Benjamin Schubert
at 2018-10-26T10:09:51Z
Merge branch 'bschubert/fix-command-sandbox' into 'master'
Check is command is a str and replace by list before checking existence
Closes #728
See merge request BuildStream/buildstream!898
-
54d89aa1
by Daniel Silverstone
at 2018-10-26T10:20:36Z
_loader/loadelement.py: Use enumerate() in dependency extraction
In cases where elements end up with large numbers of dependencies,
the n-squared performance in `_extract_depends_from_node` can produce
significant delays during project load. This patch corrects that.
In addition we were double-retrieving provenance which was unnecessary.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
e4c18ba4
by Phil Dawson
at 2018-10-26T10:43:55Z
Merge branch 'danielsilverstone-ct/use-enumerate' into 'master'
_loader/loadelement.py: Use enumerate() in dependency extraction
See merge request BuildStream/buildstream!901
-
20e5c58a
by James Ennis
at 2018-10-26T11:14:58Z
using_configuring_artifact_server.rst: Fix lines that are too long
-
320dae02
by James Ennis
at 2018-10-26T11:37:42Z
docs: Add more information to "managing the cache with systemmd" section
-
1eb993bf
by James Ennis
at 2018-10-26T12:02:49Z
Merge branch 'jennis/another_docs_patch' into 'master'
More information regarding the use of systemd to manage the cache
See merge request BuildStream/buildstream!904
-
f069d82f
by Daniel Silverstone
at 2018-10-26T12:19:58Z
various: Move _sentinel from utils.py to _yaml.py
The `_sentinel` in `utils.py` was used only for yaml node tracking.
As such, simplify matters by removing it from `utils.py` and move it
to `_yaml.py` which means that we no longer need to try and avoid a
circular import issue by means of runtime importing.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
283887a5
by James Ennis
at 2018-10-26T13:12:12Z
Merge branch 'danielsilverstone-ct/yaml-sentinel-rework' into 'master'
Move _sentinel from utils.py to _yaml.py
See merge request BuildStream/buildstream!903
-
2c6cb230
by Valentin David
at 2018-10-26T14:21:18Z
Fix some ruamel warnings during tests
-
a8250ca4
by Valentin David
at 2018-10-26T15:04:25Z
Merge branch 'valentindavid/ruamel_warnings' into 'master'
Fix some ruamel warnings during tests
See merge request BuildStream/buildstream!889
-
1b308fe3
by Daniel Silverstone
at 2018-10-26T15:50:42Z
_stream.py: Preserve stdout FD across checkout-to-stdout
In order to support things which might need to use the stdout FD
after checkout has completed writing a tarball to it, preserve it
by means of `os.dup()` and `os.dup2()`
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
65d90c31
by Daniel Silverstone
at 2018-10-26T15:50:54Z
tests/testutils/runcli.py: Support binary-mode capture of stdout
In order to test things which write tarballs to stdout correctly,
we need to capture the binary output cleanly. This ensures we're not
potentially mismatching encodings in and out.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
ed733f3e
by Daniel Silverstone
at 2018-10-26T15:51:40Z
tests/frontend/buildcheckout.py: Enable test_build_checkout_tarball_stdout
With the new support for saving the stdout FDs cleanly, and binary mode
capture in runcli, enable the tarball-to-stdout test.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
e97d2a75
by Benjamin Schubert
at 2018-10-29T10:17:07Z
Merge branch 'bschubert/fix-binary-output-capture' into 'master'
Don't close stdout when running 'checkout' to stdout
Closes #486
See merge request BuildStream/buildstream!905
-
1625de0e
by Tristan Van Berkom
at 2018-10-29T11:49:33Z
doc/source/conf.py: Added `images` directory for statically included files
-
38507b4e
by Tristan Van Berkom
at 2018-10-29T11:49:33Z
doc/source/format_intro.rst: Adding link anchor for include directives
-
26164bcd
by Tristan Van Berkom
at 2018-10-29T13:57:51Z
MANIFEST.in: Include SVG and ODG files in source distributions
-
3a23c8f6
by Tristan Van Berkom
at 2018-10-29T13:57:51Z
doc: Adding new architecture document with initial "Overview of modules" section
-
41d370f6
by Tristan Van Berkom
at 2018-10-29T13:57:51Z
doc: Adding new architecture document describing the data model
-
bed7f7ac
by Tristan Van Berkom
at 2018-10-29T13:57:51Z
doc: Adding new architecture document describing the dependency model
-
9bdf7f40
by Tristan Van Berkom
at 2018-10-29T13:57:51Z
doc: Adding new architecture document about highlevel program flow
-
6b8bdf35
by Tristan Van Berkom
at 2018-10-29T13:57:51Z
doc: Adding new architecture document about how the scheduler works.
-
b1fadaa5
by Tristan Van Berkom
at 2018-10-29T13:57:51Z
doc: Moved cachekey and sandboxing writings into the architecture section
-
baf6b578
by Tristan Van Berkom
at 2018-10-29T14:28:41Z
Merge branch 'tristan/architecture-docs' into 'master'
Tristan/architecture docs
See merge request BuildStream/buildstream!907
-
09ef8b25
by Jonathan Maw
at 2018-10-30T15:42:05Z
element.py: Always clean up the rootdir
We shouldn't need it to persist now that we cache failed build dirs.
This change breaks the test
`tests/integration/shell.py::test_sysroot_workspace_visible`.
I can no longer see a use-case for this test.
AIUI, it tested that the failed build sysroot stored in the builddir has
the workspace's files in, despite the workspace being unmounted.
I believe this behaviour is made redundant by cached buildtrees.
This fixes part of #539
-
37afde0a
by Jonathan Maw
at 2018-10-30T15:42:05Z
Element: Use cached buildtree in build shells and failure shells
This includes changes in app.py:
* Interactive failure shell no longer uses the failed build sysroot,
defaulting to the cached build tree.
Changes in element.py are:
* Errors caused by building don't store the failed build sysroot,
instead storing that a sandbox can be created to debug the error.
* When staging sources, will stage the element's cached build tree if it
exists.
Changes in _exceptions.py:
* BstError.sandbox is now a flag of whether a sandbox can be opened up
to debug the error.
Changes in widget.py:
* Don't try to print any information about the sandbox.
Changes in _message.py:
* Fix documentation so Message.sandbox is not a directory any more.
This is part of #539
-
4bc71773
by Jonathan Maw
at 2018-10-30T15:42:05Z
NEWS: Add item for cached buildtree behaviour
-
9b89564f
by Jonathan Maw
at 2018-10-30T15:42:05Z
tests: Add test that cached build trees are staged in build shells
This is related to #539
-
ec76cbe1
by Jonathan Maw
at 2018-10-30T15:42:05Z
sandbox.py: Remove redundant Sandbox.__directory
-
825963b5
by Jonathan Maw
at 2018-10-30T15:42:05Z
element: Make "--sysroot" take a bare directory
i.e. instead of taking a directory that must contain "root" and
"scratch", and treating "root" as the root, use the directory directly.
In element.py:
* __sandbox takes the `bare_sandbox` arg, to pass into the sandbox's constructor
In sandbox.py:
* If bare_sandbox, `_root` is the passed-in directory, and `__scratch`
is None.
* Trying to use `__scratch` when bare_sandbox is True is a bug.
In _mount.py:
* Don't get the value of `__scratch` if it's not needed.
This is part of #539
-
cf721838
by Jonathan Maw
at 2018-10-30T15:42:05Z
NEWS: Add item for bst shell --sysroot changes
-
79d1bb7f
by Jonathan Maw
at 2018-10-30T15:42:05Z
tests: Add tests for 'bst shell --sysroot'
This is related to #539
-
c7ac7e7d
by Jonathan Maw
at 2018-10-30T16:19:17Z
Merge branch 'jonathan/debug-remote-failed-builds' into 'master'
Jonathan/debug remote failed builds
See merge request BuildStream/buildstream!869
-
516e990e
by ctolentino8
at 2018-10-31T11:36:46Z
_frontend/app.py: Set correct element-path in interactive bst-init
-
b8a37a63
by Tristan Van Berkom
at 2018-11-01T10:16:25Z
Merge branch 'chiaratolentino/bst-init-interactive-elementpath' into 'master'
_frontend/app.py: Set correct element-path in interactive bst-init
See merge request BuildStream/buildstream!912
-
b27b592a
by Benjamin Schubert
at 2018-11-01T10:49:57Z
Remove dependency on pytest-runner
This includes a new command mimicking pytest-runner so that we
can drop this dependency
This was the only setup_requires dependency that we had and
will make like easier for people behind proxies
-
89ace5d7
by Benjamin Schubert
at 2018-11-01T11:16:36Z
Merge branch 'bschubert/remove-pytest-runner' into 'master'
Remove dependency on pytest-runner
See merge request BuildStream/buildstream!886
-
4cfabce8
by Angelos Evripiotis
at 2018-11-01T11:35:02Z
BREAK: manual.yaml: don't set any default env vars
Remove the 'MAKEFLAGS' and 'V' defaults.
Now that there is a specialised 'make' element, these make-specific
defaults don't make sense here. This element is meant to be a blank
slate for folks to build on.
Note that this is a breaking change, that will require some users to
make changes to their .bst files if they are expecting these environment
variable to be set.
_versions.py: BST_FORMAT_VERSION bumped to 18.
This fixes issue #718
-
48860aac
by Tristan Van Berkom
at 2018-11-01T12:01:04Z
Merge branch 'tristan/empty-manual-element' into 'master'
BREAK: manual.yaml: don't set any default env vars
Closes #718
See merge request BuildStream/buildstream!916
-
d868b409
by Daniel Silverstone
at 2018-11-01T13:40:24Z
_yaml.py: Implement `get()` for `ChainMap`
Since the core Python `ChainMap.get()` implements with:
self[key] if key in self else default
The double-chain-lookup is expensive. This simple change solves that
for our ChainMap derived structure. As such it improves matters
for #466 somewhat.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
7f79b9ce
by Tristan Van Berkom
at 2018-11-01T14:25:57Z
Merge branch 'danielsilverstone-ct/improve-chainmap-get' into 'master'
_yaml.py: Implement `get()` for `ChainMap`
See merge request BuildStream/buildstream!919
-
de59ebdb
by ctolentino8
at 2018-11-02T16:41:54Z
plugins/sources/pip.py: Accomodate characters '-','.','_' for packages
-
8d7cf806
by ctolentino8
at 2018-11-02T16:41:54Z
tests/integration/project/files: Remove pypi-repo
-
9c2f9bf7
by Chandan Singh
at 2018-11-02T17:09:46Z
Merge branch 'chiaratolentino/fix-pip-source-regex' into 'master'
plugins/sources/pip.py: Accomodate characters '-','.','_' for packages
See merge request BuildStream/buildstream!914
-
3788e701
by Jürg Billeter
at 2018-11-03T11:52:00Z
element.py: Pass bare_directory also to SandboxRemote()
Fixes: 825963b5 ("element: Make "--sysroot" take a bare directory")
-
82e971ef
by Jürg Billeter
at 2018-11-05T11:33:20Z
Merge branch 'juerg/remote-execution' into 'master'
element.py: Pass bare_directory also to SandboxRemote()
See merge request BuildStream/buildstream!928
-
62942bfd
by Valentin David
at 2018-11-05T12:14:20Z
Use less fetchers in overnight build to work-around limitations from servers.
Issue happens with sourceware.org server.
Fixes #742
-
442da2f9
by Javier Jardón
at 2018-11-05T12:41:54Z
Merge branch 'valentindavid/limit_fetchers_in_overnight_build' into 'master'
Use less fetchers in overnight build to work-around limitations from servers.
Closes #742
See merge request BuildStream/buildstream!929
-
0993de53
by Richard Maw
at 2018-11-05T16:41:05Z
dev-requirements: Update pytest dependency to 3.8
"_frontend/app.py: Set correct element-path in interactive bst-init"
added a dependency on the tmp_path test fixture, which was added by 3.8.
-
be8f0a54
by richardmaw-codethink
at 2018-11-05T17:08:43Z
Merge branch 'richardmaw/pytest-dep-update' into 'master'
dev-requirements: Update pytest dependency to 3.8
See merge request BuildStream/buildstream!930
-
bfb639bf
by Jürg Billeter
at 2018-11-05T17:18:12Z
element.py: Remove artifacts parameter from constructor
Get the artifact cache from the context instead.
-
ca855f91
by Jürg Billeter
at 2018-11-05T17:18:12Z
_elementfactory.py: Remove unused artifacts parameter from create()
-
15fed21c
by Jürg Billeter
at 2018-11-05T17:18:12Z
_project.py: Remove unused artifacts parameter from create_element()
-
0085d2aa
by Jürg Billeter
at 2018-11-05T17:18:12Z
element.py: Remove unused artifacts parameter from _new_from_meta()
-
f69b1117
by Jürg Billeter
at 2018-11-05T17:18:12Z
_project.py: Remove unused artifacts parameter from load_elements()
-
e398f877
by Jürg Billeter
at 2018-11-05T17:18:12Z
element.py: Do not include type name of artifact cache in cache key
The artifact cache backend does not affect build outputs and we anyway
no longer have pluggable artifact cache backends. This hardcodes
CASCache instead of removing the entry completely to avoid cache key
changes.
-
626d20ae
by Jürg Billeter
at 2018-11-05T17:18:12Z
Split up artifact cache and CAS cache
This changes CASCache from a subclass to a delegate object of
ArtifactCache. As the lower layer, CASCache no longer deals with
elements or projects.
Fixes #659.
-
ec04446b
by Jürg Billeter
at 2018-11-05T17:51:39Z
Merge branch 'juerg/cas' into 'master'
Split up artifact cache and CAS cache
Closes #659
See merge request BuildStream/buildstream!922
-
9a045080
by Josh Smith
at 2018-11-06T13:23:19Z
utils: Fix _pretty_size() for sizes > 1024T
-
d3a07e6b
by Josh Smith
at 2018-11-06T13:23:19Z
Add regression test for _pretty_size
-
0c09fb9c
by richardmaw-codethink
at 2018-11-06T14:12:20Z
Merge branch 'Qinusty/unit-test-utils' into 'master'
Fix issue with _pretty_size with large numbers of bytes
See merge request BuildStream/buildstream!799
-
b4eec489
by Jim MacArthur
at 2018-11-06T16:44:14Z
_sandboxremote.py: Use the standard SandboxError.
Replaces the custom one which was erroneously added during development.
Fixes #746.
-
1f7acf74
by Jim MacArthur
at 2018-11-06T17:16:31Z
Merge branch 'jmac/remote-sandbox-errors' into 'master'
_sandboxremote.py: Use the standard SandboxError
Closes #746
See merge request BuildStream/buildstream!931
-
8e64ccef
by James Ennis
at 2018-11-07T11:06:11Z
using_config.rst: Update our instructions for how we declare caches in user config
-
09ab676d
by James Ennis
at 2018-11-07T11:06:11Z
format_project.rst: Update instructions regarding how we declare remote caches in project.conf
-
f514124f
by James Ennis
at 2018-11-07T11:06:11Z
using_configuring_artifact_server.rst: Remove examples and redirect to user config/project config
-
029ba17d
by richardmaw-codethink
at 2018-11-07T11:32:20Z
Merge branch 'jennis/docs_patch' into 'master'
Update our instructions/examples regarding how we declare remote caches
See merge request BuildStream/buildstream!932
-
261e2cd3
by Jim MacArthur
at 2018-11-07T12:02:43Z
element.py: Use _force_rmtree instead of custom code.
There are arguments for and against using onerror - see issue #735
for details.
-
8931e42c
by Jim MacArthur
at 2018-11-07T12:26:06Z
Merge branch 'jmac/make_writable_combination' into 'master'
Use _force_rmtree instead of custom code.
Closes #735
See merge request BuildStream/buildstream!927
-
6ccfab0b
by Valentin David
at 2018-11-07T16:28:47Z
Fix cache corruption by scripts when layout and integration commands are used
Root directory was marked as a non-artifact mount, so not using
SafeHardLink. However integration commands executed with write access
to the root directory.
Fixes #749
-
f24e20e9
by richardmaw-codethink
at 2018-11-07T16:59:50Z
Merge branch 'valentindavid/script-artifact-corruption' into 'master'
Fix cache corruption by scripts when layout and integration commands are used
Closes #749
See merge request BuildStream/buildstream!934
-
e28f668a
by Valentin David
at 2018-11-08T09:34:11Z
Fix infinite recursion in default strip debug command
Fixes #645.
-
83d15350
by Valentin David
at 2018-11-08T10:01:39Z
Merge branch 'valentindavid/fix-recursive-strip-debug' into 'master'
Fix infinite recursion in default strip debug command
Closes #645
See merge request BuildStream/buildstream!918
-
9f0e12f1
by Benjamin Schubert
at 2018-11-08T10:21:12Z
Move bwrap checks in platform/linux.py
Remove the bwraps checks from _site.py and put them in platform.linux
which is the only place where they are run.
This allows the removal of a double level of caching, making
reasoning about tests easier
-
cf2e0059
by Benjamin Schubert
at 2018-11-08T10:21:12Z
conftest.py: Ensure platform is not maintained between tests
This removes the `_instance` on the platform object that we use
for caching and not recreating the object everytime at the start
of every test.
This is to ensure our tests share the least amount of state.
The performance penalty is from 5 to 10% accross the whole test suite.
The readings were done 5 times for each before and after the change
and on the same computer.
-
c51ba01b
by Benjamin Schubert
at 2018-11-08T10:21:12Z
Test that helpful messages are raised when missing dependencies
This adds a `reason` to the SandboxEror thrown in sandboxdummy
to be able to understand where the error comes from
-
f116b9b7
by Benjamin Schubert
at 2018-11-08T11:07:12Z
Merge branch 'bschubert/tests-no-chroot-linux' into 'master'
test behavior when there is no sandbox on Linux
Closes #696 and #736
See merge request BuildStream/buildstream!921
-
cc2e6ae5
by Valentin David
at 2018-11-08T12:41:36Z
Fix bug with root mounted as non-artifact in script plugin.
The issue was introduced by 6ccfab0b1b25990e406446d5cbe5aee83a5e158a.
-
e578a89f
by Valentin David
at 2018-11-08T13:11:10Z
Merge branch 'valentindavid/fix-script-install-root' into 'master'
Fix bug with root mounted as non-artifact in script plugin.
Closes #756 and #749
See merge request BuildStream/buildstream!941
-
c0a8bb66
by Angelos Evripiotis
at 2018-11-08T15:49:16Z
contributing: WIP again to be kind to reviewers
As someone coming from GitHub to GitLab, I was pleasantly surprised by
the 'filter by WIP status' option.
To make sure we get the most out of it, add a guideline to keep the
filter clean, to reduce reviewer burden.
-
f7231e90
by Angelos Evripiotis
at 2018-11-08T15:49:16Z
contributing: non-WIP should always be landable
I was recently surprised that we don't prefer addressing review comments
in 'fixup!' commits.
Coming from GitHub, I've found that fixup commits make it easier for
reviewers to see what has changed since their last review. The idea is
to use '--autosquash' before landing to clean up the history again.
It's a pleasant surprise that it's easy to keep track of what changed in
merge-requests between pushes, so we can always keep the history clean.
Document this, so folks like me can see the light sooner.
-
f7643440
by Angelos Evripiotis
at 2018-11-08T15:49:16Z
contributing: emphasise the 'why' in commits
Explain why it's useful to mention the decisions made for a change and
provide links for background, e.g. issue numbers.
My general experience of git histories is that folks don't do enough
explaining, so it's great we have a contributing section for it.
I've found it's easier for folks to stick to rules when they know the
practical reasons for them, so provide some.
-
dd5e7b04
by Angelos Evripiotis
at 2018-11-08T16:17:04Z
Merge branch 'aevri/contributing_gitlab' into 'master'
Add more to GitLab-relevant parts of contributing
See merge request BuildStream/buildstream!935
-
fe33e328
by James Ennis
at 2018-11-08T17:54:18Z
using_config.rst: Add documentation to showing how to impose quotas on the local cache
This patch partially resolves #700
-
09faf002
by James Ennis
at 2018-11-08T17:54:18Z
artifactcache.py: Fix misleading error message when using % cache quota
Due to the changed Exception message, this patch also changes
the test_parse_size_over_1024T test in misc.py
-
d153453c
by Javier Jardón
at 2018-11-08T18:22:54Z
Merge branch 'jennis/quota_declaration_fix' into 'master'
Add local cache expiry documentation and fix misleading error message when specifying a percentage cache quota
Closes #700
See merge request BuildStream/buildstream!939
-
7738f6df
by Daniel Silverstone
at 2018-11-09T11:25:44Z
plugin.py: Redirect DEBUG from `__del__` to `sys.stderr`
Previously, debug messages from `Plugin.__del__()` went to stdout
which was a problem for the test suite in extremely rare and hard to
predict circumstances. This corrects that by sending the message to
stderr where all the rest of the messages tend to go.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
62f59eaa
by Valentin David
at 2018-11-09T13:30:06Z
Merge branch 'danielsilverstone-ct/plugin-destroy-debug-to-stderr' into 'master'
plugin.py: Redirect DEBUG from `__del__` to `sys.stderr`
See merge request BuildStream/buildstream!944
-
35ec6b6a
by Jim MacArthur
at 2018-11-09T14:58:22Z
_casbaseddirectory.py: Enable direct CAS-to-CAS import.
Fixes issue #574.
-
b93100ec
by Jim MacArthur
at 2018-11-09T14:58:22Z
_casbaseddirectory: Fix order of list_modified_paths
This will now return paths in the same order as list_relative_paths.
-
78691fa0
by Jim MacArthur
at 2018-11-09T14:58:22Z
virtual_directory_import.py: New test
-
7ce6581b
by Jim MacArthur
at 2018-11-09T15:25:46Z
Merge branch 'jmac/cas_to_cas_v2' into 'master'
Direct CAS-to-CAS import
Closes #574
See merge request BuildStream/buildstream!911
-
6f4351ec
by Benjamin Schubert
at 2018-11-09T16:53:42Z
source.py: don't let StopIteration propagate to silence() contextmanager
As per PEP 0479 (https://www.python.org/dev/peps/pep-0479/),
StopIteration thrown in context managers are not valid starting from
Python 3.7.
-
e6e03451
by richardmaw-codethink
at 2018-11-12T11:05:15Z
Merge branch 'bschubert/fix-silence-stopiteration-pep-0479' into 'master'
source.py: don't let StopIteration propagate to silence() contextmanager
See merge request BuildStream/buildstream!945
-
ae5ccd76
by Phillip Smyth
at 2018-11-12T14:57:51Z
_project.py: Validate nodes early in Project._load
_project.py: Added validate_nodes() helper function to prevent duplicate lists
element-path was being used before node validation resulting in uncaught errors
-
56a54161
by Phillip Smyth
at 2018-11-12T14:57:51Z
tests/frontend: Add test for invalid element-path
-
e55a9703
by Jürg Billeter
at 2018-11-12T15:44:45Z
Merge branch 'element-path_not_validated' into 'master'
Element path not validated before use
See merge request BuildStream/buildstream!937
-
c87d6e45
by Valentin David
at 2018-11-14T11:43:33Z
Run tests on aarch64
Fixes #755
-
16b01489
by Valentin David
at 2018-11-14T11:43:33Z
Disable cachekey tests on other architectures than x86_64
-
85046b29
by Valentin David
at 2018-11-14T11:43:33Z
Disable tests on example on other architectures than x86_64
-
dd5a073b
by Valentin David
at 2018-11-14T11:43:33Z
Add support for aarch64 in a test
-
c5d72ae7
by Valentin David
at 2018-11-14T11:52:53Z
Test aarch64 only overnight
-
264a57f6
by Javier Jardón
at 2018-11-14T12:22:03Z
Merge branch 'valentindavid/integration-tests-aarch64' into 'master'
Run tests on aarch64
Closes #369 and #755
See merge request BuildStream/buildstream!948
-
da735e56
by Richard Maw
at 2018-11-14T13:30:34Z
_platform/linux.py: Move get_bwrap_version into _site.py
It's inconvenient to have to create a Linux platform to parse the bwrap version
and we want to get the version in a consistent manner.
-
e7633500
by Richard Maw
at 2018-11-14T13:30:34Z
buildstream/sandbox/_sandboxbwrap.py: Distinguish sandbox failure from command failure
If `bwrap` fails to set up the sandbox and start the payload command
it won't write an exit-code in --json-status-fd,
so we can report if it was a sandboxing failure if we don't get exit-code status
and a payload command failure if we do and it's non-zero.
Closes https://gitlab.com/BuildStream/buildstream/issues/286
-
e9e08823
by Richard Maw
at 2018-11-14T13:30:34Z
tests/testutils/site.py: Check for bwrap supporting --json-status-fd
-
90ca007e
by Richard Maw
at 2018-11-14T13:30:34Z
tests/integration/sandbox-bwrap.py: Test distinguishing sandbox exit code from command
-
327b19dd
by richardmaw-codethink
at 2018-11-14T13:59:16Z
Merge branch 'richardmaw/distinguish-sandboxing-build-fail' into 'master'
Distinguish between bubblewrap sandboxing failure and command failure
Closes #286
See merge request BuildStream/buildstream!868
-
082b3811
by Angelos Evripiotis
at 2018-11-15T19:48:12Z
contributing: add guidance on unit tests
Decrease uncertainty around whether unit tests are welcome in the
project or not.
-
b5b79056
by Jürg Billeter
at 2018-11-15T20:24:58Z
Merge branch 'aevri/unit_tests' into 'master'
contributing: add guidance on unit tests
See merge request BuildStream/buildstream!943
-
199bfff1
by Tom Pollard
at 2018-11-17T13:07:59Z
Add cli main and user config option for 'pull-buildtrees' context.
_context.py: Add pull_buildtrees global user context, the default
of which is set to False via the addition of pull-buildtrees to
userconfig.yaml cache group.
_frontend/app.py & cli.py: Add --pull-buildtrees as a bst main
option, which when passed will override the default or user defined
context for pull_buildtrees.
tests/completions/completions.py: Update for the added flag.
-
454dcaf1
by Tom Pollard
at 2018-11-17T13:07:59Z
Don't pull artifact build trees by default.
The addition of cached build trees being included in element
artifacts has led to mostly redundant download overheads when
pulling from a remote artifact server. As such the default behaviour
of pull shouldn't fetch the build tree object if available.
element.py: extend relevant pull logic for specified subdir
consideration and ensure push logic does not lead to partial
artifact pushes. Change assumption that the buildtree can be
extracted if artifact is cached. __cached_buildtree() and
__pull_directories() helpers added.
_artifactcache/: artifactcache.py & cascache.py inclusion of
helper functions for subdir artifact checking & fetching, fetch
logic extended to only pull required artifact directories.
extract & checkout updated to handle full/partial operation.
tests/: addition of integration test pullbuildtrees.py,
buildtrees.py adapted cli options, testutils/artifactshare.py
has_artifact changed to return artifact digest on true condition.
-
0d309478
by Jürg Billeter
at 2018-11-17T13:49:58Z
Merge branch 'tpollard/494' into 'master'
Don't pull artifact buildtrees by default
See merge request BuildStream/buildstream!786
-
27932739
by Benjamin Schubert
at 2018-11-19T09:46:39Z
plugins/sources/pip.py: also look for python version named "python"
In some virtual environments, we might not have the binary with the
exact python version, which would lead the current pip plugin to
fail to find a working pip version.
This adds "python" at the start of the list of valid python versions
-
ea2de561
by Benjamin Schubert
at 2018-11-19T10:19:24Z
Merge branch 'bschubert/fix-pip-python' into 'master'
plugins/sources/pip.py: also look for python version named "python"
Closes #758
See merge request BuildStream/buildstream!942
-
f23b6031
by Benjamin Schubert
at 2018-11-19T11:39:51Z
tests/frontend/buildtrack.py: check for success after invocation
Previously the result of the first call in test_build_track was not
checked to be successful, which would lead to potential erros later on
that were hard to debug.
Let's check it everytime
-
a6defc0b
by Benjamin Schubert
at 2018-11-19T11:39:51Z
Fix os.rename in git source element to correctly handle error codes
According to the documentation
(https://www.unix.com/man-page/POSIX/3posix/rename/), when the directory
already is there, either EEXIST or ENOTEMPTY could be thrown.
Previously only ENOTEMPTY was checked.
Done:
- Separated the move into its own function
- Check for both errors
- Create unit tests for it, covering most test cases
-
88089d2d
by Benjamin Schubert
at 2018-11-19T11:39:51Z
Extract atomic move function to utils.py
Moving atomically a file/directory can be tricky since different
errors might be raised for the same underlying problem.
Having a utility function to reduce this discrepancies will help
in ensuring we have correct behavior
-
fd9e46be
by Benjamin Schubert
at 2018-11-19T11:39:51Z
cascache.py: use move_atomic instead of manual error checking
-
d32e0b83
by Benjamin Schubert
at 2018-11-19T11:39:51Z
sources/pip.py: Use move_atomic instead of manual os.rename
This uses move_atomic insteand of the manual os.rename and manual error
checking and throws a SourceError for consistency with other modules.
-
6f837118
by Benjamin Schubert
at 2018-11-19T12:22:40Z
Merge branch 'bschubert/fix-atomic-move-git-repo' into 'master'
Fix os.rename in git source element to correctly handle error codes
See merge request BuildStream/buildstream!938
-
30b72244
by Chandan Singh
at 2018-11-19T12:35:01Z
buildstream/_pipeline.py: Simplify error message for assert_consistent()
Currently, `Pipeline.assert_consistent()` prints an error message like
the one below when certain elements have inconsistent sources:
```
...
Element: hello.bst is inconsistent
Source tar source at hello.bst [line 16 column 2] is missing ref
...
```
Drop the word "Source" from the beginning of the message as
"Source <source-kind> source ..." is kind of awkward to read.
-
76c5d2f8
by Chandan Singh
at 2018-11-19T12:35:01Z
Add `bst source-checkout` command
As discussed in
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
add `bst source-checkout` command. This will allow users to checkout
sources for a given target.
* _frontend/cli.py: Add source-checkout command
* _pipeline.py: Add assert_sources_cached() method
* _stream.py: Add source_checkout method, abstract out
__check_location_writable() method that used to part of checkout()
-
9f629638
by Chandan Singh
at 2018-11-19T13:03:36Z
Merge branch 'chandan/source-checkout' into 'master'
Add `bst source-checkout` command
See merge request BuildStream/buildstream!820
-
bc827cc8
by Abderrahim Kitouni
at 2018-11-19T14:55:15Z
plugins/elements/cmake.yaml: always specify variable types
cmake sometimes misinterprets relative paths as relative to the current directory
if this is not specified. See freedesktop-sdk/freedesktop-sdk#431
adjust tests/format/variables.py accordingly.
-
5fbc5f41
by Valentin David
at 2018-11-19T15:31:05Z
Merge branch 'abderrahim/cmake-variable-types' into 'master'
plugins/elements/cmake.yaml: always specify variable types
Closes #761
See merge request BuildStream/buildstream!947
-
06e28860
by Benjamin Schubert
at 2018-11-19T15:52:24Z
Don't cache sandbox errors
Sandbox errors (like missing host tools) are dependent on the host
system and rarely on what is actually done.
It is therefore better to not cache them as they are subject to
change between two runs.
Also add test to ensure sandbox failure are not cached
-
dd36cfbc
by Benjamin Schubert
at 2018-11-19T15:52:24Z
element.py: Only cache on element error or success
Reverse the way we were handling caching by only caching
when we know it's a problem with the build or the build is a
success in order not to cache transient errors.
-
eac7274d
by Benjamin Schubert
at 2018-11-19T16:29:07Z
Merge branch 'bschubert/dont-cache-errors-from-host-tools' into 'master'
Don't cache sandbox failures
Closes #727
See merge request BuildStream/buildstream!895
-
625dfe1f
by Chandan Singh
at 2018-11-19T16:41:17Z
NEWS: Add entry for the new source-checkout command
`bst source-checkout` command was recently added in
https://gitlab.com/BuildStream/buildstream/merge_requests/820. Add a
NEWS entry for the new command.
-
d4f12184
by Chandan Singh
at 2018-11-19T17:15:10Z
Merge branch 'chandan/source-checkout-news' into 'master'
NEWS: Add entry for the new source-checkout command
See merge request BuildStream/buildstream!960
-
ba6c96e8
by Richard Maw
at 2018-11-19T19:54:30Z
CasBasedDirectory: Change constructor to take a CASCache instead of a Context
The Context was only used to obtain a reference to the CASCache
and set the unused cas_directory field.
-
8a0dc3a3
by Jürg Billeter
at 2018-11-19T19:54:30Z
tests/testutils/artifactshare.py: Do not create a fake context
The fake context did not set the cache quota, triggering an error. With
CASCache now separate from ArtifactCache, we can instantiate a CASCache
without context.
-
8722aced
by Jürg Billeter
at 2018-11-19T19:54:30Z
tests/plugins/pipeline.py: Load context default values
-
fc56ffa4
by Jürg Billeter
at 2018-11-19T19:54:30Z
_context.py: Drop duplicated default values for user configuration
The default values are in userconfig.yaml, together with the
documentation. The default values should not be duplicated in
_context.py.
-
3f663d82
by Jürg Billeter
at 2018-11-19T20:36:57Z
Merge branch 'juerg/context-default-values' into 'master'
_context.py: Drop duplicated default values for user configuration
See merge request BuildStream/buildstream!953
-
b498cce7
by Angelos Evripiotis
at 2018-11-20T11:17:38Z
_yaml: document node_get()'s default_value arg
-
eb2d376f
by Angelos Evripiotis
at 2018-11-20T11:17:38Z
_context: allow 'terminate' for scheduler.on-error
Enable this option of 'terminate', which is mentioned in userconfig.yaml
and handled in _frontend/app.py:_handle_failure(). It appears to have
been left out of the valid_actions as an oversight.
Originally introduced in
https://gitlab.com/BuildStream/buildstream/commit/2622d5da9bd9fefd87436613d6e3e9770fdd0f28
-
b81c4333
by Angelos Evripiotis
at 2018-11-20T11:17:38Z
_context: refactor, extract _node_get_option_str
Use a new helper function to simplify working with nodes that can only
accept certain strings. This will be used when adding the prompt.*
config options.
In later work we can see if this function would be useful elsewhere, and
could be added to '_yaml.py'.
-
27ca6593
by Angelos Evripiotis
at 2018-11-20T11:19:33Z
Add prompt.auto-init buildstream.conf option
Provide an option in buildstream.conf to disable the 'Would you like to
...' prompt when we cannot resolve a project.
Some users prefer not to be interrupted by such prompts, so pave the way
to creating options to disable all those that might get in the way.
Follow the example of the advice.* options 'git-config', and create a
namespace for these UI options grouped by behaviour, rather than an
over-reaching 'ui.*' namespace. In later work perhaps we'll also add
'advice.*' options.
Add a NEWS item for this.
-
7ae3a3d2
by Angelos Evripiotis
at 2018-11-20T11:43:49Z
Add prompt.workspace-... options
Provide options in project.conf to disable the 'Are you sure ...'
prompts when making destructive changes:
- Add prompt.really-workspace-close-remove-dir
- Add prompt.really-workspace-reset-hard
Add a NEWS item for these.
-
8071c00c
by Angelos Evripiotis
at 2018-11-20T12:12:11Z
Merge branch 'aevri/prompt-config' into 'master'
Add 'prompt.*' config options to buildstream.conf
See merge request BuildStream/buildstream!887
-
fd41b0b5
by Raoul Hidalgo Charman
at 2018-11-20T16:32:10Z
_sandboxremote.py: Add sigterm handler that sends CancelOperation
-
c6306b88
by Jürg Billeter
at 2018-11-20T16:59:45Z
Merge branch '725-job-cancellation-on-remote-builds' into 'master'
_sandboxremote.py: Add sigterm handler that sends CancelOperation
Closes #725
See merge request BuildStream/buildstream!900
-
9fa8a881
by James Ennis
at 2018-11-20T17:00:42Z
_yaml.py: Fix incorrect error message
This patch ensures that we receive an appropriate error
message if we specify an absolute path that leads within
the project.
-
fb8860db
by Jürg Billeter
at 2018-11-20T17:32:58Z
Merge branch 'jennis/less_restrictive_pathways' into 'master'
Fix warning when using an absolute path which exists inside the project
See merge request BuildStream/buildstream!956
-
87751350
by Benjamin Schubert
at 2018-11-20T18:10:48Z
gitlab-ci.yml: Use CI_PROJECT_DIR instead of $(pwd) for cache
-
8668df9f
by Benjamin Schubert
at 2018-11-20T18:10:48Z
Export variables globally for less repetition.
This doesn't change the test behavior and enable for better
composability of tests
-
cc6a7f0d
by Benjamin Schubert
at 2018-11-20T18:10:48Z
gitlab-cy.yml: make linux-tests-template more composable.
This will allow for further de-duplication of work.
-
eb37dd89
by Benjamin Schubert
at 2018-11-20T18:10:48Z
gitlab-ci.yml: remove unused artifacts for unix tests
The unix-logs is always empty, there is therefore no reasons of keeping
it there
-
e030b48e
by Benjamin Schubert
at 2018-11-20T18:10:48Z
gitlab-ci.yml: use template for unix tests
This makes all tests use the same template, which makes reasoning
about them simpler
-
57e6358f
by Benjamin Schubert
at 2018-11-20T18:10:48Z
Ensure tests requiring bubblewrap are correctly marked
-
e70dd1f8
by Benjamin Schubert
at 2018-11-20T18:10:48Z
gitlab-ci.yml: Add linux without brwap/ostree to ensure tests behave
This allows us to make sure that tests behave nicely while missing brwap
and ostree.
-
f7cb4386
by Benjamin Schubert
at 2018-11-20T18:10:48Z
gitlab-ci.yml: Extract test command as a variable
This removes the need of having it synchronized in multiple places
-
7f316a76
by Benjamin Schubert
at 2018-11-20T18:41:03Z
Merge branch 'bschubert/add-fedora-nobwrap-tests' into 'master'
Add tests for fedora with no bubblewrap available
See merge request BuildStream/buildstream!949
-
a50c598e
by Benjamin Schubert
at 2018-11-21T09:24:15Z
utils.py: Ensure move_atomic booleans are keyword only arguments
This is to be coherent with the rest of the codebase
-
b2f604b8
by Benjamin Schubert
at 2018-11-21T09:24:15Z
utils.py: improve documentation of move_atomic
-
daca0c01
by Benjamin Schubert
at 2018-11-21T09:52:42Z
Merge branch 'bschubert/mr938-comments' into 'master'
Followup on MR 938, addressing additional comments
See merge request BuildStream/buildstream!958
-
050cd1bd
by Tom Pollard
at 2018-11-21T11:37:48Z
tests/plugin/pipeline.py: Avoid using host user conf
create_pipeline was passing no config parameter to context.load,
leading to _context.py loading in the host's buildstream user conf
when running tests locally, potentially leading to unwanted errors.
-
58353aad
by Jürg Billeter
at 2018-11-21T14:29:49Z
Merge branch 'tpollard/pipelinehostconfig' into 'master'
tests/plugin/pipeline.py: Avoid using host user conf
See merge request BuildStream/buildstream!961
-
2022c3e8
by William Salmon
at 2018-11-21T15:53:03Z
tests/frontend/workspace.py: Refactor support function
The Refactor now makes it possible to test for the forth coming multi
element workspace open commands and to test for failure.
-
7bdd44b2
by William Salmon
at 2018-11-21T15:53:03Z
Workspace CLI update
This is to update the workspace CLI to as agreed on the mailing list
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00046.html
This patch also introduces the default workspace directory.
-
6612e708
by William Salmon
at 2018-11-21T15:53:03Z
Added a news item for workspace open changes
-
4d6e2cd3
by Will Salmon
at 2018-11-21T16:20:26Z
Merge branch 'willsalmon/defaultWorkspaces' into 'master'
Updated Workspace CLI
See merge request BuildStream/buildstream!897
-
dddf0eaa
by Valentin David
at 2018-11-22T11:12:53Z
.gitlab-ci.yml: Bump version of fd.o SDK and bst_external in overnight tests
-
abef70fe
by Javier Jardón
at 2018-11-22T11:41:31Z
Merge branch 'valentindavid/overnight-tests-bump-fdo-sdk-version' into 'master'
Bump version of fd.o SDK and bst_external in overnight tests
See merge request BuildStream/buildstream!962
-
5c208698
by Jim MacArthur
at 2018-11-26T10:23:31Z
Docs: Add remote execution architecture documentation.
-
5af455a4
by Jim MacArthur
at 2018-11-26T10:55:22Z
Merge branch 'jmac/execution-architecture-docs' into 'master'
Docs: Add remote execution architecture documentation.
See merge request BuildStream/buildstream!923
-
4ca892e3
by Jim MacArthur
at 2018-11-27T09:30:13Z
tests: Avoid hangs due to exceptions in subprocesses
-
d09a18b8
by Jim MacArthur
at 2018-11-27T10:31:07Z
Merge branch 'jmac/fix-test-hangs-2' into 'master'
Avoid hanging artifact cache tests
See merge request BuildStream/buildstream!964
-
df60258a
by Jürg Billeter
at 2018-11-27T12:11:24Z
source.py: Add optional _get_local_path() method
-
42809933
by Jürg Billeter
at 2018-11-27T12:11:24Z
local.py: Implement _get_local_path()
-
eb5b10f7
by Jürg Billeter
at 2018-11-27T12:11:24Z
_loader.py: Use Source._get_local_path() for subprojects
This eliminates unneeded copies of the project directory for junctions
with a single local source.
-
b2ea208c
by Jürg Billeter
at 2018-11-27T12:50:46Z
Merge branch 'juerg/local-junctions' into 'master'
Optimization for local junctions
See merge request BuildStream/buildstream!290
-
223c0073
by Jürg Billeter
at 2018-11-27T13:41:09Z
tests/integration/manual.py: Add test for command logging
-
38d16c62
by Jürg Billeter
at 2018-11-27T13:41:09Z
sandbox: Deduplicate code to process cwd, env, and command arguments
-
85f5b574
by Jürg Billeter
at 2018-11-27T13:41:09Z
sandbox/sandbox.py: Add SandboxFlags.NONE
This improves readability.
-
bdd6925f
by Jürg Billeter
at 2018-11-27T13:41:09Z
element.py: Use SandboxFlags.NONE instead of 0
-
69005c76
by Jürg Billeter
at 2018-11-27T13:41:09Z
scriptelement.py: Use SandboxFlags.NONE instead of 0
-
f1767de2
by Jürg Billeter
at 2018-11-27T13:41:09Z
sandbox/sandbox.py: Add command batching API
This adds the batch() context manager.
-
87f7c99a
by Jürg Billeter
at 2018-11-27T13:41:09Z
element.py: Pass element object to sandbox
This allows the sandbox to output messages using the plugin id.
-
024d584d
by Jürg Billeter
at 2018-11-27T13:41:09Z
element.py: Cache artifact also on SandboxCommandError
-
bb7c3213
by Jürg Billeter
at 2018-11-27T13:41:09Z
element.py: Defer workspace.prepared setting in case of pending commands
-
04f05b1a
by Jürg Billeter
at 2018-11-27T13:41:09Z
element.py: Support command batching for integration commands
-
7827ac50
by Jürg Billeter
at 2018-11-27T13:44:49Z
element.py: Add batch_prepare_assemble() method
This allows batching of all sandbox commands aross prepare() and
assemble().
-
63db90b2
by Jürg Billeter
at 2018-11-27T13:44:49Z
buildelement.py: Support batching for integration and build commands
-
73dec86e
by Jürg Billeter
at 2018-11-27T13:44:49Z
scriptelement.py: Support batching for integration and build commands
-
79442e0a
by Jürg Billeter
at 2018-11-27T13:44:49Z
compose.py: Support batching for integration commands
-
6018b3bd
by Jürg Billeter
at 2018-11-27T13:44:49Z
sandbox/_sandboxremote.py: Implement command batching
Execute batched commands in a single shell script.
-
ec3d4486
by Jürg Billeter
at 2018-11-27T13:44:49Z
NEWS: Add entry for command batching
-
127d332f
by Jürg Billeter
at 2018-11-27T14:16:50Z
Merge branch 'juerg/command-batching' into 'master'
Command batching
Closes #675
See merge request BuildStream/buildstream!915
-
d5b5538d
by Jim MacArthur
at 2018-11-27T15:32:19Z
artifactcache: Move ArtifactCacheSpec code into CASRemoteSpec.
There is nothing in ArtifactCacheSpec that's actually specific to
artifacts, so I've made it a CAS class so we can use the same
spec for remote execution.
-
f224d4cb
by Jim MacArthur
at 2018-11-27T15:32:19Z
artifactcache.py, _context.py: Move CASCache object into context
Since the artifact cache and remote execution share the same
local CAS store, they should share the same CASCache object. Moving
this into context allows us to do this.
-
926c1446
by Jim MacArthur
at 2018-11-27T15:33:34Z
_sandboxremote: Remove references to artifact cache.
Uses the new config options introduced earlier to specify remote
exec storage and execution service.
-
d7cdb03e
by Jim MacArthur
at 2018-11-27T15:33:36Z
Documentation: Update to show new config options
-
95d8244f
by Jim MacArthur
at 2018-11-27T15:33:36Z
tests/sandboxes/remote-exec-config.py: New test.
Tests that we get a sensible error message when we specify incomplete
or wrong information in the configuration for remote execution.
-
7f2f62eb
by Jim MacArthur
at 2018-11-27T15:33:36Z
artifactcache.py: Remove unused verify_digest_pushed
-
c55ca3a5
by Jim MacArthur
at 2018-11-27T15:33:36Z
Docs: Correct URLs in artifact cache example
The port number must come after the hostname and before the path.
The path is ignored by CASCache anyway, so including it is misleading.
-
2a8a3b19
by Jim MacArthur
at 2018-11-27T18:17:03Z
Merge branch 'jmac/remote_execution_split' into 'master'
Split remote execution from artifact cache
Closes #750
See merge request BuildStream/buildstream!946
-
499c70fd
by Valentin David
at 2018-11-28T09:11:21Z
setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52
-
3513580c
by Valentin David
at 2018-11-28T14:02:32Z
Merge branch 'valentindavid/ruamel_0_15_41' into 'master'
setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52
See merge request BuildStream/buildstream!975
-
26cdee08
by Valentin David
at 2018-11-28T14:29:52Z
Use f_bavail to query available space. Not f_bfree.
f_bfree space might not be usable. In practice we see failures in big
disks because f_bfree is over 2GB and f_bavail is 0. We get ENOSPC if
writing on disk then.
-
58ca298f
by Valentin David
at 2018-11-28T14:29:52Z
Make cache clients not fail when a blob is not available.
We plan to make cache incomplete. That is some blobs are missing. For
most of cases we will delete references when requested if they are
incomplete. But there will be corner cases where objects are removed
after the reference is requested.
-
227fa26d
by Valentin David
at 2018-11-28T14:29:52Z
Fix type error in RPC messages
-
5ef19a0b
by Valentin David
at 2018-11-28T14:29:52Z
Avoid copying temporary file when adding object to CAS in server.
The file is already a temporary file and does not need copy. ENOSPC
is thrown during that copy in issue #609.
Fixes #678.
-
8d2946ff
by Valentin David
at 2018-11-28T14:29:52Z
"Fallocate" object disk space to avoid getting NOSPC error when writing
This locks the temporary object file so that cleanup does not need to
be done for every write.
-
b587953a
by Valentin David
at 2018-11-28T14:29:52Z
Update mtimes of objects for requested references.
This also remove references when some objects are missing. This is in
preparation for the move from reference to object garbage collection.
-
a64f667d
by Valentin David
at 2018-11-28T14:29:52Z
Move cas server from ref-based to object-based garbage collection.
-
353b90dd
by Valentin David
at 2018-11-28T14:29:52Z
Cleanup cache in cas server more agressively
When there is less than 2GB left, it cleans up have 10GB available.
These values are configurable.
-
ba9afa98
by Valentin David
at 2018-11-28T14:29:52Z
Lock cache cleanup in cas server
Cleaning up in parallel might slow down the cleaning process
-
9a458402
by Valentin David
at 2018-11-29T08:47:40Z
Merge branch 'valentindavid/cache_server_fill_up' into 'master'
Fix cleanup of cache in server when disk is full
Closes #678
See merge request BuildStream/buildstream!830
-
47b7a9ba
by Jim MacArthur
at 2018-11-29T09:58:48Z
_sandboxremote.py: Remove unnecessary tests.
push_message and push_directory will both raise assertion failures if
they fail to send the digest to the server. Checking the digest
returned by each only tests that the content was hashed locally, which
cannot reasonably fail.
-
4a8d0565
by Jim MacArthur
at 2018-11-29T10:42:31Z
Merge branch 'jmac/no-verify-digests' into 'master'
_sandboxremote.py: Remove unnecessary tests.
Closes #786
See merge request BuildStream/buildstream!976
-
2e78e0d1
by Valentin David
at 2018-11-29T13:18:17Z
Add support for .netrc in remote/tar/zip plugins
Fixes #723.
-
a6144100
by Valentin David
at 2018-11-29T13:18:17Z
.gitlab-ci.yml: Update image versions where forgotten
-
f6c184f5
by Valentin David
at 2018-11-29T16:15:56Z
Merge branch 'valentindavid/netrc' into 'master'
Add support for .netrc in remote/tar/zip plugins
Closes #723
See merge request BuildStream/buildstream!908
-
1f31aefc
by Angelos Evripiotis
at 2018-11-30T09:34:38Z
man/: update with changes since Apr 2018
It seems that the regenerated man pages haven't been committed for some
time. Add them all in one lump now.
Had to manually strip the version number from the generated files.
Had to manually fix the NAME field of bst.1 to not be 'bst - None'.
Oddly had to invoke click-man directly in order to get the changes to
be applied, the usual setup.py route mentioned in contributing.rst
didn't work.
Eyeballed the result of invoking man on each of the commands, seems
good.
-
033a5ad9
by Jürg Billeter
at 2018-11-30T10:09:11Z
Merge branch 'aevri/update_man' into 'master'
man/: update with changes since Apr 2018
See merge request BuildStream/buildstream!981
-
1123b9a1
by Tom Pollard
at 2018-11-30T13:30:35Z
_stream.py: Ability to pull missing buildtrees outside of pull/build
Adds helper function _buildtree_pull_required() to determine if a
pullqueue should be constructed, for commands outside of bst pull
and build where it is determined that an element's buildtree
artifact is to be required given the respective semantics and
config. Utilised in push() to attempt to mitigate skipping the push
of partial elements without the user having to have preceded it with
an explicit pull.
cli.py: Add new behaviour to push command description
element.py: Move _cached_buildtree() to be non local private method,
use _KeyStrength types to reduce duplication.
tests/integration/pullbuildtrees.py also updated to cover this
use-case.
-
ad293ed3
by Jürg Billeter
at 2018-11-30T14:10:06Z
Merge branch 'tpollard/774' into 'master'
_stream.py: Ability to pull missing buildtrees outside of pull/build
Closes #774
See merge request BuildStream/buildstream!978
-
0d634c21
by Phillip Smyth
at 2018-11-30T14:16:36Z
plugin.py: Add BAD_ELEMENT_SUFFIX warning
-
d97ef822
by Phillip Smyth
at 2018-11-30T14:16:36Z
_project.py: Check for fatal_warnings earlier
-
59d442e5
by Phillip Smyth
at 2018-11-30T15:41:57Z
loader: Added file suffix check and helper function
_loader/loader.py: Added Suffix check and warning for named elements and add helper function
-
2d6ec07d
by Phillip Smyth
at 2018-11-30T15:41:57Z
cli.py: no autocomplete on files without .bst
-
ac0bc4c0
by Phillip Smyth
at 2018-11-30T16:50:35Z
Added tests for .bst suffix and completions
buildcheckout.py: Add mandatory .bst suffix tests
completions.py: Add test for fail on invalid suffix
Added required files for testing
integration/source-determinism.py: renamed test elements to end with .bst
loader/__init__.py: initialised context properly with load and a message handler
-
d4bc662a
by Phillip Smyth
at 2018-11-30T16:50:50Z
NEWS: Added update about the Mandatory .bst change
-
7cea464a
by Jürg Billeter
at 2018-11-30T17:21:37Z
Merge branch 'mandatory_suffix' into 'master'
Mandatory .bst suffix
See merge request BuildStream/buildstream!967
-
cc5ef68c
by Tristan Van Berkom
at 2018-12-03T08:41:42Z
MANIFEST.in: Include the sources for rebuilding the documentation session HTML
This was previously missing, causing our CI to forget to rebuild the sessions
when updating the documentation.
This fixes issue #778
-
87ba19d7
by Tristan Van Berkom
at 2018-12-03T09:19:54Z
Merge branch 'tristan/fix-doc-session-rebuilds' into 'master'
MANIFEST.in: Include the sources for rebuilding the documentation session HTML
Closes #778
See merge request BuildStream/buildstream!984
-
1b24148f
by Chandan Singh
at 2018-12-03T09:46:13Z
Ensure `--deps=none` option works for `bst checkout`
Currently, `bst checkout --deps none` command always produces empty
output. Fix this issue and add regression test for the same.
- element_enums.py: Add Scope.NONE.
- element.py: Ensure Scope.NONE works correctly in addition to
Scope.RUN/Scope.ALL in Element.dependencies() and Element.search().
- tests/frontend/buildcheckout.py: Fix tests for `--deps none`.
Fixes #670.
-
8ac1e757
by Chandan Singh
at 2018-12-03T09:46:14Z
Add `--deps build` option to `bst checkout`
As discussed in
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
add `--deps build` option to `bst checkout`. This will allow users to
checkout the all build dependencies of a given element using a single
command.
- _frontend/cli.py: Add `--deps build` option for `bst checkout`.
- element.py: Support `deps='build'` in Element._prepare_sandbox().
- tests/frontend/buildcheckout.py: Ensure `--deps build` works as
expected.
-
ac60f47a
by Benjamin Schubert
at 2018-12-03T09:46:14Z
Refactor and simplify _prepare_sandbox for elements
Before we would have a intricate logics with multiple arguments
that might get ignored.
This simplifies the design and introduces a bool `shell` instead
of having two different variables concerned about scope
-
b1430fb5
by Tristan Van Berkom
at 2018-12-03T11:01:54Z
Merge branch 'chandan/bst-checkout-build' into 'master'
Add `--deps build` option to `bst checkout`
Closes #670
See merge request BuildStream/buildstream!819
-
9022a778
by Javier Jardón
at 2018-12-03T11:14:55Z
buildstream/plugins/sources/git.py: warn about checkout by default
See #783
-
4462892e
by Tristan Van Berkom
at 2018-12-03T11:45:07Z
Merge branch 'jjardon/doc_git_submodules' into 'master'
git: warn about checkout submodules by default
See merge request BuildStream/buildstream!983
-
6fcfa452
by Jonathan Maw
at 2018-12-03T13:21:05Z
_yamlcache.py: Use a project's junction name if present
This is required because if there are projects with the same name
loaded, that have elements in the same path, with the same contents,
they are considered the same, despite actually belonging to different
Projects.
This fixes buildstream issue #795
-
d0f9c724
by James Ennis
at 2018-12-03T13:21:05Z
tests/loader/junctions.py: Test that we can build a junction that is referred differently, twice.
Test required to close #795 - We should be able to build a project which refers
to the same junction twice, but with two different filenames.
-
f7581026
by Jürg Billeter
at 2018-12-03T16:27:03Z
Merge branch 'jonathan/fix-identical-element' into 'master'
_yamlcache.py: Use a project's junction name if present
Closes #795
See merge request BuildStream/buildstream!980
-
98c15463
by Tristan Van Berkom
at 2018-12-05T07:04:56Z
_scheduler/queues: Don't call update state outside of error handling harness
Commit 3fa79d8da, part of an initiative for caching of the failed builds,
introduced a call to Element._update_state() after a job completes and before
entering the error handling harness intended for handling plugin raised errors.
Element._update_state() can result in triggering plugin code to
run, so this is incorrect, and causes raised errors to crash BuildStream
if they happen here.
After analyzing the code, it appears that this additional call to
Element._update_state() is unneeded, and was only added because the
state needs to be updated for a failure as well as a success.
Instead, we now have the BuildQueue call Element._assemble_done()
unconditionally, regardless of whether the build was successful
or not, which has the same effect and also reads better.
In addition, added a FIXME comment that we are still conditionally
updating the artifact cache size from BuildQueue.done() only
if the build is successful, which is incorrect because failed builds
also increase the local artifact cache size - to fix this we need
to communicate the added artifact size through Element._assemble()
regardless of whether the build succeeded or failed.
-
5d77b871
by Tristan Van Berkom
at 2018-12-05T07:37:54Z
Merge branch 'tristan/refactor-queues-update-state' into 'master'
_scheduler/queues/queue.py: Don't call update state outside of error handling harness
See merge request BuildStream/buildstream!988
-
82513109
by Valentin David
at 2018-12-05T09:47:52Z
git source plugin: Track git tags and save them to reproduce a minimum shallow repository
Instead of tag information being fetched which can change with time,
they are tracked and saved in the projects.refs/.bst. Then we re-tag
automatically the closest tag so that `git describe` works and is
reproducible.
This new feature is opt-in with the new `track-tags` configuration,
and must be used to fix modules which are broken by our new policy
of omitting the `.git/` repository when staging git sources.
This fixes issue #487
-
ec7b62c3
by Valentin David
at 2018-12-05T09:50:15Z
NEWS: Add news about broken 'git describe' by default.
-
761e7570
by Tristan Van Berkom
at 2018-12-05T10:23:49Z
Merge branch 'valentindavid/git_describe_tracking' into 'master'
Track of git tags and save them to reproduce minimum shallow repository
Closes #487
See merge request BuildStream/buildstream!906
-
2f5966bd
by Raoul Hidalgo Charman
at 2018-12-05T10:37:10Z
platform.py: Add mapping of uname to OS-independent values
-
1ad35fcd
by Raoul Hidalgo Charman
at 2018-12-05T10:44:02Z
_config.py: Use os and architecture settings
In element, platform asks for host os and architecture to default to when
SandboxConfig is initialised.
This changes element cache keys so those have been updated in the tests.
-
9f75c274
by Raoul Hidalgo Charman
at 2018-12-05T10:44:02Z
optionarch.py: update to use same arch names as SandboxConfig
Also update tests to be consistent with this
-
a825096c
by Raoul Hidalgo Charman
at 2018-12-05T10:44:02Z
_sandboxremote.py: Add os and arch to command uploaded
These are contained within the platform field.
-
8e09a81c
by Raoul Hidalgo Charman
at 2018-12-05T10:44:02Z
optionos.py: Add new option for OS configuration
-
defec112
by Raoul Hidalgo Charman
at 2018-12-05T10:44:02Z
optionos.py: Add tests for OptionOS
Same format as OptionArch tests, but with OS's.
-
e5d8c7d8
by Raoul Hidalgo Charman
at 2018-12-05T10:44:02Z
_platform: Add checks for os and architecture
-
b0603fb0
by Raoul Hidalgo Charman
at 2018-12-05T10:44:02Z
Sandbox: use linux32 for x86-32 builds on x86-64 machines
o _platform/linux.py: Add linux32 flag to send to sandbox bwrap when the
build arch is x86-32 and the machines arch is x86-64 or similarly with
aarch32 and aarch64.
o sandbox/_sandboxbwrap.py: Use flag passed to start bwrap command with
linux32 if set.
-
f4d76996
by Raoul Hidalgo Charman
at 2018-12-05T14:58:49Z
format_project.rst: Update documentation for changes to OptionArch
-
81163a79
by Raoul Hidalgo Charman
at 2018-12-05T14:58:53Z
format_declaring.rst: Update documentation for sandbox options
-
23c27580
by Raoul Hidalgo Charman
at 2018-12-05T14:58:53Z
format_project.rst: Add docs for OptionOS
-
eb1ed410
by Jürg Billeter
at 2018-12-05T15:28:52Z
Merge branch 'raoul/775-execution-environment-reqs' into 'master'
Execution environment reqs
Closes #775
See merge request BuildStream/buildstream!969
-
9ef1a8a2
by Raoul Hidalgo Charman
at 2018-12-05T15:32:22Z
utils.py: Add message digest function
Get cascache.py to use this
-
4a56bf2a
by Raoul Hidalgo Charman
at 2018-12-05T15:33:44Z
_sandboxremote.py: Add checks to action cache before attempting to push.
Stops unneccesary pushing of builds that have already been built, just checks
the action cache to begin with.
Fixes #628
-
3641b6f6
by Raoul Hidalgo Charman
at 2018-12-05T15:33:44Z
_sandboxremote.py: Add action-cache-service endpoint
The endpoint may not necessarily be the same as execution-service endpoint.
Part of #628
-
9d77351f
by Raoul Hidalgo Charman
at 2018-12-05T15:33:44Z
format_project.rst: Add docs for action-cache-service
Part of #628
-
a53d6d1f
by Raoul Hidalgo Charman
at 2018-12-05T16:36:00Z
Merge branch 'raoul/628-RE-flow-optimisation' into 'master'
Remote-execution client flow optimisation
Closes #628
See merge request BuildStream/buildstream!982
-
4688b2c9
by Tristan Van Berkom
at 2018-12-06T07:35:59Z
tests/sources/git.py: Refactor ref-not-in-track test to use parameterization
Instead of duplicating the whole test body, run it twice while
checking for a warning or an error depending on the parameter.
-
03028c7f
by Tristan Van Berkom
at 2018-12-06T07:35:59Z
types.py: Moving CoreWarnings to the types.py module
This also ensures it is exposed via the main buildstream __init__.py
file, fixes the imports of CoreWarnings, and adjusts the documentation
links to point to the right place.
-
3b7f8df8
by Tristan Van Berkom
at 2018-12-06T07:35:59Z
Source: Fix typo in documentation, and make it a link to `bst track`
Now the commands have documentation linkability so we can link to
them when referring to them anywhere in the docs. This reference
was previously referring to a bogus non-existent `build-stream track`,
maybe this dates way back to the days before we named the frontend `bst`.
-
e5c0f067
by Tristan Van Berkom
at 2018-12-06T07:35:59Z
git source plugin: Fixing documentation linking and typo
Fixes the configurable warning part of this to properly link
to the project.conf documentation describing what configurable
warnings actually are.
Also fix the plugin raise the actual configurable warning which
it advertizes that it raises; which is 'inconsistent-submodule',
not 'inconsistent-submodules'.
-
205c27d8
by Tristan Van Berkom
at 2018-12-06T07:35:59Z
git source plugin: Rename symbol for consistency and readability
Change 'INCONSISTENT_SUBMODULE' for 'WARN_INCONSISTENT_SUBMODULE' to
make things a bit more readable and comprehensive.
-
bea4d4f5
by Tristan Van Berkom
at 2018-12-06T09:37:50Z
Merge branch 'tristan/misc-cleanup' into 'master'
Misc cleanups
See merge request BuildStream/buildstream!993
-
b1d2f001
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
source.py: Add new delegate method validate_cache()
This is guaranteed to be called only once for a given session once the
sources are known to be Consistency.CACHED, if source tracking is enabled
in the session for this source, then this will only be called if the
sources become cached after tracking completes.
-
ee7fc47f
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
git source plugin: Implementing submodule warnings
o Unlisted submodule warning
Now the git plugin will issue a configurable warning if a submodule
exists and is used (checking out the submodule is not disabled),
but is not specified in the source configuration.
o Invalid submodule warning
Now the git source plugin will issue a warning if the configuration
specified a submodule which does not exist in the underlying git repository.
As a side effect, this patch also changes the flow control of the git
plugin such that submodules which are explicitly set to not be
checked out, are also not fetched but instead ignored completely.
-
5fc9a1da
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
git source plugin: Emmit the ref-not-in-track warning from validate_cache()
Now that we have Source.validate_cache(), this is a better place to emmit
the ref-not-in-track warning, since it will be emmitted at the earliest
opportunity and not only at Source.stage() or Source.init_workspace().
This also allows us to remove the `self.tracked` local state, and cleanup
some convoluted calling paths, removing some unnecessary parameters from
the usual codepaths and making the plugin overall more readable.
-
50cb2706
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
tests/testutils/repo/git.py: Adding remove_path() helper
A function for removing paths from a git repo, can also
be used to remove submodules.
-
f139ca8f
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
tests/sources/git.py: Testing the git:unlisted-submodule warning
o Test that it is not triggered in show before fetch, because we
don't know about the unlisted submodules yet
o Test that it is triggered by a fetch command
o Test that it is triggered by `show` after having completed a
fetch command, since now we have the repository and know about
the unlisted submodule
o Test all of this under warning or error conditions (parameterized
for fatal-warnings)
-
afe1aa9f
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
tests/sources/git.py: Testing the git:invalid-submodule warning
o Test that it is not triggered in show before fetch, because we
don't know the submodules yet so we cannot know if they are
valid or not.
o Test that it is triggered by a fetch command
o Test that it is triggered by `show` after having completed a
fetch command, since now we have the repository and know which
specified submodules are invalid
o Test all of this under warning or error conditions (parameterized
for fatal-warnings)
-
57300201
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
tests/sources/git.py: Test unlisted submodules warning appearing after track
-
a0297625
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
tests/sources/git.py: Test invalid submodules warning appearing after track
-
60e96781
by Tristan Van Berkom
at 2018-12-06T15:14:37Z
Merge branch 'tristan/submodule-warnings' into 'master'
Implement submodule warnings
See merge request BuildStream/buildstream!996
-
7fe55143
by Jonathan Maw
at 2018-12-06T15:48:12Z
Fix bst source-checkout not working with open workspaces
-
2a6879a5
by Jonathan Maw
at 2018-12-06T17:46:33Z
Merge branch 'jonathan/source-checkout-workspace' into 'master'
Fix bst source-checkout not working with open workspaces
Closes #807
See merge request BuildStream/buildstream!997
-
e61e6e50
by Tristan Van Berkom
at 2018-12-07T07:57:13Z
_yaml.py: Added `allow_none` parameter to _yaml.node_get()
This allows specifying whether or not the code believes a None
value is acceptable if explicitly expressed to be None in the YAML.
A previous patch 3ba544b80f9f268be8ffe62fc8589b30212ec4a2 changed
the behavior to accept None for all values across the board, along
with allowing explicitly setting the `default_value` to `None` which
caused the code to be much more readable (this was the main motivation
of the patch, but it had the side effect of allowing None for everything).
In the majority of cases we load YAML however, it either has a value or
it is not specified in the YAML, and None is hardly ever acceptable
to be explicitly specified (it may be in the case you want to override
something with nothing in the YAML, but it is the edge case).
This had the side effect of causing unexpected crashes where the
YAML specifies an empty dictionary for instance.
Instead of forcing the caller to handle a possible None value, give
the choice to the caller if they really want to accept a None value,
and raise the LoadError otherwise.
This fixes issue #803, which is a crash due to receival of an
unexpected None value through _yaml.node_get()
-
504ecb3e
by Tristan Van Berkom
at 2018-12-07T07:57:56Z
plugin.py: Added new `allow_none` parameter to Plugin.node_get_member(), defaulting to False.
Allow plugins to conveniently leverage the new `allow_none` parameter,
and have Plugin.node_get_member() raise a LoadError in the usual
case where None is not an acceptable value to be explicitly set in
the user provided YAML.
-
642ae4e8
by Tristan Van Berkom
at 2018-12-07T07:57:56Z
tests/format/project.py: Added new regression test for empty dependency dictionaries
This adds a regression test for issue #803
-
2a0676c3
by Tristan Van Berkom
at 2018-12-07T08:33:14Z
Merge branch 'tristan/yaml-optionally-allow-none' into 'master'
Only optionally allow None values in user provided YAML
Closes #803
See merge request BuildStream/buildstream!999
-
56c2c2cc
by Javier Jardón
at 2018-12-09T16:59:26Z
buildstream/data/projectconfig.yaml: Remove default strip-binaries
They are too specific to be included by default
Recommendation is if you are building in Linux is to use the
ones begin used in the freedesktop-sdk project, for example
See #645
-
dcb6b5e8
by Javier Jardón
at 2018-12-09T16:59:26Z
NEWS: Update for removal of default strip-commands
-
ae0afce5
by Javier Jardón
at 2018-12-09T17:08:16Z
buildstream/buildelement.py: Document where to find strip commands for Linux
-
873b618c
by Javier Jardón
at 2018-12-10T09:49:52Z
Merge branch 'jjardon/strip-binaries-removal' into 'master'
Remove default strip-commands
Closes #645
See merge request BuildStream/buildstream!987
-
f2fcc2f6
by Valentin David
at 2018-12-11T12:12:08Z
Force updating tags when fetching git repository
When using aliases there are multiple remotes used in the cache
repository. When fetching, tags are not updated if the were previously
fetched from a different remote. Commits that not in a branch and only
tagged do not get fetched if the tag is not fetched.
Fixes #812
-
717c10d1
by Valentin David
at 2018-12-11T12:44:56Z
Merge branch 'valentindavid/git_force_fetch_tags' into 'master'
Force updating tags when fetching git repository
Closes #812
See merge request BuildStream/buildstream!1000
-
7892287a
by Jonathan Maw
at 2018-12-11T12:56:32Z
utils.py: Add a helper for searching upwards for files
i.e. with a given directory and filename, check parent directories until
either a directory with the filename is found, or you reach the root of
the filesystem.
This is a part of #222
-
67c7a58d
by Jonathan Maw
at 2018-12-11T12:56:32Z
Create and store data inside projects when opening workspaces
Changes to _context.py:
* Context has been extended to contain a WorkspaceProjectCache, as there
are times when we want to use it before a Workspaces can be initialised
(looking up a WorkspaceProject to find the directory that the project is
in)
Changes to _stream.py:
* Removed staging the elements from workspace_open() and workspace_reset()
Changes in _workspaces.py:
* A new WorkspaceProject contains all the information needed to refer back
to a project from its workspace (currently this is the project path and
the element used to create this workspace)
* This is stored within a new WorkspaceProjectCache object, which keeps
WorkspaceProjects around so they don't need to be loaded from disk
repeatedly.
* Workspaces has been extended to contain the WorkspaceProjectCache, and
will use it when opening and closing workspaces.
* Workspaces.create_workspace has been extended to handle the staging of
the element into the workspace, in addition to creating the equivalent
WorkspaceProject file.
This is a part of #222
-
64836b18
by Jonathan Maw
at 2018-12-11T12:56:32Z
_project.py: Rename _ensure_project_dir to _find_project_dir
This is a part of #222
-
7deeb2c3
by Jonathan Maw
at 2018-12-11T12:56:32Z
cli.py: Use utils' search upwards helper when searching for project.conf
This is a part of #222
-
496f0ab7
by Jonathan Maw
at 2018-12-11T12:56:32Z
_project.py: Find project from workspace if outside of a project
This is a part of #222
-
921f2bcb
by Jonathan Maw
at 2018-12-11T12:56:32Z
_workspaces.py: Do not include .bstproject.yaml in the cache key
This is a part of #222
-
f145a3e4
by Jonathan Maw
at 2018-12-11T12:56:32Z
cli: Interactively warn if the user is trying to close the workspace they're using to load the project
This involves changes in:
* _stream.py:
* Add the helper Stream.workspace_is_required()
* userconfig.yaml:
* Add a default value for prompt.really-workspace-close-project-inaccessible
* _context.py:
* Load the prompt 'really-workspace-close-project-inaccessible' from
user config.
* cli.py:
* If buildstream is invoked interactively, prompt the user to confirm
that they want to close the workspace they're using to load this
project.
This is a part of #222
-
494cb7c6
by Jonathan Maw
at 2018-12-11T14:12:54Z
tests: Test bst commands from an external workspace
This is a part of #222
-
fd1c5c5a
by Jonathan Maw
at 2018-12-11T14:12:54Z
NEWS: Add an entry for being able to run commands from a workspace
This is a part of #222
-
4219a6b4
by Jonathan Maw
at 2018-12-11T16:00:55Z
Merge branch 'jonathan/workspace-fragment-create' into 'master'
Support invoking buildstream from a workspace outside a project
See merge request BuildStream/buildstream!924
-
e564251b
by Jonathan Maw
at 2018-12-11T16:15:19Z
_stream.py: Fix ugly error when opening a workspace using a relative path
-
7cf83343
by Jonathan Maw
at 2018-12-11T16:15:19Z
_context.py: Store the directory buildstream was invoked on
-
a1dee91e
by Jonathan Maw
at 2018-12-11T16:15:19Z
Make specifying elements optional in bst commands
Known issues:
* `bst shell` works, but `bst shell COMMANDS...` doesn't, because click
has no way of separating optional args from variable-length args.
* `bst checkout` and `bst source-checkout`'s usage strings mark LOCATION
as an optional argument. Because click gets confused if there's an
optional argument before a mandatory argument, I had to mark LOCATION
as optional internally.
* `bst workspace open` makes no sense with element being optional, so
I skipped it.
* `bst workspace close` will probably need to be revisited when multiple
projects can own one workspace.
* `bst workspace reset` will happily delete the directory you're
currently in, requiring you to `cd $PWD` to see the contents of your
directory.
I could exclude the top-level directory of the workspace being
deleted, but it is entirely valid to run workspace commands from deeper
in the workspace.
This is a part of #222
-
9b192015
by Jonathan Maw
at 2018-12-11T16:15:19Z
tests: Add tests for guessing element names
This is a part of #222
-
e340371f
by Jonathan Maw
at 2018-12-11T16:15:19Z
Add NEWS for guessing the element in commands
This is a part of #222
-
bbf81c3d
by Jonathan Maw
at 2018-12-11T16:15:19Z
doc: Point out that element is optional when commands are run from the workspace
This is a part of #222
-
a5a53ddd
by Jonathan Maw
at 2018-12-12T13:44:09Z
Merge branch 'jonathan/workspace-fragment-guess-element' into 'master'
Guess the element when running commands in a workspace
See merge request BuildStream/buildstream!954
-
d55b9e39
by Phil Dawson
at 2018-12-12T13:55:19Z
Add --tar option to source-checkout command
This commit is part of the work towards #672
-
733aab53
by Phil Dawson
at 2018-12-12T13:55:19Z
Add --force / -f option to source-checkout command
-
c2efeba0
by Phil Dawson
at 2018-12-12T14:43:40Z
Add option to source-checkout command to generate build scripts
-
85c61894
by Phil Dawson
at 2018-12-12T14:43:40Z
Remove source bundle command
This is part of the work towards #672
-
a322d5c0
by Phil Dawson
at 2018-12-12T14:43:40Z
NEWS: Add entry about the removal of source-bundle command
-
ec909605
by Phil Dawson
at 2018-12-12T15:45:42Z
Merge branch 'phil/source-checkout-options' into 'master'
Retire bst source bundle command
Closes #672
See merge request BuildStream/buildstream!959
-
3697a611
by Richard Maw
at 2018-12-12T16:31:38Z
cli: Add support for auto-completing artifact ref names
-
b3dceb16
by Richard Maw
at 2018-12-12T16:32:41Z
cli: Add artifact command group
-
ba08a0cd
by Richard Maw
at 2018-12-12T16:32:41Z
cli: Add artifact log command
-
70fb9554
by Richard Maw
at 2018-12-12T16:32:41Z
tests: Add tests for artifact group commands
-
f773e746
by Richard Maw
at 2018-12-12T16:33:02Z
NEWS: Mention `bst artifact log`
-
b6528441
by richardmaw-codethink
at 2018-12-12T18:00:59Z
Merge branch 'richardmaw/artifact-log' into 'master'
Add artifact log command
See merge request BuildStream/buildstream!920
-
d03bf316
by Benjamin Schubert
at 2018-12-13T10:24:11Z
Mock storage space checks for tests.
Fix #530
- Extract free space computation in a function for easier
mocking
- Mock space computation during cache quota tests
- Mock cache size during cachque quota tests
- Add two more tests when the configuration would require
to much storage space
-
a116f576
by Tristan Van Berkom
at 2018-12-13T10:58:46Z
Merge branch 'BenjaminSchubert/fix-quota-tests' into 'master'
Mock storage space checks for tests.
Closes #530
See merge request BuildStream/buildstream!702
-
180fa774
by Benjamin Schubert
at 2018-12-13T12:05:15Z
element.py: fix unbounded variable in nested python exception
As per https://docs.python.org/3.7/reference/compound_stmts.html#except
variables used in the "except" clause are erased at the end of the
block.
We had a conflict with a nested exception using the same variable as
before.
This renames the nested variable to avoid the clash
-
224aa4c2
by Benjamin Schubert
at 2018-12-13T12:34:41Z
Merge branch 'bschubert/fix-unbound-variable-exception' into 'master'
element.py: fix unbounded variable in nested python exception
See merge request BuildStream/buildstream!1006
-
053beb66
by Tristan Van Berkom
at 2018-12-13T14:23:19Z
manual plugin: Support virtual directories
-
29ab271c
by Tristan Van Berkom
at 2018-12-13T14:23:19Z
makemaker plugin: Support virtual directories
-
ba955cf0
by Tristan Van Berkom
at 2018-12-13T14:23:19Z
pip element plugin: Support virtual directories
-
6010b5a4
by Tristan Van Berkom
at 2018-12-13T14:23:19Z
modulebuild plugin: Support virtual directories
-
3a6d27a4
by Tristan Van Berkom
at 2018-12-13T14:23:19Z
distutils plugin: Support virtual directories
-
4c0e602c
by Tristan Van Berkom
at 2018-12-13T14:23:19Z
BuildElement: Don't enable batching of prepare and assemble by default
Some external plugins depend derive from BuildElement and are broken
by BuildElement enabling this batching by default.
Instead, enable it in all of the individual build element plugin
implementations.
This fixes issue #800
-
60ddeeb9
by Tristan Van Berkom
at 2018-12-13T14:58:28Z
Merge branch 'tristan/dont-batch-prepare-assemble-by-default' into 'master'
Dont batch prepare assemble by default
Closes #800
See merge request BuildStream/buildstream!1009
-
bf72cc42
by Angelos Evripiotis
at 2018-12-13T17:31:17Z
contributing: mandate end-to-end tests, allow others
Update the section on adding tests, to fulfill these broad points:
- Mandate end-to-end testing, with rationale.
- Mention internal APIs as endpoints for testing.
- Warn against pitfalls of unit-testing.
It's more text than I would have liked, perhaps in later work we'll be
able to say it with less.
This change integrates feedback from the mailing list
'Guidance on Unit Tests' thread:
https://mail.gnome.org/archives/buildstream-list/2018-November/msg00045.html
-
13eb7ed2
by Angelos Evripiotis
at 2018-12-13T18:01:56Z
Merge branch 'aevri/contributing_e2e_tests' into 'master'
contributing: more clarity on testing
See merge request BuildStream/buildstream!973
-
629a6e52
by Chandan Singh
at 2018-12-14T19:34:20Z
Introduce new "source" command group
Following the message thread
https://mail.gnome.org/archives/buildstream-list/2018-November/msg00106.html,
implement a new command group called `source`. Move existing `track`,
`fetch`, and the recently added `source-checkout` commands under this
group.
For `track` and `fetch`, this is a BREAKING change, as the old commands
have been marked as obsolete. Using them will result in an error message
that refers people to use the new versions, like `bst source fetch`
instead of old `bst fetch`. `source-checkout` will now become
`source checkout` (the dash has turned into a space), and is not a
breaking change as it was added in the current development cycle.
Note that the functionality to hide commands from help output was added
only recently in Click, so the minimum version of Click that we now
require is 7.0.
Summary of changes:
* _frontend/cli.py: Add `source` command group, mark previous versions
as obsolete and hide them from the help output.
* _frontend/complete.py: Fix completion for hidden commands.
* setup.py: Bump Click minimum version to 7.0.
* tests: Update to cope with the new command names.
Fixes #814.
-
f894c0a8
by Chandan Singh
at 2018-12-14T19:34:20Z
NEWS: Add entry for new source command group
Also, change the existing entry about `source-checkout` to be
`source checkout` instead. And, while we are there, move it near the
other announcements about the source command group.
-
b23bec55
by Chandan Singh
at 2018-12-14T20:07:13Z
Merge branch 'chandan/source-subgroup' into 'master'
Introduce new "source" command group
Closes #814
See merge request BuildStream/buildstream!1003
-
d2105909
by Raoul Hidalgo Charman
at 2018-12-18T11:13:56Z
Add remote execution instance option
This is used when sending execution requests, to specify which instance of the
execution server to use.
Partial fix for #627.
-
a3bbec23
by Jim MacArthur
at 2018-12-18T11:13:56Z
sandboxremote: Add server/storage config with defaults
Adds instance name support for the remote execution storage service.
-
89219f61
by Jim MacArthur
at 2018-12-18T11:13:56Z
_cascache.py: Add instance names to GRPC calls
Adds the 'instance_name' parameter, which may be None, to most GRPC
calls in the CASCache object. ByteStream requests already have
instance_name supplied in the resource name, so do not need the
parameter.
Closes #627.
-
3dc20963
by Jim MacArthur
at 2018-12-18T11:13:56Z
Documentation: Include instance-name in remote execution documentation
-
644d8b28
by Jim MacArthur
at 2018-12-18T11:45:30Z
Merge branch 'raoul/627-RE-instance-config' into 'master'
Remote-execution instance configuration support
Closes #627
See merge request BuildStream/buildstream!952
-
e29aea36
by William Salmon
at 2018-12-19T13:23:19Z
Basic options for shell --build to use buildtrees
Fixes issue #740
-
898a23a5
by Will Salmon
at 2018-12-19T14:35:41Z
Merge branch 'willsalmon/shellBuildTrees' into 'master'
Shell --build has optional buildtrees
Closes #740
See merge request BuildStream/buildstream!986
-
a2f1d879
by Javier Jardón
at 2018-12-19T15:36:11Z
README.rst: Add license badge
-
aae5e4b3
by Javier Jardón
at 2018-12-19T16:09:48Z
Merge branch 'jjardon/license_badge' into 'master'
README.rst: Add license badge
See merge request BuildStream/buildstream!1014
-
2b767fe8
by Jürg Billeter
at 2018-12-20T10:06:11Z
Move fetch logic from FetchQueue to Element
The queue shouldn't need to know about individual sources. This is in
line with _track() and _get_consistency().
-
7a102144
by Jürg Billeter
at 2018-12-20T10:06:11Z
element.py: Do not call fetch() for cached sources
-
b325989e
by Jürg Billeter
at 2018-12-20T10:07:20Z
tests/sources: Test that fetch() is not called for cached sources
-
77d8ad45
by Jürg Billeter
at 2018-12-20T10:42:39Z
Merge branch 'juerg/fetch' into 'master'
Do not call fetch() for cached sources
See merge request BuildStream/buildstream!992
-
9835b7f1
by Chandan Singh
at 2018-12-20T12:50:02Z
Expose base class for Git source plugins
Add a `_GitSourceBase` class to act as the base class for deriving
source plugins that work with Git. The first user of this base class is
the `git` source plugin that's in core at the moment.
The contents of this base class is almost identical to the existing `GitSource`
class. The two notable differences are:
- the private methods now have leading underscores, as some were missing
it originally
- the name of the class
Note that we are exposing a private member here as we expect it to move to a
separate package soon. See the following discussion for more details:
https://gitlab.com/BuildStream/buildstream/issues/739#note_124819869
Fixes #739.
-
14da6955
by Chandan Singh
at 2018-12-20T13:34:10Z
Merge branch 'chandan/abstract-git-source' into 'master'
Expose base class for Git source plugins
Closes #739
See merge request BuildStream/buildstream!1019
-
7368f569
by Angelos Evripiotis
at 2018-12-20T13:59:23Z
_project::_find_project_dir: fix error message
Now that we are also looking for WORKSPACE_PROJECT_FILE, we should add
it to the error message. While we're there, also mention the directory
we are looking in - this could help scripters be clear about what went
wrong.
The new error message looks like this:
Error loading project: None of ['project.conf', '.bstproject.yaml']
found in '/src/temp/blah' or any of its parent directories
-
46efc91d
by Angelos Evripiotis
at 2018-12-20T13:59:23Z
BREAK: remove auto-init behaviour
In the event that the project could not be found, stop BuildStream from
asking if the user would like to create a new project. Exit with error
instead, and give a hint to the user in case they're new.
As proposed on the mailing list here:
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00082.html
The new interaction looks like this:
$ bst show nonsuch.bst
No project found. You can create a new project like so:
bst init
Error loading project: None of ['project.conf', '.bstproject.yaml']
found in '/src/temp/blah' or any of its parent directories
Fixes #826
-
e0c575c4
by Angelos Evripiotis
at 2018-12-20T14:37:38Z
Merge branch 'aevri/rm-autoinit' into 'master'
BREAK: remove auto-init behaviour
Closes #826
See merge request BuildStream/buildstream!1015
-
c05d8b4f
by Chandan Singh
at 2018-12-20T15:53:12Z
tests/frontend/logging.py: Fix regex Deprecation Warning
Use raw strings for regex searches, which is the preferred way to do
regular expressions in Python.
Without this patch, currently we get the following warnings:
```
tests/frontend/logging.py:44
/builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:44: DeprecationWarning: invalid escape sequence \[
m = re.search("\[\d\d:\d\d:\d\d\]\[\]\[\] SUCCESS Checking sources", result.stderr)
tests/frontend/logging.py:80
/builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:80: DeprecationWarning: invalid escape sequence \d
m = re.search("\d\d:\d\d:\d\d,\d\d:\d\d:\d\d.\d{6},\d\d:\d\d:\d\d,,,SUCCESS,Checking sources", result.stderr)
```
-
e8055a56
by Chandan Singh
at 2018-12-20T15:53:12Z
buildstream/utils.py: Fix regex Deprecation Warning
Specify flags at the start of the _expression_ as per the recommendation
of the standard library.
Without this patch, we currently get the following warning:
```
tests/examples/junctions.py::test_open_cross_junction_workspace
/builds/BuildStream/buildstream/dist/buildstream/buildstream/utils.py:213: DeprecationWarning: Flags not at the start of the _expression_ '\\/[^/]*\\Z(?ms)'
regexer = re.compile(_expression_)
```
-
cd4889af
by Chandan Singh
at 2018-12-20T16:34:08Z
Merge branch 'chandan/fix-warning' into 'master'
Fix Deprecation warnings from regex module
See merge request BuildStream/buildstream!1010
-
ac995236
by Tom Pollard
at 2018-12-20T17:18:02Z
_stream.py: fix _buildtree_pull_required logic
-
c3153dea
by Tom Pollard
at 2018-12-21T10:10:58Z
Merge branch 'tpollard/fixbuildtreereq' into 'master'
_stream.py: fix _buildtree_pull_required logic
See merge request BuildStream/buildstream!1021
-
caf49669
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
dev-requirements.txt: Require at least pytest 3.9
Recently we have been adding some tests which use the new tmp_path
fixture from pytest, this is new in 3.9.
-
425781db
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
_signals.py: Fixing new (simplifiable-if-_expression_) linter errors
-
1f8cc996
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
element.py: Fixing new (unnecessary-pass) and (simplifiable-if-_expression_) linter errors
-
89632150
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
source.py: Fixing new (unnecessary-pass) linter error
-
26d135ba
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
_artifactcache/casserver.py: Fixed new (wrong-import-order) linter error
-
d0988f80
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
_frontend/cli.py: Fixing new (unnecessary-pass) linter errors
-
4d54ab78
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
_frontend/complete.py: Fixing new (simplifiable-if-_expression_) linter error
-
ee0d5bf6
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
_frontend/widget.py: Fixing new (wrong-import-order) linter error
-
36295282
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
plugins/elements/import.py: Fixing new (duplicate-string-formatting-argument) linter error
-
4ffe03c6
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
buildstream/storage/_casbaseddirectory.py: Fixing new (unnecessary-pass) linter errors
-
80c137d5
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
dev-requirements.txt: Require new version of pytest-datafiles
Since Thomas Nixon fixed the datafiles plugin to be compatible
with pytest >= 3.8.0, we can now depend on it and avoid the
warnings and errors which the older datafiles plugin causes.
This fixes issue #636
-
61906a14
by Tristan Van Berkom
at 2018-12-26T19:54:05Z
dev-requirements.txt: Remove the restriction on pylint.
After having fixed the more recent linting errors, the
tests should pass with recent versions of pylint. Tested
with pylint 2.2.2.
-
94b9948c
by Tristan Van Berkom
at 2018-12-26T22:35:29Z
Merge branch 'tristan/test-fixes' into 'master'
Misc fixes to the tests and newer linter errors
Closes #636
See merge request BuildStream/buildstream!1011
-
fe4fc0b5
by Tristan Van Berkom
at 2018-12-26T23:19:07Z
_downloadablefilesource.py: Avoid crashes when HOME is unset
The python netrc module will raise OSError in the case
that HOME is not set, this was discovered while running
tests under tox.
-
c61e1793
by Tristan Van Berkom
at 2018-12-26T23:19:07Z
tests/testutils/runcli.py: Fixed broken environment handling
Treat None values in the passed dictionary as keys to be removed
from the environment, this was already happening at restoration time.
-
50c1bb7f
by Tristan Van Berkom
at 2018-12-26T23:19:07Z
tests/sources/tar.py: Test that we don't crash when HOME is unset
This happens when the netrc module is searching for a ~/.netrc
file and it doesnt find any HOME set.
-
32a101f6
by Tristan Van Berkom
at 2018-12-26T23:53:29Z
Merge branch 'tristan/fix-netrc-crasher' into 'master'
Fix netrc crasher
See merge request BuildStream/buildstream!1025
-
a080b6c7
by Tristan Van Berkom
at 2018-12-31T19:03:25Z
tests/loader/junctions.py: Use Result checking APIs instead of manually
Better to use the same API everywhere for thecking CLI results.
-
d7c8b54d
by Tristan Van Berkom
at 2018-12-31T19:14:16Z
tests/loader/junctions.py: Test the error we expect when a cross junction element is not found
-
560a6342
by Tristan Van Berkom
at 2018-12-31T19:49:24Z
Merge branch 'tristan/junction-tests' into 'master'
Junction test addition
See merge request BuildStream/buildstream!1029
-
8a020a5a
by Chandan Singh
at 2018-12-31T19:54:25Z
_loader/loader.py: Refactor warnings about element names in one method
Currently we some duplication in the way we check for invalid filenames.
To make it more robust and allow room for adding more warnings, refactor
it into a separate method `_warn_invalid_elements()` that handles just
this.
-
87568074
by Chandan Singh
at 2018-12-31T19:54:25Z
_loader/loader.py: Add warnings about invalid characters in filename
See https://mail.gnome.org/archives/buildstream-list/2018-December/msg00061.html
for some related discussion.
-
4b2992c1
by Chandan Singh
at 2018-12-31T19:54:25Z
tests/buildcheckout.py: Add tests for invalid chars in filename
-
ce9e9c2d
by Chandan Singh
at 2018-12-31T19:54:25Z
doc: Add note about valid element names
-
b848172c
by Tristan Van Berkom
at 2018-12-31T20:29:51Z
Merge branch 'chandan/element-filename-requirements' into 'master'
Add warnings about invalid characters in filename
See merge request BuildStream/buildstream!1028
-
bb9df2a1
by Chandan Singh
at 2019-01-02T11:59:35Z
doc/Makefile: Update comment about sphinx entrypoint
https://github.com/sphinx-doc/sphinx/issues/4375 has been closed as
`wontfix`. Update the comment in doc/Makefile that mentions it, based on
the discussion on the issue.
-
43eaf6ee
by Chandan Singh
at 2019-01-02T13:42:10Z
Merge branch 'chandan/update-doc-makefile-note' into 'master'
doc/Makefile: Update comment about sphinx entrypoint
See merge request BuildStream/buildstream!1026
-
3b5c8a28
by Tristan Maat
at 2019-01-02T15:57:13Z
.gitlab-ci.yml: Use new testsuite images with pycodestyle 1.4.0
We need to bump the testsuite image versions to use the new
pycodestyle update that allows excluding file paths, so that we can
exclude generated python code.
-
bd60e8a9
by Javier Jardón
at 2019-01-02T16:59:41Z
Use pycodestyle instead of pep8
pep8 package was renamed to pycodestyle to reduce confusion.
Disabled warnings:
W504 - Line break after binary operator (seems people like this)
W605 - Invalid escape sequence (some of our regexes use these)
-
32c47d1c
by Tristan Maat
at 2019-01-02T17:32:09Z
Merge branch 'jjardon/pycodestyle' into 'master'
Use pycodestyle instead pep8 python module
See merge request BuildStream/buildstream!638
-
1cbd9a73
by Chandan Singh
at 2019-01-02T19:39:58Z
Add requirements files for install, test and plugin dependencies
Add `.in` and `.txt` requirements files for BuildStream's pure python
dependencies. For each pair, the `.in` file is supposed to capture the
loose version requirements, and the corresponding `.txt` file is
supposed to have frozen requirements. We have 3 such sets:
* `requirements`: BuildStream's runtime dependencies
* `dev-requirements`: Dependencies for running tests
* `plugin-requirements`: Dependencies for core plugins
Note that the frozen requirements files will only be used for testing
purposes, and `setup.py` will continue to read loose requirements.
-
6b0cb5f3
by Chandan Singh
at 2019-01-02T19:41:21Z
Add tox.ini to enable running tests using tox
Add `tox.ini` file that will enable us to use
[tox](https://tox.readthedocs.io/) as a frontend for running tests.
Since we share the config via `setup.cfg` and requirements via
requirements files, commands like `python3 setup.py test` will continue
to work.
-
9d2d1d4f
by Chandan Singh
at 2019-01-03T03:31:52Z
.gitlab-ci.yml: Run tests using tox
Instead of invoking tests throung `setup.py`, use `tox` as a frontend in
the CI pipelines.
-
8ae04283
by Tristan Van Berkom
at 2019-01-03T03:31:52Z
CONTRIBUTING.rst: Updated to reflect running tests using tox.
-
e7e2a5c1
by Chandan Singh
at 2019-01-03T03:31:52Z
CONTRIBUTING.rst: Add steps for installing non-python build dependencies
Since we don't allow use of site packages by default in our `tox`
configuration, people will need to install non-python build dependencies
of some of our dependencies that do not provide pre-built wheels. We
have two such packages at the moment:
* psuitl: requires python C headers, compiler
* pygobject: requires python C headers, pkg-config, compiler, cairo
headers, and gobject libraries
-
afa0a369
by Chandan Singh
at 2019-01-03T03:31:52Z
Move all requirements files into "tools" directory
These new `.in` and `.txt` are making the repository look very
cluttered. Move them to a separate `tools` directory to make it look a
bit cleaner.
-
3fae3446
by Chandan Singh
at 2019-01-03T03:44:17Z
Move sphinx build functionality to tox
Currently the CI and the docs both have to duplicate the same inforation
about how to gather dependencies etc, and have to use hacky ways to run
them.
Add a new `docs` environment to our tox setup so that building docs is
as simple as running `tox -e docs`.
-
1f7bc655
by Chandan Singh
at 2019-01-03T03:44:17Z
CONTRIBUTING.rst: Add instructions to recreate tox environment
When we update our dependencies, developers will eventually need to
recreate their `tox` environments to get correct results. This happens
because `tox` isn't particularly good at recognizing changes in the
requirements files.
-
669b55b0
by Tristan Van Berkom
at 2019-01-03T15:47:27Z
Merge branch 'chandan/toxify' into 'master'
Use tox as a frontend for running tests
See merge request BuildStream/buildstream!1027
-
ca547f19
by Chandan Singh
at 2019-01-03T19:34:49Z
tests/frontend/buildcheckout.py: Fix bad filename issue for Windows
In BuildStream/buildstream!1028, we added a test specifically to test
that BuildStream correctly raises an warning when the name of an element
contains characters that are invalid on Windows. Unfortunately, we
didn't see it coming that it would make it impossible to checkout this
branch on Windows.
Fix it by generating this file, only if we are not running on Windows.
* tests/testutils/site.py: Add `IS_WINDOWS` check
* tests/frontend/buildcheckout.py: Generate file with invalid filename
on the fly
* Remove tests/frontend/project/elements/invalid-chars.
Fixes #842.
Note that this may still cause issues on WSL when running tests on a
shared filesystem, but that seems to be a generic issue on WSL with
`os.rename`.
-
01c3761c
by Chandan Singh
at 2019-01-03T20:03:32Z
Merge branch 'chandan/fix-bad-filename-mini-disaster' into 'master'
tests/frontend/buildcheckout.py: Fix bad filename issue for Windows
Closes #842
See merge request BuildStream/buildstream!1032
-
7850e4e2
by Tristan Van Berkom
at 2019-01-03T20:55:00Z
setup.cfg: Specify the pycodestyle configurations separately from pytest configuration
-
3c9d592d
by Tristan Van Berkom
at 2019-01-03T20:55:00Z
setup.cfg: Don't lint by default through setup.py test invocations
-
feba48e7
by Tristan Van Berkom
at 2019-01-03T20:55:00Z
tox.ini: Added new 'lint' environment to lint separately
-
9caa0ea0
by Tristan Van Berkom
at 2019-01-03T20:55:00Z
.gitlab-ci.yml: Update the CI to run the linter separately through tox
This should save us some cycles in CI, and also allow the
developer to more conveniently lint separately from testing.
-
91c8b634
by Tristan Van Berkom
at 2019-01-03T21:04:49Z
CONTRIBUTING.rst: Adding instructions to run the linter separately
-
892bf99e
by Tristan Van Berkom
at 2019-01-03T21:31:04Z
tools/requirements.{in,txt}: Fix warning about duplicate jinja
Apparently you need to spell Jinja2 with a capital J, otherwise
we find it listed twice in the resulting requirements.txt files.
-
7c681c42
by Tristan Van Berkom
at 2019-01-03T21:32:40Z
tools/dev-requirements.{in,txt}: Removed requirements on pylint/codestyle pytest plugins
We no longer run the linters through pytest, so we don't need these
plugins anymore.
-
d6689ed2
by Tristan Van Berkom
at 2019-01-03T22:09:02Z
Merge branch 'tristan/toxic-linting' into 'master'
Run the linter separately from the tests
See merge request BuildStream/buildstream!1033
-
e15278d7
by Thomas Coldrick
at 2019-01-03T22:14:01Z
Use collections.abc for Mapping, Iterable
In _yaml.py and _frontend/complete.py we were getting pylint warnings
for using collections.Mapping and collections.Iterable, which are
abstract classes now provided from collections.abc. This patch just
uses the classes from the right place.
-
a85da591
by Tristan Van Berkom
at 2019-01-03T22:36:16Z
Merge branch 'coldtom/collections' into 'master'
Use collections.abc for Mapping, Iterable
Closes #831
See merge request BuildStream/buildstream!1020
-
cf0e7d26
by Jürg Billeter
at 2019-01-04T07:07:59Z
_frontend/cli.py: Reinstate support for guessing targets
Moving fetch and track to the source command group accidentally dropped
the support for guessing targets for these commands when invoked from a
workspace directory. This brings it back.
-
0751fc8f
by Jürg Billeter
at 2019-01-04T11:27:58Z
Merge branch 'juerg/source-guess-target' into 'master'
_frontend/cli.py: Reinstate support for guessing targets
See merge request BuildStream/buildstream!1036
-
ebd965fe
by Chandan Singh
at 2019-01-04T15:21:35Z
Rename "tools" directory to "requirements"
This seems like a better name for the directory, as it more closely
describes the purpose of its contents.
-
bb712886
by Chandan Singh
at 2019-01-04T15:21:35Z
Add Makefile to update requirements files
-
b9792837
by Chandan Singh
at 2019-01-04T15:21:35Z
CONTRIBUTING.rst: Add instructions to update requirements files
Split the "The MANIFEST.in and setup.py" section in two:
"Managing data files" and "Updating BuildStream's Python dependencies".
Briefly explain the layout of `requirements` directory and add
instructions to use the Makefile added in the last commit.
-
45692335
by Tristan Van Berkom
at 2019-01-04T15:50:14Z
Merge branch 'chandan/update-requirements-one-liner' into 'master'
Add Makefile to update requirements files
See merge request BuildStream/buildstream!1035
-
f42dcb54
by Chandan Singh
at 2019-01-04T17:16:43Z
.gitlab-ci.yml: Remove prepare stage
As we now run tests using `tox`, we don't need to worry about manually
packing and unpacking BuildStream. So, we can remove the preapre stage
entirely.
Update `coverage` and nightly jobs to appropriately cope with this
change. Both these jobs now install all runtime dependencies from
requirements files.
-
ee29f207
by Chandan Singh
at 2019-01-04T17:41:02Z
Merge branch 'chandan/no-sdist' into 'master'
.gitlab-ci.yml: Remove prepare stage
See merge request BuildStream/buildstream!1037
-
f122cfe8
by Jürg Billeter
at 2019-01-06T17:57:33Z
_frontend/app.py: Use buildtree for interactive shell on build failure
Fixes: e29aea36 ("Basic options for shell --build to use buildtrees")
-
7dfb85b3
by Tristan Van Berkom
at 2019-01-06T18:31:25Z
Merge branch 'juerg/shell-buildtree' into 'master'
_frontend/app.py: Use buildtree for interactive shell on build failure
See merge request BuildStream/buildstream!1039
-
891fcb0e
by Tristan Van Berkom
at 2019-01-07T16:47:01Z
Fix stack traces discovered with ^C forceful termination.
* utils.py:_kill_process_tree(): Ignore NoSuchProcess errors
These are caused because we issue SIGTERM, and if the process
has not exited after a timeout, we kill it.
* _scheduler/jobs/job.py: Stop handling NoSuchProcess errors here
redundantly, they are already ignored.
It seems that we were ignoring it after sleeping when terminating
tasks from the scheduler... but we were not ignoring it when performing
the same pattern in the `Plugin.call()` -> `utils._call()` path, so
we were still getting these exceptions at termination time from host
tool processes launched by source plugins.
-
5de42d43
by Tristan Van Berkom
at 2019-01-07T18:00:37Z
Merge branch 'tristan/keyboard-interrupt-stack-trace' into 'master'
Fix stack traces discovered with ^C forceful termination.
See merge request BuildStream/buildstream!1043
-
059035b9
by Tristan Van Berkom
at 2019-01-07T18:02:00Z
_scheduler/scheduler.py: Make _schedule_jobs() private
This is not used anywhere outside of the Scheduler, currently
only the Scheduler itself is allowed to queue a job at this level.
If the highlevel business logic for automatic queueing of auxiliary
jobs moves to another location, we can make this public again.
-
b83d1b1f
by Tristan Van Berkom
at 2019-01-07T18:02:00Z
_scheduler/scheduler.py: Only run one cache size job at a time
When queuing the special cache management related cleanup and
cache size jobs, we now treat these jobs as special and do the
following:
* Avoid queueing a cleanup/cache_size job if one is already queued
We just drop redundantly queued jobs here.
* Ensure that jobs of this type only run one at a time
This could have been done with the Resources mechanics,
however as these special jobs have the same properties and
are basically owned by the Scheduler, it seemed more straight
forward to handle the behaviors of these special jobs together.
This fixes issue #753
-
16a8816f
by Tristan Van Berkom
at 2019-01-07T18:02:00Z
Scheduler: Introduced JobStatus instead of simple success boolean
This changes the deepest callback from when a Job completes to
propagate a JobStatus value instead of a simple boolean, and updates
all of the effected code paths which used to receive a boolean
to now handle the JobStatus values.
This further improves the situation for issue #753, as now we avoid
queueing cache size jobs for pull jobs which are skipped.
-
c2fc2a5e
by Tristan Van Berkom
at 2019-01-07T18:02:00Z
_scheduler/jobs/job.py: Removed 'skipped' property
This is redundant now that we report it through the JobStatus.
-
3e3984ad
by Tristan Van Berkom
at 2019-01-07T18:50:23Z
Merge branch 'tristan/one-cache-size-job' into 'master'
Only queue one cache size job
Closes #753
See merge request BuildStream/buildstream!1040
-
512c726e
by Tristan Van Berkom
at 2019-01-08T03:38:11Z
sandbox/sandbox.py: Fix regression of command logging
Since we added batch commands, the batch commands print the
text of the commands directly in the message text, but this is wrong.
The detail string is the appropriate place for text of unknown lengths
(the user can actually configure how many max lines of commands they
want to see in their log), the message text itself should be controlled
and brief enough to avoid text wrapping.
-
01171988
by Tristan Van Berkom
at 2019-01-08T04:20:14Z
Merge branch 'tristan/fix-command-status-messages' into 'master'
sandbox/sandbox.py: Fix regression of command logging
See merge request BuildStream/buildstream!1044
-
6c1d06d6
by Phil Dawson
at 2019-01-08T10:24:32Z
element.py: remove reference to source bundle command
This command has been replacved by the bst source checkout command
-
914ecb72
by Jürg Billeter
at 2019-01-08T10:54:02Z
Merge branch 'phil/remove-source-bundle-reference' into 'master'
element.py: remove documentation reference to source bundle command
See merge request BuildStream/buildstream!1041
-
81f7a964
by Tristan Van Berkom
at 2019-01-08T16:55:58Z
sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.
When code is faulty, in a plugin or in the core, we should
get a stack trace and a BUG message.
-
edcc43ed
by Tristan Van Berkom
at 2019-01-08T19:19:10Z
Merge branch 'tristan/sandbox-programming-error' into 'master'
sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.
See merge request BuildStream/buildstream!1046
-
17c5ca2d
by Valentin David
at 2019-01-09T10:45:23Z
Use relative path to project directory for remote execution certificates/keys
-
605f8d11
by Valentin David
at 2019-01-09T10:45:23Z
Add support for user remote execution configuration
Fixes #631.
-
65ea03ab
by Valentin David
at 2019-01-09T10:45:23Z
Add support for https channel to remote execution and actions servers
Fixes #780.
-
ba05c4fa
by Valentin David
at 2019-01-09T12:58:59Z
Merge branch 'valentindavid/remote_execution_configuration' into 'master'
Remote execution configuration: HTTPS and user configuration
Closes #780 and #631
See merge request BuildStream/buildstream!1030
-
eb0dbcfc
by Valentin David
at 2019-01-09T13:56:58Z
buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY
ScriptElement does not use Sandbox.get_directory. It works using it
with remote execution.
Fixes #850
-
f7681925
by Valentin David
at 2019-01-09T14:44:22Z
Merge branch 'valentindavid/script_virtual_directory' into 'master'
buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY
Closes #850
See merge request BuildStream/buildstream!1047
-
f87d1c93
by Jim MacArthur
at 2019-01-09T15:01:18Z
sandboxremote.py: Rename 'instance_name' option to 'instance-name'
It was 'instance-name' in the documentation.
-
ed8bc5bc
by Jim MacArthur
at 2019-01-09T15:30:28Z
Merge branch 'jmac/rename_instance_name' into 'master'
Rename 'instance_name' option to 'instance-name'
See merge request BuildStream/buildstream!1048
-
f29a0995
by Tristan Van Berkom
at 2019-01-09T19:51:14Z
Support running test environments in parallel with `detox`
This patch namespaces the test temp directory and the output
coverage report file with the name of the environment under test,
such that separately run tests do not access the same files.
When running tests without tox, directly through setup.py,
then the tmp directory will still be `./tmp`.
* .gitignore: Added new .coverage-reports/ directory
* .gitlab-ci.yml: Rely on tox to combine and report coverage, only
tell tox about the COVERAGE_PREFIX so that results can be namespaced
by CI job name.
This change also publishes the sources and final combined `.coverage`
file in an output gitlab artifact for inspection, and lists some missing
dependencies to the `coverage` job.
* tox.ini: Add comments and refactor main [testenv] section so that
other environments dont inherit too much unrelated cruft.
Generate the coverate reports in the respective {envtmpdir} so that
all per-process coverage files are prefixed with a full path, ensuring
that concurrent runs don't mix reports and addressing concerns
raised in #844.
Also implemented new `tox -e coverage` environment to combine
any found coverage and print a report.
* .coveragerc: Omit .tox/ directory from coverage stats
Fixes issue #844
-
d364ad02
by Tristan Van Berkom
at 2019-01-09T19:52:22Z
CONTRIBUTING.rst: Updated to mention new `coverage` tox environment.
Also point out that it is possible to run test environments in
parallel using the `detox` tool.
-
1e352434
by Tristan Van Berkom
at 2019-01-09T20:36:06Z
Merge branch 'tristan/detox-tests' into 'master'
Allow using detox, and fix/refactor collection of coverage reports
Closes #844
See merge request BuildStream/buildstream!1051
-
116c1070
by Tristan Van Berkom
at 2019-01-09T20:39:15Z
.coveragerc: Omit some things which are irrelevant to cover.
* Omit versioneer's _version.py
* Omit our __main__.py which is used only internally for generating
documentation
-
c8f3616d
by Tristan Van Berkom
at 2019-01-09T21:12:29Z
Merge branch 'tristan/refine-coverage' into 'master'
.coveragerc: Omit some things which are irrelevant to cover.
See merge request BuildStream/buildstream!1052
-
c87bb592
by Jürg Billeter
at 2019-01-10T09:43:37Z
_platform/platform.py: Add canonicalize_arch() method
-
26e33346
by Jürg Billeter
at 2019-01-10T12:50:15Z
_options/optionarch.py: Accept architecture aliases
Accept common architecture aliases for arch options instead of only
accepting the canonicalized, OS-independent architecture name. This
restores compatibility with existing projects and may simplify option
handling for projects that only target a single OS (and thus, do not
need OS-independent architecture names).
-
06deb4c4
by Jürg Billeter
at 2019-01-10T12:50:15Z
element.py: Accept architecture aliases for sandbox config
Accept common architecture aliases for the sandbox config for
consistency with arch options.
-
0c2a66b3
by Jürg Billeter
at 2019-01-10T13:06:02Z
tests/format/optionarch.py: Add tests for architecture aliases
-
f86bc760
by Jürg Billeter
at 2019-01-10T13:34:56Z
Merge branch 'juerg/arch' into 'master'
Accept architecture aliases
See merge request BuildStream/buildstream!1034
-
347eb34e
by Angelos Evripiotis
at 2019-01-10T14:38:23Z
contributing: fix 'oprtation' and some other typos
-
6bc27bc1
by Angelos Evripiotis
at 2019-01-10T15:07:46Z
Merge branch 'aevri/contrib-typos' into 'master'
contributing: fix 'oprtation' and some other typos
See merge request BuildStream/buildstream!1055
-
0b83d024
by Tristan Van Berkom
at 2019-01-10T20:02:50Z
_pipeline.py: Fix the planner to retain the original order when depth sorting
The algorithm adds elements to a dictionary and then sorts the dictionary
by the discovered depths while recursing - using an OrderedDict is enough
to ensure a stable order.
This fixes `bst show --deps plan ...` reporting different results on
each invocation.
This fixes an aspect of #824
-
630e26f1
by Tristan Van Berkom
at 2019-01-10T20:03:52Z
testutils/runcli.py: Added Result.get_start_order()
Added a function to report the list of elements which appeared in
a given queue in the order of their first appearance.
-
d1d7de57
by Tristan Van Berkom
at 2019-01-11T04:13:16Z
tests/frontend/order.py: Adding tests ensuring stable element processing order
For each sample project and expected result order, this test ensures that:
* bst show --deps plan: Always shows the expected build order.
* bst source fetch: Always executes in the expected build order.
* bst build: Always builds in the expected order (disabled).
The build part of the test is currently disabled as the scheduler
seems to be messing with the order even in a `--builders 1` scenario.
-
276b8d48
by Tristan Van Berkom
at 2019-01-11T04:42:24Z
Merge branch 'tristan/element-processing-order' into 'master'
Make build plan element list stable
See merge request BuildStream/buildstream!1058
-
fc3dcec8
by Angelos Evripiotis
at 2019-01-11T16:45:00Z
dev-requirements: pytest-cov==2.6.1 for '--no-cov'
Bump the version of pytest-cov, so that we pick up the fix for
the '--no-cov' option breaking in pytest 4.0:
https://github.com/pytest-dev/pytest-cov/pull/230
This was the error you would get prior to this version:
_pytest.warning_types.RemovedInPytest4Warning: config.warn has been
deprecated, use warnings.warn instead
Although it says 'deprecated', it did result in
termination with stack trace.
-
f86b7ff3
by Angelos Evripiotis
at 2019-01-11T16:45:00Z
contributing: '--last-failed' and '--no-cov' tips
To help new folks get to grips with pytest, add some tips on other
frequently used features.
-
d983f231
by Angelos Evripiotis
at 2019-01-11T17:19:06Z
Merge branch 'aevri/tox_no_cov' into 'master'
pytest-cov==2.6.1, contributing: '--last-failed' and '--no-cov' tips
See merge request BuildStream/buildstream!1059
-
829a2b93
by Tristan Van Berkom
at 2019-01-12T21:32:13Z
_scheduler/queues/queue.py: Put elements in the skip list, not jobs
Minor correction, looks like we're not observing this queue
otherwise we'd be seeing crashes.
-
993e30ae
by Tristan Van Berkom
at 2019-01-12T22:02:55Z
Merge branch 'tristan/fix-skipped-elements' into 'master'
_scheduler/queues/queue.py: Put elements in the skip list, not jobs
See merge request BuildStream/buildstream!1063
-
32732e01
by Javier Jardón
at 2019-01-14T09:04:01Z
.gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25
-
4f5f1184
by Valentin David
at 2019-01-14T09:40:56Z
Merge branch 'jjardon/freedesktop-sdk_latest' into 'master'
.gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25
Closes #858
See merge request BuildStream/buildstream!1064
-
bb80a2b8
by Chandan Singh
at 2019-01-14T14:30:15Z
.gitlab-ci.yml: Use newer image for overning aarch tests
During recent reorganizing of `.gitlab-ci.yml` to work with `tox`, seems
like we missed to update the image used by the overnight aarch tests,
meaning that they currently fail due to `tox` being missing from them.
While these tests will be skipped on MRs usually, here is an example of
what the tests look like if they are actually run with this change:
https://gitlab.com/BuildStream/buildstream/-/jobs/145449561.
Fixes #859.
-
10b3ee62
by Chandan Singh
at 2019-01-14T14:57:08Z
Merge branch 'chandan/fix-overnight-aarch' into 'master'
Chandan/fix overnight aarch
Closes #859
See merge request BuildStream/buildstream!1065
-
542cdaf0
by Jürg Billeter
at 2019-01-14T18:46:57Z
_frontend/cli.py: Also check original args for --config in bash completion
The path of the config file generated by testutils for completion tests
is passed as regular argument, not via COMP_WORDS. Use that config file
in complete_artifact() to ensure the test uses the right artifact
directory.
-
ff666e76
by James Ennis
at 2019-01-14T18:46:57Z
completions.py: Add a test for our artifact ref autocompletions
Since the artifact subcommand group has been created, we have been able
to tab complete both element names and artifact refs as arguments to
the artifact subcommands (e.g. bst artifact log), this commit adds a
test for this.
-
273b0f55
by Tristan Van Berkom
at 2019-01-14T20:24:57Z
Merge branch 'jennis/add_artifacts_completion' into 'master'
completions.py: Add a test for our artifact ref autocompletions
See merge request BuildStream/buildstream!1054
-
2e3c2ea2
by Javier Jardón
at 2019-01-14T22:26:04Z
.gitlab-ci.yml: Use latest bst-external
which include flatpak_repo plugin needed to build fdsdk
-
d60d2e31
by Javier Jardón
at 2019-01-15T00:39:22Z
Merge branch 'jjardon/bst_external_0_9' into 'master'
.gitlab-ci.yml: Use latest bst-external
Closes #861
See merge request BuildStream/buildstream!1068
-
605836c1
by Chandan Singh
at 2019-01-15T00:57:05Z
.gitlab-ci.yml: Add job to attempt to update dependencies
This will help us check if BuildStream is working with the latest
version of dependencies, as per our constraints. This job is allowed to
fail but its failure should signal that we need to add stricter
constraints in some of our `.in` requirements files.
-
006370af
by Chandan Singh
at 2019-01-15T01:21:33Z
Merge branch 'chandan/reqs-update-test' into 'master'
.gitlab-ci.yml: Add job to attempt to update dependencies
See merge request BuildStream/buildstream!1038
-
4b544555
by Chandan Singh
at 2019-01-15T21:28:40Z
.gitlab-ci.yml: Add tests for python 3.7
We already have tests for python 3.5 and 3.6 but not 3.7.
Fixes https://gitlab.com/BuildStream/buildstream/issues/838.
-
80fe0d9a
by Javier Jardón
at 2019-01-15T22:48:22Z
Merge branch 'chandan/python37-tests' into 'master'
.gitlab-ci.yml: Add tests for python 3.7
Closes #838
See merge request BuildStream/buildstream!1074
-
c91784ab
by Chandan Singh
at 2019-01-15T22:49:01Z
conftest.py: Don't use deprecated get_marker() function
Starting from `pytest` version 4.1.0, `Node.get_marker()` has been
removed, and hence our tests break when running with newer versions of
`pytest`. It was deprecated since a while back but it has recently been
removed completely. Use `get_closest_marker()` as a replacement that is
suggested in the changelog, and seems to work fine for our use case.
See https://github.com/pytest-dev/pytest/pull/4564 for more context on
the upstream issue.
One way of verifying this change is that this should fix the recently
added `tests-fedora-update-deps` job, that was failing before due to
this issue.
-
ecae4d73
by Javier Jardón
at 2019-01-15T23:23:41Z
Merge branch 'chandan/fix-pytest-get-marker' into 'master'
conftest.py: Don't use deprecated get_marker() function
See merge request BuildStream/buildstream!1073
-
0eac4008
by Valentin David
at 2019-01-16T10:04:57Z
buildstream/_gitsourcebase.py: Reduce git history for git describe.
Found during #833.
`git rev-list --boundary tag..HEAD` unfortunately gives boundaries
that are deeper than should when there is a merge commit between `tag`
and `HEAD`. The common ancestory of the two parents of the merge is
a boundary instead of the parent of the branch that is not traversed.
`--ancestry-path` fixes this issue by restricting `git` traversing
those branches.
-
a405e08f
by Valentin David
at 2019-01-16T10:04:57Z
buildstream/_gitsourcebase.py: Fix case where HEAD is tagged
`git rev-list --boundary HEAD..HEAD` does not return any boundary.
So in this case we need to manually tag the HEAD as a boundary.
-
c0631d48
by Valentin David
at 2019-01-16T10:33:52Z
Merge branch 'valentindavid/git-reduced-history' into 'master'
buildstream/_gitsourcebase.py: Reduce git history for git describe.
See merge request BuildStream/buildstream!1069
-
b608ac86
by James Ennis
at 2019-01-16T11:05:44Z
_context.py: Doc fix: get_toplevel_project() returns object not list
-
97a3beb6
by James Ennis
at 2019-01-16T11:05:44Z
_context.py: Add documentation to get_workspaces() command
-
d6587aa0
by James Ennis
at 2019-01-16T11:33:21Z
Merge branch 'jennis/doc_fixes_in_context' into 'master'
Small documentation/comment fixes in context.py
See merge request BuildStream/buildstream!1072
-
2683f98a
by Raoul Hidalgo Charman
at 2019-01-16T11:55:07Z
_cas: Rename artifactcache folder and move that to a root module
Other components will start to reply on cas modules, and not the artifact cache
modules so it should be organized to reflect this.
All relevant imports have been changed.
Part #802
-
d2cc4798
by Raoul Hidalgo Charman
at 2019-01-16T11:55:07Z
casremote.py: Move remote CAS classes into its own file
Part of #802
-
76f67483
by Raoul Hidalgo Charman
at 2019-01-16T11:55:07Z
cas: move remote only functions to CASRemote
List of methods moved
* Initialization check: made it a class method that is run in a subprocess, for
when checking in the main buildstream process.
* fetch_blobs
* send_blobs
* verify_digest_on_remote
* push_method
Part of #802
-
6c428bc9
by Jürg Billeter
at 2019-01-16T12:56:38Z
Merge branch 'raoul/cas-refactor' into 'master'
Cas refactor
See merge request BuildStream/buildstream!1071
-
2648b0bd
by Angelos Evripiotis
at 2019-01-16T14:44:25Z
bst-docker-import: fix Chandan's name
-
780d0f3a
by Angelos Evripiotis
at 2019-01-16T15:13:06Z
Merge branch 'chadnan' into 'master'
bst-docker-import: fix Chandan's name
See merge request BuildStream/buildstream!1076
-
4ca37dcd
by Tristan Van Berkom
at 2019-01-16T15:47:21Z
source-determinism.py integration test: Dont use integration_cache fixture
This was just deadcode, it's not really used here.
-
6286d820
by Tristan Van Berkom
at 2019-01-16T15:47:21Z
conftest.py: Use different artifact directory for integration tests
To ensure we can run integration tests in parallel, use a tempdir
for the artifact cache of each separate integration test run.
This makes it possible to run multiple full test runs including
integration tests in parallel under detox.
-
471af316
by Tristan Van Berkom
at 2019-01-16T16:32:52Z
Merge branch 'tristan/detoxing-integration-tests' into 'master'
Make integration tests parallelizable
See merge request BuildStream/buildstream!1077
-
081e8426
by James Ennis
at 2019-01-16T17:58:41Z
setup.py: Import server main from _cas/casserver.py
Since the CAS refactor, we have not been able to execute
bst-artifact-server. This commit ensures that we can.
This fix closes #867
-
e385660c
by James Ennis
at 2019-01-16T18:33:20Z
Merge branch 'jennis/fix_bst_artifact_server' into 'master'
Ensure that we can execute `bst-artifact-server`
Closes #867
See merge request BuildStream/buildstream!1079
-
6991d6d6
by Tristan Van Berkom
at 2019-01-16T19:28:14Z
_scheduler: Refactor of queues and resources.
This branch makes the following changes:
* jobs/job.py: No longer stores any interested resource list
Jobs are ephemeral again, they only ever exist while they
are running.
* queues/queue.py: Revert to only handling lists of elements
Elements pass through the queues, Queue.harvest_jobs()
replaces Queue.pop_ready_jobs() and now the Queue stops
creating jobs as soon as there are not enough resources
for the job.
Also removed unused `prepare()` abstract method.
* queues/buildqueue.py: Adapt the part where we launch a job
This part needs to be reworked anyway, just touch it up for
now so that it doesnt break with the surrounding changes.
* jobs/{cachesize,cleanup}job.py: Expose uniform complete callback
Allows the scheduler to manage resource deallocation for these
two job completions as a custom thing, at the same phase
that the Queues take care of their own resource deallocation.
* resources.py: No longer has knowledge of the job
Since jobs are ephemeral, they are not a suitable place
to store the resource identifiers, these must be provided
by the callers wherever needed.
Now the main Resources object is owned by the Scheduler
but shared with Queues, each take care of managing the
resources of the jobs they create through the same
resource API.
* scheduler.py: Reverted to only creating jobs on demand
This changes the flow of the scheduler such that whenever
jobs complete, the queues are interrogated for as many
jobs which can run at the moment but not more; and this
completely removes the waiting list.
For the internal cache management jobs, we handle this
with a little state instead of having a waiting list
and only launch when the resources permit it.
By abolishing the scheduler waiting list and creating jobs
on demand, we fix the order of element processing and consequently
fix issue #712.
-
04ba59bc
by Tristan Van Berkom
at 2019-01-16T19:28:14Z
tests/frontend/order.py: Enable the test for build and fix the fetch tests
With the scheduler changes, fetch jobs get automatically skipped
so the output is changed, using a separate repo for each element
fixes the test such that every fetch job gets a job launched.
-
32bdded8
by Tristan Van Berkom
at 2019-01-16T19:28:14Z
_artifactcache/artifactcache.py: Rephrase failure message
It was saying "There is not enough space to build the given element.",
this makes me think the error is associated to a specific element, but
this does not make sense to show up in a cleanup task.
Instead say "There is not enough space to complete the build.", which
should be more clear that even after cleaning up there is not enough
space.
-
17f6e5a8
by Tristan Van Berkom
at 2019-01-16T20:10:30Z
Merge branch 'tristan/element-processing-order' into 'master'
Scheduler refactor, fix processing order
Closes #712
See merge request BuildStream/buildstream!1067
-
19adc9e5
by Tristan Van Berkom
at 2019-01-16T20:48:10Z
element.py: Cleanup temporary staging directories on termination
Use utils._tempdir() which in turn uses _signals.terminator() for
this purpose.
This also changes utils._tempdir() to use utils._force_rmtree()
so that it is safe to use for cleaning up the staging directory.
This patch fixes orphaned temporary directories being left behind
in the artifact cache directory at forceful termination time.
-
5e9598c5
by Tristan Van Berkom
at 2019-01-16T21:17:26Z
Merge branch 'tristan/cleanup-staging-tempdir' into 'master'
element.py: Cleanup temporary staging directories on termination
See merge request BuildStream/buildstream!1080
-
5ea2f59d
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate completions test to tests/frontend
-
1eafc25c
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate `--except` testing to tests/frontend/show.py
The tests/pipeline directory will be removed, and this test
is the better of the two but also redundant with the one in
tests/frontend/show.
Renamed existing test in show.py to `test_show_except_simple`
and added the better test as `test_show_except` below it.
-
b6183123
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate virtual directory storage test to internals directory
Grouping bits of internal testing together here
-
9b6cc972
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate storage test to the internals directory
-
7e2300ce
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate yaml test to the internals directory
-
353a682f
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate utils tests into internals directory
-
3324490e
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate context test into the internals directory
-
99ea157a
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate plugin factory test to internals directory
-
e825bdd3
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate plugin loading test to internals directory
-
c7f69bcc
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Removing tests/plugins/third_party.py
This tests exactly the same thing that is tested in
tests/internals/pluginfactory.py (the new location
of tests/plugins/basics.py).
-
42150422
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Rename `plugins` directory to `elements` directory
Now that the remaining test "filter.py" in the plugins directory
tests a specific element, it makes sense to create a place for
testing elements, just like we do for sources.
-
f712aaa5
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate internal Loader basic test to internals directory
-
8d676a84
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate dependencies test to tests/format
This used to be an internal test, converted this to use the `cli` fixture.
-
bbde1c21
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate dependency order/iteration testing to the format tests
Created new `tests/format/iteration.py` which tests the order in
which elements are iterated over in various scopes in a loaded
data model.
-
8f2b4e9a
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate test for load_ref() unsupporting plugins into format/project.py
The tests/format/project.py test already has some tests about how
we error gracefully for bad plugins and bad plugin configurations,
lets put it there rather than tests/pipeline/load.py which we
will remove.
-
de881c2a
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate preflight error handling check to tests/format/project.py
This is where other load time related plugin error handling is
checked, and is the last thing to remove in the `tests/pipeline`
directory.
-
a7fbd900
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Renaming some tests in tests/format/project.py
Remove some redundancy from the test lines.
-
aa997450
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Removing tests/pipeline/load.py
The remaining test simply loads a project with one element
and asserts a value on it. This is already sufficiently tested
in tests/format/project.py.
-
561eddf1
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate protected variable handling tests to tests/format/variables.py
-
3bf40cf3
by Tristan Van Berkom
at 2019-01-16T23:35:21Z
tests: Migrate junctions test to tests/format/junctions.py
This is the directory for all things related to loading.
-
d34a4fd1
by Tristan Van Berkom
at 2019-01-17T00:26:43Z
Merge branch 'tristan/organizing-tests' into 'master'
General refactor in tests directory
See merge request BuildStream/buildstream!1062
-
d212cdfa
by Tristan Van Berkom
at 2019-01-17T00:28:03Z
sandbox/sandbox.py: Display failed commands in the detail string
We should only display commands in detail strings, not in the
message texts.
This also updates tests/integration/sandbox-bwrap.py to expect
the new message string which only contains the command exit status
and not the whole command itself, this does not alter the validity
of the text case which is checking that we can obtain the expected
return value.
-
ce91ce5d
by Tristan Van Berkom
at 2019-01-17T05:05:16Z
Merge branch 'tristan/error-message-regression' into 'master'
sandbox/sandbox.py: Display failed commands in the detail string
See merge request BuildStream/buildstream!1081
-
63b02f7c
by James Ennis
at 2019-01-17T15:13:27Z
_profile.py: Add timestamp to the logs
-
f64ee46f
by James Ennis
at 2019-01-17T17:02:28Z
_profile.py: Write binaries as well as logs
Private class methods which write the logs and write the binaries
have been added to Profile. The binaries are able to be used
by various visualisation tools.
-
2cb37a7e
by James Ennis
at 2019-01-18T09:31:24Z
Merge branch 'jennis/profiling_outputs_binaries' into 'master'
Make the profiler output binaries (which can be used to visualise the data)
See merge request BuildStream/buildstream!1082
-
56ec33cc
by Valentin David
at 2019-01-18T11:14:41Z
.gitlab-ci.yml: Add overnight tests logs to artifacts.
-
ad2df651
by Javier Jardón
at 2019-01-18T12:01:56Z
Merge branch 'valentindavid/overnight-tests-logs-artifacts' into 'master'
.gitlab-ci.yml: Add overnight tests logs to artifacts.
See merge request BuildStream/buildstream!1084
-
f874295f
by Tristan Van Berkom
at 2019-01-18T15:59:28Z
bzr source plugin: Use lock files to avoid corrupting the source cache
This patch by itself fixes #868
-
42a2fe3c
by Tristan Van Berkom
at 2019-01-18T15:59:28Z
bzr source plugins: Remove all of the atomic backup directory business logic
Follow up of last commit which uses exclusive locking to
protect bzr operations instead.
-
a8713ed2
by Tristan Van Berkom
at 2019-01-18T15:59:28Z
testutils/runcli.py: Added cli.get_element_states()
With get_element_state(), you need to invoke BuildStream once
for every element state you want to observe in a pipeline.
The new get_element_states() reports a dictionary with
the element state hashed by element name and is better to use
if you have more than one element to observe the state of.
-
a895cb2a
by Tristan Van Berkom
at 2019-01-18T15:59:28Z
tests/frontend/track.py: Use test_track_recurse() to stress test Sources
This causes multiple source instances to interact with the same
backing data store at the same time, increasing the likelyhood
of triggering issues around concurrent access.
This more reliably triggers issue #868
-
4236bcc7
by Tristan Van Berkom
at 2019-01-18T16:35:23Z
Merge branch 'tristan/fix-bzr-race' into 'master'
Fix bzr race conditions
Closes #868
See merge request BuildStream/buildstream!1083
-
951a8df1
by Angelos Evripiotis
at 2019-01-18T18:02:22Z
cli.py: add a hint about '--' to 'bst shell' help
-
9911023f
by Angelos Evripiotis
at 2019-01-18T18:56:21Z
Merge branch 'aevri/shell_separator_hint' into 'master'
cli.py: add a hint about '--' to 'bst shell' help
See merge request BuildStream/buildstream!1078
-
c9ce89d2
by Tristan Van Berkom
at 2019-01-18T19:36:26Z
_cas/cascache.py: Cleanup directories when removing refs
With out this, empty directories in the refs/heads directory just
grow unconditionally.
-
c536ab6a
by Tristan Van Berkom
at 2019-01-18T19:36:26Z
tests/artifactcache/expiry.py: Test refs directory cleanup
Enhance the test which checks removal of the extract directories
with an additional check that the ref directories are cleaned up
when removing artifacts.
-
99699ffc
by Tristan Van Berkom
at 2019-01-18T19:36:26Z
utils.py: Added _tempnamedfile()
When used in a child process in BuildStream, this should be
used instead of tempfile.NamedTemporaryFile() directly, otherwise
we fail to cleanup the file on SIGTERM.
-
8ce483d4
by Tristan Van Berkom
at 2019-01-18T19:36:26Z
_cas/cascache.py: Use utils._tempdir() and utils._tempnamedfile()
The direct usage of tempfile.TemporaryDirectory() and
tempfile.NamedTemporaryFile() here causes leakage of any temporary
data when the process operating on temporary data is terminated
with SIGTERM.
Using the utilities ensures that trash is not left behind in
~/.cache/buildstream/artifacts/tmp when the user terminates
BuildStream with ^C.
-
73c7252d
by Tristan Van Berkom
at 2019-01-18T20:57:42Z
Merge branch 'tristan/cas-cleanup-improve' into 'master'
CASCache cleanup improvements
See merge request BuildStream/buildstream!1087
-
adfb9291
by Tristan Van Berkom
at 2019-01-18T21:23:05Z
tests/testutils/python_repo.py: Use subprocess to run sdist
The current approach using setuptools.sandbox.run_setup() was
causing a spurious (but highly frequent) failure where setuptools
gets mixed up with it's manipulation of sys.modules and hits a
RuntimeError as a dictionary changes size while being iterated over.
For instance: https://gitlab.com/BuildStream/buildstream/-/jobs/147967307
Since this already happens in an isolated virtual environment created
by tox, we should not need additional sandboxing here from setuptools,
and launching this as a subprocess will be safer.
-
d114a6bd
by Tristan Van Berkom
at 2019-01-18T22:07:22Z
Merge branch 'tristan/fix-pip-source-test' into 'master'
tests/testutils/python_repo.py: Use subprocess to run sdist
See merge request BuildStream/buildstream!1090
-
bf591ade
by Chandan Singh
at 2019-01-18T22:09:04Z
Derive import plugin from Element instead of BuildElement
The `import` element is not really a build element. The main purpose of
the `BuildElement` class is to allow users to run `build-commands`,
`install-commands` etc. But, `import` does not run such commands.
Moreover, we already override all the methods provided by
`BuildElement`. So it only makes it confusing to have it derived from
`BuildElement` class when it is not a build element.
So, derive it from the base `Element` class instead.
-
2233a532
by Chandan Singh
at 2019-01-18T22:46:41Z
Merge branch 'chandan/import-is-not-buildelement' into 'master'
Derive import plugin from Element instead of BuildElement
See merge request BuildStream/buildstream!1089
-
aa3411f9
by Tristan Maat
at 2019-01-18T22:47:46Z
testutils/runcli.py: Allow removing artifacts from arbitrary dirs
`remove_artifact_from_cache` used a hard-coded path to remove
artifacts, which wasn't sufficient for integration tests.
-
fff882fe
by Tristan Maat
at 2019-01-18T22:47:46Z
widget.py: Avoid "showing 0 lines" messages when there are no lines
This happened when bst is invoked with --message-lines 0 or
--error-lines 0, and was arguably a little too verbose (the user
explicitly asked us not to show them any lines, after all).
Fixes #779
-
e230dedb
by Tristan Van Berkom
at 2019-01-18T23:23:35Z
Merge branch 'tlater/message-lines' into 'master'
Avoid "showing 0 lines" messages when we're asked to show no lines
Closes #779
See merge request BuildStream/buildstream!1031
-
43797617
by Richard Maw
at 2019-01-21T10:41:37Z
_gitsourcebase.py: Fetch with explicit refspecs
Old versions of git lack --force and --tags,
but the same effect can be had by specifying refspecs.
-
ce8dab0f
by Richard Maw
at 2019-01-21T10:41:37Z
tests/frontend/workspace.py: Skip test_open_multi_unwritable when root
The test assumes that a directory with write permission removed isn't writable,
this isn't the case if the process running the tests has CAP_DAC_OVERRIDE
which is common when running as root.
-
50165081
by Richard Maw
at 2019-01-21T10:41:37Z
tests/sources/git.py: Skip tests that assume too new a git
test_track_invalid_submodule depends on being able to remove a submodule
by `git rm $submoduledir`, but old versions of git don't update .gitmodules
so BuildStream still thinks there's a submodule present.
For expediency the test is skipped rather than changed to manually remove
the entry from .gitmodules if git hasn't done it,
since in the common case git is new enough to do that itself.
test_git_describe expects --first-parent to find another tag,
but `bst track` will gracefully degrade if the option doesn't work
so a different history will be retained with old versions of git.
It's of marginal benefit to add additional cruft
to test for different output on old versions of git that won't persist forever.
-
8677a256
by Benjamin Schubert
at 2019-01-21T10:41:37Z
Add a Centos runner
Centos is apparently different enough from fedora when running tests.
-
33782865
by Valentin David
at 2019-01-21T11:47:18Z
Merge branch 'richardmaw/centos-oldgit-test-fixes' into 'master'
Fix CentOS
Closes #833
See merge request BuildStream/buildstream!1085
-
c00b3782
by Chandan Singh
at 2019-01-21T19:37:53Z
CONTRIBUTING.rst: Fix formatting of link to pip docs
The link explaining python's requirements files was formatted
incorrectly (missing underscore at the end). This resulted in the text
being rendered literally, as opposed to being converted into a
hyperlink.
-
638ceb8a
by Chandan Singh
at 2019-01-21T20:16:38Z
Merge branch 'chandan/fix-contrib-link' into 'master'
CONTRIBUTING.rst: Fix formatting of link to pip docs
See merge request BuildStream/buildstream!1094
-
383142e3
by James Ennis
at 2019-01-22T12:32:43Z
using_commands.rst: Split out top level commands and subcommand groups
-
ac135333
by James Ennis
at 2019-01-22T12:32:43Z
using_commands.rst: Add the source checkout command
-
a3fc350f
by James Ennis
at 2019-01-22T12:32:43Z
Move push and pull to the new artifact subcommand group
This commit also ensures that if we try to use the 'old' commands,
BuildStream will fail and instruct the user to use the new command.
-
9eefe863
by James Ennis
at 2019-01-22T12:32:43Z
cli: Add artifact checkout subcommand
'artifact checkout' has slightly different behaviour from 'checkout',
that is, either '--directory' or '--tar' are now required options.
This is a step towards allowing checkout to take multiple args.
-
fbd15939
by James Ennis
at 2019-01-22T12:32:43Z
Mark 'old' checkout command as obsolete
This commit marks 'bst checkout' as a 'hidden' command. If used,
the user will be prompted to use the new 'bst artifact checkout'
command.
All tests which used 'bst checkout' have been modified to use
the new artifact sub-command.
This partially solves #822.
-
9e8034e9
by James Ennis
at 2019-01-22T12:32:43Z
Add NEWS entry describing the deprecation of checkout, pull and push
-
1bccf1d2
by James Ennis
at 2019-01-22T12:38:40Z
using_commands.rst: Split out the artifact subcommands
-
d9072371
by James Ennis
at 2019-01-22T12:38:40Z
using_commands.rst: Add artifact log subcommand
-
077d5a96
by James Ennis
at 2019-01-22T12:41:29Z
Change bst checkout/pull/push references to bst artifact checkout/pull/push
-
ea2fbe4d
by James Ennis
at 2019-01-22T13:02:14Z
man: Regenerate all of our man pages
Due to an upstream click_man issue:
https://github.com/click-contrib/click-man/issues/10
generating the man pages with our current setup.py only generates
a man page for bst-artifact-server, our first entry point.
I then had to remove this entry point from setup.py and regenerate to
obtain man pages for the commands in cli.py
-
9c2a6b93
by James Ennis
at 2019-01-22T15:08:17Z
Merge branch 'jennis/migrate_pull_push_commands' into 'master'
Move push/pull/checkout to the artifact subcommand group
See merge request BuildStream/buildstream!1045
-
35377522
by James Ennis
at 2019-01-22T15:28:13Z
cli.py: Add an obsoletion note to our deprecated commands.
Click 7.0 allows us to declare commands as 'hidden'.
However, sphinx-click and click-man still generate these hidden
commands in the documentation and man pages, respectively.
This is a stop gap solution until the upstream issues (tracked in
issues #879 and #881) have been addressed.
-
76148785
by James Ennis
at 2019-01-22T16:15:20Z
Merge branch 'jennis/add_obsoletion_note' into 'master'
Add an obsoletion note to our deprecated commands.
See merge request BuildStream/buildstream!1097
-
1443c542
by Tristan Van Berkom
at 2019-01-22T16:26:56Z
_scheduler: Fix dont display a failure for terminated jobs
This fixes a recent regression introduced in c2fc2a5ea
-
6e5c9987
by Tristan Van Berkom
at 2019-01-22T17:14:50Z
Merge branch 'tristan/fix-terminated-status' into 'master'
_scheduler: Fix dont display a failure for terminated jobs
See merge request BuildStream/buildstream!1096
-
3315b9a1
by Tristan Van Berkom
at 2019-01-22T18:46:24Z
tests/integration/pullbuildtrees.py: Fix the non-integration case.
This test has one test case which is marked as an integration test,
and the other is not an integration test, but was using the integration
cli. The integration cli does not work correctly if not run in
integration mode.
This was causing an error locally in conftest.py when trying to
create a tmpdir inside a nonexisting integration cache directory.
-
56e857f4
by Tristan Van Berkom
at 2019-01-22T19:59:57Z
Merge branch 'tristan/fix-pullbuildtrees-test' into 'master'
tests/integration/pullbuildtrees.py: Fix the non-integration case.
See merge request BuildStream/buildstream!1098
-
3895571a
by Chandan Singh
at 2019-01-23T00:45:47Z
tox.ini: Unpin sphinx dependency
https://github.com/rtfd/sphinx_rtd_theme/pull/672 has been fixed
upstream, and the newer versions of `sphinx_rtd_theme` do not break
search functionality with Sphinx >= 1.8.
-
528e8ed7
by Chandan Singh
at 2019-01-23T01:31:49Z
Merge branch 'chandan/unpin-sphinx' into 'master'
tox.ini: Unpin sphinx dependency
See merge request BuildStream/buildstream!1093
-
0098a900
by Tristan Van Berkom
at 2019-01-23T13:57:53Z
tests: Migrated cache quota test into artifactcache/cache_size.py
Instead of sitting mysteriously alone in internals/utils.py
-
9be31641
by Tristan Van Berkom
at 2019-01-23T13:57:53Z
_artifactcache.py: Raise ArtifactError() when quota size exceeds disk space.
This is not an error related to loading data, like a parse error
in the quota specification is, but a problem raised by the artifact
cache - this allows us to assert more specific machine readable
errors in test cases (instead of checking the string in stderr, which
this patch also fixes).
This also removes a typo from the error message in the said error.
* tests/artifactcache/cache_size.py
Updated test case to expect the artifact error, which consequently
changes the test case to properly assert a machine readable error
instead of asserting text in the stderr (which is the real, secret
motivation behind this patch).
* tests/artifactcache/expiry.py: Reworked test_invalid_cache_quota()
Now expect the artifact error for the tests which check configurations
which create caches too large to fit on the disk.
-
3a6e953f
by Tristan Van Berkom
at 2019-01-24T01:59:13Z
Merge branch 'tristan/insufficient-storage-error' into 'master'
Tristan/insufficient storage error
See merge request BuildStream/buildstream!1102
-
68339b19
by Tristan Van Berkom
at 2019-01-24T05:01:55Z
tests/testutils/runcli.py: Make get_element_states() take a list of targets
Instead of a single target, we can always provide a single target
in a list.
-
6ecc2b0a
by Tristan Van Berkom
at 2019-01-24T05:01:55Z
tests/artifactcache/expiry.py: Refactored to use get_element_states()
-
1140aed5
by Tristan Van Berkom
at 2019-01-24T05:01:55Z
tests/elements/filter.py: Refactored to use get_element_states()
-
b7ea8b74
by Tristan Van Berkom
at 2019-01-24T05:01:55Z
tests/sources/remote.py: Refactored to use get_element_states()
-
c2c004f9
by Tristan Van Berkom
at 2019-01-24T05:01:55Z
tests/frontend/pull.py: Refactored to use get_element_states()
-
21b2958b
by Tristan Van Berkom
at 2019-01-24T05:01:55Z
tests/frontend/push.py: Refactored to use get_element_states()
-
83fcaa9f
by Tristan Van Berkom
at 2019-01-24T05:01:56Z
tests/frontend/track.py: Refactored to use get_element_states()
-
46eb3018
by Tristan Van Berkom
at 2019-01-24T05:01:56Z
tests/frontend/workspace.py: Refactored to use get_element_states()
-
341b131b
by Tristan Van Berkom
at 2019-01-24T06:13:57Z
Merge branch 'tristan/test-element-states' into 'master'
Reduce number of calls to `bst show` in tests
See merge request BuildStream/buildstream!1103
-
ef2b4648
by Tristan Van Berkom
at 2019-01-24T06:14:20Z
tests/frontend/track.py: test_track_error_cannot_write_file() fixup
This tests how BuildStream reacts when it fails to write the tracking
results to the element files or project.refs file, which is an operation
that plugins do not play a part in.
As such, removing the per repo kind parameterization from this test
as multiple runs are redundant here.
-
40c18174
by Tristan Van Berkom
at 2019-01-24T07:00:50Z
Merge branch 'tristan/track-test-reduce' into 'master'
test_track_error_cannot_write_file() fixup
See merge request BuildStream/buildstream!1104
-
5b0bba85
by Jonathan Maw
at 2019-01-24T10:15:06Z
Test that tracking in workspaces actually works
Previously, it merely tested that buildstream did not fall other,
rather than whether it did anything useful.
-
58d7d722
by Jonathan Maw
at 2019-01-24T10:18:01Z
tests: Test that fetching an open workspace will fetch its dependencies
Previously, there was no way of detecting whether fetching happened, as
an element with an open workspace will not be fetched.
-
24bd8994
by Jürg Billeter
at 2019-01-24T13:36:33Z
Merge branch 'jonathan/test-missing-workspace-guessing' into 'master'
Add tests to cover reinstated support for guessing targets
See merge request BuildStream/buildstream!1042
-
e03dd5fc
by Jürg Billeter
at 2019-01-24T13:37:28Z
_frontend/cli.py: Guess element also for bst build --all
There is no reason to disallow guess_element() for bst build --all.
-
3850274b
by Phillip Smyth
at 2019-01-24T13:37:28Z
projectconfig.yaml: Add key for default targets
_versions.py: Bump format version
-
ab72d384
by Jürg Billeter
at 2019-01-24T13:37:28Z
_project.py: Add get_default_target() and get_default_targets() methods
_frontend/cli.py: Use new methods.
Based on patches by Phillip Smyth.
-
ee2d8434
by Jürg Billeter
at 2019-01-24T13:37:28Z
_stream.py: Add ignore_junction_targets parameter
This filters out junctions from the list of targets.
-
2f175f0a
by Jürg Billeter
at 2019-01-24T13:37:28Z
_frontend/cli.py: Ignore junctions in default targets where appropriate
Junctions cannot be built, pulled, or pushed. Specifying a junction on
the command line for these commands will result in an error. However,
junctions may be in the list of default targets, so they need to be
ignored for build, pull, and push commands.
-
5a351dee
by Jürg Billeter
at 2019-01-24T13:38:50Z
format_project.rst: Add documentation for default targets
-
77414518
by Phillip Smyth
at 2019-01-24T13:38:50Z
tests/frontend: Add default target tests for bst show and build
-
ce1c10ec
by Jürg Billeter
at 2019-01-24T13:38:50Z
tests/frontend/fetch.py: Add default target test for bst source fetch
-
99b5c0af
by Jürg Billeter
at 2019-01-24T13:38:50Z
tests/frontend/pull.py: Add default target test for bst push/pull
-
3642c8e7
by Jürg Billeter
at 2019-01-24T13:38:50Z
tests/frontend/buildcheckout.py: Add default target test with junction
Test that `bst build` does not fail in a project where the list of
default targets includes a junction (junctions cannot be built).
-
50c5159f
by Phillip Smyth
at 2019-01-24T13:38:50Z
NEWS: Add entry for default target feature
-
05587f22
by Jürg Billeter
at 2019-01-24T15:10:08Z
Merge branch 'issue-638-validate-all-files' into 'master'
Add support for default targets
See merge request BuildStream/buildstream!925
-
3e36e363
by Tristan Van Berkom
at 2019-01-24T16:55:24Z
_scheduler/resources.py: Dont error out in unregister_exclusive_interest()
Don't require the interest to be registered, just discard any interest,
this function just sets a bit in a mask, and is not intended to maintain
a balance like the reserve() function is.
-
ce01f87e
by Tristan Van Berkom
at 2019-01-24T16:55:24Z
_scheduler/scheduler.py: Run cache size exclusively at startup
When running any session that has Queues which require Resource.CACHE,
check if our loaded estimated size exceeds the quota, and if so;
lock the Resource.CACHE resource exclusively right away and run
an exclusive initial cache size job.
This ensures we cleanup first before doing anything which might
add to the cache at startup time, if deemed needed.
This is a partial fix for issue #737
-
2479e8df
by Tristan Van Berkom
at 2019-01-24T16:55:24Z
_frontend/widget.py: Render core messages more like other messages
In order to test when core activities occur by parsing the stderr
in tests, we should make the messages conform more.
At the same time, this restores alignment of columns in core
messages with the element processing related messages.
Also, _scheduler/scheduler.py is updated to make it's activity names
conform to the (current) 5 character limit for the sake of alignment.
The tests/frontend/logging.py test gets it's regexes updated for
the log lines it checks for in stderr.
-
fdb8ff65
by Tristan Van Berkom
at 2019-01-24T16:55:24Z
tests/artifactcache/expiry.py: Test that expiry happens first
-
acd0bf22
by Tristan Van Berkom
at 2019-01-24T18:01:41Z
Merge branch 'tristan/cache-management' into 'master'
Cache management fixes
See merge request BuildStream/buildstream!1091
-
41f03296
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_frontend/app.py: Initialize logging before preflighting the artifact cache
The artifact cache emits messages, and we want to allow that in preflight.
-
24ca2f46
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
utils.py: Add _get_volume_size()
We can streamline this call to os.statvfs() in a few places.
-
9fd9fbb9
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Refactored to use utils._get_volume_size()
This will benefit from a better UtilError being raised, and
and turns the artifact cache's local function into a one liner.
The loop which finds the first existing directory in the
given path has been removed, being meaningless due to the
call to os.makedirs() in ArtifactCache.__init__().
The local function was renamed to _get_cache_volume_size() and
no longer takes any arguments, which is more suitable for the
function as it serves as a testing override surface for
unittest.mock().
The following test cases which use the function to override
the ArtifactCache behavior have been updated to use the new
overridable function name:
tests/artifactcache/cache_size.py
tests/artifactcache/expiry.py
-
7ee0c579
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Added ArtifactCacheUsage()
A simple object which creates a snapshot of current
usage statistics for easy reporting in the frontend.
-
353293b6
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_context.py: Added get_artifact_cache_usage()
A frontend facing API for obtaining usage statistics.
I would have put this on Stream instead, but the Context
seems to be the de facto place for looking up the artifact cache
in general so let's put it here.
-
51ed36de
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_frontend/widget.py: Added cache usage entry in the startup heading
-
5797238b
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_frontend/status.py: Added Cache size usage indicator to status bar
This also adds some comments around the main status bar heading
rendering function.
-
8074ebf4
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Add status messages in cache management operations
Added some useful status messages when:
* Calculating a new artifact cache usage size
* Starting a cleanup
* Finishing a cleanup
Also enhanced messaging about what was cleaned up so far when
aborting a cleanup.
-
49c11bc8
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Added client progress callback to ArtifactCache.clean()
-
3616e939
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_scheduler/jobs/job.py: Allow subclasses to message the frontend
-
bcd19266
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_scheduler/jobs/cleanupjob.py: Update cache size while processing
Updates the known cache size in the main process while the cleanup
process is ongoing, so that the status indicators update live
while the cleanup happens.
-
9c33107f
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Correcting API documenting comment for remove()
This seems to have been copy/pasted from cascache, and
documents the function to possibly return None if defer_prune
was specified, but this function does not expose defer_prune.
-
a2140d74
by Tristan Van Berkom
at 2019-01-24T19:44:52Z
Merge branch 'tristan/cache-management-logging' into 'master'
Cache management logging enhancements
See merge request BuildStream/buildstream!1105
-
d5847462
by James Ennis
at 2019-01-25T16:35:21Z
_profile.py: Added a new profiling topic, scheduler
profile_start() and profile_end() calls have been incorporated into
Scheduler.run()
-
56a3954c
by James Ennis
at 2019-01-25T16:35:21Z
_profile.py: Added a new profiling topic, load-selection
profile_start() and profile_end() calls have been added to
Stream.load_selection()
-
116da6d7
by James Ennis
at 2019-01-25T16:35:21Z
_profile.py: Update copyright statement
-
137d31cd
by James Ennis
at 2019-01-25T17:59:22Z
Merge branch 'jennis/add_new_profile_topic' into 'master'
Add new 'scheduler' and 'load-selection' profiling topics
See merge request BuildStream/buildstream!1088
-
22b3a0c1
by Tristan Van Berkom
at 2019-01-25T18:35:36Z
_artifactcache.py: Don't require the quota to be available on disk.
Instead only rely on the headroom to be enough to protect against
out of space conditions. The headroom can become configurable as
a separate step is required.
The changes to achieve this are:
* Rename ArtifactCache.has_quota_exceeded() to ArtifactCache.full().
* ArtifactCache.full() now also reports True if the available
space on the artifact cache volume is smaller than the headroom.
This ensures jobs get triggered to cleanup the cache when
reaching the end of the disk.
* When loading the artifact quota, it is now only an error if
the quota exceeds the overall disk space, not if it does not
fit in the available space.
It is still a warning if the quota does not fit in the
available space on the artifact cache volume.
* Updated scheduler.py and buildqueue.py for the API rename
* tests: Updated the artifactcache/expiry.py test for its
expectations in this regard.
Added a new test to test an error when quota was specified to
exceed total disk space, and adjusted the existing tests to
expect a warning when the quota does not fit in the available
space.
This fixes issue #733 and #869.
-
27ca6cc7
by Tristan Van Berkom
at 2019-01-25T21:18:15Z
Merge branch 'tristan/cache-quota-max-only' into 'master'
_artifactcache.py: Don't require the quota to be available on disk.
Closes #869 and #733
See merge request BuildStream/buildstream!1106
-
89ba2abe
by Chandan Singh
at 2019-01-25T23:50:49Z
tox.ini: Add environment to update man pages
Previously, one had to manually install `click-man` package, and
remember to run the correct command. Now, we can simply run `tox -e man`
to update the man pages.
-
d7438688
by Chandan Singh
at 2019-01-25T23:50:49Z
man: Refresh man pages
Notable changes:
* New `source` and `artifact` command groups
* Man pages corresponding to obsolete commands, that were marked as
hidden in Click, have now been removed.
Fixes #881.
-
4edbbd27
by Chandan Singh
at 2019-01-25T23:50:49Z
setup.py, CONTRIBUTING.rst: Recommend using tox to generate man pages
Simplify our docs, by requesting users to run `tox -e man` to update man
pages, instead of manually installing `click-man` and running the
command manually.
-
a3e7aee8
by Chandan Singh
at 2019-01-25T23:50:49Z
setup.py: Do not error out when man directory is empty/missing
If the `man` directory is empty, then it won't be copied in the source
distribution, and `list_man_pages()` will throw an exception when trying
to list files in a non-existent directory. This prevents us from
installing the BuildStream package when the man pages are not there.
The most common use-case for this is when we want to re-generate the man
pages but want to install the package before re-generating them.
-
222753ac
by Tristan Van Berkom
at 2019-01-26T05:10:36Z
Merge branch 'chandan/toxic-man' into 'master'
Generate man pages using tox & update them
Closes #880 and #881
See merge request BuildStream/buildstream!1107
-
5df4105a
by Angelos Evripiotis
at 2019-01-28T10:13:40Z
news: fix 'osbolete' spelling
-
9c981eff
by Angelos Evripiotis
at 2019-01-28T10:17:57Z
Fixup refs to 'bst track'
Now that 'bst track' is obsolete, change guidance to refer to the
replacement 'bst source track' instead.
-
bef80291
by Angelos Evripiotis
at 2019-01-28T10:19:17Z
Fixup refs to 'bst fetch'
Now that 'bst fetch' is obsolete, change guidance to refer to the
replacement 'bst source fetch' instead.
-
564cb245
by Angelos Evripiotis
at 2019-01-28T11:36:16Z
Merge branch 'aevri/bst_track_guidance' into 'master'
Fixup refs to 'bst track' and 'bst fetch'
See merge request BuildStream/buildstream!1086
-
a3e2cdd2
by Tom Pollard
at 2019-01-28T12:14:40Z
_stream.py: Add use_artifact_config opt arg for load_selection()
use_artifact_config added as an optional default arg, allowing
for loading of given elements artifact remote config.
-
805baf7d
by Tom Pollard
at 2019-01-28T12:14:40Z
Download buildtrees on demand for bst shell --use-buildtree
Provide bst shell --use-buildtree the ability to attempt to
acquire missing buildtrees, given respective option, user
pull-buildtree context and remote availability.
_frontend/cli.py: Refactor logic for determining --use-buildtree
option with given opportunity to attempt pulling a non-local
buildtree. Element loaded with artifact_config to allow remote
querying.
_stream.py: With given user option and element state, construct
PullQueue to fetch remote buildtree. Continue or Error without
buildtree if cannot be attained.
tests/integration/build-tree.py: Update to support new usecases
-
38356932
by Tom Pollard
at 2019-01-28T13:44:37Z
Merge branch 'tpollard/829' into 'master'
Download buildtrees on demand for bst shell --use-buildtree
Closes #829
See merge request BuildStream/buildstream!1050
-
3590ca8c
by Abderrahim Kitouni
at 2019-01-28T14:59:19Z
requirements/requirements.in: require protobuf >= 3.6
This is needed since 0f2bc3754
-
39b952dc
by Abderrahim Kitouni
at 2019-01-28T15:19:04Z
requirements/requirements.in: require Click >= 7.0
This is needed since 629a6e524, and was lost in the conversion to requirements.in
-
80b36d0c
by Javier Jardón
at 2019-01-28T17:54:52Z
Merge branch 'abderrahim/protobuf-version' into 'master'
requirements/requirements.in: update minimum versions
Closes #884
See merge request BuildStream/buildstream!1114
-
a1ab48da
by Valentin David
at 2019-01-28T21:30:26Z
Fix crash when spawned job completes very fast
Job can complete before we return from `Job.span()` to
`Scheduler._spawn_job()`, so that `_active_jobs` would not yet contain
the job.
This would print a stack on the console and try to run a second time
the job which can have unexpected effects.
In order to reproduce the issue, in
`buildstream/_scheduler/jobs/job.py`, in `Job.spawn`,
add a call to `time.sleep()` right before call to
`asyncio.get_child_watcher()`.
This fixes issue #857.
-
2fcb4491
by Jürg Billeter
at 2019-01-28T22:36:22Z
Merge branch 'valentindavid/crash_in_scheduler_857' into 'master'
Fix crash when spawned job completes very fast
Closes #857
See merge request BuildStream/buildstream!1095
-
1c05a092
by Valentin David
at 2019-01-29T05:58:17Z
Fix type of error codes in CAS server
Fixes #882.
-
785da59c
by Jürg Billeter
at 2019-01-29T06:49:10Z
Merge branch 'valentindavid/wrong_type_in_status_code' into 'master'
Fix type of error codes in CAS server
Closes #882
See merge request BuildStream/buildstream!1099
-
ddef91ea
by Valentin David
at 2019-01-29T07:23:35Z
Make sure testing cache directory exists
Fixes #873
-
6a4c8611
by Jürg Billeter
at 2019-01-29T08:18:45Z
Merge branch 'valentindavid/make_cache_dir' into 'master'
Make sure testing cache directory exists
Closes #873
See merge request BuildStream/buildstream!1092
-
86c8e414
by Angelos Evripiotis
at 2019-01-29T10:39:29Z
BREAK:remove unconditional 'are you sure?' prompts
This is a breaking change, as it affects behaviour that people might be
relying on. An entry has been added to NEWS.
As proposed on the mailing list, this change removes the unconditional
prompts on:
o: bst workspace reset
o: bst workspace close --remove-dir
If interactive, these commands would always interrupt you with a prompt
like this:
This will remove all your changes, are you sure?
This seems like it may just save someone's work some time. It may also
condition folks to hit 'y' quickly without thinking.
This change also makes the non-interactive behaviour consistent with the
interactive behaviour in the default case. There is also the case of the
prompt configured by 'really-workspace-close-project-inaccessible',
which may be tackled in later work.
This change also removes the new config options to suppress those
prompts, and their associated news entry.
The relevant bit of the mailing list conversation is here:
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00106.html
The issue to make interactive and non-interactive behaviour consistent
is here:
https://gitlab.com/BuildStream/buildstream/issues/744
-
aae35e13
by Angelos Evripiotis
at 2019-01-29T12:08:07Z
Merge branch 'aevri/are_you_sure' into 'master'
BREAK:remove unconditional 'are you sure?' prompts
See merge request BuildStream/buildstream!1061
-
03111d39
by Dor Askayo
at 2019-01-30T10:35:06Z
filter.py: don't recurse when staging dependencies
Also bump the element's version so cached artifacts would be
invalidated.
Fixes #883
-
7256bb0c
by James Ennis
at 2019-01-30T11:34:04Z
Merge branch 'doraskayo/filter-indirect-deps-fix' into 'master'
filter.py: don't recurse when staging dependencies
Closes #883
See merge request BuildStream/buildstream!1110
-
36746730
by Chandan Singh
at 2019-01-31T10:50:05Z
tox.ini: Specify minimum version of click-man
`click-man` versions < 0.3.0 do not properly support multiple
entrypoints. Since this was added to `tox` after `0.3.0` was released,
`tox` should never be pulling older versions. But, let's add it here for
documentation purposes.
See
https://gitlab.com/BuildStream/buildstream/merge_requests/1107#note_135187046
for some background on this.
-
fa4a21ce
by Chandan Singh
at 2019-01-31T12:15:43Z
Merge branch 'chandan/min-version-click-man' into 'master'
tox.ini: Specify minimum version of click-man
See merge request BuildStream/buildstream!1120
-
dd791373
by Chandan Singh
at 2019-01-31T14:32:44Z
testutils/site.py: Support parsing more exotic git versions
We use output of `git --version` to determine if we can run some tests
that rely on features from newer git versions. Usually, we expect the
output to be like:
git version 2.17.2
On some platforms, like MacOS, there could be a suffix after the version
string, so that it looks something like:
git version 2.17.2 (Apple Git-113)
This causes things to fail like so:
ValueError: invalid literal for int() with base 10: '2 (Apple Git-113)\n'
Fix logic around `HAVE_OLD_GIT` such that we split the output of
`git --version` without limit on how many times we split. Previously we
used to split only twice so the suffixes like `(Apple Git-113)` are not
part of the parsed version.
-
96c0fbd6
by Chandan Singh
at 2019-01-31T15:39:19Z
Merge branch 'chandan/fix-git-version-mac' into 'master'
testutils/site.py: Support parsing more exotic git versions
See merge request BuildStream/buildstream!1118
-
d25e2795
by Benjamin Schubert
at 2019-01-31T17:06:23Z
Add LoadElement in dependency list for LoadElement idrectly
This removes the need for the 'Dependency' list to then be matched
with the corresponding LoadElement and will allow iterating the
graph more easily
-
2d0eebbf
by Benjamin Schubert
at 2019-01-31T17:06:23Z
Pass element directly to collect_element
This simplifies the loading
-
583bd97d
by Benjamin Schubert
at 2019-02-01T10:26:37Z
Merge branch 'bschubert/loader' into 'master'
Cleanup loader by linking LoadElements sooner
See merge request BuildStream/buildstream!1122
-
51cec3da
by Phil Dawson
at 2019-02-01T14:25:44Z
tests/cachekey: Test cache keys are independent of target elements
-
2b38aabe
by Phil Dawson
at 2019-02-01T15:33:00Z
Merge branch 'phil/cache-key-stability-test' into 'master'
tests/cachekey: Test cache keys are independent of target elements
See merge request BuildStream/buildstream!1123
-
dbb3d232
by James Ennis
at 2019-02-01T15:51:32Z
filter.py/filter.yaml: Documentation improvements
-
7e4205cb
by James Ennis
at 2019-02-01T15:51:32Z
filter.py: Add an example to the documentation
-
4109a34a
by James Ennis
at 2019-02-01T17:11:29Z
Merge branch 'jennis/filter-docs' into 'master'
Improve our filter documentation
Closes #278
See merge request BuildStream/buildstream!1112
-
c9345014
by James Ennis
at 2019-02-04T13:53:42Z
filter.py: Fail if declared domains do not exist in the parent element
This patch also uncovered the fact that our test_filter_deps_ok() test
has been inaccurate. Thus the element built in this test
(deps-permitted.bst) has been modified so that it build depends on the
input.bst element, as it should.
tests/filter.py: Ensure deps_ok test passes
-
3ab09651
by James Ennis
at 2019-02-04T14:47:43Z
Merge branch 'jennis/warn_for_nonexistent_domains' into 'master'
Fail when we explictly try to include/exclude non-existent domains in a filter element
See merge request BuildStream/buildstream!1117
-
cd8e5e27
by Tom Pollard
at 2019-02-05T11:22:01Z
Add --remote, -r option to bst build, inline with pull & push
Providing a remote will limit build's pull/push remote actions to
the given remote specifically, ignoring those defined via user or
project configuration.
-
a46650d0
by Tom Pollard
at 2019-02-05T12:41:30Z
Merge branch 'tpollard/buildremote' into 'master'
Add --remote, -r option to bst build, inline with pull & push
See merge request BuildStream/buildstream!1119
-
5c10acc4
by Angelos Evripiotis
at 2019-02-05T17:40:23Z
_includes: better provenance on recursive include
Use the provenance of the include block, instead of the whole node.
-
9746e19a
by Angelos Evripiotis
at 2019-02-05T17:40:23Z
_includes: better error on missing include
Previously, a missing include would result in an error like this:
Could not find file at not-a-file.include
Note that the file containing the include was not mentioned.
Now we get an error like this instead:
element.bst [line 7 column 5]: Include block references a file that
could not be found: 'not-a-file.include'.
-
7b7cc8d7
by Angelos Evripiotis
at 2019-02-05T17:40:23Z
_includes: better error on including directory
Previously, a missing include would result in an error like this:
mydir is a directory. bst command expects a .bst file.
Note that the file containing the include was not mentioned.
Now we get an error like this instead:
element.bst [line 12 column 0]: Include block references a
directory instead of a file: 'mydir'.
-
e3a71dca
by Angelos Evripiotis
at 2019-02-05T17:40:23Z
_includes: re-use file_path variable
Avoid an unnecessary call to os.path.join().
