-
aa0cbf5d
by Martin Blanchard
at 2018-10-23T10:54:40Z
_sandboxremote.py: Try to reopen operation steam on failure
The REAPI allows a client to reconnect to an ongoing operation stream by
providing a WaitExecution(). If implemented on server side, BuildStream
will try to recover from connection errors using it.
https://gitlab.com/BuildStream/buildstream/issues/630
-
552f5fc6
by Jim MacArthur
at 2018-10-23T11:19:48Z
Merge branch 'mablanch/630-remote-execution-reconn' into 'master'
Handle connection losses during remote build execution
See merge request BuildStream/buildstream!806
-
276d2195
by Angelos Evripiotis
at 2018-10-25T10:17:49Z
_downloadablefilesource: handle ValueError-s
-
2d012a21
by Angelos Evripiotis
at 2018-10-25T10:17:49Z
Merge branch 'aevri/unknown-uri-452' into 'master'
_downloadablefilesource: handle ValueError-s
Closes #452
See merge request BuildStream/buildstream!893
-
cf4741ce
by James Ennis
at 2018-10-25T13:05:43Z
docs: Add helpful notes to artifact server configuration instructions
-
5737bb06
by James Ennis
at 2018-10-25T13:05:43Z
using_configuring_artifact_server.rst: Fix incorrect reference to bst push (with link)
-
28317622
by James Ennis
at 2018-10-25T13:35:22Z
Merge branch 'jennis/patch_remote_cache_docs' into 'master'
Improve our documentation for configuring an artifact server
See merge request BuildStream/buildstream!899
-
d2743c52
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_context.py: Remove useless `return` from `Context.message`
This pointless bare `return` was causing modern pylint to raise an
error.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
95adf09e
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_variables.py: Fix lint issue in `find_recursive_variable`
The python `for`/`else` construct only makes sense if the `for` loop body
has a `break` statement. Otherwise the use of `else` causes pylint
to raise an error.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
9359c067
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_workspaces.py: Simplify chained comparison
In `_parse_workspace_config` there was a chained version comparison
which is simplified in this commit to quieten a pylint issue.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
796bd8c9
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_yaml.py: use `in (a,b)` to simply boolean checks
Where we use a construct `val == foo or val == bar` we can instead use
`val in (foo, bar)` which pylint prefers.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
5a3a537f
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_yaml.py: Quieten a lint in Python < 3.6
For Python before 3.6, `path.resolve()` could not take the `strict`
keyword argument. Linting on such Python versions will raise an unnecessary
issue given the check present. As such, quieten that lint.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b279c730
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_yamlcache.py: Correct bug in `YamlCache._get_filepath()`
The correct file path was computed but never returned. This was
never caught because of a test missing consumption of one of its
parameters. This commit resolves the bug mentioned in #722 leaving
the test to be amended by a future commit.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
527f3b32
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_yamlcache.py: Correct braino in variable name
This braino would mean that we'd fail to raise the right exception
if we failed to find the right project object when unpickling the
yaml cache.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
6d0c697d
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_yamlcache.py: Remove unused imports
The imports of _cachekey and utils were unused.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
32323695
by Daniel Silverstone
at 2018-10-25T14:43:52Z
buildelement.py: Quieten a lint about `__commands`
Unfortunately elements don't override `__init__` in general, so
we have to quieten pylint here.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
c5c0654d
by Daniel Silverstone
at 2018-10-25T14:43:52Z
element.py: Simplify some conditions with `in (foo, bar)`
Where we have conditions of the form `var == foo or var == bar` it
can be simplified to `var in (foo, bar)` which pylint prefers.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
4f84e2f6
by Daniel Silverstone
at 2018-10-25T14:43:52Z
element.py: Silence pylint warning about abstract method
Unfortunately pylint can't understand that we'll always use subclasses
of Element and as such complains of a no-return function (`assemble()`)
having its return value used. This quietens that warning.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b4645745
by Daniel Silverstone
at 2018-10-25T14:43:52Z
plugin.py: use dict.get() instead of an if statement
Pylint recommends the use of dict.get() rather than a multi-line if
statement for handling optional keyword arguments.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
68ec9bcf
by Daniel Silverstone
at 2018-10-25T14:43:52Z
source.py: Quiet several pylint issues with abstract methods
A number of times, abstract methods are called which pylint cannot
possibly understand will only ever happen in subclasses which implement
the abstract methods. This silences those specific warnings
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
1bf0a031
by Daniel Silverstone
at 2018-10-25T14:43:52Z
source.py: Remove unused local variable
Remove `source_kind` as it was unused.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
e9c6db1e
by Daniel Silverstone
at 2018-10-25T14:43:52Z
source.py: Remove unused variable
Remove the `context` variable which was unused.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
5a187a1e
by Daniel Silverstone
at 2018-10-25T14:43:52Z
utils.py: Disable lint warning about popen
We use `preexec_fn` in our use of Popen. Since there are threads
in the codebase, this is not necessarily safe. However we go to
great lengths to ensure that the main process doesn't spawn threads
(they're used in gRPC a lot) and so it should be safe for our use.
As such, we disable the lint here.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b368f57f
by Daniel Silverstone
at 2018-10-25T14:43:52Z
utils.py: Group stat imports
The imports of `stat` and `S_ISDIR` from `stat` should be grouped to
satisfy pylint.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b45fea19
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_artifactcache/artifactcache.py: Remove superfluous parens
The pylint report dislikes the superfluous parens which used to
be here in this assert.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
3e4a6507
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_artifactcache/artifactcache.py: Silence lint issues
The pylint tool is unable to understand that the abstract methods
in the artifact cache will only be called when there's concrete
implementations behind them. Silence the lint errors for these
specific calls.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
447780b0
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_frontend/app.py: Remove unused import of `resource`
The `resource` import was unused. Removed.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
247b9fa8
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_frontend/status.py: Reorder imports
The `curses` import is "standard" whereas `click` is not. As such, despite
alphabetical order normally being the rule, `curses` must come before `click`.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
522867c8
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_fuse/{hardlinks,mount}.py: Remove dangerous use of {} in defaults
Since default values in arguments to functions and methods are created once
at the compilation of the code, it is dangerous to include list and dict
literals in them. This changes the use of {} to None.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
31a7bce5
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_fuse/mount.py: Deal with abstract method lint issue
Unfortunately pylint can't tell this is an abstract method which will
only ever end up invoked from a class which concretises it. As such
we need to silence this warning.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
eb805c6c
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_options/optionbool.py: Refactor conditions
The pylint tool prefers that conditions of the form "var == foo or var == bar"
be refactored into "var in (foo, bar)".
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
aca9a612
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_platform/darwin.py: Remove useless __init__()
An otherwise empty __init__ passing no arguments to super.__init__
is worthless. Remove it.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
77b1506f
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_platform/darwin.py: Remove unused imports
The import of `resource` was unused.
The import of `PlatformError` was unused.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
70000b0d
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_platform/linux.py: Remove unused imports
The imports of `Message` and `MessageType` were unused.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
bab617d0
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_platform/linux.py: Simplify return statement
The linter dislikes an if/return/else/return where it can be
simplified to a return of the if condition.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
45a1d15f
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_scheduler/jobs/job.py: Correct lint issue around abstract method
The linter cannot detect abstract methods the way we use them, so
this silences the warning which ensues.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
e9154b4d
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_scheduler/queue/pullqueue.py: Remove unused return value
The `done()` method does not return anything anywhere else.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
7b082bb7
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_scheduler/queue/trackqueue.py: Remove unused import
The `SourceError` import was never used
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
f1cfc0b7
by Daniel Silverstone
at 2018-10-25T14:43:52Z
plugins/source/git.py: Rationalised unused return values
The `assert_ref_in_track()` method's return value was never used so
rationalise it to never return a value.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
7c72d25b
by Daniel Silverstone
at 2018-10-25T14:43:52Z
sandbox/_mount.py: Do not use dict literals in argument defaults
The use of dictionary literals in argument defaults is disrecommended
due to the way that they are static and thus potentially very confusing.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
4e7408ee
by Daniel Silverstone
at 2018-10-25T14:43:52Z
sandbox/_mounter.py: Remove useless inheritance on `object`
In Python 3, inheriting from `object` explicitly is pointless.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
df495d16
by Daniel Silverstone
at 2018-10-25T14:43:52Z
sandbox/_sandboxbwrap.py: Disable lint about dict.get
Sometimes `dict.get()` is preferable, in this instance it's less clear
so we'll disable that lint
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
63c6e6b1
by Daniel Silverstone
at 2018-10-25T14:43:52Z
sandbox/_sandboxchroot.py: Silence warning about unused variable
The variable was unused, remove it.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
3394741a
by Daniel Silverstone
at 2018-10-25T14:43:52Z
sandbox/_sandboxchroot.py: Silence warning about preexec_fn
We are super-careful to not use threads in places where we might use
Popen and as such this warning is save to quash.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
cd20e7a1
by Daniel Silverstone
at 2018-10-25T14:43:52Z
sandbox/_sandboxchroot.py: Silence warning about dict.get
Sometimes `dict.get()` is preferable to an `if` statement, but this time
it's clearer if we keep the structure. As such, silence the warning.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
5087f5c5
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_artifactcache/cascache.py: Refactor slightly for indentation
Correct some multi-line conditional indentation by slight refactor
if necessary.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
d73d965b
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_artifactcache/cascache.py: Silence len(SEQUENCE) as condition warnings
The lint tool dislikes the use of `if len(SEQ) == 0` type statements
because nominally `len(SEQ)` may be more expensive than `bool(SEQ)`.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
fa740402
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_artifactcache/cascache.py: Prepare attributes in __init__
Python linters prefer that attributes are defined in __init__ rather
than left to later routines.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
ad8271f0
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_artifactcache/cascache.py: Remove unused imports
The imports of `Message` and `MessageType` were unused.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
1445a901
by Daniel Silverstone
at 2018-10-25T14:43:52Z
_artifactcache/cascache.py: Cleanup lint warning about indexing
In order to quieten the linter, and to make a more correct comparison,
we switch to using the `Code.OK` enumeration from `google.rpc.Code`
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
19df07ea
by Tristan Maat
at 2018-10-25T14:43:52Z
Bump CI image tags
We need to do this so that the new linting errors pop up
-
64dbe9ae
by Daniel Silverstone
at 2018-10-25T14:43:52Z
MANIFEST: Include a lot more missing stuff in the manifest
Versioneer needs to be in the MANIFEST.in and its .pyc needs to be ignored
in the .gitignore. Also much docs were not being included, nor conftest.py.
Much of the test suite data files needed including, so changed that to a
raw include of everything in the tests/ tree.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
f6bc25dd
by Daniel Silverstone
at 2018-10-25T14:43:52Z
.gitlab-ci.yml: Change how coverage is recovered
We need to alter how coverage data is recovered in order to work with
newer coverage tooling.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
1c1d14e7
by Chandan Singh
at 2018-10-25T15:04:00Z
Merge branch 'danielsilverstone-ct/fix-lint-issues' into 'master'
Fix lint issues
See merge request BuildStream/buildstream!892
-
23140b6b
by Chandan Singh
at 2018-10-25T15:25:22Z
sources: Don't redundantly document directory configuration
Instead of redundantly documenting "directory" configuration option in
each source plugin, refer to the built-in functionality documentation of
the Source class.
* source.py: Add anchor for built-in functionality.
* sources/plugins/*.py: Remove directory config option, and add reference to
Source's built-in functionality.
Fixes #713.
-
db2a676e
by Chandan Singh
at 2018-10-25T15:26:39Z
build elements: Add reference to BuildElement's built-in config
For each element plugin derived from BuildElement, add reference to the
built-in functionality documentation of BuildElement.
* buildelement.py: Add anchor for built-in functionality.
* plugins/elements/*.py: Add refence to BuildElement's built-in
functionality for each build element.
Related to #713.
-
356d84cd
by Chandan Singh
at 2018-10-25T16:59:13Z
Merge branch 'chandan/source-doc-directory' into 'master'
plugins: Don't redundantly document common configurations
Closes #713
See merge request BuildStream/buildstream!896
-
8c41b5b8
by Daniel Silverstone
at 2018-10-26T09:02:30Z
_yaml.py: Do not insert into cache if retrieved from cache
We were unconditionally inserting into the yaml cache anything we had
retrieved from the yaml cache. This had the side-effect of being expensive
on re-runs of commands where yaml hadn't changed.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
494d7018
by Jonathan Maw
at 2018-10-26T09:26:36Z
Merge branch 'danielsilverstone-ct/do-not-reinsert-yaml' into 'master'
_yaml.py: Do not insert into cache if retrieved from cache
See merge request BuildStream/buildstream!902
-
80762ecb
by Benjamin Schubert
at 2018-10-26T09:45:39Z
Check if command is a str and replace by list before checking existence
Currently we would make sure the sandbox had a command before
converting it to a list if it was given as a string. That meant that a string
command would never exist and the check be invalid.
This also adds the same logic in the dummy sandbox for consistency.
-
f131c407
by Benjamin Schubert
at 2018-10-26T10:09:51Z
Merge branch 'bschubert/fix-command-sandbox' into 'master'
Check is command is a str and replace by list before checking existence
Closes #728
See merge request BuildStream/buildstream!898
-
54d89aa1
by Daniel Silverstone
at 2018-10-26T10:20:36Z
_loader/loadelement.py: Use enumerate() in dependency extraction
In cases where elements end up with large numbers of dependencies,
the n-squared performance in `_extract_depends_from_node` can produce
significant delays during project load. This patch corrects that.
In addition we were double-retrieving provenance which was unnecessary.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
e4c18ba4
by Phil Dawson
at 2018-10-26T10:43:55Z
Merge branch 'danielsilverstone-ct/use-enumerate' into 'master'
_loader/loadelement.py: Use enumerate() in dependency extraction
See merge request BuildStream/buildstream!901
-
20e5c58a
by James Ennis
at 2018-10-26T11:14:58Z
using_configuring_artifact_server.rst: Fix lines that are too long
-
320dae02
by James Ennis
at 2018-10-26T11:37:42Z
docs: Add more information to "managing the cache with systemmd" section
-
1eb993bf
by James Ennis
at 2018-10-26T12:02:49Z
Merge branch 'jennis/another_docs_patch' into 'master'
More information regarding the use of systemd to manage the cache
See merge request BuildStream/buildstream!904
-
f069d82f
by Daniel Silverstone
at 2018-10-26T12:19:58Z
various: Move _sentinel from utils.py to _yaml.py
The `_sentinel` in `utils.py` was used only for yaml node tracking.
As such, simplify matters by removing it from `utils.py` and move it
to `_yaml.py` which means that we no longer need to try and avoid a
circular import issue by means of runtime importing.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
283887a5
by James Ennis
at 2018-10-26T13:12:12Z
Merge branch 'danielsilverstone-ct/yaml-sentinel-rework' into 'master'
Move _sentinel from utils.py to _yaml.py
See merge request BuildStream/buildstream!903
-
ebcc348d
by Chandan Singh
at 2018-10-26T14:05:44Z
buildstream/_pipeline.py: Simplify error message for assert_consistent()
Currently, `Pipeline.assert_consistent()` prints an error message like
the one below when certain elements have inconsistent sources:
```
...
Element: hello.bst is inconsistent
Source tar source at hello.bst [line 16 column 2] is missing ref
...
```
Drop the word "Source" from the beginning of the message as
"Source <source-kind> source ..." is kind of awkward to read.
-
ce367100
by Chandan Singh
at 2018-10-26T14:05:44Z
Add `bst source-checkout` command
As discussed in
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
add `bst source-checkout` command. This will allow users to checkout
sources for a given target.
* _frontend/cli.py: Add source-checkout command
* _pipeline.py: Add assert_sources_cached() method
* _stream.py: Add source_checkout method, abstract out
__check_location_writable() method that used to part of checkout()
