-
de59ebdb
by ctolentino8
at 2018-11-02T16:41:54Z
plugins/sources/pip.py: Accomodate characters '-','.','_' for packages
-
8d7cf806
by ctolentino8
at 2018-11-02T16:41:54Z
tests/integration/project/files: Remove pypi-repo
-
9c2f9bf7
by Chandan Singh
at 2018-11-02T17:09:46Z
Merge branch 'chiaratolentino/fix-pip-source-regex' into 'master'
plugins/sources/pip.py: Accomodate characters '-','.','_' for packages
See merge request BuildStream/buildstream!914
-
3788e701
by Jürg Billeter
at 2018-11-03T11:52:00Z
element.py: Pass bare_directory also to SandboxRemote()
Fixes: 825963b5 ("element: Make "--sysroot" take a bare directory")
-
82e971ef
by Jürg Billeter
at 2018-11-05T11:33:20Z
Merge branch 'juerg/remote-execution' into 'master'
element.py: Pass bare_directory also to SandboxRemote()
See merge request BuildStream/buildstream!928
-
62942bfd
by Valentin David
at 2018-11-05T12:14:20Z
Use less fetchers in overnight build to work-around limitations from servers.
Issue happens with sourceware.org server.
Fixes #742
-
442da2f9
by Javier Jardón
at 2018-11-05T12:41:54Z
Merge branch 'valentindavid/limit_fetchers_in_overnight_build' into 'master'
Use less fetchers in overnight build to work-around limitations from servers.
Closes #742
See merge request BuildStream/buildstream!929
-
0993de53
by Richard Maw
at 2018-11-05T16:41:05Z
dev-requirements: Update pytest dependency to 3.8
"_frontend/app.py: Set correct element-path in interactive bst-init"
added a dependency on the tmp_path test fixture, which was added by 3.8.
-
be8f0a54
by richardmaw-codethink
at 2018-11-05T17:08:43Z
Merge branch 'richardmaw/pytest-dep-update' into 'master'
dev-requirements: Update pytest dependency to 3.8
See merge request BuildStream/buildstream!930
-
bfb639bf
by Jürg Billeter
at 2018-11-05T17:18:12Z
element.py: Remove artifacts parameter from constructor
Get the artifact cache from the context instead.
-
ca855f91
by Jürg Billeter
at 2018-11-05T17:18:12Z
_elementfactory.py: Remove unused artifacts parameter from create()
-
15fed21c
by Jürg Billeter
at 2018-11-05T17:18:12Z
_project.py: Remove unused artifacts parameter from create_element()
-
0085d2aa
by Jürg Billeter
at 2018-11-05T17:18:12Z
element.py: Remove unused artifacts parameter from _new_from_meta()
-
f69b1117
by Jürg Billeter
at 2018-11-05T17:18:12Z
_project.py: Remove unused artifacts parameter from load_elements()
-
e398f877
by Jürg Billeter
at 2018-11-05T17:18:12Z
element.py: Do not include type name of artifact cache in cache key
The artifact cache backend does not affect build outputs and we anyway
no longer have pluggable artifact cache backends. This hardcodes
CASCache instead of removing the entry completely to avoid cache key
changes.
-
626d20ae
by Jürg Billeter
at 2018-11-05T17:18:12Z
Split up artifact cache and CAS cache
This changes CASCache from a subclass to a delegate object of
ArtifactCache. As the lower layer, CASCache no longer deals with
elements or projects.
Fixes #659.
-
ec04446b
by Jürg Billeter
at 2018-11-05T17:51:39Z
Merge branch 'juerg/cas' into 'master'
Split up artifact cache and CAS cache
Closes #659
See merge request BuildStream/buildstream!922
-
9a045080
by Josh Smith
at 2018-11-06T13:23:19Z
utils: Fix _pretty_size() for sizes > 1024T
-
d3a07e6b
by Josh Smith
at 2018-11-06T13:23:19Z
Add regression test for _pretty_size
-
0c09fb9c
by richardmaw-codethink
at 2018-11-06T14:12:20Z
Merge branch 'Qinusty/unit-test-utils' into 'master'
Fix issue with _pretty_size with large numbers of bytes
See merge request BuildStream/buildstream!799
-
b4eec489
by Jim MacArthur
at 2018-11-06T16:44:14Z
_sandboxremote.py: Use the standard SandboxError.
Replaces the custom one which was erroneously added during development.
Fixes #746.
-
1f7acf74
by Jim MacArthur
at 2018-11-06T17:16:31Z
Merge branch 'jmac/remote-sandbox-errors' into 'master'
_sandboxremote.py: Use the standard SandboxError
Closes #746
See merge request BuildStream/buildstream!931
-
8e64ccef
by James Ennis
at 2018-11-07T11:06:11Z
using_config.rst: Update our instructions for how we declare caches in user config
-
09ab676d
by James Ennis
at 2018-11-07T11:06:11Z
format_project.rst: Update instructions regarding how we declare remote caches in project.conf
-
f514124f
by James Ennis
at 2018-11-07T11:06:11Z
using_configuring_artifact_server.rst: Remove examples and redirect to user config/project config
-
029ba17d
by richardmaw-codethink
at 2018-11-07T11:32:20Z
Merge branch 'jennis/docs_patch' into 'master'
Update our instructions/examples regarding how we declare remote caches
See merge request BuildStream/buildstream!932
-
261e2cd3
by Jim MacArthur
at 2018-11-07T12:02:43Z
element.py: Use _force_rmtree instead of custom code.
There are arguments for and against using onerror - see issue #735
for details.
-
8931e42c
by Jim MacArthur
at 2018-11-07T12:26:06Z
Merge branch 'jmac/make_writable_combination' into 'master'
Use _force_rmtree instead of custom code.
Closes #735
See merge request BuildStream/buildstream!927
-
6ccfab0b
by Valentin David
at 2018-11-07T16:28:47Z
Fix cache corruption by scripts when layout and integration commands are used
Root directory was marked as a non-artifact mount, so not using
SafeHardLink. However integration commands executed with write access
to the root directory.
Fixes #749
-
f24e20e9
by richardmaw-codethink
at 2018-11-07T16:59:50Z
Merge branch 'valentindavid/script-artifact-corruption' into 'master'
Fix cache corruption by scripts when layout and integration commands are used
Closes #749
See merge request BuildStream/buildstream!934
-
e28f668a
by Valentin David
at 2018-11-08T09:34:11Z
Fix infinite recursion in default strip debug command
Fixes #645.
-
83d15350
by Valentin David
at 2018-11-08T10:01:39Z
Merge branch 'valentindavid/fix-recursive-strip-debug' into 'master'
Fix infinite recursion in default strip debug command
Closes #645
See merge request BuildStream/buildstream!918
-
9f0e12f1
by Benjamin Schubert
at 2018-11-08T10:21:12Z
Move bwrap checks in platform/linux.py
Remove the bwraps checks from _site.py and put them in platform.linux
which is the only place where they are run.
This allows the removal of a double level of caching, making
reasoning about tests easier
-
cf2e0059
by Benjamin Schubert
at 2018-11-08T10:21:12Z
conftest.py: Ensure platform is not maintained between tests
This removes the `_instance` on the platform object that we use
for caching and not recreating the object everytime at the start
of every test.
This is to ensure our tests share the least amount of state.
The performance penalty is from 5 to 10% accross the whole test suite.
The readings were done 5 times for each before and after the change
and on the same computer.
-
c51ba01b
by Benjamin Schubert
at 2018-11-08T10:21:12Z
Test that helpful messages are raised when missing dependencies
This adds a `reason` to the SandboxEror thrown in sandboxdummy
to be able to understand where the error comes from
-
f116b9b7
by Benjamin Schubert
at 2018-11-08T11:07:12Z
Merge branch 'bschubert/tests-no-chroot-linux' into 'master'
test behavior when there is no sandbox on Linux
Closes #696 and #736
See merge request BuildStream/buildstream!921
-
cc2e6ae5
by Valentin David
at 2018-11-08T12:41:36Z
Fix bug with root mounted as non-artifact in script plugin.
The issue was introduced by 6ccfab0b1b25990e406446d5cbe5aee83a5e158a.
-
e578a89f
by Valentin David
at 2018-11-08T13:11:10Z
Merge branch 'valentindavid/fix-script-install-root' into 'master'
Fix bug with root mounted as non-artifact in script plugin.
Closes #756 and #749
See merge request BuildStream/buildstream!941
-
c0a8bb66
by Angelos Evripiotis
at 2018-11-08T15:49:16Z
contributing: WIP again to be kind to reviewers
As someone coming from GitHub to GitLab, I was pleasantly surprised by
the 'filter by WIP status' option.
To make sure we get the most out of it, add a guideline to keep the
filter clean, to reduce reviewer burden.
-
f7231e90
by Angelos Evripiotis
at 2018-11-08T15:49:16Z
contributing: non-WIP should always be landable
I was recently surprised that we don't prefer addressing review comments
in 'fixup!' commits.
Coming from GitHub, I've found that fixup commits make it easier for
reviewers to see what has changed since their last review. The idea is
to use '--autosquash' before landing to clean up the history again.
It's a pleasant surprise that it's easy to keep track of what changed in
merge-requests between pushes, so we can always keep the history clean.
Document this, so folks like me can see the light sooner.
-
f7643440
by Angelos Evripiotis
at 2018-11-08T15:49:16Z
contributing: emphasise the 'why' in commits
Explain why it's useful to mention the decisions made for a change and
provide links for background, e.g. issue numbers.
My general experience of git histories is that folks don't do enough
explaining, so it's great we have a contributing section for it.
I've found it's easier for folks to stick to rules when they know the
practical reasons for them, so provide some.
-
dd5e7b04
by Angelos Evripiotis
at 2018-11-08T16:17:04Z
Merge branch 'aevri/contributing_gitlab' into 'master'
Add more to GitLab-relevant parts of contributing
See merge request BuildStream/buildstream!935
-
fe33e328
by James Ennis
at 2018-11-08T17:54:18Z
using_config.rst: Add documentation to showing how to impose quotas on the local cache
This patch partially resolves #700
-
09faf002
by James Ennis
at 2018-11-08T17:54:18Z
artifactcache.py: Fix misleading error message when using % cache quota
Due to the changed Exception message, this patch also changes
the test_parse_size_over_1024T test in misc.py
-
d153453c
by Javier Jardón
at 2018-11-08T18:22:54Z
Merge branch 'jennis/quota_declaration_fix' into 'master'
Add local cache expiry documentation and fix misleading error message when specifying a percentage cache quota
Closes #700
See merge request BuildStream/buildstream!939
-
7738f6df
by Daniel Silverstone
at 2018-11-09T11:25:44Z
plugin.py: Redirect DEBUG from `__del__` to `sys.stderr`
Previously, debug messages from `Plugin.__del__()` went to stdout
which was a problem for the test suite in extremely rare and hard to
predict circumstances. This corrects that by sending the message to
stderr where all the rest of the messages tend to go.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
62f59eaa
by Valentin David
at 2018-11-09T13:30:06Z
Merge branch 'danielsilverstone-ct/plugin-destroy-debug-to-stderr' into 'master'
plugin.py: Redirect DEBUG from `__del__` to `sys.stderr`
See merge request BuildStream/buildstream!944
-
35ec6b6a
by Jim MacArthur
at 2018-11-09T14:58:22Z
_casbaseddirectory.py: Enable direct CAS-to-CAS import.
Fixes issue #574.
-
b93100ec
by Jim MacArthur
at 2018-11-09T14:58:22Z
_casbaseddirectory: Fix order of list_modified_paths
This will now return paths in the same order as list_relative_paths.
-
78691fa0
by Jim MacArthur
at 2018-11-09T14:58:22Z
virtual_directory_import.py: New test
-
7ce6581b
by Jim MacArthur
at 2018-11-09T15:25:46Z
Merge branch 'jmac/cas_to_cas_v2' into 'master'
Direct CAS-to-CAS import
Closes #574
See merge request BuildStream/buildstream!911
-
6f4351ec
by Benjamin Schubert
at 2018-11-09T16:53:42Z
source.py: don't let StopIteration propagate to silence() contextmanager
As per PEP 0479 (https://www.python.org/dev/peps/pep-0479/),
StopIteration thrown in context managers are not valid starting from
Python 3.7.
-
e6e03451
by richardmaw-codethink
at 2018-11-12T11:05:15Z
Merge branch 'bschubert/fix-silence-stopiteration-pep-0479' into 'master'
source.py: don't let StopIteration propagate to silence() contextmanager
See merge request BuildStream/buildstream!945
-
ae5ccd76
by Phillip Smyth
at 2018-11-12T14:57:51Z
_project.py: Validate nodes early in Project._load
_project.py: Added validate_nodes() helper function to prevent duplicate lists
element-path was being used before node validation resulting in uncaught errors
-
56a54161
by Phillip Smyth
at 2018-11-12T14:57:51Z
tests/frontend: Add test for invalid element-path
-
e55a9703
by Jürg Billeter
at 2018-11-12T15:44:45Z
Merge branch 'element-path_not_validated' into 'master'
Element path not validated before use
See merge request BuildStream/buildstream!937
-
c87d6e45
by Valentin David
at 2018-11-14T11:43:33Z
Run tests on aarch64
Fixes #755
-
16b01489
by Valentin David
at 2018-11-14T11:43:33Z
Disable cachekey tests on other architectures than x86_64
-
85046b29
by Valentin David
at 2018-11-14T11:43:33Z
Disable tests on example on other architectures than x86_64
-
dd5a073b
by Valentin David
at 2018-11-14T11:43:33Z
Add support for aarch64 in a test
-
c5d72ae7
by Valentin David
at 2018-11-14T11:52:53Z
Test aarch64 only overnight
-
264a57f6
by Javier Jardón
at 2018-11-14T12:22:03Z
Merge branch 'valentindavid/integration-tests-aarch64' into 'master'
Run tests on aarch64
Closes #369 and #755
See merge request BuildStream/buildstream!948
-
da735e56
by Richard Maw
at 2018-11-14T13:30:34Z
_platform/linux.py: Move get_bwrap_version into _site.py
It's inconvenient to have to create a Linux platform to parse the bwrap version
and we want to get the version in a consistent manner.
-
e7633500
by Richard Maw
at 2018-11-14T13:30:34Z
buildstream/sandbox/_sandboxbwrap.py: Distinguish sandbox failure from command failure
If `bwrap` fails to set up the sandbox and start the payload command
it won't write an exit-code in --json-status-fd,
so we can report if it was a sandboxing failure if we don't get exit-code status
and a payload command failure if we do and it's non-zero.
Closes https://gitlab.com/BuildStream/buildstream/issues/286
-
e9e08823
by Richard Maw
at 2018-11-14T13:30:34Z
tests/testutils/site.py: Check for bwrap supporting --json-status-fd
-
90ca007e
by Richard Maw
at 2018-11-14T13:30:34Z
tests/integration/sandbox-bwrap.py: Test distinguishing sandbox exit code from command
-
327b19dd
by richardmaw-codethink
at 2018-11-14T13:59:16Z
Merge branch 'richardmaw/distinguish-sandboxing-build-fail' into 'master'
Distinguish between bubblewrap sandboxing failure and command failure
Closes #286
See merge request BuildStream/buildstream!868
-
082b3811
by Angelos Evripiotis
at 2018-11-15T19:48:12Z
contributing: add guidance on unit tests
Decrease uncertainty around whether unit tests are welcome in the
project or not.
-
b5b79056
by Jürg Billeter
at 2018-11-15T20:24:58Z
Merge branch 'aevri/unit_tests' into 'master'
contributing: add guidance on unit tests
See merge request BuildStream/buildstream!943
-
199bfff1
by Tom Pollard
at 2018-11-17T13:07:59Z
Add cli main and user config option for 'pull-buildtrees' context.
_context.py: Add pull_buildtrees global user context, the default
of which is set to False via the addition of pull-buildtrees to
userconfig.yaml cache group.
_frontend/app.py & cli.py: Add --pull-buildtrees as a bst main
option, which when passed will override the default or user defined
context for pull_buildtrees.
tests/completions/completions.py: Update for the added flag.
-
454dcaf1
by Tom Pollard
at 2018-11-17T13:07:59Z
Don't pull artifact build trees by default.
The addition of cached build trees being included in element
artifacts has led to mostly redundant download overheads when
pulling from a remote artifact server. As such the default behaviour
of pull shouldn't fetch the build tree object if available.
element.py: extend relevant pull logic for specified subdir
consideration and ensure push logic does not lead to partial
artifact pushes. Change assumption that the buildtree can be
extracted if artifact is cached. __cached_buildtree() and
__pull_directories() helpers added.
_artifactcache/: artifactcache.py & cascache.py inclusion of
helper functions for subdir artifact checking & fetching, fetch
logic extended to only pull required artifact directories.
extract & checkout updated to handle full/partial operation.
tests/: addition of integration test pullbuildtrees.py,
buildtrees.py adapted cli options, testutils/artifactshare.py
has_artifact changed to return artifact digest on true condition.
-
0d309478
by Jürg Billeter
at 2018-11-17T13:49:58Z
Merge branch 'tpollard/494' into 'master'
Don't pull artifact buildtrees by default
See merge request BuildStream/buildstream!786
-
27932739
by Benjamin Schubert
at 2018-11-19T09:46:39Z
plugins/sources/pip.py: also look for python version named "python"
In some virtual environments, we might not have the binary with the
exact python version, which would lead the current pip plugin to
fail to find a working pip version.
This adds "python" at the start of the list of valid python versions
-
ea2de561
by Benjamin Schubert
at 2018-11-19T10:19:24Z
Merge branch 'bschubert/fix-pip-python' into 'master'
plugins/sources/pip.py: also look for python version named "python"
Closes #758
See merge request BuildStream/buildstream!942
-
f23b6031
by Benjamin Schubert
at 2018-11-19T11:39:51Z
tests/frontend/buildtrack.py: check for success after invocation
Previously the result of the first call in test_build_track was not
checked to be successful, which would lead to potential erros later on
that were hard to debug.
Let's check it everytime
-
a6defc0b
by Benjamin Schubert
at 2018-11-19T11:39:51Z
Fix os.rename in git source element to correctly handle error codes
According to the documentation
(https://www.unix.com/man-page/POSIX/3posix/rename/), when the directory
already is there, either EEXIST or ENOTEMPTY could be thrown.
Previously only ENOTEMPTY was checked.
Done:
- Separated the move into its own function
- Check for both errors
- Create unit tests for it, covering most test cases
-
88089d2d
by Benjamin Schubert
at 2018-11-19T11:39:51Z
Extract atomic move function to utils.py
Moving atomically a file/directory can be tricky since different
errors might be raised for the same underlying problem.
Having a utility function to reduce this discrepancies will help
in ensuring we have correct behavior
-
fd9e46be
by Benjamin Schubert
at 2018-11-19T11:39:51Z
cascache.py: use move_atomic instead of manual error checking
-
d32e0b83
by Benjamin Schubert
at 2018-11-19T11:39:51Z
sources/pip.py: Use move_atomic instead of manual os.rename
This uses move_atomic insteand of the manual os.rename and manual error
checking and throws a SourceError for consistency with other modules.
-
6f837118
by Benjamin Schubert
at 2018-11-19T12:22:40Z
Merge branch 'bschubert/fix-atomic-move-git-repo' into 'master'
Fix os.rename in git source element to correctly handle error codes
See merge request BuildStream/buildstream!938
-
30b72244
by Chandan Singh
at 2018-11-19T12:35:01Z
buildstream/_pipeline.py: Simplify error message for assert_consistent()
Currently, `Pipeline.assert_consistent()` prints an error message like
the one below when certain elements have inconsistent sources:
```
...
Element: hello.bst is inconsistent
Source tar source at hello.bst [line 16 column 2] is missing ref
...
```
Drop the word "Source" from the beginning of the message as
"Source <source-kind> source ..." is kind of awkward to read.
-
76c5d2f8
by Chandan Singh
at 2018-11-19T12:35:01Z
Add `bst source-checkout` command
As discussed in
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
add `bst source-checkout` command. This will allow users to checkout
sources for a given target.
* _frontend/cli.py: Add source-checkout command
* _pipeline.py: Add assert_sources_cached() method
* _stream.py: Add source_checkout method, abstract out
__check_location_writable() method that used to part of checkout()
-
9f629638
by Chandan Singh
at 2018-11-19T13:03:36Z
Merge branch 'chandan/source-checkout' into 'master'
Add `bst source-checkout` command
See merge request BuildStream/buildstream!820
-
bc827cc8
by Abderrahim Kitouni
at 2018-11-19T14:55:15Z
plugins/elements/cmake.yaml: always specify variable types
cmake sometimes misinterprets relative paths as relative to the current directory
if this is not specified. See freedesktop-sdk/freedesktop-sdk#431
adjust tests/format/variables.py accordingly.
-
5fbc5f41
by Valentin David
at 2018-11-19T15:31:05Z
Merge branch 'abderrahim/cmake-variable-types' into 'master'
plugins/elements/cmake.yaml: always specify variable types
Closes #761
See merge request BuildStream/buildstream!947
-
06e28860
by Benjamin Schubert
at 2018-11-19T15:52:24Z
Don't cache sandbox errors
Sandbox errors (like missing host tools) are dependent on the host
system and rarely on what is actually done.
It is therefore better to not cache them as they are subject to
change between two runs.
Also add test to ensure sandbox failure are not cached
-
dd36cfbc
by Benjamin Schubert
at 2018-11-19T15:52:24Z
element.py: Only cache on element error or success
Reverse the way we were handling caching by only caching
when we know it's a problem with the build or the build is a
success in order not to cache transient errors.
-
eac7274d
by Benjamin Schubert
at 2018-11-19T16:29:07Z
Merge branch 'bschubert/dont-cache-errors-from-host-tools' into 'master'
Don't cache sandbox failures
Closes #727
See merge request BuildStream/buildstream!895
-
625dfe1f
by Chandan Singh
at 2018-11-19T16:41:17Z
NEWS: Add entry for the new source-checkout command
`bst source-checkout` command was recently added in
https://gitlab.com/BuildStream/buildstream/merge_requests/820. Add a
NEWS entry for the new command.
-
d4f12184
by Chandan Singh
at 2018-11-19T17:15:10Z
Merge branch 'chandan/source-checkout-news' into 'master'
NEWS: Add entry for the new source-checkout command
See merge request BuildStream/buildstream!960
-
ba6c96e8
by Richard Maw
at 2018-11-19T19:54:30Z
CasBasedDirectory: Change constructor to take a CASCache instead of a Context
The Context was only used to obtain a reference to the CASCache
and set the unused cas_directory field.
-
8a0dc3a3
by Jürg Billeter
at 2018-11-19T19:54:30Z
tests/testutils/artifactshare.py: Do not create a fake context
The fake context did not set the cache quota, triggering an error. With
CASCache now separate from ArtifactCache, we can instantiate a CASCache
without context.
-
8722aced
by Jürg Billeter
at 2018-11-19T19:54:30Z
tests/plugins/pipeline.py: Load context default values
-
fc56ffa4
by Jürg Billeter
at 2018-11-19T19:54:30Z
_context.py: Drop duplicated default values for user configuration
The default values are in userconfig.yaml, together with the
documentation. The default values should not be duplicated in
_context.py.
-
3f663d82
by Jürg Billeter
at 2018-11-19T20:36:57Z
Merge branch 'juerg/context-default-values' into 'master'
_context.py: Drop duplicated default values for user configuration
See merge request BuildStream/buildstream!953
-
33438248
by Jim MacArthur
at 2018-11-20T11:22:44Z
Docs: Add remote execution architecture documentation.
