-
10eeb503
by Javier Jardón
at 2018-08-28T22:22:00Z
doc/source/conf.py: Fix E201, E202 warnings
-
d3c32ca2
by Javier Jardón
at 2018-08-28T22:22:00Z
doc/source/conf.py: Fix E402 warning
-
3ae5fd05
by Javier Jardón
at 2018-08-28T22:22:00Z
Fix E305 warnings
-
c6fb5ba7
by Javier Jardón
at 2018-08-28T22:22:00Z
Fix E741 warnings
-
c5eca59d
by Tristan Van Berkom
at 2018-08-29T10:45:33Z
Merge branch 'jjardon/pycodestyle_fixes' into 'master'
Some pycodestyle (PEP8) fixes
See merge request BuildStream/buildstream!746
-
63c6ee72
by Tristan Van Berkom
at 2018-08-29T10:46:30Z
doc: Adding badges.py release badge generator
This adds a step to the docs generation Makefile to generate
release.svg and snapshot.svg badges, modelled after the gitlab
badges.
This also adds the generated badges directory in docs to .gitignore
-
2d527052
by Tristan Van Berkom
at 2018-08-29T10:46:30Z
README.rst: Adding the release and snapshot badges to the readme
This is a part of #528
-
29e7eea8
by Tristan Van Berkom
at 2018-08-29T10:46:30Z
doc: Adding the release badges to the install page and the semantic versioning page
This is a part of #528
-
5d508779
by Tristan Van Berkom
at 2018-08-29T11:13:53Z
Merge branch 'tristan/docs-version-badge' into 'master'
Release badges
See merge request BuildStream/buildstream!742
-
323a5403
by Chandan Singh
at 2018-08-29T12:13:28Z
setup.py: Add more metadata useful for PyPI
- setup.py: List useful links using `project_urls`
- setup.py: Specify minumum python vresion using `python_requires`
`python_requires` is the new way of specifying requirements for python
version, as documented @
https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires.
This will also make this requirement appear nicely on the PyPI project
page.
- setup.py: Add project classifiers
PyPI will use these classifiers to categorize projects while searching
or browsing. Full list of classifiers can be found at
https://pypi.org/classifiers.
-
5fc7b5ac
by Tristan Van Berkom
at 2018-08-29T12:37:08Z
Merge branch 'chandan/setup.py-metadata' into 'master'
setup.py: Add more metadata useful for PyPI
See merge request BuildStream/buildstream!718
-
42ad937d
by Josh Smith
at 2018-08-29T12:38:15Z
Prevent ValueError on URLs without an alias
Fixes #597
-
aa3a33b3
by Tristan Van Berkom
at 2018-08-29T13:03:58Z
Merge branch 'Qinusty/597-non-alias-url-fix' into 'master'
Fix ValueError traceback for URL without alias or URI scheme
Closes #597
See merge request BuildStream/buildstream!700
-
2a2a79de
by Valentin David
at 2018-08-29T14:11:48Z
Disable round-tripping when element is not modified
-
2df7607c
by Tristan Van Berkom
at 2018-08-29T14:43:55Z
Merge branch 'valentindavid/roundtripping_only_when_modified' into 'master'
Disable round-tripping when element is not modified
See merge request BuildStream/buildstream!748
-
c146dde5
by Benjamin Schubert
at 2018-08-29T16:00:24Z
Log not-found objects in the cache as SKIPPED
-
19e87afb
by Benjamin Schubert
at 2018-08-29T16:01:12Z
Add tests that not-found objects in cache are shown as SKIPPED
-
9cc5817f
by Qinusty
at 2018-08-29T16:25:48Z
Merge branch 'bschubert/log-missed-cache' into 'master'
Log not-found objects in the cache as SKIPPED
See merge request BuildStream/buildstream!729
-
3c8646e3
by Josh Smith
at 2018-08-29T16:34:22Z
Add cyclic check within variable resolution
This aims to address #600, this will raise an exception when a resolved
variable contains a reference to the variable.
-
d41c42d6
by Josh Smith
at 2018-08-29T16:34:22Z
Add tests for cyclic variables check
Note: This modifies the docker containers used for testing to supply the
pytest-timeout package.
-
c7b72c30
by Qinusty
at 2018-08-29T16:58:48Z
Merge branch 'Qinusty/600-recursive-variables' into 'master'
Add cyclic check within variable resolution
See merge request BuildStream/buildstream!712
-
d28e1353
by Tristan Van Berkom
at 2018-08-30T06:02:41Z
tests/plugins/filter.py: Don't run redundant tests
There is no reason that the filter element codepaths can behave
differently depending on the Source implementation used in the test,
as the Source implementation does not have any filter specific
virtual methods.
Removing the redundant tests and just performing these tests with the git source.
-
e77ae07a
by Tristan Van Berkom
at 2018-08-30T06:36:41Z
Merge branch 'tristan/reduce-filter-tests' into 'master'
tests/plugins/filter.py: Don't run redundant tests
See merge request BuildStream/buildstream!753
-
9ea40e2c
by Tristan Van Berkom
at 2018-08-30T06:38:13Z
tests/frontend/mirror.py: Reenable test_mirror_fetch_upstream_absent[ostree]
This test was skipped because of issue #538, but #538 was fixed
and the test was still not reenabled.
-
8aec1102
by Tristan Van Berkom
at 2018-08-30T07:06:17Z
Merge branch 'tristan/538-reenable-ostree-test' into 'master'
tests/frontend/mirror.py: Reenable test_mirror_fetch_upstream_absent[ostree]
See merge request BuildStream/buildstream!755
-
fcc17c82
by Tristan Van Berkom
at 2018-08-30T08:44:34Z
source.py: Stylistic changes in Source.__do_fetch()
Added some comments to make the flow easier to follow, and
removed an annoying 'success' variabled in favor of a for / else
loop statement.
-
60ad044a
by Tristan Van Berkom
at 2018-08-30T08:44:34Z
source.py: Move Source.mark_download_url() to the public methods.
This was sitting in the section for abstract methods, but this
is most definitely not an abstract method to be implemented by
Sources.
-
c339f917
by Tristan Van Berkom
at 2018-08-30T08:44:34Z
plugins/source/git.py: Fix formatting of url in tracking
This was displaying the aliased URL which is pretty useless,
use the translated URL for the timed activity.
-
26565f11
by Tristan Van Berkom
at 2018-08-30T09:12:28Z
Merge branch 'tristan/source-mirroring-changes' into 'master'
Minor code changes revolving around source mirroring
See merge request BuildStream/buildstream!758
-
a38d9163
by Valentin David
at 2018-08-30T10:26:57Z
Report processing errors from tracking
Failures to write files when tracking were not reported.
Fixes #533.
-
02138181
by Tristan Van Berkom
at 2018-08-30T11:36:00Z
Merge branch 'valentindavid/post_tracking_errors' into 'master'
Report processing errors from tracking
Closes #533
See merge request BuildStream/buildstream!747
-
d23d4f3e
by Tristan Van Berkom
at 2018-08-30T12:02:27Z
_artifactcache/artifactcache.py: Write the cache_size file atomically
This is causing issues while the size file is being read and written
simultaneously.
The proper fix will be to read/add/save the file atomically and that
will require locking, but this fix is a good stop gap to existing crashes.
-
bb6aa5a5
by Tristan Van Berkom
at 2018-08-30T12:27:55Z
Merge branch 'tristan/atomic-cache-size-file' into 'master'
_artifactcache/artifactcache.py: Write the cache_size file atomically
See merge request BuildStream/buildstream!762
-
680e4fe1
by knownexus
at 2018-08-30T13:37:55Z
Replacing string 'bzr' with value from host tools
-
8cd719eb
by Phillip Smyth
at 2018-08-30T15:47:06Z
Merge branch 'bzr_fix' into 'master'
Replacing string 'bzr' with value from host tools
See merge request BuildStream/buildstream!764
-
18d0bfb4
by Tom Pollard
at 2018-08-30T19:14:33Z
buildstream/_project.py: Report if project.conf is missing name
Explicitly check that project.conf contains a name. This resolves
the issue of the provenance check from _yaml.py incorrectly reporting
the offending file as the default_config_node projectconfig.yaml
when no name key exists in the pre_config_node dict.
-
95121148
by Tiago Gomes
at 2018-08-30T20:52:29Z
Merge branch 'tpollard/591' into 'master'
buildstream/_project.py: Report if project.conf is missing name
See merge request BuildStream/buildstream!680
-
775d3fca
by Tristan Van Berkom
at 2018-08-31T06:05:32Z
_project.py: Adding FIXME comment to address #591 properly
-
0682aa1c
by Josh Smith
at 2018-08-31T10:58:12Z
job.py: Modify retry messages to be FAIL
This adjusts the message handler for the child processes to no longer
override the message type.
This also removes the ability for unhandled non BstError exceptions to
retry.
-
68eff9ba
by Josh Smith
at 2018-08-31T10:58:20Z
tests: Modify tests to ensure retry FAILURE
-
9b18e1bf
by Tristan Van Berkom
at 2018-08-31T11:22:51Z
Merge branch 'Qinusty/retries-should-fail' into 'master'
Retries log as failures
See merge request BuildStream/buildstream!766
-
e2d8fedf
by Josh Smith
at 2018-08-31T15:53:43Z
scheduler.py: Move cache_size logs into folder
This prevents the cache_size.pid.log files from cluttering the root
log directory.
-
6805a2ab
by Tristan Van Berkom
at 2018-09-01T08:24:05Z
Merge branch 'Qinusty/cache-size-directory' into 'master'
Move cache_size.pid.log files into a subdirectory of logs
See merge request BuildStream/buildstream!769
-
f8b06acc
by Tristan Van Berkom
at 2018-09-02T09:33:39Z
source.py: Document Source.get_source_fetchers() to return an iterable
Also highlight the fact that the plugin can rely on the fetcher's
fetch() method getting called before consuming the next item in the
list, which is the magick behavior that the git plugin relies on.
This is a part of #620
-
559b6fbc
by Tristan Van Berkom
at 2018-09-02T09:34:10Z
source.py: Fixing docs link formatting to be consistent.
-
43ad22d7
by Tristan Van Berkom
at 2018-09-02T09:34:10Z
source.py: Documenting that marking download URLs is mandatory
A download URL must be interpreted by the core at `Plugin.configure()`
time, even if only employed later on.
This is a part of #620
-
16462e9c
by Tristan Van Berkom
at 2018-09-02T09:36:49Z
plugin.py: Added _configure() and _get_configuring() private APIs
Keeps track of whether the plugin is currently being configured.
Adjusted Element and Source classes to call _configure() in place
of calling configure() directly.
This is a part of #620
-
1f9c4147
by Tristan Van Berkom
at 2018-09-02T09:37:12Z
source.py: Added `primary` argument to URL marking APIs
The Source must now mention whether the marked or translated
URL is "primary" or not. Even when a Source may have multiple
URLs, the auxilliary URLs are derived from the primary one, not
only is this true for git, but it is mandated by our tracking
API which assumes there is a primary URL.
This adjusts the `git` source and the test `fetch_source.py` source
to behave properly and advertize it's primary URL properly.
This is a part of #620
-
f9b2f1a9
by Tristan Van Berkom
at 2018-09-02T09:37:21Z
source.py: Track marked URLs and assert they are marked during Plugin.configure()
This cannot test for unaliased URLs, as those can be discovered later
on outside of user provided element configuration; at least we
assert that if an alias was used, we have seen it at load time.
This will cause a BUG to occur for a plugin which marks an aliased
URL (or attempts to translate one) outside of `Plugin.configure()`,
if that URL was not previously seen.
This is a part of #620
-
88460cd2
by Tristan Van Berkom
at 2018-09-03T06:23:38Z
Merge branch 'tristan/source-fetcher-changes' into 'master'
Source fetcher changes
See merge request BuildStream/buildstream!772
-
2339f0c4
by Tom Pollard
at 2018-09-03T09:44:36Z
plugins/git.py: Warn if ref is not in given track
Add a helper function assert_ref_in_track to git.py GitMirror()
which is used when staging & initing the source workspace. It
checks the element's ref exists in the track (branch/tag) if it
has been specified, raising a warning if necessary. The warning makes
use of the warning token 'REF_NOT_IN_TRACK' from the configurable
CoreWarnings. If the element has been tracked with bst, it is assumed
that the value of ref exists in the track as it was generated from it
& as such is not asserted.
-
c96fec5d
by Tom Pollard
at 2018-09-03T09:44:36Z
tests/sources/git.py: Add tests for REF_NOT_IN_TRACK
Add tests that cover assert_ref_in_track & the configurable
CoreWarnings REF_NOT_IN_TRACK warnings token.
-
6a0cdedf
by Tom Pollard
at 2018-09-03T10:16:39Z
Merge branch 'tpollard/483' into 'master'
plugins/git.py: Warn if ref is not in given track
See merge request BuildStream/buildstream!564
-
9b327eb6
by Ben Brewer
at 2018-09-04T14:41:23Z
Improve documentation for artifact cache installation
Remove ambiguity about systemd service files being separate.
Add URL for more information about systemd service files.
Add note about public keys being mandatory for self-signed certs.
Make cert/key file naming consistent throughout document.
-
3e67e64a
by Javier Jardón
at 2018-09-04T16:43:13Z
Merge branch 'benbrewer/install-artifacts-doc-improvements' into 'master'
Improve documentation for artifact cache installation
See merge request BuildStream/buildstream!777
-
3409609e
by Daniel Silverstone
at 2018-09-04T16:55:51Z
jobs.py: Reduce FD leaks from queues and process objects
The garbage collector can take too long to get around to cleaning
up the Queue and Process instances in completed Job instances. As
such, FDs tend to leak and in very large projects this can result
in running out of FDs before a build, fetch, track, or other process
can complete. This patch reduces the chance of that by only creating
the queue when it's needed, and forcing the queue and process instances
to be deleted when the parent is finished with them.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
7b32e1ec
by Tristan Maat
at 2018-09-04T17:20:55Z
Merge branch 'danielsilverstone-ct/maybe-reduce-fd-leaks' into 'master'
jobs.py: Reduce FD leaks from queues and process objects
See merge request BuildStream/buildstream!778
-
936bb93a
by Jim MacArthur
at 2018-09-07T12:57:28Z
cascache.py: Preparation for remote execution
Refactor the push() and pull() implementations so that API additions
needed for remote-execution is made easier.
https://gitlab.com/BuildStream/buildstream/issues/454
-
b0f46545
by Martin Blanchard
at 2018-09-07T12:57:29Z
cascache.py: Introduce new push helpers
Add push_directory() and push_message() helpers along with a
verify_digest_pushed().
https://gitlab.com/BuildStream/buildstream/issues/454
-
50bf3139
by Martin Blanchard
at 2018-09-07T12:57:29Z
cascache.py: Introduce new pull helpers
Add a pull_tree() helper.
https://gitlab.com/BuildStream/buildstream/issues/454
-
853e0af7
by Martin Blanchard
at 2018-09-07T12:57:29Z
_casbaseddirectory.py: Add a method for hash recalculation
https://gitlab.com/BuildStream/buildstream/issues/454
-
e7659c29
by Jim MacArthur
at 2018-09-07T12:57:29Z
sandbox.py: Allow setting the virtual directory
This is for use after remote execution has finished, since remote
execution produces a new output directory rather than modifying
the initial directory.
https://gitlab.com/BuildStream/buildstream/issues/454
-
7306a954
by Jim MacArthur
at 2018-09-07T12:57:29Z
Add "remote-execution" project configuration option
This just adds one option, "remote-execution/url". Affects multiple files.
https://gitlab.com/BuildStream/buildstream/issues/454
-
c0ef7106
by Jim MacArthur
at 2018-09-07T12:57:29Z
format_project.rst: Document remote-execution option
https://gitlab.com/BuildStream/buildstream/issues/454
-
43651af0
by Jim MacArthur
at 2018-09-07T12:57:29Z
_sandboxremote.py: Implement the REAPI client
The remote execution client is implemented as a remote sandbox that
sends sources and build commands to a REAPI server and fetches results
once remotely executed. New file.
https://gitlab.com/BuildStream/buildstream/issues/454
-
da53c005
by Jim MacArthur
at 2018-09-07T12:57:29Z
sandbox/__init__.py: Add SandboxRemote
https://gitlab.com/BuildStream/buildstream/issues/454
-
c73896bb
by Jim MacArthur
at 2018-09-07T12:57:29Z
element.py: Get the updated virtual directory after running
Executing run() on a sandbox can now replace the virtual directory,
since remote execution returns a potentially different directory rather
than an update to the existing one. Call get_virtual_directory() again
after running to accout for this.
https://gitlab.com/BuildStream/buildstream/issues/454
-
89b29b8b
by Jim MacArthur
at 2018-09-07T12:57:29Z
element.py: Switch to SandboxRemote if config option is set
https://gitlab.com/BuildStream/buildstream/issues/454
-
116f80d2
by Jim MacArthur
at 2018-09-07T12:57:29Z
autotools.py: Mark this as a BST_VIRTUAL_DIRECTORY plugin
https://gitlab.com/BuildStream/buildstream/issues/454
-
05956c71
by Martin Blanchard
at 2018-09-07T12:57:29Z
cmake.py: Mark this as a BST_VIRTUAL_DIRECTORY plugin
https://gitlab.com/BuildStream/buildstream/issues/454
-
909b3be4
by Martin Blanchard
at 2018-09-07T12:57:29Z
make.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin
https://gitlab.com/BuildStream/buildstream/issues/454
-
556f2abb
by Martin Blanchard
at 2018-09-07T12:57:30Z
meson.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin
https://gitlab.com/BuildStream/buildstream/issues/454
-
71e2b543
by Martin Blanchard
at 2018-09-07T12:57:30Z
qmake.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin
https://gitlab.com/BuildStream/buildstream/issues/454
-
d4c515d0
by Martin Blanchard
at 2018-09-07T12:57:30Z
tests/artifactcache: Add a simple test project
https://gitlab.com/BuildStream/buildstream/issues/454
-
923d443b
by Martin Blanchard
at 2018-09-07T13:22:48Z
tests/artifactcache: Add push unit-tests
https://gitlab.com/BuildStream/buildstream/issues/454
-
9ef53163
by Martin Blanchard
at 2018-09-07T13:22:57Z
tests/artifactcache: Add pull unit-tests
https://gitlab.com/BuildStream/buildstream/issues/454
-
f52169b2
by Qinusty
at 2018-09-07T16:48:02Z
Merge branch 'jmac/remote_execution_client' into 'master'
Remote execution client
See merge request BuildStream/buildstream!626
-
8326b0f7
by Thomas Coldrick
at 2018-09-08T17:54:22Z
Upstream libtool .la file removal
In freedesktop-sdk we add a script to our project.conf to remove
libtool .la files from autotools projects after install, this seems
like a sensible default, so we're attempting to send it upstream.
-
886a836e
by Valentin David
at 2018-09-08T20:07:29Z
Merge branch 'coldtom/autotools-libtool' into 'master'
Upstream freedesktop-sdk autotools config for libtool .la files
See merge request BuildStream/buildstream!683
-
2174db01
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
tests/artifactcache/expiry.py: Fix test case expectations.
-
e5847077
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
_scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE
This is what the whole resource.py thing was created for, the
cleanup job must have exclusive access to the cache, while the pull
and build jobs which result in adding new artifacts, must only require
shared access.
This is a part of #623
-
98ab2b89
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
_scheduler/resources.py: Set CACHE max resources to 0
This means there is no cap for shared resource requests.
Together with the previous commit, this causes the cleanup
job and the pull/build jobs to all require unlimited shared
access to the CACHE resource; while only the cleanup job
requires exclusive access to the resource.
This is a part of #623
-
deaecdc1
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
_scheduler/scheduler.py: Make CacheSizeJob() not require the CACHE exclusively
This runs after every pull, and does not need the cache exclusively,
only the cleanup job requires the cache exclusively.
Without this, every time a cache_size job is queued, all pull and
build jobs need to complete before cache_size job can run exclusively,
which is not good.
This is a part of #623
-
1f118ba4
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
element.py: Adding missing API documenting comment
-
26ccc1ba
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
_artifactcache: Making public methods public
The artifact cache provides the following public methods for
external callers, but was hiding them away as if they are private.
o ArtifactCache.add_artifact_size()
o ArtifactCache.set_cache_size()
Mark these properly public
-
00784059
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
_scheduler: API fixes; _check_cache_size_real() -> check_cache_size()
Here we have a very private looking _check_cache_size_real() function
which no-one would ever want to call from outside of the _scheduler,
especially given it's `_real()` prefix we should look for another
outward facing API to use.
However this is not private to the scheduler, and is intended to
be called by the `Queue` implementations.
o Renamed this to check_cache_size()
o Moved it to the public API section of the Scheduler object
o Added the missing API documenting comment
o Also added the missing API documenting comment to the private
`_run_cleanup()` callback which runs in response to completion
of the cache size calculation job.
o Also place the cleanup job logs into a cleanup subdirectory,
for better symmetry with the cache_size jobs which now have
their own subdirectory
-
a3825ba6
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
_artifactcache/artifactcache.py: Removing unused variable.
The ArtifactCache._local variable used to exist in order to
use a special hack to allow absolute paths to a remote artifact
cache, this was all for the purpose of testing.
This has all gone away with the introduction of CAS, leaving behind
a stale variable.
-
791f7dda
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
_artifactcache/artifactcache.py: Sealing away some the estimated size
Previously, the API contract was to expose the estimated_size variable
on the ArtifactCache instance for all to see, however it is only relevant
to the ArtifactCache abstract class code. Subclasses were informed to
update the estimated_size variable in their calculate_cache_size()
implementation.
To untangle this and hide away the estimated size, this commit
does the following:
o Introduces ArtifactCache.compute_cache_size() API for external
callers
o ArtifactCache.compute_cache_size() calls the abstract method
for the CasCache subclass to implement
o ArtifactCache.compute_cache_size() updates the private
estimated_size variable
o All direct callers to ArtifactCache.calculate_cache_size(),
have been updated to use the ArtifactCache.compute_cache_size()
method instead, which takes care of updating anything local
to the ArtifactCache abstract class code (the estimated_size)
-
cbfddc13
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
element.py: Remove _get_artifact_size()
There is no justification to hold onto this state here.
Instead, just make `Element._assemble()` return the size of the
artifact it cached, and localize handling of that return value in
the BuildQueue implementation where the value is observed.
-
8b748f2d
by Tristan Van Berkom
at 2018-09-10T07:53:57Z
_scheduler/jobs/cachesizejob.py: Use Platform.get_platform() public accessor
This was previously poking directly at the Platform._instance.
Also, use the name 'artifacts' to hold the artifact cache to
be consistent with other parts of the codebase, instead of
calling it 'cache'.
-
742908ea
by Tristan Van Berkom
at 2018-09-10T07:53:57Z
_scheduler/jobs/cleanupjob.py: Use Platform.get_platform() public accessor
This was previously poking directly at the Platform._instance.
Also, use the name 'artifacts' to hold the artifact cache to
be consistent with other parts of the codebase, instead of
calling it 'cache'.
-
335b2fdd
by Tristan Van Berkom
at 2018-09-10T07:53:57Z
element.py: Remove _get_artifact_cache() accessor.
The artifact cache is anyway a singleton, the Element itself
has an internal handle on the artifact cache because it is
needed very frequently; but an artifact cache is not element
specific and should not be looked up by surrounding code
on a per element basis.
Updated _scheduler/queues/queue.py, _scheduler/queues/buildqueue.py
and _scheduler/jobs/elementjob.py to get the artifact cache directly
from the Platform
-
6db54c8f
by Tristan Van Berkom
at 2018-09-10T07:53:57Z
_artifactcache: There shalt be only one cache size
This does a lot of house cleaning, finally bringing cache
cleanup logic to a level of comprehensibility.
Changes in this commit include:
o _artifactcache/artifactcache.py: _cache_size, _cache_quota and
_cache_lower_threshold are now all private variables.
get_approximate_cache_size() is now get_cache_size()
Added get_quota_exceeded() for the purpose of safely checking
if we have exceeded the quota.
set_cache_size() now asserts that the passed size is not None,
it is not acceptable to set a None size cache anymore.
o _artifactcache/cascache.py: No longer set the ArtifactCache
'cache_size' variable violently in the commit() method.
Also the calculate_cache_size() method now unconditionally
calculates the cache size, that is what it's for.
o _scheduler/jobs/cachesizejob.py & _scheduler/jobs/cleanupjob.py:
Now check the success status. Don't try to set the cache size
in the case that the job was terminated.
o _scheduler/jobs/elementjob.py & _scheduler/queues/queue.py:
No longer passing around the cache size from child tasks,
this happens only explicitly, not implicitly for all tasks.
o _scheduler/queues/buildqueue.py & _scheduler/scheduler.py:
Use get_quota_exceeded() accessor
This is a part of #623
-
958ef8ef
by Tristan Van Berkom
at 2018-09-10T09:23:06Z
Merge branch 'tristan/fix-cache-exclusivity' into 'master'
_scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE
See merge request BuildStream/buildstream!775
-
40f25512
by Richard Maw
at 2018-09-10T12:29:15Z
subprocesses: Ensure PWD is set in process environment
Naive getcwd implementations (such as in bash 4.4) can break
when bind-mounts to different paths on the same filesystem are present,
since the algorithm needs to know whether it's a mount-point
to know whether it can trust the inode value from the readdir result
or to use stat on the directory.
Less naive implementations (such as in glibc) iterate again using stat
in the case of not finding the directory because the inode in readdir was wrong,
though a Linux-specific implementation could use name_to_handle_at.
Letting the command know what directory it is in makes it unnecessary
for it to call the faulty getcwd in the first place.
-
247686dc
by richardmaw-codethink
at 2018-09-10T14:47:48Z
Merge branch 'richardmaw/subprocess-PWD' into 'master'
Ensure PWD is set in process environment
See merge request BuildStream/buildstream!782
-
ef66daf5
by Tiago Gomes
at 2018-09-10T14:57:41Z
tests: avoid setting max-jobs
Setting "max-jobs" won't be allowed anymore in a following commit.
-
a37bc6ce
by Tiago Gomes
at 2018-09-10T14:57:41Z
docs: document protected variables
And remove then from the defaults as they are dynamically set by
BuildStream.
-
af74a3f8
by Tiago Gomes
at 2018-09-10T14:57:41Z
element: validate configuration variables
Ensure that protected variables are not being redefined by the user.
-
75c55633
by Tiago Gomes
at 2018-09-10T14:57:41Z
Add tests for validating configuration variables
-
e0bb71b2
by Tiago Gomes
at 2018-09-10T15:27:53Z
Merge branch 'tiagogomes/issue-287' into 'master'
Add validation of configuration variables
See merge request BuildStream/buildstream!678
-
47f3064a
by Jürg Billeter
at 2018-09-10T15:43:25Z
_artifactcache/casserver.py: Implement Capabilities service
-
1a7fb3cb
by Jürg Billeter
at 2018-09-10T15:43:25Z
_artifactcache/casserver.py: Implement BatchReadBlobs
Fixes #632.
-
b3ffcdc8
by Jürg Billeter
at 2018-09-10T16:07:30Z
Merge branch 'juerg/cas-batch' into 'master'
_artifactcache/casserver.py: Implement BatchReadBlobs
Closes #632
See merge request BuildStream/buildstream!785
-
6f925bcb
by Javier Jardón
at 2018-09-13T07:38:48Z
source/install_source.rst: pip plugin depends on host pip
-
c6155f8d
by Javier Jardón
at 2018-09-13T08:11:54Z
Merge branch 'jjardon/pip_dependency' into 'master'
source/install_source.rst: pip plugin depends on host pip
See merge request BuildStream/buildstream!791
-
19838a07
by Chandan Singh
at 2018-09-13T10:58:38Z
README.rst: Add status badges for PyPI release and Python versions
The first badge will work fine right away while the second badge will
show "not found" until a release is made after merging this branch:
https://gitlab.com/BuildStream/buildstream/merge_requests/718.
-
3b81d451
by Chandan Singh
at 2018-09-13T12:27:59Z
Merge branch 'chandan/pypi-badge' into 'master'
README.rst: Add status badges for PyPI release and Python versions
See merge request BuildStream/buildstream!719
-
55956762
by Richard Maw
at 2018-09-13T16:50:33Z
element: Remove __cached field
This can get out of sync with other two cache states,
and we can do without it.
-
fc7f83ac
by richardmaw-codethink
at 2018-09-13T17:14:31Z
Merge branch 'richardmaw/element-cache-state-simplify' into 'master'
Simplify element state by removing `__cached`
See merge request BuildStream/buildstream!784
-
233a7d83
by Richard Maw
at 2018-09-14T08:53:14Z
Handle sockets when copying trees into artifacts
We can't include a socket in a CAS tree, but it's mostly meaningless to do so
since there can't possibly be a process serving it.
-
f06f234a
by Richard Maw
at 2018-09-14T08:53:14Z
tests: cover builds that make sockets
-
f86ab8f6
by richardmaw-codethink
at 2018-09-14T09:46:07Z
Merge branch 'richardmaw/builddir-sockets' into 'master'
Fix: While caching build artifact: "Cannot extract [path to socket file] into staging-area. Unsupported type."
See merge request BuildStream/buildstream!783
-
e7427462
by Richard Maw
at 2018-09-14T10:28:17Z
tests: Use cli.configure for custom logging test
Overriding the config with a custom config file on the command-line
prevents it merging in the test-specific configuration
and can permit it to attempt to initialise the user's cache.
-
800a8403
by Richard Maw
at 2018-09-14T10:28:17Z
Tidy relative workspace tests
They were moving the whole tmpdir to move the project repository.
This moves the cache directories etc. as well, meaning cli.run can't find them.
This was worked around by setting configure=False,
but this has the side-effect of making use of the user's caches,
which it should not be doing for reproducibility reasons.
By changing the tempdir layout to have the project in a subdirectory
we can move the project around for the relative workspace tests
without losing track of the configured state directories,
so we can leave configure=True and avoid touching the user's caches.
-
d7152ef4
by richardmaw-codethink
at 2018-09-14T10:55:16Z
Merge branch 'richardmaw/test-config-fixes' into 'master'
Fix tests that attempt to access the home directory
See merge request BuildStream/buildstream!780
-
160bb0c6
by Tristan Van Berkom
at 2018-09-14T12:07:46Z
_artifactcache/artifactcache.py: Changes to mark_required_elements()
This was previously append_required_artifacts(), which presumed that
we knew at startup time what the cache keys of all elements in the
loaded pipeline would be.
This fixes unexpected deletions of required artifacts when
dynamic tracking is enabled with `bst build --track-all target.bst`
-
39125d24
by Tristan Van Berkom
at 2018-09-14T12:07:46Z
tests/artifactcache/expiry.py: Cleanup of test for required artifacts
This commit renames test_never_delete_dependencies() to
test_never_delete_required(), renders the test more readable by renaming
some elements and reordering some statements and makes the comments more
straight forward and accurate.
-
f60558a3
by Tristan Van Berkom
at 2018-09-14T12:07:46Z
tests/artifactcache/expiry.py: Assert the expected errors
These tests were not checking that we fail for the expected reasons.
Added `res.assert_task_error(ErrorDomain.ARTIFACT, 'cache-too-full')`
where we expect to fail because the cache is too full.
-
ce68fd27
by Tristan Van Berkom
at 2018-09-14T12:07:46Z
testutils/repo/git.py: Added modify_file() method
This allows one to modify a file in an existing git repo,
as opposed to adding a new one.
-
532ec1eb
by Tristan Van Berkom
at 2018-09-14T12:07:46Z
testutils/element_generators.py: Changing sized element functions
* create_element_size()
Now uses a git Repo object instead of a local source, and
returns the repo.
* update_element_size()
Added this function which can now resize the expected output
of an element generated with create_element_size(), useful
to allow testing sized elements with the tracking feature.
-
20b797cb
by Tristan Van Berkom
at 2018-09-14T12:07:46Z
tests/artifactcache/expiry.py: Added test_never_delete_required_track()
Same test as test_never_delete_required(), except that this test ensures
that we never delete required artifacts when their cache keys are
discovered dynamically during the build.
-
c2af0d51
by Tristan Van Berkom
at 2018-09-14T12:44:42Z
Merge branch 'tristan/fix-required-artifacts' into 'master'
Don't delete required artifacts when tracking is enabled
See merge request BuildStream/buildstream!793
-
924cdc75
by Tiago Gomes
at 2018-09-14T15:32:01Z
core: remove unused variable on Context
-
510ccbfd
by Tiago Gomes
at 2018-09-14T15:32:21Z
artifactcache: fix oversight
We want to check if some file is already cached here, not the parent
directory.
-
82d4e2ac
by Tiago Gomes
at 2018-09-14T15:32:21Z
scheduler: add missing copyright
Tristan Maat created the original file, so he is added as the author.
-
18b37aab
by Tiago Gomes
at 2018-09-14T15:34:10Z
artifactcache: improve _create_tree()
* Rename it to _commit_directory() because… it is what it does; and
also for symmetry with _fetch_directory().
* Rename digest to dir_digest to make it clear this is a digest for a
directory. A following commit will also reuse the same variable name
* Document method.
-
32fad24f
by Tiago Gomes
at 2018-09-14T15:38:41Z
artifactcache: improve _fetch_directory()
* Rename tree to dir_digest to make it clear this is a Digest object,
and not a Tree object.
* Add documentation
-
46cbd889
by Tiago Gomes
at 2018-09-14T15:38:43Z
artifactcache: rename get_quota_exceeded()
-
2fa92716
by Tiago Gomes
at 2018-09-14T15:38:43Z
scheduler: minor cleanups to CleanupJob
Remove unneeded cruft.
-
c285f244
by Tiago Gomes
at 2018-09-14T16:08:07Z
Merge branch 'tiagogomes/some-cleanups' into 'master'
Bunch of cleanups
See merge request BuildStream/buildstream!798
-
662d22a9
by Tristan Van Berkom
at 2018-09-16T08:54:00Z
buildstream/_frontend/complete.py: Added missing click license compliance
Somehow I missed this when originally forking the file from the click
library, now noticing that we should have followed what was written
in: https://github.com/pallets/click/blob/master/LICENSE
-
a344407b
by Richard Maw
at 2018-09-16T10:22:59Z
sandbox: deduplicate default environment logic
-
c8fab440
by Richard Maw
at 2018-09-16T10:22:59Z
Bump Artifact Version
Since we now set PWD in the environment of builds
existing builds may behave differently so must cache differently now.
-
eb9481da
by Richard Maw
at 2018-09-16T10:22:59Z
Cachekey update helper: Document ensuring . in PYTHONPATH
The current directory isn't always in the python module search path,
so we have to ensure it is for the script to work.
Strictly speaking, the user may already have a modified PYTHONPATH
at which point PYTHONPATH=".${PYTHONPATH+:$PYTHONPATH}" is necessary,
but it's probably premature to overcomplicate the documentation like that
before we discover it's a problem.
-
9b7b2a25
by Tristan Van Berkom
at 2018-09-16T10:50:36Z
Merge branch 'richardmaw/subprocess-PWD' into 'master'
Address post-merge review of Ensure PWD is set in process environment
See merge request BuildStream/buildstream!788
-
8db62230
by Adam Jones
at 2018-09-17T07:39:06Z
Add a section on how to raise an issue
-
6f0a3e98
by Adam Jones
at 2018-09-17T07:39:20Z
Rename HACKING.rst to CONTRIBUTING.rst
-
ade19151
by Adam Jones
at 2018-09-17T07:40:45Z
update HACKING references
-
b9c5ff7d
by Tristan Van Berkom
at 2018-09-17T08:09:12Z
Merge branch 'tristan/contributing-guide' into 'master'
Update contributing guide
See merge request BuildStream/buildstream!801
-
727f2faa
by Tristan Van Berkom
at 2018-09-18T07:43:07Z
_project.py: Fix option resolution in element & source overrides
This ensures that option expressions are resolved in the project
level overrides before attempting to composite them on the instantiated
elements. Seems this is a regression from introducing the include
directive.
This fixes issue #658
-
ffa0bb36
by Tristan Van Berkom
at 2018-09-18T07:47:44Z
tests/format/optionoverrides.py: Added test for options in element overrides
This is a regression test for issue #658
-
f2ae46f8
by Tristan Van Berkom
at 2018-09-18T08:14:23Z
Merge branch 'tristan/fix-override-options' into 'master'
Fix override options
Closes #658
See merge request BuildStream/buildstream!802
-
7b117e40
by Daniel Silverstone
at 2018-09-18T08:41:32Z
_artifactcache/artifactcache.py: Ensure no double-setup of remotes
Since ArtifactCache.setup_remotes() can be expensive and should only
happen once, this commit will assert() if it is called a second time
on an artifact cache instance.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
345f5f49
by Daniel Silverstone
at 2018-09-18T08:41:32Z
tests/artifactcache/pull.py: Do not double-initialize remotes
The initialization of remotes is done by ArtifactCache.setup_remotes()
and as such it was wrong for these tests to be calling
CASCache.initialize_remotes() a second time.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
e32221b6
by Daniel Silverstone
at 2018-09-18T08:45:50Z
sandbox/_sandboxremote.py: Acquire artifact cache via Platform
The SandboxRemote used to construct its own CASCache which was
considered dangerous. This patch replaces that with acquisition of
the cache via the Platform singleton, hopefully eliminating issues
from having more than one artifact cache object in a single process.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b587579f
by Tristan Van Berkom
at 2018-09-18T09:53:26Z
Merge branch 'danielsilverstone-ct/fix-654' into 'master'
sandbox/_sandboxremote.py: Acquire cache via Platform
See merge request BuildStream/buildstream!797
-
30b41959
by Tristan Van Berkom
at 2018-09-18T09:56:45Z
_artifactcache/artifactcache.py: Error out gracefully when push remote is mal-specified
When configuring a push remote and specifying either the client-cert
or the client-key, then both must be specified. This ensures we
get an informative error instead of a stack trace and BUG.
Fixes issue #625
-
41e8dc81
by Tristan Van Berkom
at 2018-09-18T09:56:45Z
tests/artifactcache/config.py: Added test for invalid push remote configuration
Test that we get the expected error when configuring a client-cert
without client-key, or the inverse.
-
97071b6e
by Tristan Van Berkom
at 2018-09-18T10:16:43Z
Merge branch 'tristan/fix-artifact-config-crash' into 'master'
Fix artifact config crash
Closes #625
See merge request BuildStream/buildstream!804
-
081dcafa
by Richard Maw
at 2018-09-18T13:22:38Z
fuse: Report the correct device number for devices
This fixes all devices being mapped to the non-existant device 0,
which prevents being able to use even safe devices like /dev/null
through the hardlinks FUSE layer.
-
d0425608
by Richard Maw
at 2018-09-18T13:22:38Z
FUSE: Mount with -odev in chroot sandbox
This is needed to permit access to the device nodes added to /dev
on Linux when FUSE is used as root.
The chroot sandbox only works with all privileges,
so there's no explicit check for being root
or having the appropriate capabilities.
A check for whether it's running as root isn't needed on Linux with bubblewrap
because /dev or its devices are mounted on top of the FUSE layer,
so device nodes are accessed directly rather than through the FUSE layer.
-
8430fdc7
by Richard Maw
at 2018-09-18T13:22:38Z
tests: test that integration commands can use /dev
-
d8450166
by Tristan Van Berkom
at 2018-09-18T13:50:15Z
Merge branch 'richardmaw/fix-chroot-sandbox-devices' into 'master'
fix chroot sandbox devices
See merge request BuildStream/buildstream!781
-
ba4da3a4
by knownexus
at 2018-09-18T16:45:09Z
Incorrect error when malformed project.conf
bst build returns "missing project.conf" when a project.conf is invalid
This results in an existing project with malformed yaml being dismissed
and attempts to make a new project
Added new exception
-
89a16e17
by knownexus
at 2018-09-18T16:45:09Z
Adding test for Invalid Yaml
-
72b59021
by Phillip Smyth
at 2018-09-19T08:48:51Z
Merge branch 'issue-642-Invalid_project.conf_seen_as_missing' into 'master'
Incorrect error when malformed project.conf
Closes #642
See merge request BuildStream/buildstream!792
-
ca811a4d
by Josh Smith
at 2018-09-19T09:25:21Z
Rework Skipped usage
The SKIPPED message type is now used to indicate the end of a task which
was successful without having to perform the given task.
This overhauls the use of `Queue.done()` and therefore queues do not
need to provide a processed/skipped return value from `done()`. Instead
this is replaced with the action of raising a `SkipJob` exception from
within `Queue.process()`.
-
16fbc3e3
by Josh Smith
at 2018-09-19T09:25:21Z
tests.py: Test skip on push
Adds a test to ensure that BuildStream alerts the user of a skipped push
when the remote already has the artifact cached.
-
ab56f693
by Josh Smith
at 2018-09-19T09:25:21Z
element.py: Remove redundant timed_activity
This removes the timed_activity for an element _push action. This is
unnecessary as the job is already being timed elsewhere.
-
198fbfe8
by Josh Smith
at 2018-09-19T09:25:21Z
cascache.py: Move push/pull messaging to cascache
Pulled/Pushed messages will no longer be produced from within
element.py, instead they will be produced during CasCache push() and
pull() appropriately.
Message consistency has also been improved.
-
213f7794
by Josh Smith
at 2018-09-19T09:25:21Z
cascache.py: Modify messaging API calls
Both pulling and pushing INFO messages are now status messages.
Calls to the messaging API through `self.context.message()` have now
been switched to `element.info`.
-
dddd6025
by Qinusty
at 2018-09-19T10:59:48Z
Merge branch 'Qinusty/skipped-rework' into 'master'
Add SkipJob for indicating a skipped activity
See merge request BuildStream/buildstream!765
-
16d9c6e5
by Tristan Van Berkom
at 2018-09-19T11:08:33Z
_frontend/status.py: Completely remove the blessings dependency from BuildStream
This actually improves reliability of the status bar because we
now disable it completely in the case that not all of the terminal
escape sequences are supported on the given terminal.
This replaces the few functions we were using, to move the cursor
up one line, move it to the beginning of the line, and to clear a
line, with low level functions provided by the curses module in
the standard library.
This change makes it easier for downstream distro package maintainers
to package BuildStream, particularly on Fedora.
Asides from changing _frontend/status.py, this commit includes the
following changes:
* _frontend/app.py: Use python isatty() function to determine if
we are connected to a tty, instead of relying
on blessings.
* setup.py: Remove the dependency on blessings.
-
7dd74b01
by Tristan Van Berkom
at 2018-09-19T11:27:55Z
Merge branch 'tristan/remove-blessings' into 'master'
_frontend/status.py: Completely remove the blessings dependency from BuildStream
See merge request BuildStream/buildstream!808
-
335d5a45
by Adam Jones
at 2018-09-19T13:00:45Z
Fix rst links for guideline links
-
667dd4ca
by Javier Jardón
at 2018-09-19T13:25:36Z
Merge branch 'adamjones/contributing-links' into 'master'
Fix rst link formatting for guideline links
See merge request BuildStream/buildstream!811
-
68ef69e4
by Tristan Van Berkom
at 2018-09-21T05:20:46Z
source.py: Fix re-instantiation
This fixes an issue where the re-instantiated Source used with
Source mirroring enabled is not completely initialized.
Failing to load the ref from the project.refs file for instance,
will result in a crash at `fetch` time.
This fixes issue #666
-
662c729f
by Tristan Van Berkom
at 2018-09-21T05:59:30Z
Merge branch 'tristan/fix-source-reinstantiation' into 'master'
source.py: Fix re-instantiation
Closes #666
See merge request BuildStream/buildstream!817
-
461a0588
by Jim MacArthur
at 2018-09-21T10:53:11Z
element.py: Remove early warning and check directory is not None
The check for directory should have always been here. The early warning
test doesn't work for `bst checkout` because caches haven't been configured
at the point of the test, so I've had to remove it.
-
aa9caaac
by Jim MacArthur
at 2018-09-21T10:53:11Z
_sandboxremote.py: Ignore return value of push_directory
The return value is always upload_vdir.ref.
-
2aae68c7
by Jim MacArthur
at 2018-09-21T10:53:11Z
cascache.py: make push_remote raise exception if no push remotes exist
Also add docstring to method and remove return value since it was useless
-
ca1bb72c
by Jim MacArthur
at 2018-09-21T10:53:11Z
tests/artifactcache/push.py: Ignore return value of push_directory
-
55c93a82
by Jim MacArthur
at 2018-09-21T11:26:55Z
Merge branch 'jmac/remote_exec_checkout_fix' into 'master'
Remote exec: Remove early warning and check directory is not None
See merge request BuildStream/buildstream!800
-
e209beb0
by Chandan Singh
at 2018-09-21T13:10:08Z
_stream.py: Ensure source-bundle's source directory exists
Currently, `source-bundle` command is entirely broken as it tries to stage the
sources in a directory that doesn't exist. Fix it by ensuring that we create
the necessary directories before calling any methods that try to use those
directories.
This fix comes with a regression test to ensure that the basic use-case
of `source-bundle` continues to work in future.
Fixes https://gitlab.com/BuildStream/buildstream/issues/651.
-
0b000518
by Chandan Singh
at 2018-09-21T13:56:55Z
Merge branch 'chandan/fix-source-bundle' into 'master'
Fix source-bundle command
Closes #651
See merge request BuildStream/buildstream!807
-
ef26043a
by Chandan Singh
at 2018-09-21T17:14:16Z
setup.py: Make website the primary homepage
-
1b2aed40
by Chandan Singh
at 2018-09-21T17:40:11Z
Merge branch 'chandan/update-project-homepage' into 'master'
setup.py: Make website the primary homepage
See merge request BuildStream/buildstream!795
-
da7e038b
by Jürg Billeter
at 2018-09-24T13:58:55Z
element.py: Fix cache check in non-strict mode
The behavior of __assert_cached() was changed when the keystrength
parameter was introduced. This restores the previous behavior.
Fixes #607.
Fixes: 3d308894 ("element.py: Cache the result of checking whether...")
-
86ea1173
by Jürg Billeter
at 2018-09-24T13:59:59Z
tests/frontend/rebuild.py: Add strict/non-strict rebuild test
Test rebuild that requires staging of weakly cached dependency.
This is a regression test for #607.
-
a76339de
by Jürg Billeter
at 2018-09-24T16:02:38Z
Merge branch 'juerg/rebuild' into 'master'
element.py: Fix cache check in non-strict mode
Closes #607
See merge request BuildStream/buildstream!822
-
b199afe6
by Jürg Billeter
at 2018-09-25T09:01:51Z
_artifactcache/cascache.py: Add _ensure_blob helper
This adds directory objects to the local repository before downloading
files in the directory. However, artifact references are still stored
only after downloading the complete directory and thus, there won't be
dangling references. This will anyway be required for partial download
support.
-
7d199322
by Jürg Billeter
at 2018-09-25T09:01:51Z
_artifactcache/cascache.py: Increase payload size limit for uploads
gRPC can handle 1 MiB payloads. Increase size limit from 64 KiB to speed
up uploads.`
-
e2e24015
by Jürg Billeter
at 2018-09-25T09:01:51Z
_artifactcache/casserver.py: Harmonize payload size limit
Use 1 MiB as payload size limit on the server side for both individual
downloads and batch uploads.
-
697d10f2
by Jürg Billeter
at 2018-09-25T09:01:51Z
_artifactcache/cascache.py: Use BatchReadBlobs
This uses BatchReadBlobs instead of individual blob download to speed up
artifact pulling, if the server supports it.
Fixes #554.
-
81c51dbf
by Jürg Billeter
at 2018-09-25T09:31:55Z
Merge branch 'juerg/cas-batch' into 'master'
_artifactcache/cascache.py: Use BatchReadBlobs
Closes #554
See merge request BuildStream/buildstream!813
-
2df7d140
by Jim MacArthur
at 2018-09-25T10:36:37Z
sandbox.py: Stop caching get_virtual_directory if get_directory is used
Also update the documentation for get_virtual_directory to make the
implications of using both methods clearer.
-
62f59382
by Jim MacArthur
at 2018-09-25T10:36:37Z
element.py/_assemble(): Call get_virtual_directory later.
-
8cea7b17
by Jim MacArthur
at 2018-09-25T10:58:40Z
Merge branch 'jmac/stop-caching-vdirs' into 'master'
Stop caching virtual directories if get_directory is used.
See merge request BuildStream/buildstream!818
-
83ab183e
by Tiago Gomes
at 2018-09-25T16:37:30Z
CI: test building freedesktop-sdk overnight
Closes #514.
-
c8594806
by Tiago Gomes
at 2018-09-25T17:03:55Z
Merge branch 'tiagogomes/issue-514' into 'master'
CI: test building freedesktop-sdk overnight
Closes #514
See merge request BuildStream/buildstream!826
-
fdbf9be1
by Tiago Gomes
at 2018-09-26T08:17:46Z
ci: update freedesktop-sdk ref
The current one doesn't build due server being down.
-
9280b0cf
by Tiago Gomes
at 2018-09-26T15:54:53Z
ci: add ACME challenge
To finish the creation of the Let's Encrypt certificate, which will
allows us to use https://docs.buildstream.build/ for accessing the
documentation.
-
af131503
by Tiago Gomes
at 2018-09-26T16:35:58Z
Merge branch 'tiagogomes/acme-challenge' into 'master'
ci: add ACME challenge
See merge request BuildStream/buildstream!831
-
f7f14f25
by Tiago Gomes
at 2018-09-26T17:42:12Z
ci: add missing `-p` argument
-
0f2bc375
by Martin Blanchard
at 2018-09-27T08:46:27Z
_protos: Import google.rpc.Code proto
Imports code.proto from https://github.com/googleapis/googleapis and
corresponding modules generated with ./setup.py build_grpc.
https://gitlab.com/BuildStream/buildstream/issues/668
-
16cf9d5f
by Martin Blanchard
at 2018-09-27T08:46:27Z
_sandboxremote.py: Handle remote build execution exit code
https://gitlab.com/BuildStream/buildstream/issues/668
-
f5f3cb7c
by Jim MacArthur
at 2018-09-27T09:16:38Z
Merge branch 'mablanch/668-remote-build-failure' into 'master'
Better handle remote build failures
See merge request BuildStream/buildstream!825
-
b99a6740
by Jürg Billeter
at 2018-09-27T09:18:03Z
element.py: Add __use_remote_execution() helper method
-
f8bbe008
by Jürg Billeter
at 2018-09-27T09:18:03Z
_platform: Add check_sandbox_config() method
-
f9494f1f
by Jürg Billeter
at 2018-09-27T09:18:03Z
element.py: Validate sandbox config
-
b4ad84c3
by Jürg Billeter
at 2018-09-27T09:18:03Z
_platform/linux.py: Remove redundant message
Unsupported sandbox config (UID/GID) is now reported by the element.
-
966af3d2
by Jürg Billeter
at 2018-09-27T09:18:03Z
_platform/linux.py: Do not disable push globally
On systems without user namespace support, elements with unsupported
sandbox config (UID/GID) are now individually tainted, which disables
artifact push.
-
f08e5eae
by Jürg Billeter
at 2018-09-27T09:18:03Z
_artifactcache/cascache.py: Remove unused enable_push parameter
-
10ed9158
by Jürg Billeter
at 2018-09-27T09:18:03Z
_artifactcache: Add preflight() method
-
7aec8b0f
by Jürg Billeter
at 2018-09-27T09:18:03Z
Move artifact cache creation to Context
The artifact cache is no longer platform-specific.
-
a5025c33
by Jürg Billeter
at 2018-09-27T09:18:03Z
_loader: Migrate to Context.artifactcache
-
ef1cb374
by Jürg Billeter
at 2018-09-27T09:18:03Z
_scheduler: Migrate to Context.artifactcache
-
52991be1
by Jürg Billeter
at 2018-09-27T09:18:03Z
_stream.py: Migrate to Context.artifactcache
-
cb6b35dc
by Jürg Billeter
at 2018-09-27T09:18:42Z
sandbox/_sandboxremote.py: Migrate to Context.artifactcache
-
28c62b00
by Jürg Billeter
at 2018-09-27T09:18:43Z
_platform/platform.py: Remove unused artifactcache property
-
99e90918
by Jürg Billeter
at 2018-09-27T09:19:12Z
_artifactcache/casserver.py: Use Context for artifact cache creation
-
b8f920e2
by Jürg Billeter
at 2018-09-27T09:19:12Z
storage/_casbaseddirectory.py: Use Context for artifact cache creation
-
3d88c5d4
by Jürg Billeter
at 2018-09-27T09:19:12Z
tests/testutils: Use Context for artifact cache creation
-
ed878cb6
by Jürg Billeter
at 2018-09-27T09:19:12Z
tests/artifactcache/pull.py: Use Context for artifact cache creation
-
fe82c9c2
by Jürg Billeter
at 2018-09-27T09:19:12Z
tests/artifactcache/push.py: Use Context for artifact cache creation
-
0d0f700d
by Jürg Billeter
at 2018-09-27T09:19:12Z
_platform: Remove unused context
-
dd770ec3
by Jürg Billeter
at 2018-09-27T09:19:12Z
Use lazy platform instantiation
Now that the platform is independent of the context, explicit
instantiation is no longer required. This avoids issues with platform
instances used across test cases with mismatching context.
-
44da8175
by Jürg Billeter
at 2018-09-27T09:48:20Z
Merge branch 'juerg/platform' into 'master'
Make platform independent of context
See merge request BuildStream/buildstream!829
-
900e8900
by Jürg Billeter
at 2018-09-27T13:12:34Z
git.py: Make `ref` human readable
Use `git describe` to prefix `ref` with closest tag, if available.
-
107269c1
by Jürg Billeter
at 2018-09-27T13:12:34Z
testutils/repo/git.py: Add add_tag() method
-
8060ad8c
by Jürg Billeter
at 2018-09-27T13:12:34Z
tests/sources/git.py: Add track and fetch test with and without tag
-
ab1cb672
by Jürg Billeter
at 2018-09-27T13:46:41Z
Merge branch 'juerg/git-describe' into 'master'
git.py: Make `ref` human readable
See merge request BuildStream/buildstream!291
-
fcf37f9c
by knownexus
at 2018-09-27T14:22:09Z
Added `set_resource_limits()` to platform
This has been moved from app.py
As it will have different functionality depending on platform
Once the Darwin (MacOS) platform is added
Removed `resource.setrlimit()` functionality from app.py
Added `resource.setrlimit()` functionality to platform.py as function
-
80ebdd2c
by knownexus
at 2018-09-27T14:22:09Z
Added FUSE check, as WSL does not support FUSE yet
_platform/linux.py: Add FUSE check
And Ensure exception is not raised when performing sandbox availability check
Without this check, WSL would error upon finding FUSE missing
This must be bypassed to allow remote execution
-
5e81573b
by knownexus
at 2018-09-27T14:22:09Z
Max-jobs is now defined on a platform basis
This is due to Darwin (MacOS) having a different Max-Jobs limit
_platform/platform.py: Adding max-jobs call
_project.py: redirected 'max-jobs' to point at the platform
-
3c93fe97
by knownexus
at 2018-09-27T14:22:09Z
Adding Dummy sandbox
This is to allow platforms that do not support sandboxing
To error in a controlled/known way
-
0f3ef369
by knownexus
at 2018-09-27T14:22:24Z
Adding darwin.py (MacOS) platform
Adding functionality to recognise Darwin as a platform in plaform.py
-
b78ae767
by knownexus
at 2018-09-27T14:29:29Z
utils.py: Reworked safe_remove
Non-Linux platforms don't return EISDIR when attempting to unlink a
directory.
Stopped safe_remove attempting to unlink dirs
Previously safe_remove would attempt to unlink a path
Before attempting to remove it if it was a dir
Now it checks for a dir before that step
-
ab5e78b4
by James Ennis
at 2018-09-27T14:29:32Z
Import sandbox subclass when instantiating sandbox
This patch was required due to not being able to execute `bst` on
a Linux machine without FUSE available.
-
261f65ca
by Jürg Billeter
at 2018-09-27T14:54:47Z
Merge branch 'mac_fixes' into 'master'
Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412"
See merge request BuildStream/buildstream!726
-
d2ea3069
by Richard Maw
at 2018-09-27T15:16:57Z
tests: Add regression test for pushing failed builds
This test ensures that failed builds are pushed when set to on-error=quit.
Closes https://gitlab.com/BuildStream/buildstream/issues/534
-
c128429a
by richardmaw-codethink
at 2018-09-27T15:42:18Z
Merge branch 'richardmaw/push-failed-build-regression' into 'master'
tests: Add regression test for pushing failed builds
Closes #534
See merge request BuildStream/buildstream!824
-
e91eb38b
by Valentin David
at 2018-09-27T16:32:57Z
Fix outside-of-project check when project path is not canonical.
The issue happens on Silverblue where /home is a symlink to /var/home.
With this element-path is something like
/var/home/user/project/elements, when the project path is
/home/usr/project. Comparing canonical paths solves the issue.
Fixes #673
-
7babd55c
by Tiago Gomes
at 2018-09-27T16:57:12Z
Merge branch 'valentindavid/silverblue-path' into 'master'
Fix outside-of-project check when project path is not canonical.
Closes #673
See merge request BuildStream/buildstream!832
-
5cdfac81
by Mathieu Bridon
at 2018-09-28T09:20:20Z
data/userconfig.yaml: Document the cache.quota setting
-
5d3f039f
by Tom Pollard
at 2018-09-28T10:34:13Z
Merge branch 'bochecha/cache-size' into 'master'
data/userconfig.yaml: Document the cache.quota setting
See merge request BuildStream/buildstream!833
-
14e1a3b3
by Jürg Billeter
at 2018-10-01T08:18:05Z
_platform: Fix get_cpu_count() with cap=None
Comparison between None and integer is not supported.
-
232662f1
by Jürg Billeter
at 2018-10-01T08:53:06Z
Merge branch 'juerg/cpucount' into 'master'
_platform: Fix get_cpu_count() with cap=None
See merge request BuildStream/buildstream!840
-
f447aedd
by Tiago Gomes
at 2018-10-01T10:33:11Z
ci: upper case variables
For consistency.
-
682dddce
by Tiago Gomes
at 2018-10-01T10:35:12Z
ci: correct freedestop-sdk version comment
-
fafa8136
by Tiago Gomes
at 2018-10-01T10:59:54Z
Merge branch 'tiagogomes/ci-minor-tweaks' into 'master'
Minor CI tweaks
See merge request BuildStream/buildstream!842
-
26e1a3c7
by Jürg Billeter
at 2018-10-01T14:58:06Z
_artifactcache/casserver.py: Implement BatchUpdateBlobs
Fixes #676.
-
f47895c0
by Jürg Billeter
at 2018-10-01T14:58:06Z
_artifactcache/cascache.py: Use BatchUpdateBlobs
This uses BatchUpdateBlobs instead of individual blob upload to speed up
artifact pushing, if the server supports it.
Fixes #677.
-
cf00c0a1
by Jürg Billeter
at 2018-10-01T15:32:30Z
Merge branch 'juerg/cas-batch' into 'master'
CAS: Implement BatchUpdateBlobs support
Closes #677 and #676
See merge request BuildStream/buildstream!839
-
5f4ae90b
by Jürg Billeter
at 2018-10-02T06:34:02Z
sandbox/_sandboxremote.py: Use helpers to set up cwd and env
This matches the other sandbox implementations.
-
0458bc4e
by Jürg Billeter
at 2018-10-02T07:08:35Z
Merge branch 'juerg/sandbox' into 'master'
sandbox/_sandboxremote.py: Use helpers to set up cwd and env
See merge request BuildStream/buildstream!841
-
d5b396e0
by Phillip Smyth
at 2018-10-02T16:40:18Z
Forcing CI to install Sphinx 1.7.9
.gitlab-ci.yml: forced pip3 install Sphinx 1.7.9
The theme we use in our docs "sphinx_rtd_theme"
Does not support the latest version of Sphinx (>1.8)
This results in search functionality failing
-
dae842fd
by Phillip Smyth
at 2018-10-02T17:14:09Z
Merge branch 'docs_Search_not_working' into 'master'
Fixing: Search functionality in the Documentation has stopped working
Closes #681
See merge request BuildStream/buildstream!848
-
8dc16d3f
by Jürg Billeter
at 2018-10-03T05:08:21Z
_platform/linux.py: Accept all configs for dummy sandbox
If Sandbox.run() is never called, sandbox config is irrelevant. If it is
called, the build will fail anyway with the dummy sandbox.
This matches the logic in the Darwin platform.
-
66446fc3
by Jürg Billeter
at 2018-10-03T05:36:38Z
Merge branch 'juerg/dummy-sandbox' into 'master'
_platform/linux.py: Accept all configs for dummy sandbox
See merge request BuildStream/buildstream!843
-
29c19bea
by Tristan Van Berkom
at 2018-10-03T07:33:48Z
git.py: Added timed activity around fetching, displaying the URL being fetched.
This will take care of silencing the status messages while checking submodules.
-
b645881c
by Tristan Van Berkom
at 2018-10-03T07:33:48Z
source.py: Silence messages while consuming source fetchers
The source fetchers might be a list or a generator, when it is
a generator (like the git source does), then we want to ensure that
we silence the status messages which might occur as a result of
consuming a source fetcher from the generator.
This fixes the logs to be less verbose.
-
c9437616
by Tristan Van Berkom
at 2018-10-03T08:07:15Z
Merge branch 'tristan/fix-status-messages' into 'master'
fix status messages
See merge request BuildStream/buildstream!845
-
11320fe2
by Tristan Van Berkom
at 2018-10-03T09:33:39Z
_scheduler: Fix bookkeeping of terminated jobs
* Enhanced the base Job class to bookkeep which jobs have
been terminated, and to consider them as `skipped` when
asked via the `skipped` property.
* Enhanced the base Queue class to bookkeep the job statuses
more carefully.
This fixes #479
-
91271964
by Tristan Van Berkom
at 2018-10-03T09:59:40Z
Merge branch 'tristan/fix-terminated-jobs' into 'master'
_scheduler: Fix bookkeeping of terminated jobs
Closes #479
See merge request BuildStream/buildstream!850
-
3bf895d2
by Jonathan Maw
at 2018-10-03T11:48:48Z
tests: Add regression test for mirroring with project.refs
-
e4969807
by Jonathan Maw
at 2018-10-03T12:48:07Z
Merge branch 'jonathan/source-mirror-project-refs' into 'master'
tests: Add regression test for mirroring with project.refs
See merge request BuildStream/buildstream!823
-
a0814399
by Tristan Van Berkom
at 2018-10-03T13:05:52Z
_scheduler/scheduler.py: Ignore interrupt events while terminating.
For some reason, we now receive a SIGINT from the main loop even
when the SIGINT occurred with the handler disconnected in an interactive
prompt. This patch simply ignores any received SIGINT events from the
main loop in the case that we are already in the process of terminating.
This fixes issue #693
-
0a1f8e3c
by Tristan Van Berkom
at 2018-10-03T13:42:20Z
Merge branch 'tristan/fix-double-terminate-prompt' into 'master'
_scheduler/scheduler.py: Ignore interrupt events while terminating.
Closes #693
See merge request BuildStream/buildstream!852
-
11161f99
by Tristan Van Berkom
at 2018-10-03T13:44:02Z
utils.py: Give save_file_atomic() a tempdir argument
Allow callers to decide where the temporary file will be created.
-
3e797bb9
by Tristan Van Berkom
at 2018-10-03T13:44:02Z
_artifactcache/cascache.py: Don't create temporary files in the CAS storage
Use the designated tempdir when creating refs, we expect that temporary
files are not created in the storage directory ever, they should be
only ever created in the designated temporary directory.
This fixes race conditions where utils._get_dir_size() throws an
unhandled exception when attempting to stat the file which inadvertantly
disappears.
-
d9020e43
by Tristan Van Berkom
at 2018-10-03T13:44:02Z
utils.py: Document _get_dir_size() expectations.
This function assumes that files do not disappear while
walking the given directory.
-
3e5ff5a9
by Tristan Van Berkom
at 2018-10-03T14:09:51Z
Merge branch 'tristan/fix-cache-size-race' into 'master'
fix cache size race
See merge request BuildStream/buildstream!854
-
e8179c34
by Jim MacArthur
at 2018-10-03T16:06:59Z
element.py: Put the temporary staging directory in artifactdir
-
3cf38c8e
by Jim MacArthur
at 2018-10-03T16:44:02Z
Merge branch 'jmac/temporaries-inside-cachedir' into 'master'
Move the temporary staging directory to artifactdir
See merge request BuildStream/buildstream!856
-
59c92bda
by Daniel Silverstone
at 2018-10-04T08:12:20Z
sandbox/_sandboxdummy.py: Take a reason for use
Since the SandboxDummy sandbox is only used in very specific
circumstances and with good reason, this adjusts the SandboxDummy
class to take a reason and to return it if `.run()` has to
raise an exception because this is a dummy sandbox.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b9ddcd0e
by Daniel Silverstone
at 2018-10-04T08:12:20Z
_platform/darwin.py: Give reason for use of dummy sandbox
Since Darwin is limited to the dummy sandbox for now due to OSXFUSE
being unsupported as yet, and there being no suitable sandboxing option
for use on OSX, give this as the reason so that it can be reported if
the user attempts a local build.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
df0d5a8b
by Daniel Silverstone
at 2018-10-04T09:04:21Z
_platform/linux.py: Refactor checks for sandboxing
To better report issues in the absence of a suitable bwrap, or the
FUSE devices, this refactors the checks for sandboxing in the Linux
platform to cover the various possibilities. The reasons are then
collated and passed to the dummy sandbox for later reporting to the
user if a local build is attempted.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b8421a9c
by Daniel Silverstone
at 2018-10-04T09:04:21Z
_site.py: Protect against failure running `bwrap --version`
On some potentially broken systems, running `bwrap --version` might
fail with an error code. This patch corrects the oversight ensuring
that we cleanly return False for version checking in such cases.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
c74bfbe5
by Daniel Silverstone
at 2018-10-04T09:04:21Z
_site.py: Reduce complexity of bwrap version comparison
Instead of an if/else ladder which is quite complex, this patch
uses the fact that Python supports by-component tuple comparison
to simply compare two (major, minor, patch) tuples
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
c46a7e87
by Daniel Silverstone
at 2018-10-04T09:13:04Z
setup.py: Change bwrap assertion to a warning
Since there are use-cases where BuildStream could be installed
onto systems which do not have BubbleWrap (e.g. for remote-build-only
scenarios) it is not correct to assert a dependency on bwrap during
installation. This patch makes the assertion a warning, and also
clarifies the message somewhat. This should fix #644
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
fd6a9573
by Jürg Billeter
at 2018-10-04T09:45:58Z
Merge branch 'danielsilverstone-ct/bwrap-check-runtime-only' into 'master'
Make bwrap check runtime only
Closes #644
See merge request BuildStream/buildstream!847
-
c5778941
by Josh Smith
at 2018-10-04T13:43:10Z
element.py: Do not save workspace on failed build
This fixes #634, BuildStream no longer saves workspace configuration on
a failed build.
-
788cde6a
by Josh Smith
at 2018-10-04T13:43:10Z
testing: Add regression test for workspace bug
-
8630bac4
by Tristan Van Berkom
at 2018-10-04T14:11:27Z
Merge branch 'Qinusty/634-workspace-failed-builds' into 'master'
Do not save workspace on failed build
Closes #634
See merge request BuildStream/buildstream!812
-
a7984218
by Valentin David
at 2018-10-04T14:56:09Z
Catch correct exception from shutil.rmtree
Python documentation is not clear on what shutil.rmtree is supposed to
raise. However from the source code, it is obvious that OSError are
raised, but never shutil.Error.
It is not easy to test in normal condition because issues happen
usually in combination with a FUSE filesystem, a probably a race
condition where `fstatat` returns an error just before the filesystem
being unmounted.
Fixes #153.
-
b7b20d9e
by Valentin David
at 2018-10-04T15:31:22Z
Merge branch 'valentindavid/rmtree_oserror' into 'master'
Catch correct exception from shutil.rmtree
Closes #153
See merge request BuildStream/buildstream!849
-
42ea143d
by Chandan Singh
at 2018-10-05T12:40:48Z
Add contrib script to generate Docker images from bst checkout
This script can be useful to generate a Docker image from `bst checkout`
of an element as a single command. While this script does not eliminate
the need for a proper Docker/OCI element plugin that would probably also
support layering, it provides a cheap way to export the element into a
Docker image.
-
b38f9f9d
by Jonathan Maw
at 2018-10-05T13:05:38Z
Merge branch 'chandan/bst-docker-import' into 'master'
Add contrib script to generate Docker images from bst checkout
See merge request BuildStream/buildstream!857
-
47df1e36
by Jürg Billeter
at 2018-10-05T13:30:38Z
element.py: Prepare local sandbox for bst checkout and bst shell
A remote execution sandbox is not suitable for these commands.
-
d3921ab8
by Jonathan Maw
at 2018-10-05T13:54:17Z
Merge branch 'juerg/remote' into 'master'
element.py: Prepare local sandbox for bst checkout and bst shell
See merge request BuildStream/buildstream!860
-
53d9f977
by Tristan Van Berkom
at 2018-10-06T14:40:46Z
CONTRIBUTING.rst: Updating CONTRIBUTING guidelines
This is almost a complete rewrite of the CONTRIBUTING guide.
* The patch submission guidelines have become less ambiguous and more strict
* Some general restructuring and reordering of the file took place
* The codeing guidelines have changed completely.
o There is much less room for ambiguity here now
o More emphasis on consistency in the codebase
o Added some more abstract points which should be considered
when writing and reviewing patches
* The policy on public/private symbols has been greatly clarified
* Added new section about adding new core plugins and what needs
to be done as a consequence of that
-
5587715c
by Tristan Van Berkom
at 2018-10-07T06:19:03Z
CONTRIBUTING.rst: Some fixes in grammer and minor corrections
-
43c7ac23
by Angelos Evripiotis
at 2018-10-07T15:19:58Z
contributing.rst: fix typo
-
3f0c919f
by Angelos Evripiotis
at 2018-10-07T15:52:52Z
Merge branch 'aevri/fixup-contributing-typos' into 'master'
contributing.rst: fix typos
See merge request BuildStream/buildstream!865
-
9aa466ed
by Javier Jardón
at 2018-10-07T17:57:42Z
Depend on fusepy package instead internal copy