-
2339f0c4
by Tom Pollard
at 2018-09-03T09:44:36Z
plugins/git.py: Warn if ref is not in given track
Add a helper function assert_ref_in_track to git.py GitMirror()
which is used when staging & initing the source workspace. It
checks the element's ref exists in the track (branch/tag) if it
has been specified, raising a warning if necessary. The warning makes
use of the warning token 'REF_NOT_IN_TRACK' from the configurable
CoreWarnings. If the element has been tracked with bst, it is assumed
that the value of ref exists in the track as it was generated from it
& as such is not asserted.
-
c96fec5d
by Tom Pollard
at 2018-09-03T09:44:36Z
tests/sources/git.py: Add tests for REF_NOT_IN_TRACK
Add tests that cover assert_ref_in_track & the configurable
CoreWarnings REF_NOT_IN_TRACK warnings token.
-
6a0cdedf
by Tom Pollard
at 2018-09-03T10:16:39Z
Merge branch 'tpollard/483' into 'master'
plugins/git.py: Warn if ref is not in given track
See merge request BuildStream/buildstream!564
-
9b327eb6
by Ben Brewer
at 2018-09-04T14:41:23Z
Improve documentation for artifact cache installation
Remove ambiguity about systemd service files being separate.
Add URL for more information about systemd service files.
Add note about public keys being mandatory for self-signed certs.
Make cert/key file naming consistent throughout document.
-
3e67e64a
by Javier Jardón
at 2018-09-04T16:43:13Z
Merge branch 'benbrewer/install-artifacts-doc-improvements' into 'master'
Improve documentation for artifact cache installation
See merge request BuildStream/buildstream!777
-
3409609e
by Daniel Silverstone
at 2018-09-04T16:55:51Z
jobs.py: Reduce FD leaks from queues and process objects
The garbage collector can take too long to get around to cleaning
up the Queue and Process instances in completed Job instances. As
such, FDs tend to leak and in very large projects this can result
in running out of FDs before a build, fetch, track, or other process
can complete. This patch reduces the chance of that by only creating
the queue when it's needed, and forcing the queue and process instances
to be deleted when the parent is finished with them.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
7b32e1ec
by Tristan Maat
at 2018-09-04T17:20:55Z
Merge branch 'danielsilverstone-ct/maybe-reduce-fd-leaks' into 'master'
jobs.py: Reduce FD leaks from queues and process objects
See merge request BuildStream/buildstream!778
-
936bb93a
by Jim MacArthur
at 2018-09-07T12:57:28Z
cascache.py: Preparation for remote execution
Refactor the push() and pull() implementations so that API additions
needed for remote-execution is made easier.
https://gitlab.com/BuildStream/buildstream/issues/454
-
b0f46545
by Martin Blanchard
at 2018-09-07T12:57:29Z
cascache.py: Introduce new push helpers
Add push_directory() and push_message() helpers along with a
verify_digest_pushed().
https://gitlab.com/BuildStream/buildstream/issues/454
-
50bf3139
by Martin Blanchard
at 2018-09-07T12:57:29Z
cascache.py: Introduce new pull helpers
Add a pull_tree() helper.
https://gitlab.com/BuildStream/buildstream/issues/454
-
853e0af7
by Martin Blanchard
at 2018-09-07T12:57:29Z
_casbaseddirectory.py: Add a method for hash recalculation
https://gitlab.com/BuildStream/buildstream/issues/454
-
e7659c29
by Jim MacArthur
at 2018-09-07T12:57:29Z
sandbox.py: Allow setting the virtual directory
This is for use after remote execution has finished, since remote
execution produces a new output directory rather than modifying
the initial directory.
https://gitlab.com/BuildStream/buildstream/issues/454
-
7306a954
by Jim MacArthur
at 2018-09-07T12:57:29Z
Add "remote-execution" project configuration option
This just adds one option, "remote-execution/url". Affects multiple files.
https://gitlab.com/BuildStream/buildstream/issues/454
-
c0ef7106
by Jim MacArthur
at 2018-09-07T12:57:29Z
format_project.rst: Document remote-execution option
https://gitlab.com/BuildStream/buildstream/issues/454
-
43651af0
by Jim MacArthur
at 2018-09-07T12:57:29Z
_sandboxremote.py: Implement the REAPI client
The remote execution client is implemented as a remote sandbox that
sends sources and build commands to a REAPI server and fetches results
once remotely executed. New file.
https://gitlab.com/BuildStream/buildstream/issues/454
-
da53c005
by Jim MacArthur
at 2018-09-07T12:57:29Z
sandbox/__init__.py: Add SandboxRemote
https://gitlab.com/BuildStream/buildstream/issues/454
-
c73896bb
by Jim MacArthur
at 2018-09-07T12:57:29Z
element.py: Get the updated virtual directory after running
Executing run() on a sandbox can now replace the virtual directory,
since remote execution returns a potentially different directory rather
than an update to the existing one. Call get_virtual_directory() again
after running to accout for this.
https://gitlab.com/BuildStream/buildstream/issues/454
-
89b29b8b
by Jim MacArthur
at 2018-09-07T12:57:29Z
element.py: Switch to SandboxRemote if config option is set
https://gitlab.com/BuildStream/buildstream/issues/454
-
116f80d2
by Jim MacArthur
at 2018-09-07T12:57:29Z
autotools.py: Mark this as a BST_VIRTUAL_DIRECTORY plugin
https://gitlab.com/BuildStream/buildstream/issues/454
-
05956c71
by Martin Blanchard
at 2018-09-07T12:57:29Z
cmake.py: Mark this as a BST_VIRTUAL_DIRECTORY plugin
https://gitlab.com/BuildStream/buildstream/issues/454
-
909b3be4
by Martin Blanchard
at 2018-09-07T12:57:29Z
make.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin
https://gitlab.com/BuildStream/buildstream/issues/454
-
556f2abb
by Martin Blanchard
at 2018-09-07T12:57:30Z
meson.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin
https://gitlab.com/BuildStream/buildstream/issues/454
-
71e2b543
by Martin Blanchard
at 2018-09-07T12:57:30Z
qmake.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin
https://gitlab.com/BuildStream/buildstream/issues/454
-
d4c515d0
by Martin Blanchard
at 2018-09-07T12:57:30Z
tests/artifactcache: Add a simple test project
https://gitlab.com/BuildStream/buildstream/issues/454
-
923d443b
by Martin Blanchard
at 2018-09-07T13:22:48Z
tests/artifactcache: Add push unit-tests
https://gitlab.com/BuildStream/buildstream/issues/454
-
9ef53163
by Martin Blanchard
at 2018-09-07T13:22:57Z
tests/artifactcache: Add pull unit-tests
https://gitlab.com/BuildStream/buildstream/issues/454
-
f52169b2
by Qinusty
at 2018-09-07T16:48:02Z
Merge branch 'jmac/remote_execution_client' into 'master'
Remote execution client
See merge request BuildStream/buildstream!626
-
8326b0f7
by Thomas Coldrick
at 2018-09-08T17:54:22Z
Upstream libtool .la file removal
In freedesktop-sdk we add a script to our project.conf to remove
libtool .la files from autotools projects after install, this seems
like a sensible default, so we're attempting to send it upstream.
-
886a836e
by Valentin David
at 2018-09-08T20:07:29Z
Merge branch 'coldtom/autotools-libtool' into 'master'
Upstream freedesktop-sdk autotools config for libtool .la files
See merge request BuildStream/buildstream!683
-
2174db01
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
tests/artifactcache/expiry.py: Fix test case expectations.
-
e5847077
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
_scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE
This is what the whole resource.py thing was created for, the
cleanup job must have exclusive access to the cache, while the pull
and build jobs which result in adding new artifacts, must only require
shared access.
This is a part of #623
-
98ab2b89
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
_scheduler/resources.py: Set CACHE max resources to 0
This means there is no cap for shared resource requests.
Together with the previous commit, this causes the cleanup
job and the pull/build jobs to all require unlimited shared
access to the CACHE resource; while only the cleanup job
requires exclusive access to the resource.
This is a part of #623
-
deaecdc1
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
_scheduler/scheduler.py: Make CacheSizeJob() not require the CACHE exclusively
This runs after every pull, and does not need the cache exclusively,
only the cleanup job requires the cache exclusively.
Without this, every time a cache_size job is queued, all pull and
build jobs need to complete before cache_size job can run exclusively,
which is not good.
This is a part of #623
-
1f118ba4
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
element.py: Adding missing API documenting comment
-
26ccc1ba
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
_artifactcache: Making public methods public
The artifact cache provides the following public methods for
external callers, but was hiding them away as if they are private.
o ArtifactCache.add_artifact_size()
o ArtifactCache.set_cache_size()
Mark these properly public
-
00784059
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
_scheduler: API fixes; _check_cache_size_real() -> check_cache_size()
Here we have a very private looking _check_cache_size_real() function
which no-one would ever want to call from outside of the _scheduler,
especially given it's `_real()` prefix we should look for another
outward facing API to use.
However this is not private to the scheduler, and is intended to
be called by the `Queue` implementations.
o Renamed this to check_cache_size()
o Moved it to the public API section of the Scheduler object
o Added the missing API documenting comment
o Also added the missing API documenting comment to the private
`_run_cleanup()` callback which runs in response to completion
of the cache size calculation job.
o Also place the cleanup job logs into a cleanup subdirectory,
for better symmetry with the cache_size jobs which now have
their own subdirectory
-
a3825ba6
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
_artifactcache/artifactcache.py: Removing unused variable.
The ArtifactCache._local variable used to exist in order to
use a special hack to allow absolute paths to a remote artifact
cache, this was all for the purpose of testing.
This has all gone away with the introduction of CAS, leaving behind
a stale variable.
-
791f7dda
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
_artifactcache/artifactcache.py: Sealing away some the estimated size
Previously, the API contract was to expose the estimated_size variable
on the ArtifactCache instance for all to see, however it is only relevant
to the ArtifactCache abstract class code. Subclasses were informed to
update the estimated_size variable in their calculate_cache_size()
implementation.
To untangle this and hide away the estimated size, this commit
does the following:
o Introduces ArtifactCache.compute_cache_size() API for external
callers
o ArtifactCache.compute_cache_size() calls the abstract method
for the CasCache subclass to implement
o ArtifactCache.compute_cache_size() updates the private
estimated_size variable
o All direct callers to ArtifactCache.calculate_cache_size(),
have been updated to use the ArtifactCache.compute_cache_size()
method instead, which takes care of updating anything local
to the ArtifactCache abstract class code (the estimated_size)
-
cbfddc13
by Tristan Van Berkom
at 2018-09-10T07:53:56Z
element.py: Remove _get_artifact_size()
There is no justification to hold onto this state here.
Instead, just make `Element._assemble()` return the size of the
artifact it cached, and localize handling of that return value in
the BuildQueue implementation where the value is observed.
-
8b748f2d
by Tristan Van Berkom
at 2018-09-10T07:53:57Z
_scheduler/jobs/cachesizejob.py: Use Platform.get_platform() public accessor
This was previously poking directly at the Platform._instance.
Also, use the name 'artifacts' to hold the artifact cache to
be consistent with other parts of the codebase, instead of
calling it 'cache'.
-
742908ea
by Tristan Van Berkom
at 2018-09-10T07:53:57Z
_scheduler/jobs/cleanupjob.py: Use Platform.get_platform() public accessor
This was previously poking directly at the Platform._instance.
Also, use the name 'artifacts' to hold the artifact cache to
be consistent with other parts of the codebase, instead of
calling it 'cache'.
-
335b2fdd
by Tristan Van Berkom
at 2018-09-10T07:53:57Z
element.py: Remove _get_artifact_cache() accessor.
The artifact cache is anyway a singleton, the Element itself
has an internal handle on the artifact cache because it is
needed very frequently; but an artifact cache is not element
specific and should not be looked up by surrounding code
on a per element basis.
Updated _scheduler/queues/queue.py, _scheduler/queues/buildqueue.py
and _scheduler/jobs/elementjob.py to get the artifact cache directly
from the Platform
-
6db54c8f
by Tristan Van Berkom
at 2018-09-10T07:53:57Z
_artifactcache: There shalt be only one cache size
This does a lot of house cleaning, finally bringing cache
cleanup logic to a level of comprehensibility.
Changes in this commit include:
o _artifactcache/artifactcache.py: _cache_size, _cache_quota and
_cache_lower_threshold are now all private variables.
get_approximate_cache_size() is now get_cache_size()
Added get_quota_exceeded() for the purpose of safely checking
if we have exceeded the quota.
set_cache_size() now asserts that the passed size is not None,
it is not acceptable to set a None size cache anymore.
o _artifactcache/cascache.py: No longer set the ArtifactCache
'cache_size' variable violently in the commit() method.
Also the calculate_cache_size() method now unconditionally
calculates the cache size, that is what it's for.
o _scheduler/jobs/cachesizejob.py & _scheduler/jobs/cleanupjob.py:
Now check the success status. Don't try to set the cache size
in the case that the job was terminated.
o _scheduler/jobs/elementjob.py & _scheduler/queues/queue.py:
No longer passing around the cache size from child tasks,
this happens only explicitly, not implicitly for all tasks.
o _scheduler/queues/buildqueue.py & _scheduler/scheduler.py:
Use get_quota_exceeded() accessor
This is a part of #623
-
958ef8ef
by Tristan Van Berkom
at 2018-09-10T09:23:06Z
Merge branch 'tristan/fix-cache-exclusivity' into 'master'
_scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE
See merge request BuildStream/buildstream!775
-
40f25512
by Richard Maw
at 2018-09-10T12:29:15Z
subprocesses: Ensure PWD is set in process environment
Naive getcwd implementations (such as in bash 4.4) can break
when bind-mounts to different paths on the same filesystem are present,
since the algorithm needs to know whether it's a mount-point
to know whether it can trust the inode value from the readdir result
or to use stat on the directory.
Less naive implementations (such as in glibc) iterate again using stat
in the case of not finding the directory because the inode in readdir was wrong,
though a Linux-specific implementation could use name_to_handle_at.
Letting the command know what directory it is in makes it unnecessary
for it to call the faulty getcwd in the first place.
-
247686dc
by richardmaw-codethink
at 2018-09-10T14:47:48Z
Merge branch 'richardmaw/subprocess-PWD' into 'master'
Ensure PWD is set in process environment
See merge request BuildStream/buildstream!782
-
ef66daf5
by Tiago Gomes
at 2018-09-10T14:57:41Z
tests: avoid setting max-jobs
Setting "max-jobs" won't be allowed anymore in a following commit.
-
a37bc6ce
by Tiago Gomes
at 2018-09-10T14:57:41Z
docs: document protected variables
And remove then from the defaults as they are dynamically set by
BuildStream.
-
af74a3f8
by Tiago Gomes
at 2018-09-10T14:57:41Z
element: validate configuration variables
Ensure that protected variables are not being redefined by the user.
-
75c55633
by Tiago Gomes
at 2018-09-10T14:57:41Z
Add tests for validating configuration variables
-
e0bb71b2
by Tiago Gomes
at 2018-09-10T15:27:53Z
Merge branch 'tiagogomes/issue-287' into 'master'
Add validation of configuration variables
See merge request BuildStream/buildstream!678
-
47f3064a
by Jürg Billeter
at 2018-09-10T15:43:25Z
_artifactcache/casserver.py: Implement Capabilities service
-
1a7fb3cb
by Jürg Billeter
at 2018-09-10T15:43:25Z
_artifactcache/casserver.py: Implement BatchReadBlobs
Fixes #632.
-
b3ffcdc8
by Jürg Billeter
at 2018-09-10T16:07:30Z
Merge branch 'juerg/cas-batch' into 'master'
_artifactcache/casserver.py: Implement BatchReadBlobs
Closes #632
See merge request BuildStream/buildstream!785
-
6f925bcb
by Javier Jardón
at 2018-09-13T07:38:48Z
source/install_source.rst: pip plugin depends on host pip
-
c6155f8d
by Javier Jardón
at 2018-09-13T08:11:54Z
Merge branch 'jjardon/pip_dependency' into 'master'
source/install_source.rst: pip plugin depends on host pip
See merge request BuildStream/buildstream!791
-
19838a07
by Chandan Singh
at 2018-09-13T10:58:38Z
README.rst: Add status badges for PyPI release and Python versions
The first badge will work fine right away while the second badge will
show "not found" until a release is made after merging this branch:
https://gitlab.com/BuildStream/buildstream/merge_requests/718.