[Notes] [Git][BuildStream/buildstream][jonathan/expose-downloadablefiles

Jonathan Maw pushed to branch jonathan/expose-downloadablefilesource at BuildStream / buildstream

Commits:

6047a575

by Tristan Van Berkom at 2018-08-28T13:41:36Z

tests/frontend/workspaces.py: Removing some redundant tests

Removed redundant tests from recently merged !740, this new
test does not need to run for every different source kind.

7535fda8

by Jürg Billeter at 2018-08-28T14:25:35Z

Merge branch 'tristan/remove-redundant-test' into 'master'

tests/frontend/workspaces.py: Removing some redundant tests

See merge request BuildStream/buildstream!741

10eeb503
by Javier Jardón at 2018-08-28T22:22:00Z
```
doc/source/conf.py: Fix E201, E202 warnings
```
d3c32ca2
by Javier Jardón at 2018-08-28T22:22:00Z
```
doc/source/conf.py: Fix E402 warning
```
3ae5fd05
by Javier Jardón at 2018-08-28T22:22:00Z
```
Fix E305 warnings
```
c6fb5ba7
by Javier Jardón at 2018-08-28T22:22:00Z
```
Fix E741 warnings
```

c5eca59d

by Tristan Van Berkom at 2018-08-29T10:45:33Z

Merge branch 'jjardon/pycodestyle_fixes' into 'master'

Some pycodestyle (PEP8) fixes

See merge request BuildStream/buildstream!746

63c6ee72

by Tristan Van Berkom at 2018-08-29T10:46:30Z

doc: Adding badges.py release badge generator

This adds a step to the docs generation Makefile to generate
release.svg and snapshot.svg badges, modelled after the gitlab
badges.

This also adds the generated badges directory in docs to .gitignore

2d527052

by Tristan Van Berkom at 2018-08-29T10:46:30Z

README.rst: Adding the release and snapshot badges to the readme

This is a part of #528

29e7eea8

by Tristan Van Berkom at 2018-08-29T10:46:30Z

doc: Adding the release badges to the install page and the semantic versioning page

This is a part of #528

5d508779

by Tristan Van Berkom at 2018-08-29T11:13:53Z

Merge branch 'tristan/docs-version-badge' into 'master'

Release badges

See merge request BuildStream/buildstream!742

323a5403

by Chandan Singh at 2018-08-29T12:13:28Z

setup.py: Add more metadata useful for PyPI

- setup.py: List useful links using `project_urls`

- setup.py: Specify minumum python vresion using `python_requires`

  `python_requires` is the new way of specifying requirements for python
  version, as documented @
  https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires.

  This will also make this requirement appear nicely on the PyPI project
  page.

- setup.py: Add project classifiers

  PyPI will use these classifiers to categorize projects while searching
  or browsing. Full list of classifiers can be found at
  https://pypi.org/classifiers.

5fc7b5ac

by Tristan Van Berkom at 2018-08-29T12:37:08Z

Merge branch 'chandan/setup.py-metadata' into 'master'

setup.py: Add more metadata useful for PyPI

See merge request BuildStream/buildstream!718

42ad937d

by Josh Smith at 2018-08-29T12:38:15Z

Prevent ValueError on URLs without an alias

Fixes #597

aa3a33b3

by Tristan Van Berkom at 2018-08-29T13:03:58Z

Merge branch 'Qinusty/597-non-alias-url-fix' into 'master'

Fix ValueError traceback for URL without alias or URI scheme

Closes #597

See merge request BuildStream/buildstream!700

2a2a79de

by Valentin David at 2018-08-29T14:11:48Z

Disable round-tripping when element is not modified

2df7607c

by Tristan Van Berkom at 2018-08-29T14:43:55Z

Merge branch 'valentindavid/roundtripping_only_when_modified' into 'master'

Disable round-tripping when element is not modified

See merge request BuildStream/buildstream!748

c146dde5
by Benjamin Schubert at 2018-08-29T16:00:24Z
```
Log not-found objects in the cache as SKIPPED
```

19e87afb

by Benjamin Schubert at 2018-08-29T16:01:12Z

Add tests that not-found objects in cache are shown as SKIPPED

9cc5817f

by Qinusty at 2018-08-29T16:25:48Z

Merge branch 'bschubert/log-missed-cache' into 'master'

Log not-found objects in the cache as SKIPPED

See merge request BuildStream/buildstream!729

3c8646e3

by Josh Smith at 2018-08-29T16:34:22Z

Add cyclic check within variable resolution

This aims to address #600, this will raise an exception when a resolved
variable contains a reference to the variable.

d41c42d6

by Josh Smith at 2018-08-29T16:34:22Z

Add tests for cyclic variables check

Note: This modifies the docker containers used for testing to supply the
pytest-timeout package.

c7b72c30

by Qinusty at 2018-08-29T16:58:48Z

Merge branch 'Qinusty/600-recursive-variables' into 'master'

Add cyclic check within variable resolution

See merge request BuildStream/buildstream!712

d28e1353

by Tristan Van Berkom at 2018-08-30T06:02:41Z

tests/plugins/filter.py: Don't run redundant tests

There is no reason that the filter element codepaths can behave
differently depending on the Source implementation used in the test,
as the Source implementation does not have any filter specific
virtual methods.

Removing the redundant tests and just performing these tests with the git source.

e77ae07a

by Tristan Van Berkom at 2018-08-30T06:36:41Z

Merge branch 'tristan/reduce-filter-tests' into 'master'

tests/plugins/filter.py: Don't run redundant tests

See merge request BuildStream/buildstream!753

9ea40e2c

by Tristan Van Berkom at 2018-08-30T06:38:13Z

tests/frontend/mirror.py: Reenable test_mirror_fetch_upstream_absent[ostree]

This test was skipped because of issue #538, but #538 was fixed
and the test was still not reenabled.

8aec1102

by Tristan Van Berkom at 2018-08-30T07:06:17Z

Merge branch 'tristan/538-reenable-ostree-test' into 'master'

tests/frontend/mirror.py: Reenable test_mirror_fetch_upstream_absent[ostree]

See merge request BuildStream/buildstream!755

fcc17c82

by Tristan Van Berkom at 2018-08-30T08:44:34Z

source.py: Stylistic changes in Source.__do_fetch()

Added some comments to make the flow easier to follow, and
removed an annoying 'success' variabled in favor of a for / else
loop statement.

60ad044a

by Tristan Van Berkom at 2018-08-30T08:44:34Z

source.py: Move Source.mark_download_url() to the public methods.

This was sitting in the section for abstract methods, but this
is most definitely not an abstract method to be implemented by
Sources.

c339f917

by Tristan Van Berkom at 2018-08-30T08:44:34Z

plugins/source/git.py: Fix formatting of url in tracking

This was displaying the aliased URL which is pretty useless,
use the translated URL for the timed activity.

26565f11

by Tristan Van Berkom at 2018-08-30T09:12:28Z

Merge branch 'tristan/source-mirroring-changes' into 'master'

Minor code changes revolving around source mirroring

See merge request BuildStream/buildstream!758

a38d9163

by Valentin David at 2018-08-30T10:26:57Z

Report processing errors from tracking

Failures to write files when tracking were not reported.

Fixes #533.

02138181

by Tristan Van Berkom at 2018-08-30T11:36:00Z

Merge branch 'valentindavid/post_tracking_errors' into 'master'

Report processing errors from tracking

Closes #533

See merge request BuildStream/buildstream!747

d23d4f3e

by Tristan Van Berkom at 2018-08-30T12:02:27Z

_artifactcache/artifactcache.py: Write the cache_size file atomically

This is causing issues while the size file is being read and written
simultaneously.

The proper fix will be to read/add/save the file atomically and that
will require locking, but this fix is a good stop gap to existing crashes.

bb6aa5a5

by Tristan Van Berkom at 2018-08-30T12:27:55Z

Merge branch 'tristan/atomic-cache-size-file' into 'master'

_artifactcache/artifactcache.py: Write the cache_size file atomically

See merge request BuildStream/buildstream!762

680e4fe1

by knownexus at 2018-08-30T13:37:55Z

Replacing string 'bzr' with value from host tools

8cd719eb

by Phillip Smyth at 2018-08-30T15:47:06Z

Merge branch 'bzr_fix' into 'master'

Replacing string 'bzr' with value from host tools

See merge request BuildStream/buildstream!764

18d0bfb4

by Tom Pollard at 2018-08-30T19:14:33Z

buildstream/_project.py: Report if project.conf is missing name

Explicitly check that project.conf contains a name. This resolves
the issue of the provenance check from _yaml.py incorrectly reporting
the offending file as the default_config_node projectconfig.yaml
when no name key exists in the pre_config_node dict.

95121148

by Tiago Gomes at 2018-08-30T20:52:29Z

Merge branch 'tpollard/591' into 'master'

buildstream/_project.py: Report if project.conf is missing name

See merge request BuildStream/buildstream!680

775d3fca

by Tristan Van Berkom at 2018-08-31T06:05:32Z

_project.py: Adding FIXME comment to address #591 properly

0682aa1c

by Josh Smith at 2018-08-31T10:58:12Z

job.py: Modify retry messages to be FAIL

This adjusts the message handler for the child processes to no longer
override the message type.

This also removes the ability for unhandled non BstError exceptions to
retry.

68eff9ba

by Josh Smith at 2018-08-31T10:58:20Z

tests: Modify tests to ensure retry FAILURE

9b18e1bf

by Tristan Van Berkom at 2018-08-31T11:22:51Z

Merge branch 'Qinusty/retries-should-fail' into 'master'

Retries log as failures

See merge request BuildStream/buildstream!766

e2d8fedf

by Josh Smith at 2018-08-31T15:53:43Z

scheduler.py: Move cache_size logs into folder

This prevents the cache_size.pid.log files from cluttering the root
log directory.

6805a2ab

by Tristan Van Berkom at 2018-09-01T08:24:05Z

Merge branch 'Qinusty/cache-size-directory' into 'master'

Move cache_size.pid.log files into a subdirectory of logs

See merge request BuildStream/buildstream!769

f8b06acc

by Tristan Van Berkom at 2018-09-02T09:33:39Z

source.py: Document Source.get_source_fetchers() to return an iterable

Also highlight the fact that the plugin can rely on the fetcher's
fetch() method getting called before consuming the next item in the
list, which is the magick behavior that the git plugin relies on.

This is a part of #620

559b6fbc

by Tristan Van Berkom at 2018-09-02T09:34:10Z

source.py: Fixing docs link formatting to be consistent.

43ad22d7

by Tristan Van Berkom at 2018-09-02T09:34:10Z

source.py: Documenting that marking download URLs is mandatory

A download URL must be interpreted by the core at `Plugin.configure()`
time, even if only employed later on.

This is a part of #620

16462e9c

by Tristan Van Berkom at 2018-09-02T09:36:49Z

plugin.py: Added _configure() and _get_configuring() private APIs

Keeps track of whether the plugin is currently being configured.

Adjusted Element and Source classes to call _configure() in place
of calling configure() directly.

This is a part of #620

1f9c4147

by Tristan Van Berkom at 2018-09-02T09:37:12Z

source.py: Added `primary` argument to URL marking APIs

The Source must now mention whether the marked or translated
URL is "primary" or not. Even when a Source may have multiple
URLs, the auxilliary URLs are derived from the primary one, not
only is this true for git, but it is mandated by our tracking
API which assumes there is a primary URL.

This adjusts the `git` source and the test `fetch_source.py` source
to behave properly and advertize it's primary URL properly.

This is a part of #620

f9b2f1a9

by Tristan Van Berkom at 2018-09-02T09:37:21Z

source.py: Track marked URLs and assert they are marked during Plugin.configure()

This cannot test for unaliased URLs, as those can be discovered later
on outside of user provided element configuration; at least we
assert that if an alias was used, we have seen it at load time.

This will cause a BUG to occur for a plugin which marks an aliased
URL (or attempts to translate one) outside of `Plugin.configure()`,
if that URL was not previously seen.

This is a part of #620

88460cd2

by Tristan Van Berkom at 2018-09-03T06:23:38Z

Merge branch 'tristan/source-fetcher-changes' into 'master'

Source fetcher changes

See merge request BuildStream/buildstream!772

2339f0c4

by Tom Pollard at 2018-09-03T09:44:36Z

plugins/git.py: Warn if ref is not in given track

Add a helper function assert_ref_in_track to git.py GitMirror()
which is used when staging & initing the source workspace. It
checks the element's ref exists in the track (branch/tag) if it
has been specified, raising a warning if necessary. The warning makes
use of the warning token 'REF_NOT_IN_TRACK' from the configurable
CoreWarnings. If the element has been tracked with bst, it is assumed
that the value of ref exists in the track as it was generated from it
& as such is not asserted.

c96fec5d

by Tom Pollard at 2018-09-03T09:44:36Z

tests/sources/git.py: Add tests for REF_NOT_IN_TRACK

Add tests that cover assert_ref_in_track & the configurable
CoreWarnings REF_NOT_IN_TRACK warnings token.

6a0cdedf

by Tom Pollard at 2018-09-03T10:16:39Z

Merge branch 'tpollard/483' into 'master'

plugins/git.py: Warn if ref is not in given track

See merge request BuildStream/buildstream!564

9b327eb6

by Ben Brewer at 2018-09-04T14:41:23Z

Improve documentation for artifact cache installation

Remove ambiguity about systemd service files being separate.
Add URL for more information about systemd service files.
Add note about public keys being mandatory for self-signed certs.
Make cert/key file naming consistent throughout document.

3e67e64a

by Javier Jardón at 2018-09-04T16:43:13Z

Merge branch 'benbrewer/install-artifacts-doc-improvements' into 'master'

Improve documentation for artifact cache installation

See merge request BuildStream/buildstream!777

3409609e

by Daniel Silverstone at 2018-09-04T16:55:51Z

jobs.py: Reduce FD leaks from queues and process objects

The garbage collector can take too long to get around to cleaning
up the Queue and Process instances in completed Job instances.  As
such, FDs tend to leak and in very large projects this can result
in running out of FDs before a build, fetch, track, or other process
can complete.  This patch reduces the chance of that by only creating
the queue when it's needed, and forcing the queue and process instances
to be deleted when the parent is finished with them.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

7b32e1ec

by Tristan Maat at 2018-09-04T17:20:55Z

Merge branch 'danielsilverstone-ct/maybe-reduce-fd-leaks' into 'master'

jobs.py: Reduce FD leaks from queues and process objects

See merge request BuildStream/buildstream!778

936bb93a

by Jim MacArthur at 2018-09-07T12:57:28Z

cascache.py: Preparation for remote execution

Refactor the push() and pull() implementations so that API additions
needed for remote-execution is made easier.

https://gitlab.com/BuildStream/buildstream/issues/454

b0f46545

by Martin Blanchard at 2018-09-07T12:57:29Z

cascache.py: Introduce new push helpers

Add push_directory() and push_message() helpers along with a
verify_digest_pushed().

https://gitlab.com/BuildStream/buildstream/issues/454

50bf3139

by Martin Blanchard at 2018-09-07T12:57:29Z

cascache.py: Introduce new pull helpers

Add a pull_tree() helper.

https://gitlab.com/BuildStream/buildstream/issues/454

853e0af7

by Martin Blanchard at 2018-09-07T12:57:29Z

_casbaseddirectory.py: Add a method for hash recalculation

https://gitlab.com/BuildStream/buildstream/issues/454

e7659c29

by Jim MacArthur at 2018-09-07T12:57:29Z

sandbox.py: Allow setting the virtual directory

This is for use after remote execution has finished, since remote
execution produces a new output directory rather than modifying
the initial directory.

https://gitlab.com/BuildStream/buildstream/issues/454

7306a954

by Jim MacArthur at 2018-09-07T12:57:29Z

Add "remote-execution" project configuration option

This just adds one option, "remote-execution/url". Affects multiple files.

https://gitlab.com/BuildStream/buildstream/issues/454

c0ef7106

by Jim MacArthur at 2018-09-07T12:57:29Z

format_project.rst: Document remote-execution option

https://gitlab.com/BuildStream/buildstream/issues/454

43651af0

by Jim MacArthur at 2018-09-07T12:57:29Z

_sandboxremote.py: Implement the REAPI client

The remote execution client is implemented as a remote sandbox that
sends sources and build commands to a REAPI server and fetches results
once remotely executed. New file.

https://gitlab.com/BuildStream/buildstream/issues/454

da53c005

by Jim MacArthur at 2018-09-07T12:57:29Z

sandbox/__init__.py: Add SandboxRemote

https://gitlab.com/BuildStream/buildstream/issues/454

c73896bb

by Jim MacArthur at 2018-09-07T12:57:29Z

element.py: Get the updated virtual directory after running

Executing run() on a sandbox can now replace the virtual directory,
since remote execution returns a potentially different directory rather
than an update to the existing one. Call get_virtual_directory() again
after running to accout for this.

https://gitlab.com/BuildStream/buildstream/issues/454

89b29b8b

by Jim MacArthur at 2018-09-07T12:57:29Z

element.py: Switch to SandboxRemote if config option is set

https://gitlab.com/BuildStream/buildstream/issues/454

116f80d2

by Jim MacArthur at 2018-09-07T12:57:29Z

autotools.py: Mark this as a BST_VIRTUAL_DIRECTORY plugin

https://gitlab.com/BuildStream/buildstream/issues/454

05956c71

by Martin Blanchard at 2018-09-07T12:57:29Z

cmake.py: Mark this as a BST_VIRTUAL_DIRECTORY plugin

https://gitlab.com/BuildStream/buildstream/issues/454

909b3be4

by Martin Blanchard at 2018-09-07T12:57:29Z

make.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin

https://gitlab.com/BuildStream/buildstream/issues/454

556f2abb

by Martin Blanchard at 2018-09-07T12:57:30Z

meson.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin

https://gitlab.com/BuildStream/buildstream/issues/454

71e2b543

by Martin Blanchard at 2018-09-07T12:57:30Z

qmake.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin

https://gitlab.com/BuildStream/buildstream/issues/454

d4c515d0

by Martin Blanchard at 2018-09-07T12:57:30Z

tests/artifactcache: Add a simple test project

https://gitlab.com/BuildStream/buildstream/issues/454

923d443b

by Martin Blanchard at 2018-09-07T13:22:48Z

tests/artifactcache: Add push unit-tests

https://gitlab.com/BuildStream/buildstream/issues/454

9ef53163

by Martin Blanchard at 2018-09-07T13:22:57Z

tests/artifactcache: Add pull unit-tests

https://gitlab.com/BuildStream/buildstream/issues/454

f52169b2

by Qinusty at 2018-09-07T16:48:02Z

Merge branch 'jmac/remote_execution_client' into 'master'

Remote execution client

See merge request BuildStream/buildstream!626

8326b0f7

by Thomas Coldrick at 2018-09-08T17:54:22Z

Upstream libtool .la file removal

In freedesktop-sdk we add a script to our project.conf to remove
libtool .la files from autotools projects after install, this seems
like a sensible default, so we're attempting to send it upstream.

886a836e

by Valentin David at 2018-09-08T20:07:29Z

Merge branch 'coldtom/autotools-libtool' into 'master'

Upstream freedesktop-sdk autotools config for libtool .la files

See merge request BuildStream/buildstream!683

2174db01

by Tristan Van Berkom at 2018-09-10T07:53:56Z

tests/artifactcache/expiry.py: Fix test case expectations.

e5847077

by Tristan Van Berkom at 2018-09-10T07:53:56Z

_scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE

This is what the whole resource.py thing was created for, the
cleanup job must have exclusive access to the cache, while the pull
and build jobs which result in adding new artifacts, must only require
shared access.

This is a part of #623

98ab2b89

by Tristan Van Berkom at 2018-09-10T07:53:56Z

_scheduler/resources.py: Set CACHE max resources to 0

This means there is no cap for shared resource requests.

Together with the previous commit, this causes the cleanup
job and the pull/build jobs to all require unlimited shared
access to the CACHE resource; while only the cleanup job
requires exclusive access to the resource.

This is a part of #623

deaecdc1

by Tristan Van Berkom at 2018-09-10T07:53:56Z

_scheduler/scheduler.py: Make CacheSizeJob() not require the CACHE exclusively

This runs after every pull, and does not need the cache exclusively,
only the cleanup job requires the cache exclusively.

Without this, every time a cache_size job is queued, all pull and
build jobs need to complete before cache_size job can run exclusively,
which is not good.

This is a part of #623

1f118ba4

by Tristan Van Berkom at 2018-09-10T07:53:56Z

element.py: Adding missing API documenting comment

26ccc1ba

by Tristan Van Berkom at 2018-09-10T07:53:56Z

_artifactcache: Making public methods public

The artifact cache provides the following public methods for
external callers, but was hiding them away as if they are private.

  o ArtifactCache.add_artifact_size()

  o ArtifactCache.set_cache_size()

Mark these properly public

00784059

by Tristan Van Berkom at 2018-09-10T07:53:56Z

_scheduler: API fixes; _check_cache_size_real() -> check_cache_size()

Here we have a very private looking _check_cache_size_real() function
which no-one would ever want to call from outside of the _scheduler,
especially given it's `_real()` prefix we should look for another
outward facing API to use.

However this is not private to the scheduler, and is intended to
be called by the `Queue` implementations.

  o Renamed this to check_cache_size()

  o Moved it to the public API section of the Scheduler object

  o Added the missing API documenting comment

  o Also added the missing API documenting comment to the private
    `_run_cleanup()` callback which runs in response to completion
    of the cache size calculation job.

  o Also place the cleanup job logs into a cleanup subdirectory,
    for better symmetry with the cache_size jobs which now have
    their own subdirectory

a3825ba6

by Tristan Van Berkom at 2018-09-10T07:53:56Z

_artifactcache/artifactcache.py: Removing unused variable.

The ArtifactCache._local variable used to exist in order to
use a special hack to allow absolute paths to a remote artifact
cache, this was all for the purpose of testing.

This has all gone away with the introduction of CAS, leaving behind
a stale variable.

791f7dda

by Tristan Van Berkom at 2018-09-10T07:53:56Z

_artifactcache/artifactcache.py: Sealing away some the estimated size

Previously, the API contract was to expose the estimated_size variable
on the ArtifactCache instance for all to see, however it is only relevant
to the ArtifactCache abstract class code. Subclasses were informed to
update the estimated_size variable in their calculate_cache_size()
implementation.

To untangle this and hide away the estimated size, this commit
does the following:

  o Introduces ArtifactCache.compute_cache_size() API for external
    callers

  o ArtifactCache.compute_cache_size() calls the abstract method
    for the CasCache subclass to implement

  o ArtifactCache.compute_cache_size() updates the private
    estimated_size variable

  o All direct callers to ArtifactCache.calculate_cache_size(),
    have been updated to use the ArtifactCache.compute_cache_size()
    method instead, which takes care of updating anything local
    to the ArtifactCache abstract class code (the estimated_size)

cbfddc13

by Tristan Van Berkom at 2018-09-10T07:53:56Z

element.py: Remove _get_artifact_size()

There is no justification to hold onto this state here.

Instead, just make `Element._assemble()` return the size of the
artifact it cached, and localize handling of that return value in
the BuildQueue implementation where the value is observed.

8b748f2d

by Tristan Van Berkom at 2018-09-10T07:53:57Z

_scheduler/jobs/cachesizejob.py: Use Platform.get_platform() public accessor

This was previously poking directly at the Platform._instance.

Also, use the name 'artifacts' to hold the artifact cache to
be consistent with other parts of the codebase, instead of
calling it 'cache'.

742908ea

by Tristan Van Berkom at 2018-09-10T07:53:57Z

_scheduler/jobs/cleanupjob.py: Use Platform.get_platform() public accessor

This was previously poking directly at the Platform._instance.

Also, use the name 'artifacts' to hold the artifact cache to
be consistent with other parts of the codebase, instead of
calling it 'cache'.

335b2fdd

by Tristan Van Berkom at 2018-09-10T07:53:57Z

element.py: Remove _get_artifact_cache() accessor.

The artifact cache is anyway a singleton, the Element itself
has an internal handle on the artifact cache because it is
needed very frequently; but an artifact cache is not element
specific and should not be looked up by surrounding code
on a per element basis.

Updated _scheduler/queues/queue.py, _scheduler/queues/buildqueue.py
and _scheduler/jobs/elementjob.py to get the artifact cache directly
from the Platform

6db54c8f

by Tristan Van Berkom at 2018-09-10T07:53:57Z

_artifactcache: There shalt be only one cache size

This does a lot of house cleaning, finally bringing cache
cleanup logic to a level of comprehensibility.

Changes in this commit include:

  o _artifactcache/artifactcache.py: _cache_size, _cache_quota and
    _cache_lower_threshold are now all private variables.

    get_approximate_cache_size() is now get_cache_size()

    Added get_quota_exceeded() for the purpose of safely checking
    if we have exceeded the quota.

    set_cache_size() now asserts that the passed size is not None,
    it is not acceptable to set a None size cache anymore.

  o _artifactcache/cascache.py: No longer set the ArtifactCache
    'cache_size' variable violently in the commit() method.

    Also the calculate_cache_size() method now unconditionally
    calculates the cache size, that is what it's for.

  o _scheduler/jobs/cachesizejob.py & _scheduler/jobs/cleanupjob.py:

    Now check the success status. Don't try to set the cache size
    in the case that the job was terminated.

  o _scheduler/jobs/elementjob.py & _scheduler/queues/queue.py:

    No longer passing around the cache size from child tasks,
    this happens only explicitly, not implicitly for all tasks.

  o _scheduler/queues/buildqueue.py & _scheduler/scheduler.py:

    Use get_quota_exceeded() accessor

This is a part of #623

958ef8ef

by Tristan Van Berkom at 2018-09-10T09:23:06Z

Merge branch 'tristan/fix-cache-exclusivity' into 'master'

_scheduler/queues: Mark build and pull queue as requiring shared access to the CACHE

See merge request BuildStream/buildstream!775

40f25512

by Richard Maw at 2018-09-10T12:29:15Z

subprocesses: Ensure PWD is set in process environment

Naive getcwd implementations (such as in bash 4.4) can break
when bind-mounts to different paths on the same filesystem are present,
since the algorithm needs to know whether it's a mount-point
to know whether it can trust the inode value from the readdir result
or to use stat on the directory.

Less naive implementations (such as in glibc) iterate again using stat
in the case of not finding the directory because the inode in readdir was wrong,
though a Linux-specific implementation could use name_to_handle_at.

Letting the command know what directory it is in makes it unnecessary
for it to call the faulty getcwd in the first place.

247686dc

by richardmaw-codethink at 2018-09-10T14:47:48Z

Merge branch 'richardmaw/subprocess-PWD' into 'master'

Ensure PWD is set in process environment

See merge request BuildStream/buildstream!782

ef66daf5

by Tiago Gomes at 2018-09-10T14:57:41Z

tests: avoid setting max-jobs

Setting "max-jobs" won't be allowed anymore in a following commit.

a37bc6ce

by Tiago Gomes at 2018-09-10T14:57:41Z

docs: document protected variables

And remove then from the defaults as they are dynamically set by
BuildStream.

af74a3f8

by Tiago Gomes at 2018-09-10T14:57:41Z

element: validate configuration variables

Ensure that protected variables are not being redefined by the user.

75c55633

by Tiago Gomes at 2018-09-10T14:57:41Z

Add tests for validating configuration variables

e0bb71b2

by Tiago Gomes at 2018-09-10T15:27:53Z

Merge branch 'tiagogomes/issue-287' into 'master'

Add validation of configuration variables

See merge request BuildStream/buildstream!678

47f3064a

by Jürg Billeter at 2018-09-10T15:43:25Z

_artifactcache/casserver.py: Implement Capabilities service

1a7fb3cb

by Jürg Billeter at 2018-09-10T15:43:25Z

_artifactcache/casserver.py: Implement BatchReadBlobs

Fixes #632.

b3ffcdc8

by Jürg Billeter at 2018-09-10T16:07:30Z

Merge branch 'juerg/cas-batch' into 'master'

_artifactcache/casserver.py: Implement BatchReadBlobs

Closes #632

See merge request BuildStream/buildstream!785

6f925bcb

by Javier Jardón at 2018-09-13T07:38:48Z

source/install_source.rst: pip plugin depends on host pip

c6155f8d

by Javier Jardón at 2018-09-13T08:11:54Z

Merge branch 'jjardon/pip_dependency' into 'master'

source/install_source.rst: pip plugin depends on host pip

See merge request BuildStream/buildstream!791

19838a07

by Chandan Singh at 2018-09-13T10:58:38Z

README.rst: Add status badges for PyPI release and Python versions

The first badge will work fine right away while the second badge will
show "not found" until a release is made after merging this branch:
https://gitlab.com/BuildStream/buildstream/merge_requests/718.

3b81d451

by Chandan Singh at 2018-09-13T12:27:59Z

Merge branch 'chandan/pypi-badge' into 'master'

README.rst: Add status badges for PyPI release and Python versions

See merge request BuildStream/buildstream!719

55956762

by Richard Maw at 2018-09-13T16:50:33Z

element: Remove __cached field

This can get out of sync with other two cache states,
and we can do without it.

fc7f83ac

by richardmaw-codethink at 2018-09-13T17:14:31Z

Merge branch 'richardmaw/element-cache-state-simplify' into 'master'

Simplify element state by removing `__cached`

See merge request BuildStream/buildstream!784

233a7d83

by Richard Maw at 2018-09-14T08:53:14Z

Handle sockets when copying trees into artifacts

We can't include a socket in a CAS tree, but it's mostly meaningless to do so
since there can't possibly be a process serving it.

f06f234a
by Richard Maw at 2018-09-14T08:53:14Z
```
tests: cover builds that make sockets
```

f86ab8f6

by richardmaw-codethink at 2018-09-14T09:46:07Z

Merge branch 'richardmaw/builddir-sockets' into 'master'

Fix: While caching build artifact: "Cannot extract [path to socket file] into staging-area. Unsupported type."

See merge request BuildStream/buildstream!783

e7427462

by Richard Maw at 2018-09-14T10:28:17Z

tests: Use cli.configure for custom logging test

Overriding the config with a custom config file on the command-line
prevents it merging in the test-specific configuration
and can permit it to attempt to initialise the user's cache.

800a8403

by Richard Maw at 2018-09-14T10:28:17Z

Tidy relative workspace tests

They were moving the whole tmpdir to move the project repository.
This moves the cache directories etc. as well, meaning cli.run can't find them.
This was worked around by setting configure=False,
but this has the side-effect of making use of the user's caches,
which it should not be doing for reproducibility reasons.

By changing the tempdir layout to have the project in a subdirectory
we can move the project around for the relative workspace tests
without losing track of the configured state directories,
so we can leave configure=True and avoid touching the user's caches.

d7152ef4

by richardmaw-codethink at 2018-09-14T10:55:16Z

Merge branch 'richardmaw/test-config-fixes' into 'master'

Fix tests that attempt to access the home directory

See merge request BuildStream/buildstream!780

160bb0c6

by Tristan Van Berkom at 2018-09-14T12:07:46Z

_artifactcache/artifactcache.py: Changes to mark_required_elements()

This was previously append_required_artifacts(), which presumed that
we knew at startup time what the cache keys of all elements in the
loaded pipeline would be.

This fixes unexpected deletions of required artifacts when
dynamic tracking is enabled with `bst build --track-all target.bst`

39125d24

by Tristan Van Berkom at 2018-09-14T12:07:46Z

tests/artifactcache/expiry.py: Cleanup of test for required artifacts

This commit renames test_never_delete_dependencies() to
test_never_delete_required(), renders the test more readable by renaming
some elements and reordering some statements and makes the comments more
straight forward and accurate.

f60558a3

by Tristan Van Berkom at 2018-09-14T12:07:46Z

tests/artifactcache/expiry.py: Assert the expected errors

These tests were not checking that we fail for the expected reasons.

Added `res.assert_task_error(ErrorDomain.ARTIFACT, 'cache-too-full')`
where we expect to fail because the cache is too full.

ce68fd27

by Tristan Van Berkom at 2018-09-14T12:07:46Z

testutils/repo/git.py: Added modify_file() method

This allows one to modify a file in an existing git repo,
as opposed to adding a new one.

532ec1eb

by Tristan Van Berkom at 2018-09-14T12:07:46Z

testutils/element_generators.py: Changing sized element functions

 * create_element_size()

   Now uses a git Repo object instead of a local source, and
   returns the repo.

 * update_element_size()

   Added this function which can now resize the expected output
   of an element generated with create_element_size(), useful
   to allow testing sized elements with the tracking feature.

20b797cb

by Tristan Van Berkom at 2018-09-14T12:07:46Z

tests/artifactcache/expiry.py: Added test_never_delete_required_track()

Same test as test_never_delete_required(), except that this test ensures
that we never delete required artifacts when their cache keys are
discovered dynamically during the build.

c2af0d51

by Tristan Van Berkom at 2018-09-14T12:44:42Z

Merge branch 'tristan/fix-required-artifacts' into 'master'

Don't delete required artifacts when tracking is enabled

See merge request BuildStream/buildstream!793

924cdc75
by Tiago Gomes at 2018-09-14T15:32:01Z
```
core: remove unused variable on Context
```

510ccbfd

by Tiago Gomes at 2018-09-14T15:32:21Z

artifactcache: fix oversight

We want to check if some file is already cached here, not the parent
directory.

82d4e2ac

by Tiago Gomes at 2018-09-14T15:32:21Z

scheduler: add missing copyright

Tristan Maat created the original file, so he is added as the author.

18b37aab

by Tiago Gomes at 2018-09-14T15:34:10Z

artifactcache: improve _create_tree()

* Rename it to _commit_directory() because… it is what it does; and
  also for symmetry with _fetch_directory().
* Rename digest to dir_digest to make it clear this is a digest for a
  directory. A following commit will also reuse the same variable name
* Document method.

32fad24f

by Tiago Gomes at 2018-09-14T15:38:41Z

artifactcache: improve _fetch_directory()

* Rename tree to dir_digest to make it clear this is a Digest object,
  and not a Tree object.
* Add documentation

46cbd889
by Tiago Gomes at 2018-09-14T15:38:43Z
```
artifactcache: rename get_quota_exceeded()
```

2fa92716

by Tiago Gomes at 2018-09-14T15:38:43Z

scheduler: minor cleanups to CleanupJob

Remove unneeded cruft.

c285f244

by Tiago Gomes at 2018-09-14T16:08:07Z

Merge branch 'tiagogomes/some-cleanups' into 'master'

Bunch of cleanups

See merge request BuildStream/buildstream!798

662d22a9

by Tristan Van Berkom at 2018-09-16T08:54:00Z

buildstream/_frontend/complete.py: Added missing click license compliance

Somehow I missed this when originally forking the file from the click
library, now noticing that we should have followed what was written
in: https://github.com/pallets/click/blob/master/LICENSE

a344407b

by Richard Maw at 2018-09-16T10:22:59Z

sandbox: deduplicate default environment logic

c8fab440

by Richard Maw at 2018-09-16T10:22:59Z

Bump Artifact Version

Since we now set PWD in the environment of builds
existing builds may behave differently so must cache differently now.

eb9481da

by Richard Maw at 2018-09-16T10:22:59Z

Cachekey update helper: Document ensuring . in PYTHONPATH

The current directory isn't always in the python module search path,
so we have to ensure it is for the script to work.

Strictly speaking, the user may already have a modified PYTHONPATH
at which point PYTHONPATH=".${PYTHONPATH+:$PYTHONPATH}" is necessary,
but it's probably premature to overcomplicate the documentation like that
before we discover it's a problem.

9b7b2a25

by Tristan Van Berkom at 2018-09-16T10:50:36Z

Merge branch 'richardmaw/subprocess-PWD' into 'master'

Address post-merge review of Ensure PWD is set in process environment

See merge request BuildStream/buildstream!788

8db62230
by Adam Jones at 2018-09-17T07:39:06Z
```
Add a section on how to raise an issue
```
6f0a3e98
by Adam Jones at 2018-09-17T07:39:20Z
```
Rename HACKING.rst to CONTRIBUTING.rst
```
ade19151
by Adam Jones at 2018-09-17T07:40:45Z
```
update HACKING references
```

b9c5ff7d

by Tristan Van Berkom at 2018-09-17T08:09:12Z

Merge branch 'tristan/contributing-guide' into 'master'

Update contributing guide

See merge request BuildStream/buildstream!801

727f2faa

by Tristan Van Berkom at 2018-09-18T07:43:07Z

_project.py: Fix option resolution in element & source overrides

This ensures that option expressions are resolved in the project
level overrides before attempting to composite them on the instantiated
elements. Seems this is a regression from introducing the include
directive.

This fixes issue #658

ffa0bb36

by Tristan Van Berkom at 2018-09-18T07:47:44Z

tests/format/optionoverrides.py: Added test for options in element overrides

This is a regression test for issue #658

f2ae46f8

by Tristan Van Berkom at 2018-09-18T08:14:23Z

Merge branch 'tristan/fix-override-options' into 'master'

Fix override options

Closes #658

See merge request BuildStream/buildstream!802

7b117e40

by Daniel Silverstone at 2018-09-18T08:41:32Z

_artifactcache/artifactcache.py: Ensure no double-setup of remotes

Since ArtifactCache.setup_remotes() can be expensive and should only
happen once, this commit will assert() if it is called a second time
on an artifact cache instance.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

345f5f49

by Daniel Silverstone at 2018-09-18T08:41:32Z

tests/artifactcache/pull.py: Do not double-initialize remotes

The initialization of remotes is done by ArtifactCache.setup_remotes()
and as such it was wrong for these tests to be calling
CASCache.initialize_remotes() a second time.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

e32221b6

by Daniel Silverstone at 2018-09-18T08:45:50Z

sandbox/_sandboxremote.py: Acquire artifact cache via Platform

The SandboxRemote used to construct its own CASCache which was
considered dangerous.  This patch replaces that with acquisition of
the cache via the Platform singleton, hopefully eliminating issues
from having more than one artifact cache object in a single process.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

b587579f

by Tristan Van Berkom at 2018-09-18T09:53:26Z

Merge branch 'danielsilverstone-ct/fix-654' into 'master'

sandbox/_sandboxremote.py: Acquire cache via Platform

See merge request BuildStream/buildstream!797

30b41959

by Tristan Van Berkom at 2018-09-18T09:56:45Z

_artifactcache/artifactcache.py: Error out gracefully when push remote is mal-specified

When configuring a push remote and specifying either the client-cert
or the client-key, then both must be specified. This ensures we
get an informative error instead of a stack trace and BUG.

Fixes issue #625

41e8dc81

by Tristan Van Berkom at 2018-09-18T09:56:45Z

tests/artifactcache/config.py: Added test for invalid push remote configuration

Test that we get the expected error when configuring a client-cert
without client-key, or the inverse.

97071b6e

by Tristan Van Berkom at 2018-09-18T10:16:43Z

Merge branch 'tristan/fix-artifact-config-crash' into 'master'

Fix artifact config crash

Closes #625

See merge request BuildStream/buildstream!804

081dcafa

by Richard Maw at 2018-09-18T13:22:38Z

fuse: Report the correct device number for devices

This fixes all devices being mapped to the non-existant device 0,
which prevents being able to use even safe devices like /dev/null
through the hardlinks FUSE layer.

d0425608

by Richard Maw at 2018-09-18T13:22:38Z

FUSE: Mount with -odev in chroot sandbox

This is needed to permit access to the device nodes added to /dev
on Linux when FUSE is used as root.

The chroot sandbox only works with all privileges,
so there's no explicit check for being root
or having the appropriate capabilities.

A check for whether it's running as root isn't needed on Linux with bubblewrap
because /dev or its devices are mounted on top of the FUSE layer,
so device nodes are accessed directly rather than through the FUSE layer.

8430fdc7

by Richard Maw at 2018-09-18T13:22:38Z

tests: test that integration commands can use /dev

d8450166

by Tristan Van Berkom at 2018-09-18T13:50:15Z

Merge branch 'richardmaw/fix-chroot-sandbox-devices' into 'master'

fix chroot sandbox devices

See merge request BuildStream/buildstream!781

ba4da3a4

by knownexus at 2018-09-18T16:45:09Z

Incorrect error when malformed project.conf

bst build returns "missing project.conf" when a project.conf is invalid
This results in an existing project with malformed yaml being dismissed
and attempts to make a new project

Added new exception

89a16e17
by knownexus at 2018-09-18T16:45:09Z
```
Adding test for Invalid Yaml
```

72b59021

by Phillip Smyth at 2018-09-19T08:48:51Z

Merge branch 'issue-642-Invalid_project.conf_seen_as_missing' into 'master'

Incorrect error when malformed project.conf

Closes #642

See merge request BuildStream/buildstream!792

ca811a4d

by Josh Smith at 2018-09-19T09:25:21Z

Rework Skipped usage

The SKIPPED message type is now used to indicate the end of a task which
was successful without having to perform the given task.

This overhauls the use of `Queue.done()` and therefore queues do not
need to provide a processed/skipped return value from `done()`. Instead
this is replaced with the action of raising a `SkipJob` exception from
within `Queue.process()`.

16fbc3e3

by Josh Smith at 2018-09-19T09:25:21Z

tests.py: Test skip on push

Adds a test to ensure that BuildStream alerts the user of a skipped push
when the remote already has the artifact cached.

ab56f693

by Josh Smith at 2018-09-19T09:25:21Z

element.py: Remove redundant timed_activity

This removes the timed_activity for an element _push action. This is
unnecessary as the job is already being timed elsewhere.

198fbfe8

by Josh Smith at 2018-09-19T09:25:21Z

cascache.py: Move push/pull messaging to cascache

Pulled/Pushed messages will no longer be produced from within
element.py, instead they will be produced during CasCache push() and
pull() appropriately.

Message consistency has also been improved.

213f7794

by Josh Smith at 2018-09-19T09:25:21Z

cascache.py: Modify messaging API calls

Both pulling and pushing INFO messages are now status messages.

Calls to the messaging API through `self.context.message()` have now
been switched to `element.info`.

dddd6025

by Qinusty at 2018-09-19T10:59:48Z

Merge branch 'Qinusty/skipped-rework' into 'master'

Add SkipJob for indicating a skipped activity

See merge request BuildStream/buildstream!765

16d9c6e5

by Tristan Van Berkom at 2018-09-19T11:08:33Z

_frontend/status.py: Completely remove the blessings dependency from BuildStream

This actually improves reliability of the status bar because we
now disable it completely in the case that not all of the terminal
escape sequences are supported on the given terminal.

This replaces the few functions we were using, to move the cursor
up one line, move it to the beginning of the line, and to clear a
line, with low level functions provided by the curses module in
the standard library.

This change makes it easier for downstream distro package maintainers
to package BuildStream, particularly on Fedora.

Asides from changing _frontend/status.py, this commit includes the
following changes:

  * _frontend/app.py: Use python isatty() function to determine if
                      we are connected to a tty, instead of relying
		      on blessings.

  * setup.py: Remove the dependency on blessings.

7dd74b01

by Tristan Van Berkom at 2018-09-19T11:27:55Z

Merge branch 'tristan/remove-blessings' into 'master'

_frontend/status.py: Completely remove the blessings dependency from BuildStream

See merge request BuildStream/buildstream!808

335d5a45
by Adam Jones at 2018-09-19T13:00:45Z
```
Fix rst links for guideline links
```

667dd4ca

by Javier Jardón at 2018-09-19T13:25:36Z

Merge branch 'adamjones/contributing-links' into 'master'

Fix rst link formatting for guideline links

See merge request BuildStream/buildstream!811

68ef69e4

by Tristan Van Berkom at 2018-09-21T05:20:46Z

source.py: Fix re-instantiation

This fixes an issue where the re-instantiated Source used with
Source mirroring enabled is not completely initialized.

Failing to load the ref from the project.refs file for instance,
will result in a crash at `fetch` time.

This fixes issue #666

662c729f

by Tristan Van Berkom at 2018-09-21T05:59:30Z

Merge branch 'tristan/fix-source-reinstantiation' into 'master'

source.py: Fix re-instantiation

Closes #666

See merge request BuildStream/buildstream!817

461a0588

by Jim MacArthur at 2018-09-21T10:53:11Z

element.py: Remove early warning and check directory is not None

The check for directory should have always been here. The early warning
test doesn't work for `bst checkout` because caches haven't been configured
at the point of the test, so I've had to remove it.

aa9caaac

by Jim MacArthur at 2018-09-21T10:53:11Z

_sandboxremote.py: Ignore return value of push_directory

The return value is always upload_vdir.ref.

2aae68c7

by Jim MacArthur at 2018-09-21T10:53:11Z

cascache.py: make push_remote raise exception if no push remotes exist

Also add docstring to method and remove return value since it was useless

ca1bb72c

by Jim MacArthur at 2018-09-21T10:53:11Z

tests/artifactcache/push.py: Ignore return value of push_directory

55c93a82

by Jim MacArthur at 2018-09-21T11:26:55Z

Merge branch 'jmac/remote_exec_checkout_fix' into 'master'

Remote exec: Remove early warning and check directory is not None

See merge request BuildStream/buildstream!800

e209beb0

by Chandan Singh at 2018-09-21T13:10:08Z

_stream.py: Ensure source-bundle's source directory exists

Currently, `source-bundle` command is entirely broken as it tries to stage the
sources in a directory that doesn't exist. Fix it by ensuring that we create
the necessary directories before calling any methods that try to use those
directories.

This fix comes with a regression test to ensure that the basic use-case
of `source-bundle` continues to work in future.

Fixes https://gitlab.com/BuildStream/buildstream/issues/651.

0b000518

by Chandan Singh at 2018-09-21T13:56:55Z

Merge branch 'chandan/fix-source-bundle' into 'master'

Fix source-bundle command

Closes #651

See merge request BuildStream/buildstream!807

ef26043a
by Chandan Singh at 2018-09-21T17:14:16Z
```
setup.py: Make website the primary homepage
```

1b2aed40

by Chandan Singh at 2018-09-21T17:40:11Z

Merge branch 'chandan/update-project-homepage' into 'master'

setup.py: Make website the primary homepage

See merge request BuildStream/buildstream!795

da7e038b

by Jürg Billeter at 2018-09-24T13:58:55Z

element.py: Fix cache check in non-strict mode

The behavior of __assert_cached() was changed when the keystrength
parameter was introduced. This restores the previous behavior.

Fixes #607.

Fixes: 3d308894 ("element.py: Cache the result of checking whether...")

86ea1173

by Jürg Billeter at 2018-09-24T13:59:59Z

tests/frontend/rebuild.py: Add strict/non-strict rebuild test

Test rebuild that requires staging of weakly cached dependency.

This is a regression test for #607.

a76339de

by Jürg Billeter at 2018-09-24T16:02:38Z

Merge branch 'juerg/rebuild' into 'master'

element.py: Fix cache check in non-strict mode

Closes #607

See merge request BuildStream/buildstream!822

b199afe6

by Jürg Billeter at 2018-09-25T09:01:51Z

_artifactcache/cascache.py: Add _ensure_blob helper

This adds directory objects to the local repository before downloading
files in the directory. However, artifact references are still stored
only after downloading the complete directory and thus, there won't be
dangling references. This will anyway be required for partial download
support.

7d199322

by Jürg Billeter at 2018-09-25T09:01:51Z

_artifactcache/cascache.py: Increase payload size limit for uploads

gRPC can handle 1 MiB payloads. Increase size limit from 64 KiB to speed
up uploads.`

e2e24015

by Jürg Billeter at 2018-09-25T09:01:51Z

_artifactcache/casserver.py: Harmonize payload size limit

Use 1 MiB as payload size limit on the server side for both individual
downloads and batch uploads.

697d10f2

by Jürg Billeter at 2018-09-25T09:01:51Z

_artifactcache/cascache.py: Use BatchReadBlobs

This uses BatchReadBlobs instead of individual blob download to speed up
artifact pulling, if the server supports it.

Fixes #554.

81c51dbf

by Jürg Billeter at 2018-09-25T09:31:55Z

Merge branch 'juerg/cas-batch' into 'master'

_artifactcache/cascache.py: Use BatchReadBlobs

Closes #554

See merge request BuildStream/buildstream!813

2df7d140

by Jim MacArthur at 2018-09-25T10:36:37Z

sandbox.py: Stop caching get_virtual_directory if get_directory is used

Also update the documentation for get_virtual_directory to make the
implications of using both methods clearer.

62f59382

by Jim MacArthur at 2018-09-25T10:36:37Z

element.py/_assemble(): Call get_virtual_directory later.

8cea7b17

by Jim MacArthur at 2018-09-25T10:58:40Z

Merge branch 'jmac/stop-caching-vdirs' into 'master'

Stop caching virtual directories if get_directory is used.

See merge request BuildStream/buildstream!818

83ab183e

by Tiago Gomes at 2018-09-25T16:37:30Z

CI: test building freedesktop-sdk overnight

Closes #514.

c8594806

by Tiago Gomes at 2018-09-25T17:03:55Z

Merge branch 'tiagogomes/issue-514' into 'master'

CI: test building freedesktop-sdk overnight

Closes #514

See merge request BuildStream/buildstream!826

fdbf9be1

by Tiago Gomes at 2018-09-26T08:17:46Z

ci: update freedesktop-sdk ref

The current one doesn't build due server being down.

9280b0cf

by Tiago Gomes at 2018-09-26T15:54:53Z

ci: add ACME challenge

To finish the creation of the Let's Encrypt certificate, which will
allows us to use https://docs.buildstream.build/ for accessing the
documentation.

af131503

by Tiago Gomes at 2018-09-26T16:35:58Z

Merge branch 'tiagogomes/acme-challenge' into 'master'

ci: add ACME challenge

See merge request BuildStream/buildstream!831

f7f14f25
by Tiago Gomes at 2018-09-26T17:42:12Z
```
ci: add missing `-p` argument
```

0f2bc375

by Martin Blanchard at 2018-09-27T08:46:27Z

_protos: Import google.rpc.Code proto

Imports code.proto from https://github.com/googleapis/googleapis and
corresponding modules generated with ./setup.py build_grpc.

https://gitlab.com/BuildStream/buildstream/issues/668

16cf9d5f

by Martin Blanchard at 2018-09-27T08:46:27Z

_sandboxremote.py: Handle remote build execution exit code

https://gitlab.com/BuildStream/buildstream/issues/668

f5f3cb7c

by Jim MacArthur at 2018-09-27T09:16:38Z

Merge branch 'mablanch/668-remote-build-failure' into 'master'

Better handle remote build failures

See merge request BuildStream/buildstream!825

b99a6740

by Jürg Billeter at 2018-09-27T09:18:03Z

element.py: Add __use_remote_execution() helper method

f8bbe008
by Jürg Billeter at 2018-09-27T09:18:03Z
```
_platform: Add check_sandbox_config() method
```
f9494f1f
by Jürg Billeter at 2018-09-27T09:18:03Z
```
element.py: Validate sandbox config
```

b4ad84c3

by Jürg Billeter at 2018-09-27T09:18:03Z

_platform/linux.py: Remove redundant message

Unsupported sandbox config (UID/GID) is now reported by the element.

966af3d2

by Jürg Billeter at 2018-09-27T09:18:03Z

_platform/linux.py: Do not disable push globally

On systems without user namespace support, elements with unsupported
sandbox config (UID/GID) are now individually tainted, which disables
artifact push.

f08e5eae

by Jürg Billeter at 2018-09-27T09:18:03Z

_artifactcache/cascache.py: Remove unused enable_push parameter

10ed9158
by Jürg Billeter at 2018-09-27T09:18:03Z
```
_artifactcache: Add preflight() method
```

7aec8b0f

by Jürg Billeter at 2018-09-27T09:18:03Z

Move artifact cache creation to Context

The artifact cache is no longer platform-specific.

a5025c33
by Jürg Billeter at 2018-09-27T09:18:03Z
```
_loader: Migrate to Context.artifactcache
```
ef1cb374
by Jürg Billeter at 2018-09-27T09:18:03Z
```
_scheduler: Migrate to Context.artifactcache
```
52991be1
by Jürg Billeter at 2018-09-27T09:18:03Z
```
_stream.py: Migrate to Context.artifactcache
```

cb6b35dc

by Jürg Billeter at 2018-09-27T09:18:42Z

sandbox/_sandboxremote.py: Migrate to Context.artifactcache

28c62b00

by Jürg Billeter at 2018-09-27T09:18:43Z

_platform/platform.py: Remove unused artifactcache property

99e90918

by Jürg Billeter at 2018-09-27T09:19:12Z

_artifactcache/casserver.py: Use Context for artifact cache creation

b8f920e2

by Jürg Billeter at 2018-09-27T09:19:12Z

storage/_casbaseddirectory.py: Use Context for artifact cache creation

3d88c5d4

by Jürg Billeter at 2018-09-27T09:19:12Z

tests/testutils: Use Context for artifact cache creation

ed878cb6

by Jürg Billeter at 2018-09-27T09:19:12Z

tests/artifactcache/pull.py: Use Context for artifact cache creation

fe82c9c2

by Jürg Billeter at 2018-09-27T09:19:12Z

tests/artifactcache/push.py: Use Context for artifact cache creation

0d0f700d
by Jürg Billeter at 2018-09-27T09:19:12Z
```
_platform: Remove unused context
```

dd770ec3

by Jürg Billeter at 2018-09-27T09:19:12Z

Use lazy platform instantiation

Now that the platform is independent of the context, explicit
instantiation is no longer required. This avoids issues with platform
instances used across test cases with mismatching context.

44da8175

by Jürg Billeter at 2018-09-27T09:48:20Z

Merge branch 'juerg/platform' into 'master'

Make platform independent of context

See merge request BuildStream/buildstream!829

900e8900

by Jürg Billeter at 2018-09-27T13:12:34Z

git.py: Make `ref` human readable

Use `git describe` to prefix `ref` with closest tag, if available.

107269c1
by Jürg Billeter at 2018-09-27T13:12:34Z
```
testutils/repo/git.py: Add add_tag() method
```

8060ad8c

by Jürg Billeter at 2018-09-27T13:12:34Z

tests/sources/git.py: Add track and fetch test with and without tag

ab1cb672

by Jürg Billeter at 2018-09-27T13:46:41Z

Merge branch 'juerg/git-describe' into 'master'

git.py: Make `ref` human readable

See merge request BuildStream/buildstream!291

fcf37f9c

by knownexus at 2018-09-27T14:22:09Z

Added `set_resource_limits()` to platform

This has been moved from app.py
As it will have different functionality depending on platform
Once the Darwin (MacOS) platform is added

Removed `resource.setrlimit()` functionality from app.py
Added `resource.setrlimit()` functionality to platform.py as function

80ebdd2c

by knownexus at 2018-09-27T14:22:09Z

Added FUSE check, as WSL does not support FUSE yet

_platform/linux.py: Add FUSE check
And Ensure exception is not raised when performing sandbox availability check
Without this check, WSL would error upon finding FUSE missing
This must be bypassed to allow remote execution

5e81573b

by knownexus at 2018-09-27T14:22:09Z

Max-jobs is now defined on a platform basis

This is due to Darwin (MacOS) having a different Max-Jobs limit
_platform/platform.py: Adding max-jobs call

_project.py: redirected 'max-jobs' to point at the platform

3c93fe97

by knownexus at 2018-09-27T14:22:09Z

Adding Dummy sandbox

This is to allow platforms that do not support sandboxing
To error in a controlled/known way

0f3ef369

by knownexus at 2018-09-27T14:22:24Z

Adding darwin.py (MacOS) platform

Adding functionality to recognise Darwin as a platform in plaform.py

b78ae767

by knownexus at 2018-09-27T14:29:29Z

utils.py: Reworked safe_remove

Non-Linux platforms don't return EISDIR when attempting to unlink a
directory.

Stopped safe_remove attempting to unlink dirs
Previously safe_remove would attempt to unlink a path
Before attempting to remove it if it was a dir

Now it checks for a dir before that step

ab5e78b4

by James Ennis at 2018-09-27T14:29:32Z

Import sandbox subclass when instantiating sandbox

This patch was required due to not being able to execute `bst` on
a Linux machine without FUSE available.

261f65ca

by Jürg Billeter at 2018-09-27T14:54:47Z

Merge branch 'mac_fixes' into 'master'

Implement compatibility fixes for MacOSX and WSL Blocks #411 and #412"

See merge request BuildStream/buildstream!726

d2ea3069

by Richard Maw at 2018-09-27T15:16:57Z

tests: Add regression test for pushing failed builds

This test ensures that failed builds are pushed when set to on-error=quit.

Closes https://gitlab.com/BuildStream/buildstream/issues/534

c128429a

by richardmaw-codethink at 2018-09-27T15:42:18Z

Merge branch 'richardmaw/push-failed-build-regression' into 'master'

tests: Add regression test for pushing failed builds

Closes #534

See merge request BuildStream/buildstream!824

e91eb38b

by Valentin David at 2018-09-27T16:32:57Z

Fix outside-of-project check when project path is not canonical.

The issue happens on Silverblue where /home is a symlink to /var/home.
With this element-path is something like
/var/home/user/project/elements, when the project path is
/home/usr/project. Comparing canonical paths solves the issue.

Fixes #673

7babd55c

by Tiago Gomes at 2018-09-27T16:57:12Z

Merge branch 'valentindavid/silverblue-path' into 'master'

Fix outside-of-project check when project path is not canonical.

Closes #673

See merge request BuildStream/buildstream!832

5cdfac81

by Mathieu Bridon at 2018-09-28T09:20:20Z

data/userconfig.yaml: Document the cache.quota setting

5d3f039f

by Tom Pollard at 2018-09-28T10:34:13Z

Merge branch 'bochecha/cache-size' into 'master'

data/userconfig.yaml: Document the cache.quota setting

See merge request BuildStream/buildstream!833

14e1a3b3

by Jürg Billeter at 2018-10-01T08:18:05Z

_platform: Fix get_cpu_count() with cap=None

Comparison between None and integer is not supported.

232662f1

by Jürg Billeter at 2018-10-01T08:53:06Z

Merge branch 'juerg/cpucount' into 'master'

_platform: Fix get_cpu_count() with cap=None

See merge request BuildStream/buildstream!840

f447aedd
by Tiago Gomes at 2018-10-01T10:33:11Z
```
ci: upper case variables

For consistency.
```
682dddce
by Tiago Gomes at 2018-10-01T10:35:12Z
```
ci: correct freedestop-sdk version comment
```

fafa8136

by Tiago Gomes at 2018-10-01T10:59:54Z

Merge branch 'tiagogomes/ci-minor-tweaks' into 'master'

Minor CI tweaks

See merge request BuildStream/buildstream!842

26e1a3c7

by Jürg Billeter at 2018-10-01T14:58:06Z

_artifactcache/casserver.py: Implement BatchUpdateBlobs

Fixes #676.

f47895c0

by Jürg Billeter at 2018-10-01T14:58:06Z

_artifactcache/cascache.py: Use BatchUpdateBlobs

This uses BatchUpdateBlobs instead of individual blob upload to speed up
artifact pushing, if the server supports it.

Fixes #677.

cf00c0a1

by Jürg Billeter at 2018-10-01T15:32:30Z

Merge branch 'juerg/cas-batch' into 'master'

CAS: Implement BatchUpdateBlobs support

Closes #677 and #676

See merge request BuildStream/buildstream!839

5f4ae90b

by Jürg Billeter at 2018-10-02T06:34:02Z

sandbox/_sandboxremote.py: Use helpers to set up cwd and env

This matches the other sandbox implementations.

0458bc4e

by Jürg Billeter at 2018-10-02T07:08:35Z

Merge branch 'juerg/sandbox' into 'master'

sandbox/_sandboxremote.py: Use helpers to set up cwd and env

See merge request BuildStream/buildstream!841

d5b396e0

by Phillip Smyth at 2018-10-02T16:40:18Z

Forcing CI to install Sphinx 1.7.9

.gitlab-ci.yml: forced pip3 install Sphinx 1.7.9
The theme we use in our docs "sphinx_rtd_theme"
Does not support the latest version of Sphinx (>1.8)
This results in search functionality failing

dae842fd

by Phillip Smyth at 2018-10-02T17:14:09Z

Merge branch 'docs_Search_not_working' into 'master'

Fixing: Search functionality in the Documentation has stopped working

Closes #681

See merge request BuildStream/buildstream!848

8dc16d3f

by Jürg Billeter at 2018-10-03T05:08:21Z

_platform/linux.py: Accept all configs for dummy sandbox

If Sandbox.run() is never called, sandbox config is irrelevant. If it is
called, the build will fail anyway with the dummy sandbox.

This matches the logic in the Darwin platform.

66446fc3

by Jürg Billeter at 2018-10-03T05:36:38Z

Merge branch 'juerg/dummy-sandbox' into 'master'

_platform/linux.py: Accept all configs for dummy sandbox

See merge request BuildStream/buildstream!843

29c19bea

by Tristan Van Berkom at 2018-10-03T07:33:48Z

git.py: Added timed activity around fetching, displaying the URL being fetched.

This will take care of silencing the status messages while checking submodules.

b645881c

by Tristan Van Berkom at 2018-10-03T07:33:48Z

source.py: Silence messages while consuming source fetchers

The source fetchers might be a list or a generator, when it is
a generator (like the git source does), then we want to ensure that
we silence the status messages which might occur as a result of
consuming a source fetcher from the generator.

This fixes the logs to be less verbose.

c9437616

by Tristan Van Berkom at 2018-10-03T08:07:15Z

Merge branch 'tristan/fix-status-messages' into 'master'

fix status messages

See merge request BuildStream/buildstream!845

11320fe2

by Tristan Van Berkom at 2018-10-03T09:33:39Z

_scheduler: Fix bookkeeping of terminated jobs

 * Enhanced the base Job class to bookkeep which jobs have
   been terminated, and to consider them as `skipped` when
   asked via the `skipped` property.

 * Enhanced the base Queue class to bookkeep the job statuses
   more carefully.

This fixes #479

91271964

by Tristan Van Berkom at 2018-10-03T09:59:40Z

Merge branch 'tristan/fix-terminated-jobs' into 'master'

_scheduler: Fix bookkeeping of terminated jobs

Closes #479

See merge request BuildStream/buildstream!850

3bf895d2

by Jonathan Maw at 2018-10-03T11:48:48Z

tests: Add regression test for mirroring with project.refs

e4969807

by Jonathan Maw at 2018-10-03T12:48:07Z

Merge branch 'jonathan/source-mirror-project-refs' into 'master'

tests: Add regression test for mirroring with project.refs

See merge request BuildStream/buildstream!823

a0814399

by Tristan Van Berkom at 2018-10-03T13:05:52Z

_scheduler/scheduler.py: Ignore interrupt events while terminating.

For some reason, we now receive a SIGINT from the main loop even
when the SIGINT occurred with the handler disconnected in an interactive
prompt. This patch simply ignores any received SIGINT events from the
main loop in the case that we are already in the process of terminating.

This fixes issue #693

0a1f8e3c

by Tristan Van Berkom at 2018-10-03T13:42:20Z

Merge branch 'tristan/fix-double-terminate-prompt' into 'master'

_scheduler/scheduler.py: Ignore interrupt events while terminating.

Closes #693

See merge request BuildStream/buildstream!852

11161f99

by Tristan Van Berkom at 2018-10-03T13:44:02Z

utils.py: Give save_file_atomic() a tempdir argument

Allow callers to decide where the temporary file will be created.

3e797bb9

by Tristan Van Berkom at 2018-10-03T13:44:02Z

_artifactcache/cascache.py: Don't create temporary files in the CAS storage

Use the designated tempdir when creating refs, we expect that temporary
files are not created in the storage directory ever, they should be
only ever created in the designated temporary directory.

This fixes race conditions where utils._get_dir_size() throws an
unhandled exception when attempting to stat the file which inadvertantly
disappears.

d9020e43

by Tristan Van Berkom at 2018-10-03T13:44:02Z

utils.py: Document _get_dir_size() expectations.

This function assumes that files do not disappear while
walking the given directory.

3e5ff5a9

by Tristan Van Berkom at 2018-10-03T14:09:51Z

Merge branch 'tristan/fix-cache-size-race' into 'master'

fix cache size race

See merge request BuildStream/buildstream!854

e8179c34

by Jim MacArthur at 2018-10-03T16:06:59Z

element.py: Put the temporary staging directory in artifactdir

3cf38c8e

by Jim MacArthur at 2018-10-03T16:44:02Z

Merge branch 'jmac/temporaries-inside-cachedir' into 'master'

Move the temporary staging directory to artifactdir

See merge request BuildStream/buildstream!856

59c92bda

by Daniel Silverstone at 2018-10-04T08:12:20Z

sandbox/_sandboxdummy.py: Take a reason for use

Since the SandboxDummy sandbox is only used in very specific
circumstances and with good reason, this adjusts the SandboxDummy
class to take a reason and to return it if `.run()` has to
raise an exception because this is a dummy sandbox.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

b9ddcd0e

by Daniel Silverstone at 2018-10-04T08:12:20Z

_platform/darwin.py: Give reason for use of dummy sandbox

Since Darwin is limited to the dummy sandbox for now due to OSXFUSE
being unsupported as yet, and there being no suitable sandboxing option
for use on OSX, give this as the reason so that it can be reported if
the user attempts a local build.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

df0d5a8b

by Daniel Silverstone at 2018-10-04T09:04:21Z

_platform/linux.py: Refactor checks for sandboxing

To better report issues in the absence of a suitable bwrap, or the
FUSE devices, this refactors the checks for sandboxing in the Linux
platform to cover the various possibilities.  The reasons are then
collated and passed to the dummy sandbox for later reporting to the
user if a local build is attempted.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

b8421a9c

by Daniel Silverstone at 2018-10-04T09:04:21Z

_site.py: Protect against failure running `bwrap --version`

On some potentially broken systems, running `bwrap --version` might
fail with an error code.  This patch corrects the oversight ensuring
that we cleanly return False for version checking in such cases.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

c74bfbe5

by Daniel Silverstone at 2018-10-04T09:04:21Z

_site.py: Reduce complexity of bwrap version comparison

Instead of an if/else ladder which is quite complex, this patch
uses the fact that Python supports by-component tuple comparison
to simply compare two (major, minor, patch) tuples

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

c46a7e87

by Daniel Silverstone at 2018-10-04T09:13:04Z

setup.py: Change bwrap assertion to a warning

Since there are use-cases where BuildStream could be installed
onto systems which do not have BubbleWrap (e.g. for remote-build-only
scenarios) it is not correct to assert a dependency on bwrap during
installation.  This patch makes the assertion a warning, and also
clarifies the message somewhat.  This should fix #644

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

fd6a9573

by Jürg Billeter at 2018-10-04T09:45:58Z

Merge branch 'danielsilverstone-ct/bwrap-check-runtime-only' into 'master'

Make bwrap check runtime only

Closes #644

See merge request BuildStream/buildstream!847

c5778941

by Josh Smith at 2018-10-04T13:43:10Z

element.py: Do not save workspace on failed build

This fixes #634, BuildStream no longer saves workspace configuration on
a failed build.

788cde6a

by Josh Smith at 2018-10-04T13:43:10Z

testing: Add regression test for workspace bug

8630bac4

by Tristan Van Berkom at 2018-10-04T14:11:27Z

Merge branch 'Qinusty/634-workspace-failed-builds' into 'master'

Do not save workspace on failed build

Closes #634

See merge request BuildStream/buildstream!812

a7984218

by Valentin David at 2018-10-04T14:56:09Z

Catch correct exception from shutil.rmtree

Python documentation is not clear on what shutil.rmtree is supposed to
raise. However from the source code, it is obvious that OSError are
raised, but never shutil.Error.

It is not easy to test in normal condition because issues happen
usually in combination with a FUSE filesystem, a probably a race
condition where `fstatat` returns an error just before the filesystem
being unmounted.

Fixes #153.

b7b20d9e

by Valentin David at 2018-10-04T15:31:22Z

Merge branch 'valentindavid/rmtree_oserror' into 'master'

Catch correct exception from shutil.rmtree

Closes #153

See merge request BuildStream/buildstream!849

42ea143d

by Chandan Singh at 2018-10-05T12:40:48Z

Add contrib script to generate Docker images from bst checkout

This script can be useful to generate a Docker image from `bst checkout`
of an element as a single command. While this script does not eliminate
the need for a proper Docker/OCI element plugin that would probably also
support layering, it provides a cheap way to export the element into a
Docker image.

b38f9f9d

by Jonathan Maw at 2018-10-05T13:05:38Z

Merge branch 'chandan/bst-docker-import' into 'master'

Add contrib script to generate Docker images from bst checkout

See merge request BuildStream/buildstream!857

47df1e36

by Jürg Billeter at 2018-10-05T13:30:38Z

element.py: Prepare local sandbox for bst checkout and bst shell

A remote execution sandbox is not suitable for these commands.

d3921ab8

by Jonathan Maw at 2018-10-05T13:54:17Z

Merge branch 'juerg/remote' into 'master'

element.py: Prepare local sandbox for bst checkout and bst shell

See merge request BuildStream/buildstream!860

53d9f977

by Tristan Van Berkom at 2018-10-06T14:40:46Z

CONTRIBUTING.rst: Updating CONTRIBUTING guidelines

This is almost a complete rewrite of the CONTRIBUTING guide.

  * The patch submission guidelines have become less ambiguous and more strict

  * Some general restructuring and reordering of the file took place

  * The codeing guidelines have changed completely.

    o There is much less room for ambiguity here now
    o More emphasis on consistency in the codebase
    o Added some more abstract points which should be considered
      when writing and reviewing patches

  * The policy on public/private symbols has been greatly clarified

  * Added new section about adding new core plugins and what needs
    to be done as a consequence of that

5587715c

by Tristan Van Berkom at 2018-10-07T06:19:03Z

CONTRIBUTING.rst: Some fixes in grammer and minor corrections

43c7ac23
by Angelos Evripiotis at 2018-10-07T15:19:58Z
```
contributing.rst: fix typo
```

3f0c919f

by Angelos Evripiotis at 2018-10-07T15:52:52Z

Merge branch 'aevri/fixup-contributing-typos' into 'master'

contributing.rst: fix typos

See merge request BuildStream/buildstream!865

6f5f795e

by Tristan Van Berkom at 2018-10-08T08:46:11Z

CONTRIBUTING.rst: Added section about minimizing API surfaces

And modified some titles so that the titles in CONTRIBUTING.rst actually
follow the documentation guidelines for naming of section titles.

abb9fb98

by Tristan Van Berkom at 2018-10-08T08:59:18Z

CONTRIBUTING.rst: Added section on file naming convention

ba905e3a

by Tristan Van Berkom at 2018-10-08T09:05:18Z

CONTRIBUTING.rst: Correcting `Context` example in the "imports" section

This example is very outdated and comes from pre 1.0 when the Context
object was going to be public API. Just corrected the example made about
imports to import the `Context` object from `._context` instead of `.context`.

674afe94

by Tristan Van Berkom at 2018-10-08T09:07:45Z

CONTRIBUTING.rst: Added a note that public API objects should be exposed from __init__.py

350c6796

by Tristan Van Berkom at 2018-10-08T10:44:47Z

CONTRIBUTING.rst: Added section in PEP-8 coding style about line lengths.

We have a long line limit in order to handle the exceptions without making
code unreadable as a result, this long line length limit is not an invitation
to fill up the limit as much as possible.

3ca487b8

by Chandan Singh at 2018-10-08T18:25:39Z

bst-docker-import: Consistently use stderr for all logs

In !857, we added `contrib/bst-docker-import` script but it has a small
issue that some of logs go to stdout while others go to stderr. Fix it
so that all logging is done on stderr.

49df3d75

by Chandan Singh at 2018-10-08T18:58:54Z

Merge branch 'chandan/fix-bst-docker-import-logging' into 'master'

bst-docker-import: Consistently use stderr for all logs

See merge request BuildStream/buildstream!863

d827dfae
by Angelos Evripiotis at 2018-10-09T09:12:45Z
```
contributing.rst: end lines with punctuation
```

9590e8ae

by Angelos Evripiotis at 2018-10-09T09:12:45Z

contributing.rst: fix whitespace

No tabs, no lines of only horizontal whitespace.

6d02e269
by Angelos Evripiotis at 2018-10-09T09:12:45Z
```
contributing.rst: fix typo of 'get_count'
```

3ed26a47

by Angelos Evripiotis at 2018-10-09T09:12:45Z

contributing.rst: de-paren subclass example

It seems easier to read when separated into two sentences.

5b22d850
by Angelos Evripiotis at 2018-10-09T09:12:45Z
```
contributing.rst: no spaces before '?'
```

ac0776f8

by Tristan Van Berkom at 2018-10-09T09:36:33Z

Merge branch 'aevri/contributing_fixups' into 'master'

Minor fixups to contributing.rst

See merge request BuildStream/buildstream!866

885bd946

by Tristan Van Berkom at 2018-10-09T09:39:24Z

CONTRIBUTING.rst: Added note about sphinx supporting docstrings on instance variables

Python does not natively support this, but sphinx does parse them and includes
these in the generated documentation.

120d8c73

by Tristan Van Berkom at 2018-10-09T09:45:23Z

CONTRIBUTING.rst: Fix typos in previous commit.

Ooops, that was done far too quickly.

cda1354c

by Chandan Singh at 2018-10-10T00:17:51Z

doc: Add new document about Docker integrations

Add a new section under "Additional writings" section about how
BuildStream integrates with Docker. At present, this section includes
only two sections:

- "Run BuildStream inside Docker", that refers to existing instructions
for doing so
- "Generate Docker images", that adds instructions for using the newly
added `contrib/bst-docker-import` script

In future, this document can be extended to include the following
things:

- Importing Docker sources via `docker` source plugin - if/when the
Docker source plguin is moved from bst-external to BuildStream proper
- A Docker sandbox, if/when that is added

dd472d95

by Chandan Singh at 2018-10-10T00:18:17Z

NEWS: Add entry about contrib/bst-docker-import

As suggested in
https://gitlab.com/BuildStream/buildstream/merge_requests/857#note_106809743,
add a NEWS entry for `contrib/bst-docker-import` that was added
recently.

35ac26a7

by Chandan Singh at 2018-10-10T00:59:17Z

Merge branch 'chandan/bst-and-docker' into 'master'

Add documentation and NEWS entry about bst-docker-import

See merge request BuildStream/buildstream!864

7d96333f
by Laurence Urhegyi at 2018-10-10T19:24:14Z
```
README: Update to add link to website
```

39492db8

by devcurmudgeon at 2018-10-10T19:24:14Z

Merge branch 'patch-2' into 'master'

README: Update to add link to website

See merge request BuildStream/buildstream!859

4f0bfb4a

by Tristan Van Berkom at 2018-10-11T15:37:51Z

Rename element_enums.py -> types.py

This will be the place to store low level data types used
throughout the core, for now this includes public and private
types.

9dc10cc7

by Tristan Van Berkom at 2018-10-11T15:45:36Z

doc/source/core_framework.rst: Include the foundation types in the API docs.

efc0d4cc

by Tristan Van Berkom at 2018-10-11T15:48:44Z

types.py: Moved Consistency definition here from source.py

And slightly touch up the doc strings for Consistency and Scope.

a0712ead

by Tristan Van Berkom at 2018-10-11T16:23:54Z

Merge branch 'tristan/refactor-types-api' into 'master'

Refactor types api

See merge request BuildStream/buildstream!870

ce55b9a0

by Tiago Gomes at 2018-10-12T10:15:46Z

doc: updates considering website being live now

* Add a link to the website on the main page.
* Remove install instructions as they are now on the website.
* Remove Resources section as that information can be found at the
  website, and also looks bad.
* Move artifact server setup from the no longer existing Install section
  to the Using section.

10b092e1

by Tristan Van Berkom at 2018-10-12T10:21:27Z

doc/source/additional_docker.rst: Fix link to refer to website

The docker guide, which is part of the install guide, has moved
to the website.

1c7ba9a0

by Tristan Van Berkom at 2018-10-12T10:24:31Z

doc/source/index.rst: Moved references to the website below the simple ToC.

3738dd06

by Tristan Van Berkom at 2018-10-12T10:49:39Z

Merge branch 'tristan/remove-install-guide' into 'master'

Remove install guide

See merge request BuildStream/buildstream!872

c31ed138

by Tristan Van Berkom at 2018-10-14T15:24:53Z

CONTRIBUTING.rst: Added more guidelines about documenting the user guide

b39d87b4

by Jürg Billeter at 2018-10-15T11:07:05Z

_artifactcache: Fix crash in push_directory() without artifact server

Fixes #708.

4e3ec89e

by Jürg Billeter at 2018-10-15T11:48:16Z

Merge branch 'juerg/remote-config' into 'master'

_artifactcache: Fix crash in push_directory() without artifact server

Closes #708

See merge request BuildStream/buildstream!871

d15ed8fc

by Jonathan Maw at 2018-10-15T15:54:59Z

yaml: Add a cache of parsed and provenanced yaml

Note that the ProvenanceFile's names will be incorrect after loading
from the cache, but this is currently only used for writeback, which
isn't used in junctions.

f4eb685a
by Jonathan Maw at 2018-10-15T15:54:59Z
```
yamlcache.py: Add yaml cache tests
```

26dfa882

by Lachlan Mackenzie at 2018-10-15T15:54:59Z

yaml.py: Add loader for yaml test files cache option

* Fix to CWD issue provided by Jonathan Maw

0fbff08b

by Lachlan Mackenzie at 2018-10-15T15:54:59Z

yaml.py: Add yaml cache testing to yaml list composition test

3d58d7c5

by Lachlan Mackenzie at 2018-10-15T15:54:59Z

yaml.py: Add yaml cache test to yaml list composition twice

363cfc39

by Lachlan Mackenzie at 2018-10-15T15:54:59Z

yamlcache.py: Add YAML cache changed file test

* Test in same style as test_yamlcache_used
* Move of project not required so removed

086c47c4

by Jonathan Maw at 2018-10-15T16:19:01Z

Merge branch 'lachlan/pickle-yaml-test-list-composite' into 'master'

Add YAM cache testing to yaml tests and further YAML cache refinements

See merge request BuildStream/buildstream!836

6298e409

by William Salmon at 2018-10-16T12:39:14Z

Sandbox: CWD was not being created for workspaces

The code was creating the cwd folder but when the workspace was
mounted in to the buildroot it was hiding the folder created in it
behind the bind mounted workspace.

However by using the bubblewarp `--dir` directive to ensure that cwd
exists we can cover both workspace and non workspace situations with
the same method.

For issue #512 in Gitlab.

bbcb9bd5

by William Salmon at 2018-10-16T12:39:14Z

New Integration test for workspace command-subdir

Build stream was only creating command-subdir for non workspace builds
this has now been fixed, and this regression test should support the
feature.

For issue #512 in Gitlab.

5e7cc645

by William Salmon at 2018-10-16T12:39:14Z

Add conf-root variable to builds

Adding the conf-root variable makes creating out of source builds
slightly easier.

For issue #512 in Gitlab.

0d04e1b7

by William Salmon at 2018-10-16T12:39:14Z

Added tests for conf-root variable

For issue #512 in Gitlab.

b2767ac0

by William Salmon at 2018-10-16T12:39:14Z

Extended the Documentation to cover out of source builds

For issue #512 in Gitlab.

118704a9

by William Salmon at 2018-10-16T12:39:14Z

Added a news item for building out of the source directory

For issue #512 in Gitlab.

0a5db229

by Phil Dawson at 2018-10-16T13:19:56Z

Merge branch 'willsalmon/outOfSourecBuild' into 'master'

Out of source builds

Closes #512

See merge request BuildStream/buildstream!776

1fd38fb7

by Jim MacArthur at 2018-10-16T14:54:09Z

bst2html.py: Name the second argument to avoid an error

b023d5c0

by Jim MacArthur at 2018-10-16T15:28:15Z

Merge branch 'jmac/doc-argument-fix' into 'master'

Add named argument in bst2html.py to fix documentation

See merge request BuildStream/buildstream!874

ac5b482d
by Richard Dale at 2018-10-16T19:40:39Z
```
Fix spelling of it's and its
```

3d9c4eca

by Jürg Billeter at 2018-10-16T20:17:39Z

Merge branch 'rdale/spelling' into 'master'

Fix spelling of it's and its

See merge request BuildStream/buildstream!877

edc07536

by Tiago Gomes at 2018-10-17T13:49:15Z

README: use correct links for documentation website

155296a7

by Tiago Gomes at 2018-10-17T13:49:15Z

setup.py: use correct link for documentation website

131f95eb

by Tiago Gomes at 2018-10-17T14:15:38Z

Merge branch 'tiagogones/update-links-to-documentation' into 'master'

Update links to documentation website

See merge request BuildStream/buildstream!876

c327d1f3
by Valentin David at 2018-10-17T14:33:40Z
```
Change URL to the Alpine tarball
```

eeb334f2

by Javier Jardón at 2018-10-17T15:03:17Z

Merge branch 'valentindavid/move_alpine_tarball' into 'master'

Change URL to the Alpine tarball

See merge request BuildStream/buildstream!880

b8f69946

by Daniel Silverstone at 2018-10-18T09:32:09Z

_loader/loader.py: Be more explanatory in CircDep exception

When a circular dependency is detected, the reported error hilights
only one part of the loop in question.  This change ensures that
the entire loop is listed so that the user has a greater chance of
easily determining the correct fix.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

892866e1

by richardmaw-codethink at 2018-10-18T10:04:32Z

Merge branch 'danielsilverstone-ct/report-loop' into 'master'

_loader/loader.py: Be more explanatory in CircDep exception

See merge request BuildStream/buildstream!878

f1aa0974

by Javier Jardón at 2018-10-18T14:13:35Z

Fix python warnings: Use collections.abc instead collections

Since python 3.3, collections has been moved to collections.abc module.
For backwards compatibility, they continue to be visible in this module through Python 3.7.
Subsequently, they will be removed entirely.

See https://docs.python.org/3/library/collections.html

f1550c8e

by Valentin David at 2018-10-18T16:02:45Z

Merge branch 'jjardon/warnings_collections' into 'master'

Fix python warnings: Use collections.abc instead collections

See merge request BuildStream/buildstream!883

1c24979a

by Angelos Evripiotis at 2018-10-19T21:17:01Z

Spelling fixes: 'affect', not 'effect' as verb

Fix most instances of 'effect' being used as a verb instead of a noun.
Skipped generated instance in doc/source/conf.py.

43a2eee3

by Angelos Evripiotis at 2018-10-19T21:48:59Z

Merge branch 'aevri/affect' into 'master'

Spelling fixes: 'affect', not 'effect' as verb

See merge request BuildStream/buildstream!885

12719f0d

by Jürg Billeter at 2018-10-22T17:05:41Z

_artifactcache: Rename update_atime() to update_mtime()

os.utime() updates both, however, list_artifacts() sorts refs by mtime,
i.e., atime is irrelevant.

a7a28d14

by Jürg Billeter at 2018-10-22T17:05:41Z

_artifactcache: Fix ref in update_mtime()

get_artifact_fullname() is required to construct the ref. The cache key
alone does not suffice.

be29e0f5

by Jürg Billeter at 2018-10-22T17:53:26Z

Merge branch 'juerg/cas-mtime' into 'master'

_artifactcache: Fix ref in update_mtime()

See merge request BuildStream/buildstream!882

b74aca1a

by Jürg Billeter at 2018-10-23T09:22:19Z

tests/testutils/artifactshare.py: Fix has_artifact() to match core code

Replace path separator with '-' to match Element.normal_name.

c7dda150

by Jürg Billeter at 2018-10-23T09:48:00Z

Merge branch 'juerg/test-artifactshare' into 'master'

tests/testutils/artifactshare.py: Fix has_artifact() to match core code

See merge request BuildStream/buildstream!890

74c115b9

by Angelos Evripiotis at 2018-10-23T10:07:31Z

plugins/sources: refactor, rm needless ExitStack-s

Remove two uses of ExitStack that aren't necessary and make the code
a little harder to follow.

ecb58b42

by Phil Dawson at 2018-10-23T10:33:47Z

Merge branch 'aevri/rm-exitstack' into 'master'

plugins/sources: refactor, rm needless ExitStack-s

See merge request BuildStream/buildstream!888

aa0cbf5d

by Martin Blanchard at 2018-10-23T10:54:40Z

_sandboxremote.py: Try to reopen operation steam on failure

The REAPI allows a client to reconnect to an ongoing operation stream by
providing a WaitExecution(). If implemented on server side, BuildStream
will try to recover from connection errors using it.

https://gitlab.com/BuildStream/buildstream/issues/630

552f5fc6

by Jim MacArthur at 2018-10-23T11:19:48Z

Merge branch 'mablanch/630-remote-execution-reconn' into 'master'

Handle connection losses during remote build execution

See merge request BuildStream/buildstream!806

276d2195
by Angelos Evripiotis at 2018-10-25T10:17:49Z
```
_downloadablefilesource: handle ValueError-s
```

2d012a21

by Angelos Evripiotis at 2018-10-25T10:17:49Z

Merge branch 'aevri/unknown-uri-452' into 'master'

_downloadablefilesource: handle ValueError-s

Closes #452

See merge request BuildStream/buildstream!893

cf4741ce

by James Ennis at 2018-10-25T13:05:43Z

docs: Add helpful notes to artifact server configuration instructions

5737bb06

by James Ennis at 2018-10-25T13:05:43Z

using_configuring_artifact_server.rst: Fix incorrect reference to bst push (with link)

28317622

by James Ennis at 2018-10-25T13:35:22Z

Merge branch 'jennis/patch_remote_cache_docs' into 'master'

Improve our documentation for configuring an artifact server

See merge request BuildStream/buildstream!899

d2743c52