[Notes] [Git][BuildStream/buildstream][tiagogomes/issue-514] 316 commits

Tiago Gomes pushed to branch tiagogomes/issue-514 at BuildStream / buildstream

Commits:

8aa33e23

by Valentin David at 2018-08-09T12:49:17Z

Keep original flags for create in SafeHardlinks.

When open(2) is used with flags O_CREAT|O_RDWR, the file descriptor
must be readable. Unfortunately O_RDWR was not passed which made
read fail with EBADF and mmap to signal SIGBUS.

This issue happened with man-db for example.

Fixes #143.

ef7810f3

by Valentin David at 2018-08-09T14:06:50Z

Merge branch 'valentindavid/fuse-create-flags' into 'master'

Keep original flags for create in SafeHardlinks.

Closes #143

See merge request BuildStream/buildstream!624

2d061173

by Javier Jardón at 2018-08-09T16:31:15Z

buildstream/_project.py:

Multiprocessing with n_jobs given by multiprocessing.cpu_count() is not optimal for
systems where not all CPU can be used (in particular, Docker, CI etc).

As suggested in the multiprocessing docs
(http://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count)
using len(os.sched_getaffinity(0)) might be better.

4e1488ee

by Javier Jardón at 2018-08-09T16:31:15Z

buildstream/_project.py: Restrict max-jobs

even if the machine have mores cores available

Patch taken from YBD: https://gitlab.com/baserock/ybd/blob/master/ybd/app.py#L227

de955834

by Javier Jardón at 2018-08-09T17:22:24Z

Merge branch 'jjardon/max-jobs' into 'master'

buildstream/_project.py: Restrict max-jobs

See merge request BuildStream/buildstream!620

97595585
by Jim MacArthur at 2018-08-09T17:35:19Z
```
setup.py: Pin ruamel.yaml version to <= 0.15
```

a602365c

by Tristan Van Berkom at 2018-08-09T20:03:52Z

Merge branch 'jmac/pin-ruamel-version' into 'master'

Pin ruamel.yaml version to <= 0.15

See merge request BuildStream/buildstream!630

db0478ab

by Phillip Smyth at 2018-08-10T10:17:47Z

Implementing relative workspaces

This fixes #191

A note has been added to NEWS explaining backwards
compatibility issues

04cee9a9

by Phillip Smyth at 2018-08-10T11:12:31Z

Merge branch 'relative_workspaces' into 'master'

Patch for issue #191 support relative workspaces

Closes #191

See merge request BuildStream/buildstream!504

5dcecbad

by Valentin David at 2018-08-10T12:18:55Z

Set environment in bwrap command line instead of its environment

Fixes #498

2e8db54e

by Valentin David at 2018-08-10T12:53:19Z

Merge branch 'valentindavid/498_bwrap_environment' into 'master'

Set environment in bwrap command line instead of its environment

Closes #498

See merge request BuildStream/buildstream!565

95920f48

by Chandan Singh at 2018-08-11T15:46:05Z

Move development reqirements to dev-requirements.txt

In some cases, such as when working inside a virtual environment, it can
be desirable to install all dependencies for running tests using `pip`.
This is currently not possible since setuptools does not support
installing these dependencies in a virtual environment (by design).
(See https://stackoverflow.com/a/21003259.)

To circumvent this issue, move such requirements to
`dev-requirements.txt` file that can be used easily with
`pip install -r`. This also enables tests to be run directly using
`pytest`, which can be more convenient than `-addopts` approach when one
needs to add multiple options.

This will also be useful in creating better testuite images, and fix
some of the issues noticed in
https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/56.

a9f63c5e

by Javier Jardón at 2018-08-11T16:53:39Z

Merge branch 'chandan/dev-requirements' into 'master'

Move development reqirements to dev-requirements.txt

See merge request BuildStream/buildstream!637

48c7c0a5
by Chandan Singh at 2018-08-11T23:41:05Z
```
Fix typo in dev-requirements.txt
```

88115648

by Chandan Singh at 2018-08-12T00:38:36Z

Merge branch 'chandan/fix-dev-reqs-typo' into 'master'

Fix typo in dev-requirements.txt

See merge request BuildStream/buildstream!641

0b8beb42
by Phil Dawson at 2018-08-12T00:40:57Z
```
.gitlab-ci-yml: Add ubuntu 18.04 test
```

499df6a5

by Javier Jardón at 2018-08-12T01:37:12Z

Merge branch 'phil/add-ubuntu-ci-job' into 'master'

.gitlab-ci-yml: Add ubuntu 18.04 test

See merge request BuildStream/buildstream!523

60c2873b
by Javier Jardón at 2018-08-12T01:38:29Z
```
.gitlab-ci.yml: Run tests in fedora28 image
```

2f27d428

by Javier Jardón at 2018-08-12T01:38:29Z

.gitlab-ci.yml: Use the same version of the docker image for all jobs

1819a288

by Javier Jardón at 2018-08-12T03:37:04Z

dev-requirements.txt: Remove obsolete comments

This also fixes an error with pylint in setup.py:
setup.py:226:19: R1718: Consider using a set comprehension (consider-using-set-comprehension)

48c715e3

by Javier Jardón at 2018-08-12T04:42:45Z

Merge branch 'jjardon/ci_fedora28' into 'master'

Add job to run tests in fedora 28

See merge request BuildStream/buildstream!643

ccf6e479

by Valentin David at 2018-08-12T05:54:36Z

buildstream/plugins/sources/local.py: Make staging deterministic.

Instead of copying metadata on files staged by local, we manually set
mode to 0755 or 0644 depending on whether user execution was enabled
on source file.

This makes file modes deterministic independently on the way source
was distributed.

Non-deterministic mode copying all metadata can still be enabled by
disable 'deterministic' Boolean configuration on the plugin.

Fixes #527.

322cab58

by Valentin David at 2018-08-12T05:54:36Z

Use deterministic umask when staging sources.

This fix is applied to plugins bzr, git, patch.

Fixes #543 #544 #555.

d08699bd

by Valentin David at 2018-08-12T05:54:36Z

buildstream/plugins/sources/zip.py: Fix non-determism in staging.

Staging could end-up with file with different rights depending on the
umask. The extracted files need to get their access rights fixed.

70fb7f17

by Valentin David at 2018-08-12T05:54:36Z

Add some integration tests for source plugin determinism.

67df3904

by Valentin David at 2018-08-12T05:54:36Z

Bump BST_CORE_ARTIFACT_VERSION for deterministic source plugins

c1fdebb3

by Tristan Van Berkom at 2018-08-12T06:59:46Z

Merge branch 'valentindavid/deterministic-source' into 'master'

Deterministic staging

Closes #543, #544, #555, and #527

See merge request BuildStream/buildstream!616

3e5aa8ff

by Chandan Singh at 2018-08-12T10:57:15Z

doc: Fix install instructions for Fedora

`python3-arpy` package is not available on Fedora so it has to be
installed using `pip`. Update instructions accordingly.

For reference, here is a pipeline that failed when trying to install
`arpy` using current instructions:
https://gitlab.com/BuildStream/buildstream-docker-images/-/jobs/88553245.

7a13c834

by Javier Jardón at 2018-08-12T20:11:12Z

Merge branch 'chandan/update-fedora-install-docs' into 'master'

doc: Fix install instructions for Fedora

See merge request BuildStream/buildstream!640

c2cc62e7

by Chandan Singh at 2018-08-13T00:17:21Z

remote source: Add cachekey test

Add cachekey tests for the recently added `remote` source plugin to
ensure that future changes do not break API compatibility.

a791e09c

by Tristan Van Berkom at 2018-08-13T07:43:52Z

Merge branch 'chandan/remote-source-cachekey-test' into 'master'

remote source: Add cachekey test

See merge request BuildStream/buildstream!636

353a6cc2

by Valentin David at 2018-08-13T09:29:40Z

Fix tracking of junctions used in project.conf.

Stream._load() now returns early without resolving build pipeline when
only tracking. Resolving track pipelines does not require to fully
load project configurations when when elements to track are only
junctions.

However build pipelines require to fully load project
configurations. This might not be possible in the case a project
configuration includes a file from a junction that yet needs to be
tracked.

Fixes #565.

5a667107

by Tristan Van Berkom at 2018-08-13T10:58:12Z

Merge branch 'valentindavid/fix_included_junction_track' into 'master'

Fix tracking of junctions used in project.conf

Closes #565

See merge request BuildStream/buildstream!619

6e759d8a

by Valentin David at 2018-08-13T12:39:23Z

Set version of ruaml.yaml to strictly less than 0.15.52.

This allows to use version 0.15.51 rather than 0.15.0 which is
required for Python 3.

Fixes #571.

56f2b2e4

by Valentin David at 2018-08-13T13:36:45Z

Merge branch 'valentindavid/ruamel-version' into 'master'

Set version of ruaml.yaml to at least 0.15.41 but strictly less than 0.15.52.

Closes #571

See merge request BuildStream/buildstream!646

4f6881d6

by Jürg Billeter at 2018-08-14T08:14:32Z

NEWS: Move recent entries to a new 1.3.1 section

Caching failed builds and relative workspaces were not part of the 1.1.5
developer snapshot.

0c8d0b0b

by Jonathan Maw at 2018-08-14T08:54:31Z

loader: Add 'build-depends' and 'runtime-depends' fields to elements

95798475

by Jonathan Maw at 2018-08-14T08:54:31Z

tests: Add tests for loading builddeps and runtime deps

476fcfa6

by Jonathan Maw at 2018-08-14T08:54:31Z

versions: Bump format version to 14

Format version raised because of a change to the core element format.

c51bbff3

by Jonathan Maw at 2018-08-14T08:54:31Z

docs: Add documentation of build-depends and runtime-depends fields

45117239
by Jonathan Maw at 2018-08-14T08:58:03Z
```
Add NEWS
```

9d074d7f

by Jonathan Maw at 2018-08-14T11:52:59Z

Merge branch '463-make-dependency-type-default-to-build-2' into 'master'

Resolve "Make dependency type default to build"

Closes #463

See merge request BuildStream/buildstream!633

69c5ccde

by Valentin David at 2018-08-14T13:31:03Z

buildstream/plugins/sources/git.py: Fix clone side effect in getting fetchers

We now delay refreshing submodules until we have a clone, that is after
we have fetched the main fetcher.

Fixes #537

820af29d

by Valentin David at 2018-08-14T13:31:03Z

tests/frontend/mirror.py: Re-enable a test for git

ca349fb4

by Jonathan Maw at 2018-08-14T13:31:03Z

tests: Test that fetching passes when upstream is absent

cb468469

by Jonathan Maw at 2018-08-14T13:31:03Z

tests: Add a test of the git source with submodules

dab5bb14

by Valentin David at 2018-08-14T13:31:03Z

Test git mirroring fallback on submodules when main repo is not mirrored.

1a3c4814

by Valentin David at 2018-08-14T13:31:03Z

Test we can discovered submodules on fallback mirrored git repositories

c52f6f9a

by Tristan Van Berkom at 2018-08-15T09:28:35Z

Merge branch 'valentindavid/fallback_mirror_git' into 'master'

Delay refreshing git submodule until we have a clone

Closes #537

See merge request BuildStream/buildstream!656

e6457725

by Josh Smith at 2018-08-15T10:17:29Z

examples: Switch ftp.gnu.org to mirror

This should avoid current issues accessing the necessary sources which
are blocking CI currently.

9f675c64

by Qinusty at 2018-08-15T11:15:08Z

Merge branch 'Qinusty/gnu-mirror' into 'master'

Fix CI - ftp.gnu.org unreachable

See merge request BuildStream/buildstream!665

f3d58233

by Josh Smith at 2018-08-15T11:15:58Z

_project.py: Add fatal-warnings configuration item

This allows for users to configure fatal-warnings to be either a list of
warnings.

This commit deprecates the use of fail-on-overlap within project.conf,
this will now use the fatal-warnings configuration item.

element.py: Cache key calculation now takes into account all of the
fatal-warnings

tests: This modifys the tests/frontend/overlaps.py tests to support the
new fatal-warnings configuration. Backwards compatibility is also
tested for `fail-on-overlap`

_versions.py: BST_FORMAT_VERSION bumped to 15 for fatal-warnings
	      BST_CORE_ARTIFACT_VERSION bumpted to 5 for fatal-warnings

Fixes: #526

4669309e

by Josh Smith at 2018-08-15T11:15:58Z

git.py: Add configurable warning for inconsistent-submodule

This follows the implementation of configurable warnings.

c9dd51f2

by Josh Smith at 2018-08-15T11:15:58Z

docs: Add documentation for Configurable Warnings

This includes detailing the use of `fatal-warnings` within project.conf

03527d5a

by Josh Smith at 2018-08-15T11:15:58Z

tests: Add tests for configurable warnings

This adds multiple tests for custom plugin warnings and core warnings,
providing checks for both cases which should cause warnings and errors
when configured as fatal.

Also adds tests for cache key calculations.

76f34a63

by Tristan Van Berkom at 2018-08-15T12:16:04Z

Merge branch 'Qinusty/526-fail-on-warnings' into 'master'

Configurable Warnings

Closes #526

See merge request BuildStream/buildstream!627

8b84a51a

by Jürg Billeter at 2018-08-15T13:02:32Z

_artifactcache/casserver.py: Fix resource_name format for blobs

Continue to accept requests from non-conforming BuildStream clients for
the time being to ease migration.

1202ef8a

by Jürg Billeter at 2018-08-15T13:02:32Z

_artifactcache/cascache.py: Fix resource_name format for blobs

This requires an updated server.

Fixes #572.

6a9d737e

by Jürg Billeter at 2018-08-15T13:02:32Z

_artifactcache/casserver.py: Improve ByteStream error handling

Replace assertions with gRPC error responses.

00762442

by Jürg Billeter at 2018-08-15T14:13:08Z

Merge branch 'juerg/cas' into 'master'

CAS: Fix resource_name format for blobs

Closes #572

See merge request BuildStream/buildstream!660

88ccff6e

by Jim MacArthur at 2018-08-15T14:33:05Z

Convert uses of external_directory to get_underlying_directory()

b84218a4

by Jim MacArthur at 2018-08-15T14:33:05Z

_filebaseddirectory/directory.py: Move VirtualDirectoryError to Directory

3341948a

by Jim MacArthur at 2018-08-15T14:33:05Z

Move KeyStrength enum out to element_enums.py

70f35b6c

by Jim MacArthur at 2018-08-15T14:33:05Z

Initial implementation of _casbaseddirectory.py

ef8525c2
by Jim MacArthur at 2018-08-15T14:33:05Z
```
Directory API: add mark_changed
```

00632108

by Jim MacArthur at 2018-08-15T14:33:05Z

filebaseddirectory: Update index after descend

0213f68d

by Jim MacArthur at 2018-08-15T14:33:05Z

Sandbox: Return a CasBasedDirectory when an environment variable is set

136deb2e
by Jim MacArthur at 2018-08-15T14:33:05Z
```
Add basic storage test 'storage-test.py'
```

12920046

by Jim MacArthur at 2018-08-15T17:03:26Z

Merge branch 'jmac/cas_virtual_directory' into 'master'

CAS-backed virtual directory implementation

See merge request BuildStream/buildstream!481

9d2442b2

by Chandan Singh at 2018-08-15T17:29:07Z

Allow source plugins to access previous sources

Source plugin implementations can now specify that they need access to
previously staged sources by specifying
`BST_REQUIRES_PREVIOUS_SOURCES_TRACK` and/or
`BST_REQUIRES_PREVIOUS_SOURCES_FETCH`, corresponding to access at `track`
and `fetch` times respectively.

Fixes #381.

Replaces !505. For relevant discussion, see this discussion:
https://gitlab.com/BuildStream/buildstream/merge_requests/505#note_83780747

a7a82650

by Chandan Singh at 2018-08-15T18:23:04Z

Add pip source plugin

`pip` source plugin can stage python packages that are either specified
directly in the element definition or picked up from `requirements.txt`
from previous sources. In order to support the latter use-case
(which is also the primary motivation for this plugin), this plugin
requires access to previous sources and hence is an example of a
Source Transform source.

Also, bump `BST_FORMAT_VERSION` as this patch adds a new core plugin.

4a2dd6af

by Chandan Singh at 2018-08-15T18:58:58Z

Add NEWS entry for Source Transform and pip source

d4706096

by Chandan Singh at 2018-08-15T20:21:01Z

Merge branch 'chandan/sourcetransform' into 'master'

Allow source plugins to access previous sources

Closes #381

See merge request BuildStream/buildstream!568

c0431f15

by Valentin David at 2018-08-16T07:43:07Z

Fix ostree repository mirroring

Ostree mirrors were not sharing the same local repository, so it was
impossible the request refs from the right local repository when data
was fetched from a mirror rather than upstream.

Instead of having several repository with one remote each, we now
have one repository with several remotes.

This fixes #538.

04f83679

by Tristan Van Berkom at 2018-08-16T08:54:27Z

Merge branch 'valentindavid/fallback_mirror_ostree' into 'master'

Fix ostree repository mirroring

Closes #538

See merge request BuildStream/buildstream!658

177c4264

by William Salmon at 2018-08-16T09:52:38Z

Trying to mitigate a mtime granularity braking the cache tests

This patch mitigates the granularity of the mtimes used on the default
gitlab runners, Allowing the test suite to pass on these runners.

01c4ac57

by Tom Pollard at 2018-08-16T11:01:45Z

Merge branch 'willsalmon/CacheExpiryTest' into 'master'

Trying to mitigate a file system issue

See merge request BuildStream/buildstream!595

0066d701

by Tiago Gomes at 2018-08-16T12:33:01Z

Use http instead of https for the gnu ftp mirror

https seems broken on the Debian image:

START   autotools/hello/42930621-fetch.499.log
START   Fetching https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz
FAILURE Fetching https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz
FAILURE tar source at hello.bst [line 16 column 2]: Error mirroring https://ftpmirror.gnu.org/gnu/automake/automake-1.16.tar.gz: <urlopen error [SSL: SSLV3_ALERT_HANDSHAKE_FAILURE] sslv3 alert handshake failure (_ssl.c:720)>

898aa8a0

by Tiago Gomes at 2018-08-16T13:43:06Z

Merge branch 'tiagogomes/disable-https-gnu-repo' into 'master'

Use http instead of https for the gnu ftp mirror

See merge request BuildStream/buildstream!676

b62c361c

by Tiago Gomes at 2018-08-16T13:54:54Z

cascache: use errno module

os.errno does no longer work with Python 3.7

Closes #577.

1e3e2a93

by Tiago Gomes at 2018-08-16T15:00:12Z

Merge branch 'tiagogomes/issue-577' into 'master'

cascache: use errno module

Closes #577

See merge request BuildStream/buildstream!674

10f27f3f

by Tristan Van Berkom at 2018-08-17T06:20:40Z

_frontend/app.py: Notify session completions

Use the optionally implemented desktop notification method
to notify when a session completes.

Previously, this used to only notify when an interactive prompt
appears.

This is an attempt to fix #385

c102b92f

by Tristan Van Berkom at 2018-08-17T08:52:07Z

Merge branch 'tristan/notifications' into 'master'

_frontend/app.py: Notify session completions

Closes #385

See merge request BuildStream/buildstream!672

9b6fac5a

by William Salmon at 2018-08-17T09:48:01Z

Add Error to git and ostree sources configure

Raise a error at configure time if the track and ref properties are
not present in the sources.

This is to address https://gitlab.com/BuildStream/buildstream/issues/471
that documented unhelpful behaviour when tracking git sources. However
the issue was also identified in ostree.

d5ed4f45

by William Salmon at 2018-08-17T09:49:23Z

Improve error message for build if there are refs missing

c43af4eb

by Will Salmon at 2018-08-17T10:51:38Z

Merge branch 'willsalmon/APIFix' into 'master'

Updating the no track or ref fix to match the buildstream style

See merge request BuildStream/buildstream!628

7263ea8c

by Josh Smith at 2018-08-17T13:19:23Z

job.py: Prevent terminated jobs retrying

Fixes #531: Jobs were retrying when terminated, this lead to the process
being spawned again and starting up fresh.

30f696f2

by Qinusty at 2018-08-17T14:29:45Z

Merge branch 'Qinusty/531-fetch-retries-on-terminate' into 'master'

Prevent jobs retrying on terminate

Closes #531

See merge request BuildStream/buildstream!662

c15cb951

by Tristan Van Berkom at 2018-08-20T10:19:03Z

setup.cfg: Add tests/integration/project to norecursedirs

This is causing pytest to try to import the recently added app1.py
from the pip related tests. This fails because it's not meant to be
imported outside of the sandbox environment.

372abed5

by Tristan Van Berkom at 2018-08-20T11:47:53Z

Merge branch 'tristan/exclude-project-py-from-tests' into 'master'

setup.cfg: Add tests/integration/project to norecursedirs

See merge request BuildStream/buildstream!682

1f88a1e9

by Jonathan Maw at 2018-08-20T12:36:02Z

Use ArtifactCache's get_cache_size when calculating the quota

It would have been more convenient to store the quota size in the
ArtifactCache, except that the scheduler needs the quota size and
doesn't have access to the ArtifactCache.

d9507a9e

by Jonathan Maw at 2018-08-20T12:36:02Z

Fix get_cache_size to store it in the ArtifactCache, not the context

a9b81dc4

by Jonathan Maw at 2018-08-20T12:36:02Z

artifactcache: Read and write cache size to disk

a67906e6
by Jonathan Maw at 2018-08-20T13:06:51Z
```
Add tests that cache size works
```

92e34ccd

by Jonathan Maw at 2018-08-20T16:27:18Z

Merge branch 'jonathan/cache-cache-size' into 'master'

Jonathan/cache cache size

See merge request BuildStream/buildstream!679

f1a619b5

by Jonathan Maw at 2018-08-20T16:34:03Z

CASCache: Fix cell-var-from-loop linting error

ba74a128

by Jonathan Maw at 2018-08-20T17:53:33Z

Merge branch 'jonathan/cascache-cell-var-from-loop' into 'master'

Jonathan/cascache cell var from loop

See merge request BuildStream/buildstream!686

18acd3ea

by Valentin David at 2018-08-21T08:44:34Z

Fix broken indentation after tracking.

Issue was introduced by 171e803f (include directive) and the fix was
found courtesy of @Qinusty. This fixes also the include
feature. Because elements are to be serialized, the included fragments
need to use copy_tree when loaded.

Related to #470.

6bd688cf

by Valentin David at 2018-08-21T08:44:34Z

buildstream/_includes.py: Fix caching of included fragments.

4cab18e3

by Qinusty at 2018-08-21T09:38:39Z

Merge branch 'valentindavid/fix-broken-indentation-after-track' into 'master'

Fix broken indentation after track

See merge request BuildStream/buildstream!622

f3fbdac0

by Jonathan Maw at 2018-08-21T11:33:34Z

Pipeline: Skip except_elements logic if no elements to except

3a4c147a

by Jonathan Maw at 2018-08-21T12:43:41Z

Merge branch 'jonathan/faster-except-elements' into 'master'

Pipeline: Skip except_elements logic if no elements to except

See merge request BuildStream/buildstream!688

d1bd8c0b

by Chandan Singh at 2018-08-21T12:46:57Z

.gitlab-ci.yml: Bump tags for testsuite images

These new images come with pinned versions of `pytest` and `pylint`:

- `pylint`: `== 2.1.1`
- `pytest`: `>= 3.7`

12d1cec2

by Chandan Singh at 2018-08-21T12:46:57Z

dev-requirements.txt: Pin versions of pytest and pylint

These packages should only be updated as a deliberate change and the
code should be tested with the new versions to ensure that the test
still pass.

Any future updates should also happen in sync with the
buildstream-docker-images repository so that the testsuite base images
also enforce the same restrictions for the development requirements.

Replaces https://gitlab.com/BuildStream/buildstream/merge_requests/684.

320b9562

by Tristan Van Berkom at 2018-08-21T13:57:00Z

Merge branch 'chandan/update-ci-tags' into 'master'

Restrict pylint/pytest versions and bump CI tags

See merge request BuildStream/buildstream!689

ef606229

by Josh Smith at 2018-08-21T14:49:57Z

plugin.py: Stop printing None for fatal-warnings

Providing warn with detail=None (default) will no longer print None when
a fatal warning is triggered.

d8256c74

by Qinusty at 2018-08-21T15:46:14Z

Merge branch 'Qinusty/598-fatal-warning-none-detail' into 'master'

Fatal warnings no longer print None when detail=None

See merge request BuildStream/buildstream!694

ab1caac1

by Chandan Singh at 2018-08-21T23:47:32Z

setup.py: Add fields required for publishing on PyPi

Fill out those fields in `setup.py` that are required for publishing a
package on PyPi.

Part of https://gitlab.com/BuildStream/buildstream/issues/587.

81278f84

by Tristan Van Berkom at 2018-08-22T05:23:33Z

Merge branch 'chandan/pypi-prep' into 'master'

setup.py: Add fields required for publishing on PyPi

See merge request BuildStream/buildstream!695

3d4e649d

by Tristan Van Berkom at 2018-08-22T05:24:42Z

_frontend/linuxapp.py: Fix special casing around desktop notification escape sequence

Now we allow the notification to happen on any TERM
which starts with 'xterm' or 'vte', and we only do it
if the VTE_VERSION is >= 4600, where we know for sure that
VTE will not print garbage on the terminal.

Fixes #385

c8bafb68

by Tristan Van Berkom at 2018-08-22T06:24:12Z

Merge branch 'tristan/notifications' into 'master'

_frontend/linuxapp.py: Fix special casing around desktop notification escape sequence

Closes #385

See merge request BuildStream/buildstream!692

eeaddbac

by William Salmon at 2018-08-22T08:05:28Z

Catch Non Numeric versions

This patch just displays a better message than the default stack trace
but dose not try to fix the problem. A further patch will be created but
it effects versioneer so may take longer to land as it may need to go
via versioneer mainline.

e15f5fa3

by William Salmon at 2018-08-22T08:05:28Z

Search for tags with the *.*.* patten for version

This has been done in a way that I hope might be able to be upstreamed
in to versioneer. This is not garanteed but it would be good if it
could be.

Please see https://github.com/warner/python-versioneer/pull/186 and
https://github.com/wsalmonct/python-versioneer/tree/feature/regexTags
for further details.

ff52ae18

by Will Salmon at 2018-08-22T09:33:04Z

Merge branch 'willsalmon/versionTagRegrex' into 'master'

Search for tags with the *.*.* patten for version

See merge request BuildStream/buildstream!601

f4e51cf1

by Sam Thursfield at 2018-08-22T10:38:41Z

Don't strip pathname from debug symbol files

When GDB looks for debug symbols it looks for the full path of the file
inside the configured debug-file-directory. For example, if the
debug-file-directory is set to a default of /usr/lib/debug, and you are
debugging /usr/bin/python3, GDB will look for its debug symbols at this
location: /usr/lib/debug/usr/bin/python3.

BuildStream has been putting all debug files inside /usr/lib/debug
under their $(basename), so in the above example GDB would fail to find
any debug symbols for /usr/bin/python3 because they would be in the
incorrect locatoin of /usr/lib/debug/python3.

5b79bbb7

by Tristan Van Berkom at 2018-08-22T11:39:32Z

Merge branch 'tristan/debug-symbols-location' into 'master'

Don't strip pathname from debug symbol files

See merge request BuildStream/buildstream!697

91d62d2f

by Chandan Singh at 2018-08-22T18:38:01Z

MANIFEST.in: Fix include for dev-requirements.txt

Currently, running any setuptools commands using `setup.py` generates a
warning like so:
        warning: manifest_maker: MANIFEST.in, line 21: unknown action 'dev-requirements.txt'

The syntax is invalid and got accidentally introduced in !637 but it's a
shame that this is not a warning and no obvious way to make it an error
if there are invalid things in MANIFEST.in file.

cd0775eb

by Chandan Singh at 2018-08-22T23:44:36Z

Merge branch 'chandan/fix-mainfest-dev-requirements' into 'master'

MANIFEST.in: Fix include for dev-requirements.txt

See merge request BuildStream/buildstream!704

b0d1aa83

by Tristan Van Berkom at 2018-08-23T05:54:05Z

_frontend/linuxapp.py: Fixing fallout from !693

When fixing terminal notifications, I had introduced a bug
with accesses to `os.environ` which triggered KeyError, this
patch fixes it.

2215859c

by Valentin David at 2018-08-23T06:40:19Z

Improve error message for deleted open workspaces

Fixes #576.

e37ac3bc

by Tristan Van Berkom at 2018-08-23T07:45:49Z

Merge branch 'valentindavid/inconsistant-workspace' into 'master'

Improve error message for deleted open workspaces

Closes #576

See merge request BuildStream/buildstream!703

2ac654a8

by Valentin David at 2018-08-23T08:02:50Z

Remove artifact extracts when artifact expires in cache

Fixes #561

176aa09f

by Tristan Van Berkom at 2018-08-23T09:10:51Z

Merge branch 'valentindavid/extract-expiry' into 'master'

Remove artifact extracts when artifact expires in cache

Closes #561

See merge request BuildStream/buildstream!685

f1e3f10f

by Mathieu Bridon at 2018-08-23T09:11:47Z

deps: Specify the minimum version required for blessings

Buildstream uses the Terminal.does_styling attribute, which was only
added in blessings 1.6.

Blessings helpfully returns an empty string when calling an nonexistent
attribute on the Terminal object, instead of failing.

Because Fedora has blessings 1.5, Buildstream thought that my terminal
couldn't handle styling, and just didn't print the status bar, silently.

Specifying the minimum version avoids this silent failure.

bea01531
by Tristan Van Berkom at 2018-08-23T09:14:40Z
```
.gitlab-ci.yml: Avoid running tests in post-merge
```

da19bcf1

by Tristan Van Berkom at 2018-08-23T09:14:40Z

README.rst: Removing the badges

Now that we only run pre-merge CI, it makes no sense to show a badge for
the coverage and pipeline status of the last run of master.

It would make sense to show a badge for the last pipeline which
was run for the last merge request that was merged to master, but
I don't see any feature to get that information.

8216ff8d

by Tristan Van Berkom at 2018-08-23T10:13:23Z

Merge branch 'tristan/reduce-gitlab-ci' into 'master'

.gitlab-ci.yml: Avoid running tests in post-merge

See merge request BuildStream/buildstream!709

56096b60

by Chandan Singh at 2018-08-24T02:12:03Z

.gitlab-ci.yml: Verify that source tarball installs correctly

Add a check to verify that the source distribution tarball that we
generate installs corrctly. It is useful to do this check as this is the
same tarball will be used when installing BuildStream via PyPI.

This check also happens indirectly in the coverage job where we install
BuildStream but this check makes it more explicit and will also cause
the CI to fail early if the tarball is not correct.

86e4729d

by Tristan Van Berkom at 2018-08-24T06:42:03Z

Merge branch 'chandan/ci-sdist-test' into 'master'

.gitlab-ci.yml: Verify that source tarball installs correctly

See merge request BuildStream/buildstream!720

d88568ef

by Valentin David at 2018-08-24T06:48:08Z

tests/integration/source-determinism.py: Use cli_integration.

f7ef151b

by Tristan Van Berkom at 2018-08-24T07:36:19Z

Merge branch 'valentindavid/cli_integration_source_determinism' into 'master'

tests/integration/source-determinism.py: Use cli_integration.

See merge request BuildStream/buildstream!715

fa476d79

by Tristan Van Berkom at 2018-08-24T07:55:43Z

tests/integration/project: Use the same alias for the alpine tarball

This makes the integration tests use the same 'alpine' alias for the
tests as we use in the examples, this avoids a redundant download
of an extra alpine tarball in integration tests.

This is a part of #603

255f9ee3

by Tristan Van Berkom at 2018-08-24T08:39:19Z

Merge branch 'tristan/avoid-redundant-alpine' into 'master'

tests/integration/project: Use the same alias for the alpine tarball

See merge request BuildStream/buildstream!722

e1ed09aa

by Jim MacArthur at 2018-08-24T09:51:27Z

element.py: chmod 777 directories if delete fails.

Staging may produce directories with less than 'rwx' permissions which
will cause tempfile to throw an exception on deleting the temporary
directory if not changed.

01fe504b

by James Ennis at 2018-08-24T09:51:27Z

tar tests: Add tarball which contains a read-only dir

a3755c9a

by James Ennis at 2018-08-24T09:51:27Z

tar.py: Add test for a tarball containing a read-only dir

1bfcca1a

by James Ennis at 2018-08-24T10:31:41Z

Merge branch 'jmac/tempfile-extraction-bug' into 'master'

Correct crash after staging tars with read-only directories

See merge request BuildStream/buildstream!713

744925f3

by James Ennis at 2018-08-24T10:34:57Z

HACKING.rst: Remove -k flag for individual tests as this is not required

dfee6d75

by James Ennis at 2018-08-24T11:15:45Z

Merge branch 'jennis/correct_HACKING' into 'master'

'-k' flag not needed to run individual tests

See merge request BuildStream/buildstream!714

923bfaec

by Chandan Singh at 2018-08-24T11:22:33Z

doc: Add instructions to install BuildStream via PyPI

Add instructions to install and update BuildStream python package via
PyPI, and also make it the recommended method.

Part of https://gitlab.com/BuildStream/buildstream/issues/587.

2c3a6ab8

by Chandan Singh at 2018-08-24T12:01:56Z

Merge branch 'chandan/pip-install-instructions' into 'master'

doc: Add instructions to install BuildStream via PyPI

See merge request BuildStream/buildstream!717

f4745600

by Josh Smith at 2018-08-24T12:20:05Z

setup.cfg: Show timing of the 20 slowest tests

This helps identifying what tests are slow or if the CI got stuck on
some of them

4c40a027

by Javier Jardón at 2018-08-24T13:04:41Z

Merge branch 'jjardon/ci_show_timings' into 'master'

.gitlab-ci.yml: Show timing of the 20 slowest tests

See merge request BuildStream/buildstream!716

0301f74e

by Benjamin Schubert at 2018-08-24T14:52:49Z

buildelement.py: remove double logging (again) as in #507

This was broken again by d14d8ee29b5fcaadcfd05934de71b46221243fcd,
this commits re-applies the fix from 12fea82e12a27eccb34ef4efbaa7c6874787eb4c

f991644c

by Qinusty at 2018-08-24T15:33:48Z

Merge branch 'bschubert15/fix-double-log' into 'master'

buildelement.py: remove double logging (again) as in #507

See merge request BuildStream/buildstream!730

f4c2bf19

by Phil Dawson at 2018-08-25T08:43:09Z

workspaces.rst: Move terminal sessions to doc/source/sessions-stored/

23b01f24

by Phil Dawson at 2018-08-25T08:43:09Z

workspaces.rst: fix note which is not rendering correctly

369bec0c

by Tristan Van Berkom at 2018-08-25T08:44:26Z

doc/source/developing/workspaces.rst: Some whitespace fixes

Some whitespace changes to follow the style regarding empty
lines between sections.

24cb3037

by Tristan Van Berkom at 2018-08-25T09:10:53Z

.gitignore: Ignore doc/source/sessions directory of generated html files

A preventative measure against people committing the generated
html files to the wrong location.

4f8fe977

by Tristan Van Berkom at 2018-08-25T09:10:53Z

HACKING.rst: Documenting the new "shell" configuration in the bst2html input file format

d819fb64

by Tristan Van Berkom at 2018-08-25T11:41:15Z

Merge branch 'tristan/fixup-workspace-docs' into 'master'

Fixing some fallout in the workspace documentation additions

Closes #553

See merge request BuildStream/buildstream!732

ba675863

by Tristan Van Berkom at 2018-08-25T11:42:08Z

doc/source/main_install.rst: Use one level depth toctree

This main section is just a choice of subsections, lets
make this more clear with a single level depth toctree.

52fce701

by Tristan Van Berkom at 2018-08-25T11:42:08Z

doc/source/main_install.rst: Move the main install link target here

This is referred to from the artifact install guide, and was
previously pointing directly to the source install guide.

f7289b6b

by Tristan Van Berkom at 2018-08-25T11:42:08Z

doc/source/install_source.rst: Splitting up install instructions

Instead of having all of this on one page, we now have a much
simpler page for installing on distros where BuildStream is packaged
and a separate page for installing distro specific system dependencies
and installing from source code (either from PyPI or git).

a1fa504d

by Tristan Van Berkom at 2018-08-25T11:42:08Z

Add section about current releases

This is not ideal as the releases are hardcoded, but I think is better
than nothing until we got a solution to automate this

This commit was originally by Javier Jardón <jjardon gnome org> from
merge request !661, and then later reworked into an install guide refactor.

08632568

by Tristan Van Berkom at 2018-08-25T11:42:08Z

doc/source/install_source.rst: Explain about release tag in git install instructions.

This obsoletes Laurence's explanation in merge request !657, and
is a part of the initiative of issue #528 to clarify which versions
users should be installing.

a7e046d5

by Tristan Van Berkom at 2018-08-25T12:22:51Z

Merge branch 'tristan/install-guide-refactoring' into 'master'

Install guide refactoring

See merge request BuildStream/buildstream!733

c53d190b

by Javier Jardón at 2018-08-26T06:04:58Z

doc/bst2html.py: Fix deprecation warning

This fixes the following:
"
/home/jjardon/buildstream/buildstream/doc/bst2html.py:32: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Mapping
"

a9390efd

by Tristan Van Berkom at 2018-08-26T07:27:50Z

Merge branch 'jjardon/doc_fix_deprecations' into 'master'

doc/bst2html.py: Fix deprecation warning

See merge request BuildStream/buildstream!727

5fb772b5

by Tristan Van Berkom at 2018-08-26T07:30:58Z

.gitlab-ci.yml: Trying to fix code quality reports

This hasnt been working for a while, it might be because GitLab
doesnt quite grasp that we have multiple branches to merge MRs to,
or it might be because they have change the API and broken back compat
in some way.

See: https://docs.gitlab.com/ee/ci/examples/code_quality.html

fa696307

by Tristan Van Berkom at 2018-08-26T08:10:59Z

Merge branch 'tristan/fix-code-quality' into 'master'

.gitlab-ci.yml: Trying to fix code quality reports

See merge request BuildStream/buildstream!735

0f329276

by Tristan Van Berkom at 2018-08-26T08:11:56Z

README.rst: Use stronger language to follow the installation guide.

And also refer to the user guide section instead of directly to the
tutorial, the tutorial will still be the first section of the user
guide so this is a better link target for getting started.

This is related to #528

4b6f3d0d

by Tristan Van Berkom at 2018-08-26T08:57:05Z

Merge branch 'tristan/readme-version' into 'master'

README.rst: Use stronger language to follow the installation guide.

See merge request BuildStream/buildstream!736

7c6e91f5

by Tristan Van Berkom at 2018-08-27T06:51:20Z

Revert ".gitlab-ci.yml: Avoid running tests in post-merge"

This reverts commit bea01531b984522940562866e904c83e8f9aed50.

539963e3

by Tristan Van Berkom at 2018-08-27T06:51:47Z

Revert "README.rst: Removing the badges"

This reverts commit da19bcf1de75bf930d0f233addc858b4073d0173.

5a5b6699

by Tristan Van Berkom at 2018-08-27T06:53:55Z

.gitlab-ci.yml: Moving code_quality job into tests phase

This job always takes a long time and does not really depend
on the artifacts in the 'test' phase, moving it to the 'test'
phase will reduce overall CI time.

c6ba8932

by Tristan Van Berkom at 2018-08-27T07:27:06Z

Merge branch 'tristan/restore-post-merge-CI' into 'master'

Tristan/restore post merge ci

See merge request BuildStream/buildstream!737

9cb69b0b

by Tristan Van Berkom at 2018-08-27T07:46:22Z

doc/source/main_install.rst: Clarify that BuildStream cannot be run on Windows

The previous wording was misleading and caused one to think you can run
BuildStream on Windows, this commit fixes that and rewords the main section
heading to be a bit more abstract.

Fixes issue #608

8c7fadd8

by Tristan Van Berkom at 2018-08-27T08:12:52Z

Merge branch 'tristan/install-guide-main' into 'master'

doc/source/main_install.rst: Clarify that BuildStream cannot be run on Windows

Closes #608

See merge request BuildStream/buildstream!739

ddbf7bc2

by Jürg Billeter at 2018-08-28T08:56:06Z

element.py: Schedule assemble for key of workspaced elements

For uncached workspaced elements, assemble is required even just to
calculate the cache key. As dynamic scheduling relies on cache keys,
schedule assemble for uncached workspace elements even if they have not
been marked as required yet.

Fixes #461.

ffc556ff

by Valentin David at 2018-08-28T09:01:05Z

tests/frontend/workspace.py: Add test for workspaced dependencies

This adds a regression test for #461.

dd90d53d

by Jürg Billeter at 2018-08-28T09:01:05Z

element.py: Do not attempt to pull workspaced elements

6a5a8e7a

by Tristan Van Berkom at 2018-08-28T11:29:22Z

Merge branch 'juerg/workspaced-dependencies' into 'master'

Fix key invalidation for workspaced dependencies

Closes #461

See merge request BuildStream/buildstream!740

6047a575

by Tristan Van Berkom at 2018-08-28T13:41:36Z

tests/frontend/workspaces.py: Removing some redundant tests

Removed redundant tests from recently merged !740, this new
test does not need to run for every different source kind.

7535fda8

by Jürg Billeter at 2018-08-28T14:25:35Z

Merge branch 'tristan/remove-redundant-test' into 'master'

tests/frontend/workspaces.py: Removing some redundant tests

See merge request BuildStream/buildstream!741

10eeb503
by Javier Jardón at 2018-08-28T22:22:00Z
```
doc/source/conf.py: Fix E201, E202 warnings
```
d3c32ca2
by Javier Jardón at 2018-08-28T22:22:00Z
```
doc/source/conf.py: Fix E402 warning
```
3ae5fd05
by Javier Jardón at 2018-08-28T22:22:00Z
```
Fix E305 warnings
```
c6fb5ba7
by Javier Jardón at 2018-08-28T22:22:00Z
```
Fix E741 warnings
```

c5eca59d

by Tristan Van Berkom at 2018-08-29T10:45:33Z

Merge branch 'jjardon/pycodestyle_fixes' into 'master'

Some pycodestyle (PEP8) fixes

See merge request BuildStream/buildstream!746

63c6ee72

by Tristan Van Berkom at 2018-08-29T10:46:30Z

doc: Adding badges.py release badge generator

This adds a step to the docs generation Makefile to generate
release.svg and snapshot.svg badges, modelled after the gitlab
badges.

This also adds the generated badges directory in docs to .gitignore

2d527052

by Tristan Van Berkom at 2018-08-29T10:46:30Z

README.rst: Adding the release and snapshot badges to the readme

This is a part of #528

29e7eea8

by Tristan Van Berkom at 2018-08-29T10:46:30Z

doc: Adding the release badges to the install page and the semantic versioning page

This is a part of #528

5d508779

by Tristan Van Berkom at 2018-08-29T11:13:53Z

Merge branch 'tristan/docs-version-badge' into 'master'

Release badges

See merge request BuildStream/buildstream!742

323a5403

by Chandan Singh at 2018-08-29T12:13:28Z

setup.py: Add more metadata useful for PyPI

- setup.py: List useful links using `project_urls`

- setup.py: Specify minumum python vresion using `python_requires`

  `python_requires` is the new way of specifying requirements for python
  version, as documented @
  https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires.

  This will also make this requirement appear nicely on the PyPI project
  page.

- setup.py: Add project classifiers

  PyPI will use these classifiers to categorize projects while searching
  or browsing. Full list of classifiers can be found at
  https://pypi.org/classifiers.

5fc7b5ac

by Tristan Van Berkom at 2018-08-29T12:37:08Z

Merge branch 'chandan/setup.py-metadata' into 'master'

setup.py: Add more metadata useful for PyPI

See merge request BuildStream/buildstream!718

42ad937d

by Josh Smith at 2018-08-29T12:38:15Z

Prevent ValueError on URLs without an alias

Fixes #597

aa3a33b3

by Tristan Van Berkom at 2018-08-29T13:03:58Z

Merge branch 'Qinusty/597-non-alias-url-fix' into 'master'

Fix ValueError traceback for URL without alias or URI scheme

Closes #597

See merge request BuildStream/buildstream!700

2a2a79de

by Valentin David at 2018-08-29T14:11:48Z

Disable round-tripping when element is not modified

2df7607c

by Tristan Van Berkom at 2018-08-29T14:43:55Z

Merge branch 'valentindavid/roundtripping_only_when_modified' into 'master'

Disable round-tripping when element is not modified

See merge request BuildStream/buildstream!748

c146dde5
by Benjamin Schubert at 2018-08-29T16:00:24Z
```
Log not-found objects in the cache as SKIPPED
```

19e87afb

by Benjamin Schubert at 2018-08-29T16:01:12Z

Add tests that not-found objects in cache are shown as SKIPPED

9cc5817f

by Qinusty at 2018-08-29T16:25:48Z

Merge branch 'bschubert/log-missed-cache' into 'master'

Log not-found objects in the cache as SKIPPED

See merge request BuildStream/buildstream!729

3c8646e3

by Josh Smith at 2018-08-29T16:34:22Z

Add cyclic check within variable resolution

This aims to address #600, this will raise an exception when a resolved
variable contains a reference to the variable.

d41c42d6

by Josh Smith at 2018-08-29T16:34:22Z

Add tests for cyclic variables check

Note: This modifies the docker containers used for testing to supply the
pytest-timeout package.

c7b72c30

by Qinusty at 2018-08-29T16:58:48Z

Merge branch 'Qinusty/600-recursive-variables' into 'master'

Add cyclic check within variable resolution

See merge request BuildStream/buildstream!712

d28e1353

by Tristan Van Berkom at 2018-08-30T06:02:41Z

tests/plugins/filter.py: Don't run redundant tests

There is no reason that the filter element codepaths can behave
differently depending on the Source implementation used in the test,
as the Source implementation does not have any filter specific
virtual methods.

Removing the redundant tests and just performing these tests with the git source.

e77ae07a

by Tristan Van Berkom at 2018-08-30T06:36:41Z

Merge branch 'tristan/reduce-filter-tests' into 'master'

tests/plugins/filter.py: Don't run redundant tests

See merge request BuildStream/buildstream!753

9ea40e2c

by Tristan Van Berkom at 2018-08-30T06:38:13Z

tests/frontend/mirror.py: Reenable test_mirror_fetch_upstream_absent[ostree]

This test was skipped because of issue #538, but #538 was fixed
and the test was still not reenabled.

8aec1102

by Tristan Van Berkom at 2018-08-30T07:06:17Z

Merge branch 'tristan/538-reenable-ostree-test' into 'master'

tests/frontend/mirror.py: Reenable test_mirror_fetch_upstream_absent[ostree]

See merge request BuildStream/buildstream!755

fcc17c82

by Tristan Van Berkom at 2018-08-30T08:44:34Z

source.py: Stylistic changes in Source.__do_fetch()

Added some comments to make the flow easier to follow, and
removed an annoying 'success' variabled in favor of a for / else
loop statement.

60ad044a

by Tristan Van Berkom at 2018-08-30T08:44:34Z

source.py: Move Source.mark_download_url() to the public methods.

This was sitting in the section for abstract methods, but this
is most definitely not an abstract method to be implemented by
Sources.

c339f917

by Tristan Van Berkom at 2018-08-30T08:44:34Z

plugins/source/git.py: Fix formatting of url in tracking

This was displaying the aliased URL which is pretty useless,
use the translated URL for the timed activity.

26565f11

by Tristan Van Berkom at 2018-08-30T09:12:28Z

Merge branch 'tristan/source-mirroring-changes' into 'master'

Minor code changes revolving around source mirroring

See merge request BuildStream/buildstream!758

a38d9163

by Valentin David at 2018-08-30T10:26:57Z

Report processing errors from tracking

Failures to write files when tracking were not reported.

Fixes #533.

02138181

by Tristan Van Berkom at 2018-08-30T11:36:00Z

Merge branch 'valentindavid/post_tracking_errors' into 'master'

Report processing errors from tracking

Closes #533

See merge request BuildStream/buildstream!747

d23d4f3e

by Tristan Van Berkom at 2018-08-30T12:02:27Z

_artifactcache/artifactcache.py: Write the cache_size file atomically

This is causing issues while the size file is being read and written
simultaneously.

The proper fix will be to read/add/save the file atomically and that
will require locking, but this fix is a good stop gap to existing crashes.

bb6aa5a5

by Tristan Van Berkom at 2018-08-30T12:27:55Z

Merge branch 'tristan/atomic-cache-size-file' into 'master'

_artifactcache/artifactcache.py: Write the cache_size file atomically

See merge request BuildStream/buildstream!762

680e4fe1

by knownexus at 2018-08-30T13:37:55Z

Replacing string 'bzr' with value from host tools

8cd719eb

by Phillip Smyth at 2018-08-30T15:47:06Z

Merge branch 'bzr_fix' into 'master'

Replacing string 'bzr' with value from host tools

See merge request BuildStream/buildstream!764

18d0bfb4

by Tom Pollard at 2018-08-30T19:14:33Z

buildstream/_project.py: Report if project.conf is missing name

Explicitly check that project.conf contains a name. This resolves
the issue of the provenance check from _yaml.py incorrectly reporting
the offending file as the default_config_node projectconfig.yaml
when no name key exists in the pre_config_node dict.

95121148

by Tiago Gomes at 2018-08-30T20:52:29Z

Merge branch 'tpollard/591' into 'master'

buildstream/_project.py: Report if project.conf is missing name

See merge request BuildStream/buildstream!680

775d3fca

by Tristan Van Berkom at 2018-08-31T06:05:32Z

_project.py: Adding FIXME comment to address #591 properly

0682aa1c

by Josh Smith at 2018-08-31T10:58:12Z

job.py: Modify retry messages to be FAIL

This adjusts the message handler for the child processes to no longer
override the message type.

This also removes the ability for unhandled non BstError exceptions to
retry.

68eff9ba

by Josh Smith at 2018-08-31T10:58:20Z

tests: Modify tests to ensure retry FAILURE

9b18e1bf

by Tristan Van Berkom at 2018-08-31T11:22:51Z

Merge branch 'Qinusty/retries-should-fail' into 'master'

Retries log as failures

See merge request BuildStream/buildstream!766

e2d8fedf

by Josh Smith at 2018-08-31T15:53:43Z

scheduler.py: Move cache_size logs into folder

This prevents the cache_size.pid.log files from cluttering the root
log directory.

6805a2ab

by Tristan Van Berkom at 2018-09-01T08:24:05Z

Merge branch 'Qinusty/cache-size-directory' into 'master'

Move cache_size.pid.log files into a subdirectory of logs

See merge request BuildStream/buildstream!769

f8b06acc

by Tristan Van Berkom at 2018-09-02T09:33:39Z

source.py: Document Source.get_source_fetchers() to return an iterable

Also highlight the fact that the plugin can rely on the fetcher's
fetch() method getting called before consuming the next item in the
list, which is the magick behavior that the git plugin relies on.

This is a part of #620

559b6fbc

by Tristan Van Berkom at 2018-09-02T09:34:10Z

source.py: Fixing docs link formatting to be consistent.

43ad22d7

by Tristan Van Berkom at 2018-09-02T09:34:10Z

source.py: Documenting that marking download URLs is mandatory

A download URL must be interpreted by the core at `Plugin.configure()`
time, even if only employed later on.

This is a part of #620

16462e9c

by Tristan Van Berkom at 2018-09-02T09:36:49Z

plugin.py: Added _configure() and _get_configuring() private APIs

Keeps track of whether the plugin is currently being configured.

Adjusted Element and Source classes to call _configure() in place
of calling configure() directly.

This is a part of #620

1f9c4147

by Tristan Van Berkom at 2018-09-02T09:37:12Z

source.py: Added `primary` argument to URL marking APIs

The Source must now mention whether the marked or translated
URL is "primary" or not. Even when a Source may have multiple
URLs, the auxilliary URLs are derived from the primary one, not
only is this true for git, but it is mandated by our tracking
API which assumes there is a primary URL.

This adjusts the `git` source and the test `fetch_source.py` source
to behave properly and advertize it's primary URL properly.

This is a part of #620

f9b2f1a9

by Tristan Van Berkom at 2018-09-02T09:37:21Z

source.py: Track marked URLs and assert they are marked during Plugin.configure()

This cannot test for unaliased URLs, as those can be discovered later
on outside of user provided element configuration; at least we
assert that if an alias was used, we have seen it at load time.

This will cause a BUG to occur for a plugin which marks an aliased
URL (or attempts to translate one) outside of `Plugin.configure()`,
if that URL was not previously seen.

This is a part of #620

88460cd2

by Tristan Van Berkom at 2018-09-03T06:23:38Z

Merge branch 'tristan/source-fetcher-changes' into 'master'

Source fetcher changes

See merge request BuildStream/buildstream!772

2339f0c4

by Tom Pollard at 2018-09-03T09:44:36Z

plugins/git.py: Warn if ref is not in given track

Add a helper function assert_ref_in_track to git.py GitMirror()
which is used when staging & initing the source workspace. It
checks the element's ref exists in the track (branch/tag) if it
has been specified, raising a warning if necessary. The warning makes
use of the warning token 'REF_NOT_IN_TRACK' from the configurable
CoreWarnings. If the element has been tracked with bst, it is assumed
that the value of ref exists in the track as it was generated from it
& as such is not asserted.

c96fec5d

by Tom Pollard at 2018-09-03T09:44:36Z

tests/sources/git.py: Add tests for REF_NOT_IN_TRACK

Add tests that cover assert_ref_in_track & the configurable
CoreWarnings REF_NOT_IN_TRACK warnings token.

6a0cdedf

by Tom Pollard at 2018-09-03T10:16:39Z

Merge branch 'tpollard/483' into 'master'

plugins/git.py: Warn if ref is not in given track

See merge request BuildStream/buildstream!564

9b327eb6

by Ben Brewer at 2018-09-04T14:41:23Z

Improve documentation for artifact cache installation

Remove ambiguity about systemd service files being separate.
Add URL for more information about systemd service files.
Add note about public keys being mandatory for self-signed certs.
Make cert/key file naming consistent throughout document.

3e67e64a

by Javier Jardón at 2018-09-04T16:43:13Z

Merge branch 'benbrewer/install-artifacts-doc-improvements' into 'master'

Improve documentation for artifact cache installation

See merge request BuildStream/buildstream!777

3409609e

by Daniel Silverstone at 2018-09-04T16:55:51Z

jobs.py: Reduce FD leaks from queues and process objects

The garbage collector can take too long to get around to cleaning
up the Queue and Process instances in completed Job instances.  As
such, FDs tend to leak and in very large projects this can result
in running out of FDs before a build, fetch, track, or other process
can complete.  This patch reduces the chance of that by only creating
the queue when it's needed, and forcing the queue and process instances
to be deleted when the parent is finished with them.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

7b32e1ec

by Tristan Maat at 2018-09-04T17:20:55Z

Merge branch 'danielsilverstone-ct/maybe-reduce-fd-leaks' into 'master'

jobs.py: Reduce FD leaks from queues and process objects

See merge request BuildStream/buildstream!778

936bb93a

by Jim MacArthur at 2018-09-07T12:57:28Z

cascache.py: Preparation for remote execution

Refactor the push() and pull() implementations so that API additions
needed for remote-execution is made easier.

https://gitlab.com/BuildStream/buildstream/issues/454

b0f46545

by Martin Blanchard at 2018-09-07T12:57:29Z

cascache.py: Introduce new push helpers

Add push_directory() and push_message() helpers along with a
verify_digest_pushed().

https://gitlab.com/BuildStream/buildstream/issues/454

50bf3139

by Martin Blanchard at 2018-09-07T12:57:29Z

cascache.py: Introduce new pull helpers

Add a pull_tree() helper.

https://gitlab.com/BuildStream/buildstream/issues/454

853e0af7

by Martin Blanchard at 2018-09-07T12:57:29Z

_casbaseddirectory.py: Add a method for hash recalculation

https://gitlab.com/BuildStream/buildstream/issues/454

e7659c29

by Jim MacArthur at 2018-09-07T12:57:29Z

sandbox.py: Allow setting the virtual directory

This is for use after remote execution has finished, since remote
execution produces a new output directory rather than modifying
the initial directory.

https://gitlab.com/BuildStream/buildstream/issues/454

7306a954

by Jim MacArthur at 2018-09-07T12:57:29Z

Add "remote-execution" project configuration option

This just adds one option, "remote-execution/url". Affects multiple files.

https://gitlab.com/BuildStream/buildstream/issues/454

c0ef7106

by Jim MacArthur at 2018-09-07T12:57:29Z

format_project.rst: Document remote-execution option

https://gitlab.com/BuildStream/buildstream/issues/454

43651af0

by Jim MacArthur at 2018-09-07T12:57:29Z

_sandboxremote.py: Implement the REAPI client

The remote execution client is implemented as a remote sandbox that
sends sources and build commands to a REAPI server and fetches results
once remotely executed. New file.

https://gitlab.com/BuildStream/buildstream/issues/454

da53c005

by Jim MacArthur at 2018-09-07T12:57:29Z

sandbox/__init__.py: Add SandboxRemote

https://gitlab.com/BuildStream/buildstream/issues/454

c73896bb

by Jim MacArthur at 2018-09-07T12:57:29Z

element.py: Get the updated virtual directory after running

Executing run() on a sandbox can now replace the virtual directory,
since remote execution returns a potentially different directory rather
than an update to the existing one. Call get_virtual_directory() again
after running to accout for this.

https://gitlab.com/BuildStream/buildstream/issues/454

89b29b8b

by Jim MacArthur at 2018-09-07T12:57:29Z

element.py: Switch to SandboxRemote if config option is set

https://gitlab.com/BuildStream/buildstream/issues/454

116f80d2

by Jim MacArthur at 2018-09-07T12:57:29Z

autotools.py: Mark this as a BST_VIRTUAL_DIRECTORY plugin

https://gitlab.com/BuildStream/buildstream/issues/454

05956c71

by Martin Blanchard at 2018-09-07T12:57:29Z

cmake.py: Mark this as a BST_VIRTUAL_DIRECTORY plugin

https://gitlab.com/BuildStream/buildstream/issues/454

909b3be4

by Martin Blanchard at 2018-09-07T12:57:29Z

make.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin

https://gitlab.com/BuildStream/buildstream/issues/454

556f2abb

by Martin Blanchard at 2018-09-07T12:57:30Z

meson.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin

https://gitlab.com/BuildStream/buildstream/issues/454

71e2b543

by Martin Blanchard at 2018-09-07T12:57:30Z

qmake.py: Mark this as a BST_VIRTUAL_DIRECTORY‧plugin

https://gitlab.com/BuildStream/buildstream/issues/454

d4c515d0

by Martin Blanchard at 2018-09-07T12:57:30Z

tests/artifactcache: Add a simple test project

https://gitlab.com/BuildStream/buildstream/issues/454

923d443b

by Martin Blanchard at 2018-09-07T13:22:48Z

tests/artifactcache: Add push unit-tests

https://gitlab.com/BuildStream/buildstream/issues/454

9ef53163

by Martin Blanchard at 2018-09-07T13:22:57Z

tests/artifactcache: Add pull unit-tests

https://gitlab.com/BuildStream/buildstream/issues/454

f52169b2

by Qinusty at 2018-09-07T16:48:02Z

Merge branch 'jmac/remote_execution_client' into 'master'

Remote execution client

See merge request BuildStream/buildstream!626

8326b0f7

by Thomas Coldrick at 2018-09-08T17:54:22Z

Upstream libtool .la file removal

In freedesktop-sdk we add a script to our project.conf to remove
libtool .la files from autotools projects after install, this seems
like a sensible default, so we're attempting to send it upstream.

886a836e

by Valentin David at 2018-09-08T20:07:29Z

Merge branch 'coldtom/autotools-libtool' into 'master'

Upstream freedesktop-sdk autotools config for libtool .la files

See merge request BuildStream/buildstream!683

2174db01