[Notes] [Git][BuildStream/buildstream][valentindavid/fix-broken-indentat

Valentin David pushed to branch valentindavid/fix-broken-indentation-after-track at BuildStream / buildstream

Commits:

c68dcab8

by Tiago Gomes at 2018-08-09T09:18:43Z

cascache: move tmp directory one level up

The CAS uses a temp directory while manipulating the cache, temporary
files can be added and removed while adding artifacts to the cache here.

Since calculation of the cache size happens in parallel to artifact
cache additions, this causes race conditions in the size calculation
job, as we end up calling `stat` on temporary files which are being
removed in parallel.

Handle this by moving the temporary directory out of the way, and avoid
considering the tmp directory when calculating the cache size

35ab0335

by Tiago Gomes at 2018-08-09T10:16:24Z

Merge branch 'tiagogomes/issue-520' into 'master'

Fix race condition when calculating disk usage

See merge request BuildStream/buildstream!600

8aa33e23

by Valentin David at 2018-08-09T12:49:17Z

Keep original flags for create in SafeHardlinks.

When open(2) is used with flags O_CREAT|O_RDWR, the file descriptor
must be readable. Unfortunately O_RDWR was not passed which made
read fail with EBADF and mmap to signal SIGBUS.

This issue happened with man-db for example.

Fixes #143.

ef7810f3

by Valentin David at 2018-08-09T14:06:50Z

Merge branch 'valentindavid/fuse-create-flags' into 'master'

Keep original flags for create in SafeHardlinks.

Closes #143

See merge request BuildStream/buildstream!624

2d061173

by Javier Jardón at 2018-08-09T16:31:15Z

buildstream/_project.py:

Multiprocessing with n_jobs given by multiprocessing.cpu_count() is not optimal for
systems where not all CPU can be used (in particular, Docker, CI etc).

As suggested in the multiprocessing docs
(http://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count)
using len(os.sched_getaffinity(0)) might be better.

4e1488ee

by Javier Jardón at 2018-08-09T16:31:15Z

buildstream/_project.py: Restrict max-jobs

even if the machine have mores cores available

Patch taken from YBD: https://gitlab.com/baserock/ybd/blob/master/ybd/app.py#L227

de955834

by Javier Jardón at 2018-08-09T17:22:24Z

Merge branch 'jjardon/max-jobs' into 'master'

buildstream/_project.py: Restrict max-jobs

See merge request BuildStream/buildstream!620

97595585
by Jim MacArthur at 2018-08-09T17:35:19Z
```
setup.py: Pin ruamel.yaml version to <= 0.15
```

a602365c

by Tristan Van Berkom at 2018-08-09T20:03:52Z

Merge branch 'jmac/pin-ruamel-version' into 'master'

Pin ruamel.yaml version to <= 0.15

See merge request BuildStream/buildstream!630

db0478ab

by Phillip Smyth at 2018-08-10T10:17:47Z

Implementing relative workspaces

This fixes #191

A note has been added to NEWS explaining backwards
compatibility issues

04cee9a9

by Phillip Smyth at 2018-08-10T11:12:31Z

Merge branch 'relative_workspaces' into 'master'

Patch for issue #191 support relative workspaces

Closes #191

See merge request BuildStream/buildstream!504

5dcecbad

by Valentin David at 2018-08-10T12:18:55Z

Set environment in bwrap command line instead of its environment

Fixes #498

2e8db54e

by Valentin David at 2018-08-10T12:53:19Z

Merge branch 'valentindavid/498_bwrap_environment' into 'master'

Set environment in bwrap command line instead of its environment

Closes #498

See merge request BuildStream/buildstream!565

95920f48

by Chandan Singh at 2018-08-11T15:46:05Z

Move development reqirements to dev-requirements.txt

In some cases, such as when working inside a virtual environment, it can
be desirable to install all dependencies for running tests using `pip`.
This is currently not possible since setuptools does not support
installing these dependencies in a virtual environment (by design).
(See https://stackoverflow.com/a/21003259.)

To circumvent this issue, move such requirements to
`dev-requirements.txt` file that can be used easily with
`pip install -r`. This also enables tests to be run directly using
`pytest`, which can be more convenient than `-addopts` approach when one
needs to add multiple options.

This will also be useful in creating better testuite images, and fix
some of the issues noticed in
https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/56.

a9f63c5e

by Javier Jardón at 2018-08-11T16:53:39Z

Merge branch 'chandan/dev-requirements' into 'master'

Move development reqirements to dev-requirements.txt

See merge request BuildStream/buildstream!637

48c7c0a5
by Chandan Singh at 2018-08-11T23:41:05Z
```
Fix typo in dev-requirements.txt
```

88115648

by Chandan Singh at 2018-08-12T00:38:36Z

Merge branch 'chandan/fix-dev-reqs-typo' into 'master'

Fix typo in dev-requirements.txt

See merge request BuildStream/buildstream!641

0b8beb42
by Phil Dawson at 2018-08-12T00:40:57Z
```
.gitlab-ci-yml: Add ubuntu 18.04 test
```

499df6a5

by Javier Jardón at 2018-08-12T01:37:12Z

Merge branch 'phil/add-ubuntu-ci-job' into 'master'

.gitlab-ci-yml: Add ubuntu 18.04 test

See merge request BuildStream/buildstream!523

60c2873b
by Javier Jardón at 2018-08-12T01:38:29Z
```
.gitlab-ci.yml: Run tests in fedora28 image
```

2f27d428

by Javier Jardón at 2018-08-12T01:38:29Z

.gitlab-ci.yml: Use the same version of the docker image for all jobs

1819a288

by Javier Jardón at 2018-08-12T03:37:04Z

dev-requirements.txt: Remove obsolete comments

This also fixes an error with pylint in setup.py:
setup.py:226:19: R1718: Consider using a set comprehension (consider-using-set-comprehension)

48c715e3

by Javier Jardón at 2018-08-12T04:42:45Z

Merge branch 'jjardon/ci_fedora28' into 'master'

Add job to run tests in fedora 28

See merge request BuildStream/buildstream!643

ccf6e479

by Valentin David at 2018-08-12T05:54:36Z

buildstream/plugins/sources/local.py: Make staging deterministic.

Instead of copying metadata on files staged by local, we manually set
mode to 0755 or 0644 depending on whether user execution was enabled
on source file.

This makes file modes deterministic independently on the way source
was distributed.

Non-deterministic mode copying all metadata can still be enabled by
disable 'deterministic' Boolean configuration on the plugin.

Fixes #527.

322cab58

by Valentin David at 2018-08-12T05:54:36Z

Use deterministic umask when staging sources.

This fix is applied to plugins bzr, git, patch.

Fixes #543 #544 #555.

d08699bd

by Valentin David at 2018-08-12T05:54:36Z

buildstream/plugins/sources/zip.py: Fix non-determism in staging.

Staging could end-up with file with different rights depending on the
umask. The extracted files need to get their access rights fixed.

70fb7f17

by Valentin David at 2018-08-12T05:54:36Z

Add some integration tests for source plugin determinism.

67df3904

by Valentin David at 2018-08-12T05:54:36Z

Bump BST_CORE_ARTIFACT_VERSION for deterministic source plugins

c1fdebb3

by Tristan Van Berkom at 2018-08-12T06:59:46Z

Merge branch 'valentindavid/deterministic-source' into 'master'

Deterministic staging

Closes #543, #544, #555, and #527

See merge request BuildStream/buildstream!616

3e5aa8ff

by Chandan Singh at 2018-08-12T10:57:15Z

doc: Fix install instructions for Fedora

`python3-arpy` package is not available on Fedora so it has to be
installed using `pip`. Update instructions accordingly.

For reference, here is a pipeline that failed when trying to install
`arpy` using current instructions:
https://gitlab.com/BuildStream/buildstream-docker-images/-/jobs/88553245.

7a13c834

by Javier Jardón at 2018-08-12T20:11:12Z

Merge branch 'chandan/update-fedora-install-docs' into 'master'

doc: Fix install instructions for Fedora

See merge request BuildStream/buildstream!640

c2cc62e7

by Chandan Singh at 2018-08-13T00:17:21Z

remote source: Add cachekey test

Add cachekey tests for the recently added `remote` source plugin to
ensure that future changes do not break API compatibility.

a791e09c

by Tristan Van Berkom at 2018-08-13T07:43:52Z

Merge branch 'chandan/remote-source-cachekey-test' into 'master'

remote source: Add cachekey test

See merge request BuildStream/buildstream!636

353a6cc2

by Valentin David at 2018-08-13T09:29:40Z

Fix tracking of junctions used in project.conf.

Stream._load() now returns early without resolving build pipeline when
only tracking. Resolving track pipelines does not require to fully
load project configurations when when elements to track are only
junctions.

However build pipelines require to fully load project
configurations. This might not be possible in the case a project
configuration includes a file from a junction that yet needs to be
tracked.

Fixes #565.

5a667107

by Tristan Van Berkom at 2018-08-13T10:58:12Z

Merge branch 'valentindavid/fix_included_junction_track' into 'master'

Fix tracking of junctions used in project.conf

Closes #565

See merge request BuildStream/buildstream!619

6e759d8a

by Valentin David at 2018-08-13T12:39:23Z

Set version of ruaml.yaml to strictly less than 0.15.52.

This allows to use version 0.15.51 rather than 0.15.0 which is
required for Python 3.

Fixes #571.

56f2b2e4

by Valentin David at 2018-08-13T13:36:45Z

Merge branch 'valentindavid/ruamel-version' into 'master'

Set version of ruaml.yaml to at least 0.15.41 but strictly less than 0.15.52.

Closes #571

See merge request BuildStream/buildstream!646

4f6881d6

by Jürg Billeter at 2018-08-14T08:14:32Z

NEWS: Move recent entries to a new 1.3.1 section

Caching failed builds and relative workspaces were not part of the 1.1.5
developer snapshot.

0c8d0b0b

by Jonathan Maw at 2018-08-14T08:54:31Z

loader: Add 'build-depends' and 'runtime-depends' fields to elements

95798475

by Jonathan Maw at 2018-08-14T08:54:31Z

tests: Add tests for loading builddeps and runtime deps

476fcfa6

by Jonathan Maw at 2018-08-14T08:54:31Z

versions: Bump format version to 14

Format version raised because of a change to the core element format.

c51bbff3

by Jonathan Maw at 2018-08-14T08:54:31Z

docs: Add documentation of build-depends and runtime-depends fields

45117239
by Jonathan Maw at 2018-08-14T08:58:03Z
```
Add NEWS
```

9d074d7f

by Jonathan Maw at 2018-08-14T11:52:59Z

Merge branch '463-make-dependency-type-default-to-build-2' into 'master'

Resolve "Make dependency type default to build"

Closes #463

See merge request BuildStream/buildstream!633

69c5ccde

by Valentin David at 2018-08-14T13:31:03Z

buildstream/plugins/sources/git.py: Fix clone side effect in getting fetchers

We now delay refreshing submodules until we have a clone, that is after
we have fetched the main fetcher.

Fixes #537

820af29d

by Valentin David at 2018-08-14T13:31:03Z

tests/frontend/mirror.py: Re-enable a test for git

ca349fb4

by Jonathan Maw at 2018-08-14T13:31:03Z

tests: Test that fetching passes when upstream is absent

cb468469

by Jonathan Maw at 2018-08-14T13:31:03Z

tests: Add a test of the git source with submodules

dab5bb14

by Valentin David at 2018-08-14T13:31:03Z

Test git mirroring fallback on submodules when main repo is not mirrored.

1a3c4814

by Valentin David at 2018-08-14T13:31:03Z

Test we can discovered submodules on fallback mirrored git repositories

c52f6f9a

by Tristan Van Berkom at 2018-08-15T09:28:35Z

Merge branch 'valentindavid/fallback_mirror_git' into 'master'

Delay refreshing git submodule until we have a clone

Closes #537

See merge request BuildStream/buildstream!656

e6457725

by Josh Smith at 2018-08-15T10:17:29Z

examples: Switch ftp.gnu.org to mirror

This should avoid current issues accessing the necessary sources which
are blocking CI currently.

9f675c64

by Qinusty at 2018-08-15T11:15:08Z

Merge branch 'Qinusty/gnu-mirror' into 'master'

Fix CI - ftp.gnu.org unreachable

See merge request BuildStream/buildstream!665

f3d58233

by Josh Smith at 2018-08-15T11:15:58Z

_project.py: Add fatal-warnings configuration item

This allows for users to configure fatal-warnings to be either a list of
warnings.

This commit deprecates the use of fail-on-overlap within project.conf,
this will now use the fatal-warnings configuration item.

element.py: Cache key calculation now takes into account all of the
fatal-warnings

tests: This modifys the tests/frontend/overlaps.py tests to support the
new fatal-warnings configuration. Backwards compatibility is also
tested for `fail-on-overlap`

_versions.py: BST_FORMAT_VERSION bumped to 15 for fatal-warnings
	      BST_CORE_ARTIFACT_VERSION bumpted to 5 for fatal-warnings

Fixes: #526

4669309e

by Josh Smith at 2018-08-15T11:15:58Z

git.py: Add configurable warning for inconsistent-submodule

This follows the implementation of configurable warnings.

c9dd51f2

by Josh Smith at 2018-08-15T11:15:58Z

docs: Add documentation for Configurable Warnings

This includes detailing the use of `fatal-warnings` within project.conf

03527d5a

by Josh Smith at 2018-08-15T11:15:58Z

tests: Add tests for configurable warnings

This adds multiple tests for custom plugin warnings and core warnings,
providing checks for both cases which should cause warnings and errors
when configured as fatal.

Also adds tests for cache key calculations.

76f34a63

by Tristan Van Berkom at 2018-08-15T12:16:04Z

Merge branch 'Qinusty/526-fail-on-warnings' into 'master'

Configurable Warnings

Closes #526

See merge request BuildStream/buildstream!627

8b84a51a

by Jürg Billeter at 2018-08-15T13:02:32Z

_artifactcache/casserver.py: Fix resource_name format for blobs

Continue to accept requests from non-conforming BuildStream clients for
the time being to ease migration.

1202ef8a

by Jürg Billeter at 2018-08-15T13:02:32Z

_artifactcache/cascache.py: Fix resource_name format for blobs

This requires an updated server.

Fixes #572.

6a9d737e

by Jürg Billeter at 2018-08-15T13:02:32Z

_artifactcache/casserver.py: Improve ByteStream error handling

Replace assertions with gRPC error responses.

00762442

by Jürg Billeter at 2018-08-15T14:13:08Z

Merge branch 'juerg/cas' into 'master'

CAS: Fix resource_name format for blobs

Closes #572

See merge request BuildStream/buildstream!660

88ccff6e