[Notes] [Git][BuildStream/buildstream][jmac/cas_virtual

Jim MacArthur pushed to branch jmac/cas_virtual_directory at BuildStream / buildstream

Commits:

c68dcab8

by Tiago Gomes at 2018-08-09T09:18:43Z

cascache: move tmp directory one level up

The CAS uses a temp directory while manipulating the cache, temporary
files can be added and removed while adding artifacts to the cache here.

Since calculation of the cache size happens in parallel to artifact
cache additions, this causes race conditions in the size calculation
job, as we end up calling `stat` on temporary files which are being
removed in parallel.

Handle this by moving the temporary directory out of the way, and avoid
considering the tmp directory when calculating the cache size

35ab0335

by Tiago Gomes at 2018-08-09T10:16:24Z

Merge branch 'tiagogomes/issue-520' into 'master'

Fix race condition when calculating disk usage

See merge request BuildStream/buildstream!600

8aa33e23

by Valentin David at 2018-08-09T12:49:17Z

Keep original flags for create in SafeHardlinks.

When open(2) is used with flags O_CREAT|O_RDWR, the file descriptor
must be readable. Unfortunately O_RDWR was not passed which made
read fail with EBADF and mmap to signal SIGBUS.

This issue happened with man-db for example.

Fixes #143.

ef7810f3

by Valentin David at 2018-08-09T14:06:50Z

Merge branch 'valentindavid/fuse-create-flags' into 'master'

Keep original flags for create in SafeHardlinks.

Closes #143

See merge request BuildStream/buildstream!624

2d061173

by Javier Jardón at 2018-08-09T16:31:15Z

buildstream/_project.py:

Multiprocessing with n_jobs given by multiprocessing.cpu_count() is not optimal for
systems where not all CPU can be used (in particular, Docker, CI etc).

As suggested in the multiprocessing docs
(http://docs.python.org/3/library/multiprocessing.html#multiprocessing.cpu_count)
using len(os.sched_getaffinity(0)) might be better.

4e1488ee

by Javier Jardón at 2018-08-09T16:31:15Z

buildstream/_project.py: Restrict max-jobs

even if the machine have mores cores available

Patch taken from YBD: https://gitlab.com/baserock/ybd/blob/master/ybd/app.py#L227

de955834

by Javier Jardón at 2018-08-09T17:22:24Z

Merge branch 'jjardon/max-jobs' into 'master'

buildstream/_project.py: Restrict max-jobs

See merge request BuildStream/buildstream!620

97595585
by Jim MacArthur at 2018-08-09T17:35:19Z
```
setup.py: Pin ruamel.yaml version to <= 0.15
```

a602365c

by Tristan Van Berkom at 2018-08-09T20:03:52Z

Merge branch 'jmac/pin-ruamel-version' into 'master'

Pin ruamel.yaml version to <= 0.15

See merge request BuildStream/buildstream!630

db0478ab

by Phillip Smyth at 2018-08-10T10:17:47Z

Implementing relative workspaces

This fixes #191

A note has been added to NEWS explaining backwards
compatibility issues

04cee9a9

by Phillip Smyth at 2018-08-10T11:12:31Z

Merge branch 'relative_workspaces' into 'master'

Patch for issue #191 support relative workspaces

Closes #191

See merge request BuildStream/buildstream!504

5dcecbad

by Valentin David at 2018-08-10T12:18:55Z

Set environment in bwrap command line instead of its environment

Fixes #498

2e8db54e

by Valentin David at 2018-08-10T12:53:19Z

Merge branch 'valentindavid/498_bwrap_environment' into 'master'

Set environment in bwrap command line instead of its environment

Closes #498

See merge request BuildStream/buildstream!565

95920f48

by Chandan Singh at 2018-08-11T15:46:05Z

Move development reqirements to dev-requirements.txt

In some cases, such as when working inside a virtual environment, it can
be desirable to install all dependencies for running tests using `pip`.
This is currently not possible since setuptools does not support
installing these dependencies in a virtual environment (by design).
(See https://stackoverflow.com/a/21003259.)

To circumvent this issue, move such requirements to
`dev-requirements.txt` file that can be used easily with
`pip install -r`. This also enables tests to be run directly using
`pytest`, which can be more convenient than `-addopts` approach when one
needs to add multiple options.

This will also be useful in creating better testuite images, and fix
some of the issues noticed in
https://gitlab.com/BuildStream/buildstream-docker-images/merge_requests/56.

a9f63c5e

by Javier Jardón at 2018-08-11T16:53:39Z

Merge branch 'chandan/dev-requirements' into 'master'

Move development reqirements to dev-requirements.txt

See merge request BuildStream/buildstream!637

48c7c0a5
by Chandan Singh at 2018-08-11T23:41:05Z
```
Fix typo in dev-requirements.txt
```

88115648

by Chandan Singh at 2018-08-12T00:38:36Z

Merge branch 'chandan/fix-dev-reqs-typo' into 'master'

Fix typo in dev-requirements.txt

See merge request BuildStream/buildstream!641

0b8beb42
by Phil Dawson at 2018-08-12T00:40:57Z
```
.gitlab-ci-yml: Add ubuntu 18.04 test
```

499df6a5

by Javier Jardón at 2018-08-12T01:37:12Z

Merge branch 'phil/add-ubuntu-ci-job' into 'master'

.gitlab-ci-yml: Add ubuntu 18.04 test

See merge request BuildStream/buildstream!523

60c2873b
by Javier Jardón at 2018-08-12T01:38:29Z
```
.gitlab-ci.yml: Run tests in fedora28 image
```

2f27d428

by Javier Jardón at 2018-08-12T01:38:29Z

.gitlab-ci.yml: Use the same version of the docker image for all jobs

1819a288

by Javier Jardón at 2018-08-12T03:37:04Z

dev-requirements.txt: Remove obsolete comments

This also fixes an error with pylint in setup.py:
setup.py:226:19: R1718: Consider using a set comprehension (consider-using-set-comprehension)

48c715e3

by Javier Jardón at 2018-08-12T04:42:45Z

Merge branch 'jjardon/ci_fedora28' into 'master'

Add job to run tests in fedora 28

See merge request BuildStream/buildstream!643

ccf6e479

by Valentin David at 2018-08-12T05:54:36Z

buildstream/plugins/sources/local.py: Make staging deterministic.

Instead of copying metadata on files staged by local, we manually set
mode to 0755 or 0644 depending on whether user execution was enabled
on source file.

This makes file modes deterministic independently on the way source
was distributed.

Non-deterministic mode copying all metadata can still be enabled by
disable 'deterministic' Boolean configuration on the plugin.

Fixes #527.

322cab58

by Valentin David at 2018-08-12T05:54:36Z

Use deterministic umask when staging sources.

This fix is applied to plugins bzr, git, patch.

Fixes #543 #544 #555.

d08699bd

by Valentin David at 2018-08-12T05:54:36Z

buildstream/plugins/sources/zip.py: Fix non-determism in staging.

Staging could end-up with file with different rights depending on the
umask. The extracted files need to get their access rights fixed.

70fb7f17

by Valentin David at 2018-08-12T05:54:36Z

Add some integration tests for source plugin determinism.

67df3904

by Valentin David at 2018-08-12T05:54:36Z

Bump BST_CORE_ARTIFACT_VERSION for deterministic source plugins

c1fdebb3

by Tristan Van Berkom at 2018-08-12T06:59:46Z

Merge branch 'valentindavid/deterministic-source' into 'master'

Deterministic staging

Closes #543, #544, #555, and #527

See merge request BuildStream/buildstream!616

3e5aa8ff

by Chandan Singh at 2018-08-12T10:57:15Z

doc: Fix install instructions for Fedora

`python3-arpy` package is not available on Fedora so it has to be
installed using `pip`. Update instructions accordingly.

For reference, here is a pipeline that failed when trying to install
`arpy` using current instructions:
https://gitlab.com/BuildStream/buildstream-docker-images/-/jobs/88553245.

7a13c834

by Javier Jardón at 2018-08-12T20:11:12Z

Merge branch 'chandan/update-fedora-install-docs' into 'master'

doc: Fix install instructions for Fedora

See merge request BuildStream/buildstream!640

c2cc62e7

by Chandan Singh at 2018-08-13T00:17:21Z

remote source: Add cachekey test

Add cachekey tests for the recently added `remote` source plugin to
ensure that future changes do not break API compatibility.

a791e09c

by Tristan Van Berkom at 2018-08-13T07:43:52Z

Merge branch 'chandan/remote-source-cachekey-test' into 'master'

remote source: Add cachekey test

See merge request BuildStream/buildstream!636

353a6cc2

by Valentin David at 2018-08-13T09:29:40Z

Fix tracking of junctions used in project.conf.

Stream._load() now returns early without resolving build pipeline when
only tracking. Resolving track pipelines does not require to fully
load project configurations when when elements to track are only
junctions.

However build pipelines require to fully load project
configurations. This might not be possible in the case a project
configuration includes a file from a junction that yet needs to be
tracked.

Fixes #565.

5a667107

by Tristan Van Berkom at 2018-08-13T10:58:12Z

Merge branch 'valentindavid/fix_included_junction_track' into 'master'

Fix tracking of junctions used in project.conf

Closes #565

See merge request BuildStream/buildstream!619

38466b1f