[Notes] [Git][BuildStream/buildstream][bschubert/pipeline] 121 commits:

Benjamin Schubert pushed to branch bschubert/pipeline at BuildStream / buildstream

Commits:

03111d39

by Dor Askayo at 2019-01-30T10:35:06Z

filter.py: don't recurse when staging dependencies

Also bump the element's version so cached artifacts would be
invalidated.

Fixes #883

7256bb0c

by James Ennis at 2019-01-30T11:34:04Z

Merge branch 'doraskayo/filter-indirect-deps-fix' into 'master'

filter.py: don't recurse when staging dependencies

Closes #883

See merge request BuildStream/buildstream!1110

36746730

by Chandan Singh at 2019-01-31T10:50:05Z

tox.ini: Specify minimum version of click-man

`click-man` versions < 0.3.0 do not properly support multiple
entrypoints. Since this was added to `tox` after `0.3.0` was released,
`tox` should never be pulling older versions. But, let's add it here for
documentation purposes.

See
https://gitlab.com/BuildStream/buildstream/merge_requests/1107#note_135187046
for some background on this.

fa4a21ce

by Chandan Singh at 2019-01-31T12:15:43Z

Merge branch 'chandan/min-version-click-man' into 'master'

tox.ini: Specify minimum version of click-man

See merge request BuildStream/buildstream!1120

dd791373

by Chandan Singh at 2019-01-31T14:32:44Z

testutils/site.py: Support parsing more exotic git versions

We use output of `git --version` to determine if we can run some tests
that rely on features from newer git versions. Usually, we expect the
output to be like:

    git version 2.17.2

On some platforms, like MacOS, there could be a suffix after the version
string, so that it looks something like:

    git version 2.17.2 (Apple Git-113)

This causes things to fail like so:

    ValueError: invalid literal for int() with base 10: '2 (Apple Git-113)\n'

Fix logic around `HAVE_OLD_GIT` such that we split the output of
`git --version` without limit on how many times we split. Previously we
used to split only twice so the suffixes like `(Apple Git-113)` are not
part of the parsed version.

96c0fbd6

by Chandan Singh at 2019-01-31T15:39:19Z

Merge branch 'chandan/fix-git-version-mac' into 'master'

testutils/site.py: Support parsing more exotic git versions

See merge request BuildStream/buildstream!1118

d25e2795

by Benjamin Schubert at 2019-01-31T17:06:23Z

Add LoadElement in dependency list for LoadElement idrectly

This removes the need for the 'Dependency' list to then be matched
with the corresponding LoadElement and will allow iterating the
graph more easily

2d0eebbf

by Benjamin Schubert at 2019-01-31T17:06:23Z

Pass element directly to collect_element

This simplifies the loading

583bd97d

by Benjamin Schubert at 2019-02-01T10:26:37Z

Merge branch 'bschubert/loader' into 'master'

Cleanup loader by linking LoadElements sooner

See merge request BuildStream/buildstream!1122

51cec3da

by Phil Dawson at 2019-02-01T14:25:44Z

tests/cachekey: Test cache keys are independent of target elements

2b38aabe

by Phil Dawson at 2019-02-01T15:33:00Z

Merge branch 'phil/cache-key-stability-test' into 'master'

tests/cachekey: Test cache keys are independent of target elements

See merge request BuildStream/buildstream!1123

dbb3d232

by James Ennis at 2019-02-01T15:51:32Z

filter.py/filter.yaml: Documentation improvements

7e4205cb

by James Ennis at 2019-02-01T15:51:32Z

filter.py: Add an example to the documentation

4109a34a

by James Ennis at 2019-02-01T17:11:29Z

Merge branch 'jennis/filter-docs' into 'master'

Improve our filter documentation

Closes #278

See merge request BuildStream/buildstream!1112

c9345014

by James Ennis at 2019-02-04T13:53:42Z

filter.py: Fail if declared domains do not exist in the parent element

This patch also uncovered the fact that our test_filter_deps_ok() test
has been inaccurate. Thus the element built in this test
(deps-permitted.bst) has been modified so that it build depends on the
input.bst element, as it should.

tests/filter.py: Ensure deps_ok test passes

3ab09651

by James Ennis at 2019-02-04T14:47:43Z

Merge branch 'jennis/warn_for_nonexistent_domains' into 'master'

Fail when we explictly try to include/exclude non-existent domains in a filter element

See merge request BuildStream/buildstream!1117

cd8e5e27

by Tom Pollard at 2019-02-05T11:22:01Z

Add --remote, -r option to bst build, inline with pull & push

Providing a remote will limit build's pull/push remote actions to
the given remote specifically, ignoring those defined via user or
project configuration.

a46650d0

by Tom Pollard at 2019-02-05T12:41:30Z

Merge branch 'tpollard/buildremote' into 'master'

Add --remote, -r option to bst build, inline with pull & push

See merge request BuildStream/buildstream!1119

9c94e8e5

by Daniel Silverstone at 2019-02-05T13:15:24Z

requirements: Add roaringbitmap to the requirements

In order to support use of roaring bitmaps in the loader, we need to
depend on it here.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

7682ef49

by Daniel Silverstone at 2019-02-05T13:15:24Z

loadelement.py: Use roaring bitmaps in dep cache

The dependency cache in LoadElement causes the peak RAM consumption of
the loader to be exceedingly large, upwards of 25GB for a test of a
Debian stack.  By switching from the old dict cache to a roaring bitmap
based cache, we reduce the cost of the loader cache in RAM terms to
around 5.5GB for the same stack, which makes it plausible for the short
term.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

0e471144

by Benjamin Schubert at 2019-02-05T15:45:37Z

Merge branch 'danielsilverstone-ct/roaring-bitmaps' into 'master'

Switch to roaring bitmaps for the loader dependency caches

See merge request BuildStream/buildstream!1128

f17ea6a6

by Jonathan Maw at 2019-02-06T10:32:05Z

gitlab-ci: Add commands to be run by WSL runners

d4d2f897
by Jonathan Maw at 2019-02-06T10:45:34Z
```
tests: Skip tests that use sandboxes on WSL
```

f4c8a0ec

by Jonathan Maw at 2019-02-06T10:45:34Z

tests: Fix test failures caused by unix sockets' path length limit

The unix specification for unix sockets only allocates ~100 characters
for the length of the path. This may be longer than the path for a CI
runner, or the path to a user's buildstream directory.

256bcde5

by Jonathan Maw at 2019-02-06T15:29:18Z

Merge branch 'jonathan/wsl-tests' into 'master'

Add pre-merge tests that use a WSL runner

Closes #852

See merge request BuildStream/buildstream!1108

335ef2b8

by Daniel Silverstone at 2019-02-06T17:30:13Z

loadelement.py: Switch from roaringbitmap to pyroaring

The licence for roaringbitmap was not compatible.  pyroaring is
a similar project which is MIT licensed and is slightly better on
memory too.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

8f90be91
by Benjamin Schubert at 2019-02-06T18:36:17Z
```
Update testing images to include c++ compiler
```

1a75b252

by Benjamin Schubert at 2019-02-06T19:34:09Z

Merge branch 'danielsilverstone-ct/other-roaring' into 'master'

loadelement.py: Switch from roaringbitmap to pyroaring

Closes #899

See merge request BuildStream/buildstream!1133

7285e87e

by William Salmon at 2019-02-07T09:42:36Z

Add more log formatting options

The 'wallclock-us' is a option that we thought we already had.

e61f4713

by Will Salmon at 2019-02-07T10:58:54Z

Merge branch 'willsalmon/log_formating' into 'master'

Add more log formatting options

See merge request BuildStream/buildstream!1125

b4d4c4f5

by Phil Dawson at 2019-02-08T14:27:54Z

Expose basic api for testing external plugins.

We want external plugins to be able to make use of the core testing utils.
This commit exposes the basic utilities which are currently in use in
bst-external plugins. If necessary, more utilities could be exposed in the
future.

Moves the following files from tests/testutils/ to
buildstream/plugintestingutils/:
o runcli.py
o integration.py

As part of this, this commit makes the following changes to runcli.py
and integration.py:
o runcli.py: Fix linting errors
o runcli.py: Add user facing documentation
o Integration.py: Add user facing documentation

77ce0c16

by Phil Dawson at 2019-02-08T16:28:05Z

Merge branch 'phil/plugin-testing-api' into 'master'

Expose basic api for testing external plugins.

Closes #847

See merge request BuildStream/buildstream!1075

1ee4a4ba

by Abderrahim Kitouni at 2019-02-08T16:42:23Z

_artifactcache.py: don't leak the project specific remote caches

the code for initializing remotes added the project specific remote
caches to the global list instead of making a copy.

Fixes #618

24c0de16

by Abderrahim Kitouni at 2019-02-08T16:42:23Z

_project.py: use artifact caches from the parent project for junctions

This makes a junction use the artifact cache of the parent project
before the ones defined for the junction

Fixes #401

a937f99a

by Javier Jardón at 2019-02-08T20:29:05Z

Merge branch 'abderrahim/artifact-cache-junction' into 'master'

Use artifact cache specs from the parent project before those defined in junctions

Closes #618 and #401

See merge request BuildStream/buildstream!1113

da1560e4

by Javier Jardón at 2019-02-09T09:10:01Z

.gitlab-ci.yml: Remove testing on Fedora 27

Fedora 27 is EOL since 2018-11-30

See https://fedoraproject.org/wiki/End_of_life

79fbab61

by Jürg Billeter at 2019-02-09T10:30:20Z

Merge branch 'jjardon/fedora_27' into 'master'

.gitlab-ci.yml: Remove testing on Fedora 27

See merge request BuildStream/buildstream!1136

354c563d
by Jürg Billeter at 2019-02-10T20:03:46Z
```
import.py: Validate config node

Fixes #662.
```

5e1be71f

by Jürg Billeter at 2019-02-11T05:10:56Z

Merge branch 'juerg/import' into 'master'

import.py: Validate config node

Closes #662

See merge request BuildStream/buildstream!1141

99e1be45

by Jürg Billeter at 2019-02-11T05:12:25Z

local.py: Do not follow symlinks in local directories

isdir() follows symlinks on the host, resulting in potential host
contamination. This change reorders the file checks to avoid this issue.

f95e222e

by Jürg Billeter at 2019-02-11T05:12:25Z

sandbox/sandbox.py: Do not follow symlinks in _has_command()

This is required to ensure symlinks are not resolved on the host.

89973fb3

by Jürg Billeter at 2019-02-11T05:12:25Z

utils.py: Remove list_dirs parameter from list_relative_paths()

list_dirs was always True in the BuildStream code base. There was also a
bug in the list_dirs=False code path as it did not return symlinks in
`dirnames`.

This is an API break, however, there are no known external callers.

d1da3fb0

by Jürg Billeter at 2019-02-11T05:12:25Z

utils.py: Fix sorting of symlinks to directories

os.walk() resolves symlinks to check whether they point to a directory
even when followlinks is set to False. We already work around that
broken behavior by extracting symlinks from `dirnames`. However, the
sort order was still incorrect as we returned symlinks in dirnames
before files and other symlinks. This change fixes this, sorting all
files and symlinks in a single list.

7cf67ed3

by Jürg Billeter at 2019-02-11T05:12:25Z

_casbaseddirectory.py: Do not mimic os.walk() in list_relative_paths()

This matches the change in utils.list_relative_paths() that now sorts
all symlinks as files, instead of following the broken behavior of
os.walk().

7ec0bb5e

by Jürg Billeter at 2019-02-11T05:44:20Z

tests/sources/local.py: Add directory symlink test

c07cc967

by Jürg Billeter at 2019-02-11T07:13:28Z

Merge branch 'juerg/symlinks' into 'master'

Symlink fixes

See merge request BuildStream/buildstream!1138

9e3d3e05

by Angelos Evripiotis at 2019-02-11T07:14:50Z

contributing: snakeviz replaces pyflame+flamegraph

Replace the instructions for pyflame+flamegraph with simpler ones for
snakeviz. For our general use-case this seems to be easier and better.
Usage of this tool was demonstrated at the 2019 BuildStream Gathering in
January by Daniel Silverstone, when presenting the aggregate results of
profiling on many target environments.

Here is the relevant mailing list thread:

"Profiling before the gathering"
https://mail.gnome.org/archives/buildstream-list/2019-January/msg00057.html

bb6a692d

by Jürg Billeter at 2019-02-11T09:20:34Z

Merge branch 'snakeviz' into 'master'

contributing: snakeviz replaces pyflame+flamegraph

See merge request BuildStream/buildstream!1129

1ed63e54

by Angelos Evripiotis at 2019-02-11T09:24:48Z

_includes: better provenance on recursive include

Use the provenance of the include block, instead of the whole node.

02e48209

by Angelos Evripiotis at 2019-02-11T09:24:48Z

_includes: better error on missing include

Previously, a missing include would result in an error like this:

    Could not find file at not-a-file.include

Note that the file containing the include was not mentioned.

Now we get an error like this instead:

    element.bst [line 7 column 5]: Include block references a file that
    could not be found: 'not-a-file.include'.

4336e3bf

by Angelos Evripiotis at 2019-02-11T09:24:48Z

_includes: better error on including directory

Previously, include a directory result in an error like this:

    mydir is a directory. bst command expects a .bst file.

Note that the file containing the include was not mentioned.

Now we get an error like this instead:

    element.bst [line 12 column 0]: Include block references a
    directory instead of a file: 'mydir'.

3f6c5000

by Angelos Evripiotis at 2019-02-11T09:24:48Z

_includes: re-use file_path variable

Avoid an unnecessary call to os.path.join().

adde0c94

by Angelos Evripiotis at 2019-02-11T09:24:48Z

tests/format/include: remove unused tmpdir's

Don't create and remove temp dirs unnecessarily when they are not used,
looks like these were just copy-pastes without intended side-effects.

a66f8379

by Jürg Billeter at 2019-02-11T13:52:54Z

Merge branch 'aevri/include-error' into 'master'

More user-friendly reporting on include errors

See merge request BuildStream/buildstream!891

a482b008

by Javier Jardón at 2019-02-11T19:23:02Z

.gitlab-ci.yml: Do not automatically run "test-wsl" until runner problems are sorted out

0816b8b1

by Jürg Billeter at 2019-02-11T21:29:29Z

Merge branch 'jjardon/allow_fail_WSL' into 'master'

.gitlab-ci.yml: allow "test-wsl" to fail until runner problems are sorted out

See merge request BuildStream/buildstream!1145

a7aed65a

by Jürg Billeter at 2019-02-12T05:52:07Z

utils.py: Increase buffer size in sha256sum()

Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
factor 4, according to a very simple benchmark.

8b9e1d24

by Jürg Billeter at 2019-02-12T05:52:07Z

_cas/cascache.py: Increase buffer size in add_object()

Increasing buffer size from 4 kB to 64 kB speeds up read() bandwidth by
factor 4, according to a very simple benchmark.

86a9048a

by Jürg Billeter at 2019-02-12T07:08:18Z

Merge branch 'juerg/buffer-size' into 'master'

Increase read buffer size to improve performance

See merge request BuildStream/buildstream!1142

9b6c18e4

by Chandan Singh at 2019-02-12T10:20:52Z

_frontend: Allow printing dependencies using `bst show`

At present, there isn't an easy way to print anything from `bst show`
that would give the users an idea of what the dependency graph looks
like. One could use things like `--deps build`, but that will just print
a list, without any information about the dependency edges.

Add `%{deps}`, `%{build-deps}` and `%{runtime-deps}` format strings to
`bst show` that would simply print the list of all dependencies, build
dependencies and runtime dependencies respectively.

Summary of changes:

  * buildstream/_frontend/cli.py: Add help for new format symbols.
  * buildstream/_frontend/widget.py: Add support for new format symbols
                                     for dependencies.
  * tests/frontend/show.py: Add tests for new format symbols.

126bc006
by Chandan Singh at 2019-02-12T10:23:44Z
```
Re-generate man pages
```

dbce3434

by Chandan Singh at 2019-02-12T12:19:57Z

Merge branch 'chandan/deps' into 'master'

_frontend: Allow printing dependencies using `bst show`

Closes #890

See merge request BuildStream/buildstream!1121

24d29a17

by Javier Jardón at 2019-02-12T12:27:42Z

.gitlab-ci.yml: Add job to run test in current Fedora:29

80dcf65d

by Javier Jardón at 2019-02-12T12:27:42Z

.gitlab-ci.yml: run all generic fedora test in current stable version:29

58b84905

by Javier Jardón at 2019-02-12T12:27:42Z

.gitlab-ci.yml: Remove "tests-python-3.7-stretch" job

Fedora 29 includes python 3.7 so this is not needed anymore

95d9b9ae

by Javier Jardón at 2019-02-12T13:31:56Z

Merge branch 'jjardon/fedora_29' into 'master'

.gitlab-ci.yml: Test with current fedora release: 29

See merge request BuildStream/buildstream!1137

a39c4767

by Daniel Silverstone at 2019-02-12T15:12:30Z

_yaml.py: Only retrieve provenance in node_get() when needed

We were indiscriminately retrieving the node's provenance data in the
`node_get()` function which was accounting for approximately a third of
the total runtime of `node_get()` which dominates pre-scheduler time in
`bst build`.  This change ameliorates that situation by only retrieving
the provenance data when it's actually needed.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

cda03313

by Daniel Silverstone at 2019-02-12T15:12:30Z

_yaml.py: Reduce use of `isinstance()` in `node_sanitize()`

We know that nodes are typically one of:
str, list, dict, bool, tuple, NoneType or our ChainMap
Of these, dict and ChainMap are Mapping, only list is list
and the rest are returned unchanged.  We can reduce/defer our use
of isinstance here, dramatically, improving performance.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

b6f08e1b

by Daniel Silverstone at 2019-02-12T15:12:30Z

_yaml.py: Remove use of isinstance() in `{node,list}_{chain_,}copy`

A non-trivial proportion of time pre-scheduler in `bst build` is spent
copying (or chain-copying) nodes.  Approximately a quarter of the time
spent in that effort is in `isinstance()`.  This removes that CPU load.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

69ee11c6

by Daniel Silverstone at 2019-02-12T15:12:30Z

_yaml.py: Reduce cost of node_final_assertions

By re-using the isinstance replacements from earlier commits and
using a tuple of the string constants for checking for composition
markers, we reduce the cost of node_final_assertions by two thirds
in basic testing.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

0928e570

by Daniel Silverstone at 2019-02-12T15:12:30Z

_context.py: Cache result of get_strict()

While get_strict() doesn't look expensive per-se, it is called so many
times that it is valuable to cache the result once computed.  Since I
don't think it can change once it is computable, cache it immediately
that becomes possible and we save 20s in my test case.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

022a59f0

by Benjamin Schubert at 2019-02-12T16:14:05Z

Merge branch 'danielsilverstone-ct/further-optimisations' into 'master'

Further optimisations

See merge request BuildStream/buildstream!1131

352f4ad9

by Jonathan Maw at 2019-02-12T16:44:35Z

loader.py: Stage junctions into .bst instead of a tmpdir

62396af9

by Jonathan Maw at 2019-02-12T16:44:35Z

loader.py: Load junction from workspace if one's open

f1e9cb66

by Jonathan Maw at 2019-02-12T16:44:35Z

Remove cleanup functionality from loader

Since junction check-outs are now persistent, there is no need to clean
up temporary directories once finished.
Now, junction checkouts are stored within the top-level project dir and
will be cleaned-up when that directory is removed.

This involves changes in:
* loader.py: Remove cleanup logic and passing-around of tmpdir.
* _project.py: Remove passing around tmpdir and calling loader's
               cleanup.

e51116d5

by Jonathan Maw at 2019-02-12T17:59:25Z

Merge branch 'jonathan/junction-no-tmpdir' into 'master'

Stage junctions into .bst instead of a tmpdir

Closes #895

See merge request BuildStream/buildstream!1134

478e5c47

by Valentin David at 2019-02-12T18:06:50Z

buildstream/_cas/cascache.py: Set 0644 rights to pulled files

This was broken by 5ef19a0b31df84caed1e41719ef7ea5c6bd8a8bc.

6de65306

by Valentin David at 2019-02-12T19:13:43Z

Merge branch 'valentindavid/pull-chmod-bug' into 'master'

buildstream/_cas/cascache.py: Set 0644 rights to pulled files

See merge request BuildStream/buildstream!1144

9b140fa0