[Notes] [Git][BuildStream/buildstream][valentindavid/absolute-whitelist]

Valentin David pushed to branch valentindavid/absolute-whitelist at BuildStream / buildstream

Commits:

5c208698

by Jim MacArthur at 2018-11-26T10:23:31Z

Docs: Add remote execution architecture documentation.

5af455a4

by Jim MacArthur at 2018-11-26T10:55:22Z

Merge branch 'jmac/execution-architecture-docs' into 'master'

Docs: Add remote execution architecture documentation.

See merge request BuildStream/buildstream!923

4ca892e3

by Jim MacArthur at 2018-11-27T09:30:13Z

tests: Avoid hangs due to exceptions in subprocesses

d09a18b8

by Jim MacArthur at 2018-11-27T10:31:07Z

Merge branch 'jmac/fix-test-hangs-2' into 'master'

Avoid hanging artifact cache tests

See merge request BuildStream/buildstream!964

df60258a

by Jürg Billeter at 2018-11-27T12:11:24Z

source.py: Add optional _get_local_path() method

42809933
by Jürg Billeter at 2018-11-27T12:11:24Z
```
local.py: Implement _get_local_path()
```

eb5b10f7

by Jürg Billeter at 2018-11-27T12:11:24Z

_loader.py: Use Source._get_local_path() for subprojects

This eliminates unneeded copies of the project directory for junctions
with a single local source.

b2ea208c

by Jürg Billeter at 2018-11-27T12:50:46Z

Merge branch 'juerg/local-junctions' into 'master'

Optimization for local junctions

See merge request BuildStream/buildstream!290

223c0073

by Jürg Billeter at 2018-11-27T13:41:09Z

tests/integration/manual.py: Add test for command logging

38d16c62

by Jürg Billeter at 2018-11-27T13:41:09Z

sandbox: Deduplicate code to process cwd, env, and command arguments

85f5b574

by Jürg Billeter at 2018-11-27T13:41:09Z

sandbox/sandbox.py: Add SandboxFlags.NONE

This improves readability.

bdd6925f

by Jürg Billeter at 2018-11-27T13:41:09Z

element.py: Use SandboxFlags.NONE instead of 0

69005c76

by Jürg Billeter at 2018-11-27T13:41:09Z

scriptelement.py: Use SandboxFlags.NONE instead of 0

f1767de2

by Jürg Billeter at 2018-11-27T13:41:09Z

sandbox/sandbox.py: Add command batching API

This adds the batch() context manager.

87f7c99a

by Jürg Billeter at 2018-11-27T13:41:09Z

element.py: Pass element object to sandbox

This allows the sandbox to output messages using the plugin id.

024d584d

by Jürg Billeter at 2018-11-27T13:41:09Z

element.py: Cache artifact also on SandboxCommandError

bb7c3213

by Jürg Billeter at 2018-11-27T13:41:09Z

element.py: Defer workspace.prepared setting in case of pending commands

04f05b1a

by Jürg Billeter at 2018-11-27T13:41:09Z

element.py: Support command batching for integration commands

7827ac50

by Jürg Billeter at 2018-11-27T13:44:49Z

element.py: Add batch_prepare_assemble() method

This allows batching of all sandbox commands aross prepare() and
assemble().

63db90b2

by Jürg Billeter at 2018-11-27T13:44:49Z

buildelement.py: Support batching for integration and build commands

73dec86e

by Jürg Billeter at 2018-11-27T13:44:49Z

scriptelement.py: Support batching for integration and build commands

79442e0a

by Jürg Billeter at 2018-11-27T13:44:49Z

compose.py: Support batching for integration commands

6018b3bd

by Jürg Billeter at 2018-11-27T13:44:49Z

sandbox/_sandboxremote.py: Implement command batching

Execute batched commands in a single shell script.

ec3d4486
by Jürg Billeter at 2018-11-27T13:44:49Z
```
NEWS: Add entry for command batching
```

127d332f

by Jürg Billeter at 2018-11-27T14:16:50Z

Merge branch 'juerg/command-batching' into 'master'

Command batching

Closes #675

See merge request BuildStream/buildstream!915

d5b5538d

by Jim MacArthur at 2018-11-27T15:32:19Z

artifactcache: Move ArtifactCacheSpec code into CASRemoteSpec.

There is nothing in ArtifactCacheSpec that's actually specific to
artifacts, so I've made it a CAS class so we can use the same
spec for remote execution.

f224d4cb

by Jim MacArthur at 2018-11-27T15:32:19Z

artifactcache.py, _context.py: Move CASCache object into context

Since the artifact cache and remote execution share the same
local CAS store, they should share the same CASCache object. Moving
this into context allows us to do this.

926c1446

by Jim MacArthur at 2018-11-27T15:33:34Z

_sandboxremote: Remove references to artifact cache.

Uses the new config options introduced earlier to specify remote
exec storage and execution service.

d7cdb03e

by Jim MacArthur at 2018-11-27T15:33:36Z

Documentation: Update to show new config options

95d8244f

by Jim MacArthur at 2018-11-27T15:33:36Z

tests/sandboxes/remote-exec-config.py: New test.

Tests that we get a sensible error message when we specify incomplete
or wrong information in the configuration for remote execution.

7f2f62eb

by Jim MacArthur at 2018-11-27T15:33:36Z

artifactcache.py: Remove unused verify_digest_pushed

c55ca3a5

by Jim MacArthur at 2018-11-27T15:33:36Z

Docs: Correct URLs in artifact cache example

The port number must come after the hostname and before the path.
The path is ignored by CASCache anyway, so including it is misleading.

2a8a3b19

by Jim MacArthur at 2018-11-27T18:17:03Z

Merge branch 'jmac/remote_execution_split' into 'master'

Split remote execution from artifact cache

Closes #750

See merge request BuildStream/buildstream!946

499c70fd

by Valentin David at 2018-11-28T09:11:21Z

setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52

3513580c

by Valentin David at 2018-11-28T14:02:32Z

Merge branch 'valentindavid/ruamel_0_15_41' into 'master'

setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52

See merge request BuildStream/buildstream!975

26cdee08

by Valentin David at 2018-11-28T14:29:52Z

Use f_bavail to query available space. Not f_bfree.

f_bfree space might not be usable. In practice we see failures in big
disks because f_bfree is over 2GB and f_bavail is 0. We get ENOSPC if
writing on disk then.

58ca298f

by Valentin David at 2018-11-28T14:29:52Z

Make cache clients not fail when a blob is not available.

We plan to make cache incomplete. That is some blobs are missing.  For
most of cases we will delete references when requested if they are
incomplete. But there will be corner cases where objects are removed
after the reference is requested.

227fa26d
by Valentin David at 2018-11-28T14:29:52Z
```
Fix type error in RPC messages
```

5ef19a0b

by Valentin David at 2018-11-28T14:29:52Z

Avoid copying temporary file when adding object to CAS in server.

The file is already a temporary file and does not need copy.  ENOSPC
is thrown during that copy in issue #609.

Fixes #678.

8d2946ff

by Valentin David at 2018-11-28T14:29:52Z

"Fallocate" object disk space to avoid getting NOSPC error when writing

This locks the temporary object file so that cleanup does not need to
be done for every write.

b587953a

by Valentin David at 2018-11-28T14:29:52Z

Update mtimes of objects for requested references.

This also remove references when some objects are missing. This is in
preparation for the move from reference to object garbage collection.

a64f667d

by Valentin David at 2018-11-28T14:29:52Z

Move cas server from ref-based to object-based garbage collection.

353b90dd

by Valentin David at 2018-11-28T14:29:52Z

Cleanup cache in cas server more agressively

When there is less than 2GB left, it cleans up have 10GB available.
These values are configurable.

ba9afa98

by Valentin David at 2018-11-28T14:29:52Z

Lock cache cleanup in cas server

Cleaning up in parallel might slow down the cleaning process

9a458402

by Valentin David at 2018-11-29T08:47:40Z

Merge branch 'valentindavid/cache_server_fill_up' into 'master'

Fix cleanup of cache in server when disk is full

Closes #678

See merge request BuildStream/buildstream!830

47b7a9ba

by Jim MacArthur at 2018-11-29T09:58:48Z

_sandboxremote.py: Remove unnecessary tests.

push_message and push_directory will both raise assertion failures if
they fail to send the digest to the server. Checking the digest
returned by each only tests that the content was hashed locally, which
cannot reasonably fail.

4a8d0565

by Jim MacArthur at 2018-11-29T10:42:31Z

Merge branch 'jmac/no-verify-digests' into 'master'

_sandboxremote.py: Remove unnecessary tests.

Closes #786

See merge request BuildStream/buildstream!976

2e78e0d1

by Valentin David at 2018-11-29T13:18:17Z

Add support for .netrc in remote/tar/zip plugins

Fixes #723.

a6144100

by Valentin David at 2018-11-29T13:18:17Z

.gitlab-ci.yml: Update image versions where forgotten

f6c184f5

by Valentin David at 2018-11-29T16:15:56Z

Merge branch 'valentindavid/netrc' into 'master'

Add support for .netrc in remote/tar/zip plugins

Closes #723

See merge request BuildStream/buildstream!908

1f31aefc

by Angelos Evripiotis at 2018-11-30T09:34:38Z

man/: update with changes since Apr 2018

It seems that the regenerated man pages haven't been committed for some
time. Add them all in one lump now.

Had to manually strip the version number from the generated files.

Had to manually fix the NAME field of bst.1 to not be 'bst - None'.

Oddly had to invoke click-man directly in order to get the changes to
be applied, the usual setup.py route mentioned in contributing.rst
didn't work.

Eyeballed the result of invoking man on each of the commands, seems
good.

033a5ad9

by Jürg Billeter at 2018-11-30T10:09:11Z

Merge branch 'aevri/update_man' into 'master'

man/: update with changes since Apr 2018

See merge request BuildStream/buildstream!981

1123b9a1

by Tom Pollard at 2018-11-30T13:30:35Z

_stream.py: Ability to pull missing buildtrees outside of pull/build

Adds helper function _buildtree_pull_required() to determine if a
pullqueue should be constructed, for commands outside of bst pull
and build where it is determined that an element's buildtree
artifact is to be required given the respective semantics and
config. Utilised in push() to attempt to mitigate skipping the push
of partial elements without the user having to have preceded it with
an explicit pull.

cli.py: Add new behaviour to push command description

element.py: Move _cached_buildtree() to be non local private method,
use _KeyStrength types to reduce duplication.

tests/integration/pullbuildtrees.py also updated to cover this
use-case.

ad293ed3

by Jürg Billeter at 2018-11-30T14:10:06Z

Merge branch 'tpollard/774' into 'master'

_stream.py: Ability to pull missing buildtrees outside of pull/build

Closes #774

See merge request BuildStream/buildstream!978

0d634c21
by Phillip Smyth at 2018-11-30T14:16:36Z
```
plugin.py: Add BAD_ELEMENT_SUFFIX warning
```

d97ef822

by Phillip Smyth at 2018-11-30T14:16:36Z

_project.py: Check for fatal_warnings earlier

59d442e5

by Phillip Smyth at 2018-11-30T15:41:57Z

loader: Added file suffix check and helper function

_loader/loader.py: Added Suffix check and warning for named elements and add helper function

2d6ec07d

by Phillip Smyth at 2018-11-30T15:41:57Z

cli.py: no autocomplete on files without .bst

ac0bc4c0

by Phillip Smyth at 2018-11-30T16:50:35Z

Added tests for .bst suffix and completions

buildcheckout.py: Add mandatory .bst suffix tests
completions.py: Add test for fail on invalid suffix
Added required files for testing
integration/source-determinism.py: renamed test elements to end with .bst
loader/__init__.py: initialised context properly with load and a message handler

d4bc662a

by Phillip Smyth at 2018-11-30T16:50:50Z

NEWS: Added update about the Mandatory .bst change

7cea464a

by Jürg Billeter at 2018-11-30T17:21:37Z

Merge branch 'mandatory_suffix' into 'master'

Mandatory .bst suffix

See merge request BuildStream/buildstream!967

cc5ef68c

by Tristan Van Berkom at 2018-12-03T08:41:42Z

MANIFEST.in: Include the sources for rebuilding the documentation session HTML

This was previously missing, causing our CI to forget to rebuild the sessions
when updating the documentation.

This fixes issue #778

87ba19d7

by Tristan Van Berkom at 2018-12-03T09:19:54Z

Merge branch 'tristan/fix-doc-session-rebuilds' into 'master'

MANIFEST.in: Include the sources for rebuilding the documentation session HTML

Closes #778

See merge request BuildStream/buildstream!984

1b24148f

by Chandan Singh at 2018-12-03T09:46:13Z

Ensure `--deps=none` option works for `bst checkout`

Currently, `bst checkout --deps none` command always produces empty
output. Fix this issue and add regression test for the same.

- element_enums.py: Add Scope.NONE.

- element.py: Ensure Scope.NONE works correctly in addition to
  Scope.RUN/Scope.ALL in Element.dependencies() and Element.search().

- tests/frontend/buildcheckout.py: Fix tests for `--deps none`.

Fixes #670.

8ac1e757

by Chandan Singh at 2018-12-03T09:46:14Z

Add `--deps build` option to `bst checkout`

As discussed in
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
add `--deps build` option to `bst checkout`. This will allow users to
checkout the all build dependencies of a given element using a single
command.

- _frontend/cli.py: Add `--deps build` option for `bst checkout`.

- element.py: Support `deps='build'` in Element._prepare_sandbox().

- tests/frontend/buildcheckout.py: Ensure `--deps build` works as
  expected.

ac60f47a

by Benjamin Schubert at 2018-12-03T09:46:14Z

Refactor and simplify _prepare_sandbox for elements

Before we would have a intricate logics with multiple arguments
that might get ignored.

This simplifies the design and introduces a bool `shell` instead
of having two different variables concerned about scope

b1430fb5

by Tristan Van Berkom at 2018-12-03T11:01:54Z

Merge branch 'chandan/bst-checkout-build' into 'master'

Add `--deps build` option to `bst checkout`

Closes #670

See merge request BuildStream/buildstream!819

9022a778

by Javier Jardón at 2018-12-03T11:14:55Z

buildstream/plugins/sources/git.py: warn about checkout by default

See #783

4462892e

by Tristan Van Berkom at 2018-12-03T11:45:07Z

Merge branch 'jjardon/doc_git_submodules' into 'master'

git: warn about checkout submodules  by default

See merge request BuildStream/buildstream!983

6fcfa452

by Jonathan Maw at 2018-12-03T13:21:05Z

_yamlcache.py: Use a project's junction name if present

This is required because if there are projects with the same name
loaded, that have elements in the same path, with the same contents,
they are considered the same, despite actually belonging to different
Projects.

This fixes buildstream issue #795

d0f9c724

by James Ennis at 2018-12-03T13:21:05Z

tests/loader/junctions.py: Test that we can build a junction that is referred differently, twice.

Test required to close #795 - We should be able to build a project which refers
to the same junction twice, but with two different filenames.

f7581026

by Jürg Billeter at 2018-12-03T16:27:03Z

Merge branch 'jonathan/fix-identical-element' into 'master'

_yamlcache.py: Use a project's junction name if present

Closes #795

See merge request BuildStream/buildstream!980

98c15463

by Tristan Van Berkom at 2018-12-05T07:04:56Z

_scheduler/queues: Don't call update state outside of error handling harness

Commit 3fa79d8da, part of an initiative for caching of the failed builds,
introduced a call to Element._update_state() after a job completes and before
entering the error handling harness intended for handling plugin raised errors.

Element._update_state() can result in triggering plugin code to
run, so this is incorrect, and causes raised errors to crash BuildStream
if they happen here.

After analyzing the code, it appears that this additional call to
Element._update_state() is unneeded, and was only added because the
state needs to be updated for a failure as well as a success.

Instead, we now have the BuildQueue call Element._assemble_done()
unconditionally, regardless of whether the build was successful
or not, which has the same effect and also reads better.

In addition, added a FIXME comment that we are still conditionally
updating the artifact cache size from BuildQueue.done() only
if the build is successful, which is incorrect because failed builds
also increase the local artifact cache size - to fix this we need
to communicate the added artifact size through Element._assemble()
regardless of whether the build succeeded or failed.

5d77b871

by Tristan Van Berkom at 2018-12-05T07:37:54Z

Merge branch 'tristan/refactor-queues-update-state' into 'master'

_scheduler/queues/queue.py: Don't call update state outside of error handling harness

See merge request BuildStream/buildstream!988

82513109

by Valentin David at 2018-12-05T09:47:52Z

git source plugin: Track git tags and save them to reproduce a minimum shallow repository

Instead of tag information being fetched which can change with time,
they are tracked and saved in the projects.refs/.bst. Then we re-tag
automatically the closest tag so that `git describe` works and is
reproducible.

This new feature is opt-in with the new `track-tags` configuration,
and must be used to fix modules which are broken by our new policy
of omitting the `.git/` repository when staging git sources.

This fixes issue #487

ec7b62c3

by Valentin David at 2018-12-05T09:50:15Z

NEWS: Add news about broken 'git describe' by default.

761e7570

by Tristan Van Berkom at 2018-12-05T10:23:49Z

Merge branch 'valentindavid/git_describe_tracking' into 'master'

Track of git tags and save them to reproduce minimum shallow repository

Closes #487

See merge request BuildStream/buildstream!906

2f5966bd

by Raoul Hidalgo Charman at 2018-12-05T10:37:10Z

platform.py: Add mapping of uname to OS-independent values

1ad35fcd

by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

_config.py: Use os and architecture settings

In element, platform asks for host os and architecture to default to when
SandboxConfig is initialised.
This changes element cache keys so those have been updated in the tests.

9f75c274

by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

optionarch.py: update to use same arch names as SandboxConfig

Also update tests to be consistent with this

a825096c

by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

_sandboxremote.py: Add os and arch to command uploaded

These are contained within the platform field.

8e09a81c
by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z
```
optionos.py: Add new option for OS configuration
```

defec112

by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

optionos.py: Add tests for OptionOS

Same format as OptionArch tests, but with OS's.

e5d8c7d8
by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z
```
_platform: Add checks for os and architecture
```

b0603fb0

by Raoul Hidalgo Charman at 2018-12-05T10:44:02Z

Sandbox: use linux32 for x86-32 builds on x86-64 machines

  o _platform/linux.py: Add linux32 flag to send to sandbox bwrap when the
    build arch is x86-32 and the machines arch is x86-64 or similarly with
    aarch32 and aarch64.
  o sandbox/_sandboxbwrap.py: Use flag passed to start bwrap command with
    linux32 if set.

f4d76996

by Raoul Hidalgo Charman at 2018-12-05T14:58:49Z

format_project.rst: Update documentation for changes to OptionArch

81163a79

by Raoul Hidalgo Charman at 2018-12-05T14:58:53Z

format_declaring.rst: Update documentation for sandbox options

23c27580
by Raoul Hidalgo Charman at 2018-12-05T14:58:53Z
```
format_project.rst: Add docs for OptionOS
```

eb1ed410

by Jürg Billeter at 2018-12-05T15:28:52Z

Merge branch 'raoul/775-execution-environment-reqs' into 'master'

Execution environment reqs

Closes #775

See merge request BuildStream/buildstream!969

9ef1a8a2

by Raoul Hidalgo Charman at 2018-12-05T15:32:22Z

utils.py: Add message digest function

Get cascache.py to use this

4a56bf2a

by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

_sandboxremote.py: Add checks to action cache before attempting to push.

Stops unneccesary pushing of builds that have already been built, just checks
the action cache to begin with.

Fixes #628

3641b6f6

by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

_sandboxremote.py: Add action-cache-service endpoint

The endpoint may not necessarily be the same as execution-service endpoint.

Part of #628

9d77351f

by Raoul Hidalgo Charman at 2018-12-05T15:33:44Z

format_project.rst: Add docs for action-cache-service

Part of #628

a53d6d1f

by Raoul Hidalgo Charman at 2018-12-05T16:36:00Z

Merge branch 'raoul/628-RE-flow-optimisation' into 'master'

Remote-execution client flow optimisation

Closes #628

See merge request BuildStream/buildstream!982

4688b2c9

by Tristan Van Berkom at 2018-12-06T07:35:59Z

tests/sources/git.py: Refactor ref-not-in-track test to use parameterization

Instead of duplicating the whole test body, run it twice while
checking for a warning or an error depending on the parameter.

03028c7f

by Tristan Van Berkom at 2018-12-06T07:35:59Z

types.py: Moving CoreWarnings to the types.py module

This also ensures it is exposed via the main buildstream __init__.py
file, fixes the imports of CoreWarnings, and adjusts the documentation
links to point to the right place.

3b7f8df8

by Tristan Van Berkom at 2018-12-06T07:35:59Z

Source: Fix typo in documentation, and make it a link to `bst track`

Now the commands have documentation linkability so we can link to
them when referring to them anywhere in the docs. This reference
was previously referring to a bogus non-existent `build-stream track`,
maybe this dates way back to the days before we named the frontend `bst`.

e5c0f067

by Tristan Van Berkom at 2018-12-06T07:35:59Z

git source plugin: Fixing documentation linking and typo

Fixes the configurable warning part of this to properly link
to the project.conf documentation describing what configurable
warnings actually are.

Also fix the plugin raise the actual configurable warning which
it advertizes that it raises; which is 'inconsistent-submodule',
not 'inconsistent-submodules'.

205c27d8

by Tristan Van Berkom at 2018-12-06T07:35:59Z

git source plugin: Rename symbol for consistency and readability

Change 'INCONSISTENT_SUBMODULE' for 'WARN_INCONSISTENT_SUBMODULE' to
make things a bit more readable and comprehensive.

bea4d4f5

by Tristan Van Berkom at 2018-12-06T09:37:50Z

Merge branch 'tristan/misc-cleanup' into 'master'

Misc cleanups

See merge request BuildStream/buildstream!993

b1d2f001

by Tristan Van Berkom at 2018-12-06T14:18:31Z

source.py: Add new delegate method validate_cache()

This is guaranteed to be called only once for a given session once the
sources are known to be Consistency.CACHED, if source tracking is enabled
in the session for this source, then this will only be called if the
sources become cached after tracking completes.

ee7fc47f

by Tristan Van Berkom at 2018-12-06T14:18:31Z

git source plugin: Implementing submodule warnings

 o Unlisted submodule warning

   Now the git plugin will issue a configurable warning if a submodule
   exists and is used (checking out the submodule is not disabled),
   but is not specified in the source configuration.

 o Invalid submodule warning

   Now the git source plugin will issue a warning if the configuration
   specified a submodule which does not exist in the underlying git repository.

As a side effect, this patch also changes the flow control of the git
plugin such that submodules which are explicitly set to not be
checked out, are also not fetched but instead ignored completely.

5fc9a1da

by Tristan Van Berkom at 2018-12-06T14:18:31Z

git source plugin: Emmit the ref-not-in-track warning from validate_cache()

Now that we have Source.validate_cache(), this is a better place to emmit
the ref-not-in-track warning, since it will be emmitted at the earliest
opportunity and not only at Source.stage() or Source.init_workspace().

This also allows us to remove the `self.tracked` local state, and cleanup
some convoluted calling paths, removing some unnecessary parameters from
the usual codepaths and making the plugin overall more readable.

50cb2706

by Tristan Van Berkom at 2018-12-06T14:18:31Z

tests/testutils/repo/git.py: Adding remove_path() helper

A function for removing paths from a git repo, can also
be used to remove submodules.

f139ca8f

by Tristan Van Berkom at 2018-12-06T14:18:31Z

tests/sources/git.py: Testing the git:unlisted-submodule warning

 o Test that it is not triggered in show before fetch, because we
   don't know about the unlisted submodules yet

 o Test that it is triggered by a fetch command

 o Test that it is triggered by `show` after having completed a
   fetch command, since now we have the repository and know about
   the unlisted submodule

 o Test all of this under warning or error conditions (parameterized
   for fatal-warnings)

afe1aa9f

by Tristan Van Berkom at 2018-12-06T14:18:31Z

tests/sources/git.py: Testing the git:invalid-submodule warning

   o Test that it is not triggered in show before fetch, because we
     don't know the submodules yet so we cannot know if they are
     valid or not.

   o Test that it is triggered by a fetch command

   o Test that it is triggered by `show` after having completed a
     fetch command, since now we have the repository and know which
     specified submodules are invalid

   o Test all of this under warning or error conditions (parameterized
     for fatal-warnings)

57300201

by Tristan Van Berkom at 2018-12-06T14:18:31Z

tests/sources/git.py: Test unlisted submodules warning appearing after track

a0297625

by Tristan Van Berkom at 2018-12-06T14:18:31Z

tests/sources/git.py: Test invalid submodules warning appearing after track

60e96781

by Tristan Van Berkom at 2018-12-06T15:14:37Z

Merge branch 'tristan/submodule-warnings' into 'master'

Implement submodule warnings

See merge request BuildStream/buildstream!996

7fe55143

by Jonathan Maw at 2018-12-06T15:48:12Z

Fix bst source-checkout not working with open workspaces

2a6879a5

by Jonathan Maw at 2018-12-06T17:46:33Z

Merge branch 'jonathan/source-checkout-workspace' into 'master'

Fix bst source-checkout not working with open workspaces

Closes #807

See merge request BuildStream/buildstream!997

e61e6e50

by Tristan Van Berkom at 2018-12-07T07:57:13Z

_yaml.py: Added `allow_none` parameter to _yaml.node_get()

This allows specifying whether or not the code believes a None
value is acceptable if explicitly expressed to be None in the YAML.

A previous patch 3ba544b80f9f268be8ffe62fc8589b30212ec4a2 changed
the behavior to accept None for all values across the board, along
with allowing explicitly setting the `default_value` to `None` which
caused the code to be much more readable (this was the main motivation
of the patch, but it had the side effect of allowing None for everything).

In the majority of cases we load YAML however, it either has a value or
it is not specified in the YAML, and None is hardly ever acceptable
to be explicitly specified (it may be in the case you want to override
something with nothing in the YAML, but it is the edge case).

This had the side effect of causing unexpected crashes where the
YAML specifies an empty dictionary for instance.

Instead of forcing the caller to handle a possible None value, give
the choice to the caller if they really want to accept a None value,
and raise the LoadError otherwise.

This fixes issue #803, which is a crash due to receival of an
unexpected None value through _yaml.node_get()

504ecb3e

by Tristan Van Berkom at 2018-12-07T07:57:56Z

plugin.py: Added new `allow_none` parameter to Plugin.node_get_member(), defaulting to False.

Allow plugins to conveniently leverage the new `allow_none` parameter,
and have Plugin.node_get_member() raise a LoadError in the usual
case where None is not an acceptable value to be explicitly set in
the user provided YAML.

642ae4e8

by Tristan Van Berkom at 2018-12-07T07:57:56Z

tests/format/project.py: Added new regression test for empty dependency dictionaries

This adds a regression test for issue #803

2a0676c3

by Tristan Van Berkom at 2018-12-07T08:33:14Z

Merge branch 'tristan/yaml-optionally-allow-none' into 'master'

Only optionally allow None values in user provided YAML

Closes #803

See merge request BuildStream/buildstream!999

56c2c2cc

by Javier Jardón at 2018-12-09T16:59:26Z

buildstream/data/projectconfig.yaml: Remove default strip-binaries

They are too specific to be included by default

Recommendation is if you are building in Linux is to use the
ones begin used in the freedesktop-sdk project, for example

See #645

dcb6b5e8

by Javier Jardón at 2018-12-09T16:59:26Z

NEWS: Update for removal of default strip-commands

ae0afce5

by Javier Jardón at 2018-12-09T17:08:16Z

buildstream/buildelement.py: Document where to find strip commands for Linux

873b618c

by Javier Jardón at 2018-12-10T09:49:52Z

Merge branch 'jjardon/strip-binaries-removal' into 'master'

Remove default strip-commands

Closes #645

See merge request BuildStream/buildstream!987

f2fcc2f6

by Valentin David at 2018-12-11T12:12:08Z

Force updating tags when fetching git repository

When using aliases there are multiple remotes used in the cache
repository. When fetching, tags are not updated if the were previously
fetched from a different remote. Commits that not in a branch and only
tagged do not get fetched if the tag is not fetched.

Fixes #812

717c10d1

by Valentin David at 2018-12-11T12:44:56Z

Merge branch 'valentindavid/git_force_fetch_tags' into 'master'

Force updating tags when fetching git repository

Closes #812

See merge request BuildStream/buildstream!1000

7892287a

by Jonathan Maw at 2018-12-11T12:56:32Z

utils.py: Add a helper for searching upwards for files

i.e. with a given directory and filename, check parent directories until
either a directory with the filename is found, or you reach the root of
the filesystem.

This is a part of #222

67c7a58d

by Jonathan Maw at 2018-12-11T12:56:32Z

Create and store data inside projects when opening workspaces

Changes to _context.py:
* Context has been extended to contain a WorkspaceProjectCache, as there
  are times when we want to use it before a Workspaces can be initialised
  (looking up a WorkspaceProject to find the directory that the project is
  in)

Changes to _stream.py:
* Removed staging the elements from workspace_open() and workspace_reset()

Changes in _workspaces.py:
* A new WorkspaceProject contains all the information needed to refer back
  to a project from its workspace (currently this is the project path and
  the element used to create this workspace)
* This is stored within a new WorkspaceProjectCache object, which keeps
  WorkspaceProjects around so they don't need to be loaded from disk
  repeatedly.
* Workspaces has been extended to contain the WorkspaceProjectCache, and
  will use it when opening and closing workspaces.
* Workspaces.create_workspace has been extended to handle the staging of
  the element into the workspace, in addition to creating the equivalent
  WorkspaceProject file.

This is a part of #222

64836b18

by Jonathan Maw at 2018-12-11T12:56:32Z

_project.py: Rename _ensure_project_dir to _find_project_dir

This is a part of #222

7deeb2c3

by Jonathan Maw at 2018-12-11T12:56:32Z

cli.py: Use utils' search upwards helper when searching for project.conf

This is a part of #222

496f0ab7

by Jonathan Maw at 2018-12-11T12:56:32Z

_project.py: Find project from workspace if outside of a project

This is a part of #222

921f2bcb

by Jonathan Maw at 2018-12-11T12:56:32Z

_workspaces.py: Do not include .bstproject.yaml in the cache key

This is a part of #222

f145a3e4

by Jonathan Maw at 2018-12-11T12:56:32Z

cli: Interactively warn if the user is trying to close the workspace they're using to load the project

This involves changes in:
* _stream.py:
  * Add the helper Stream.workspace_is_required()
* userconfig.yaml:
  * Add a default value for prompt.really-workspace-close-project-inaccessible
* _context.py:
  * Load the prompt 'really-workspace-close-project-inaccessible' from
    user config.
* cli.py:
  * If buildstream is invoked interactively, prompt the user to confirm
    that they want to close the workspace they're using to load this
    project.

This is a part of #222

494cb7c6

by Jonathan Maw at 2018-12-11T14:12:54Z

tests: Test bst commands from an external workspace

This is a part of #222

fd1c5c5a

by Jonathan Maw at 2018-12-11T14:12:54Z

NEWS: Add an entry for being able to run commands from a workspace

This is a part of #222

4219a6b4

by Jonathan Maw at 2018-12-11T16:00:55Z

Merge branch 'jonathan/workspace-fragment-create' into 'master'

Support invoking buildstream from a workspace outside a project

See merge request BuildStream/buildstream!924

e564251b

by Jonathan Maw at 2018-12-11T16:15:19Z

_stream.py: Fix ugly error when opening a workspace using a relative path

7cf83343

by Jonathan Maw at 2018-12-11T16:15:19Z

_context.py: Store the directory buildstream was invoked on

a1dee91e

by Jonathan Maw at 2018-12-11T16:15:19Z

Make specifying elements optional in bst commands

Known issues:
* `bst shell` works, but `bst shell COMMANDS...` doesn't, because click
  has no way of separating optional args from variable-length args.
* `bst checkout` and `bst source-checkout`'s usage strings mark LOCATION
  as an optional argument. Because click gets confused if there's an
  optional argument before a mandatory argument, I had to mark LOCATION
  as optional internally.
* `bst workspace open` makes no sense with element being optional, so
  I skipped it.
* `bst workspace close` will probably need to be revisited when multiple
  projects can own one workspace.
* `bst workspace reset` will happily delete the directory you're
  currently in, requiring you to `cd $PWD` to see the contents of your
  directory.
  I could exclude the top-level directory of the workspace being
  deleted, but it is entirely valid to run workspace commands from deeper
  in the workspace.

This is a part of #222

9b192015

by Jonathan Maw at 2018-12-11T16:15:19Z

tests: Add tests for guessing element names

This is a part of #222

e340371f

by Jonathan Maw at 2018-12-11T16:15:19Z

Add NEWS for guessing the element in commands

This is a part of #222

bbf81c3d

by Jonathan Maw at 2018-12-11T16:15:19Z

doc: Point out that element is optional when commands are run from the workspace

This is a part of #222

a5a53ddd

by Jonathan Maw at 2018-12-12T13:44:09Z

Merge branch 'jonathan/workspace-fragment-guess-element' into 'master'

Guess the element when running commands in a workspace

See merge request BuildStream/buildstream!954

d55b9e39

by Phil Dawson at 2018-12-12T13:55:19Z

Add --tar option to source-checkout command

This commit is part of the work towards #672

733aab53

by Phil Dawson at 2018-12-12T13:55:19Z

Add --force / -f option to source-checkout command

c2efeba0

by Phil Dawson at 2018-12-12T14:43:40Z

Add option to source-checkout command to generate build scripts

85c61894

by Phil Dawson at 2018-12-12T14:43:40Z

Remove source bundle command

This is part of the work towards #672

a322d5c0

by Phil Dawson at 2018-12-12T14:43:40Z

NEWS: Add entry about the removal of source-bundle command

ec909605

by Phil Dawson at 2018-12-12T15:45:42Z

Merge branch 'phil/source-checkout-options' into 'master'

Retire bst source bundle command

Closes #672

See merge request BuildStream/buildstream!959

3697a611

by Richard Maw at 2018-12-12T16:31:38Z

cli: Add support for auto-completing artifact ref names

b3dceb16
by Richard Maw at 2018-12-12T16:32:41Z
```
cli: Add artifact command group
```
ba08a0cd
by Richard Maw at 2018-12-12T16:32:41Z
```
cli: Add artifact log command
```

70fb9554

by Richard Maw at 2018-12-12T16:32:41Z

tests: Add tests for artifact group commands

f773e746
by Richard Maw at 2018-12-12T16:33:02Z
```
NEWS: Mention `bst artifact log`
```

b6528441

by richardmaw-codethink at 2018-12-12T18:00:59Z

Merge branch 'richardmaw/artifact-log' into 'master'

Add artifact log command

See merge request BuildStream/buildstream!920

d03bf316

by Benjamin Schubert at 2018-12-13T10:24:11Z

Mock storage space checks for tests.

Fix #530

- Extract free space computation in a function for easier
mocking
- Mock space computation during cache quota tests
- Mock cache size during cachque quota tests
- Add two more tests when the configuration would require
to much storage space

a116f576

by Tristan Van Berkom at 2018-12-13T10:58:46Z

Merge branch 'BenjaminSchubert/fix-quota-tests' into 'master'

Mock storage space checks for tests.

Closes #530

See merge request BuildStream/buildstream!702

180fa774

by Benjamin Schubert at 2018-12-13T12:05:15Z

element.py: fix unbounded variable in nested python exception

As per https://docs.python.org/3.7/reference/compound_stmts.html#except
variables used in the "except" clause are erased at the end of the
block.

We had a conflict with a nested exception using the same variable as
before.

This renames the nested variable to avoid the clash

224aa4c2

by Benjamin Schubert at 2018-12-13T12:34:41Z

Merge branch 'bschubert/fix-unbound-variable-exception' into 'master'

element.py: fix unbounded variable in nested python exception

See merge request BuildStream/buildstream!1006

053beb66
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
manual plugin: Support virtual directories
```
29ab271c
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
makemaker plugin: Support virtual directories
```
ba955cf0
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
pip element plugin: Support virtual directories
```
6010b5a4
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
modulebuild plugin: Support virtual directories
```
3a6d27a4
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
distutils plugin: Support virtual directories
```

4c0e602c

by Tristan Van Berkom at 2018-12-13T14:23:19Z

BuildElement: Don't enable batching of prepare and assemble by default

Some external plugins depend derive from BuildElement and are broken
by BuildElement enabling this batching by default.

Instead, enable it in all of the individual build element plugin
implementations.

This fixes issue #800

60ddeeb9

by Tristan Van Berkom at 2018-12-13T14:58:28Z

Merge branch 'tristan/dont-batch-prepare-assemble-by-default' into 'master'

Dont batch prepare assemble by default

Closes #800

See merge request BuildStream/buildstream!1009

bf72cc42

by Angelos Evripiotis at 2018-12-13T17:31:17Z

contributing: mandate end-to-end tests, allow others

Update the section on adding tests, to fulfill these broad points:

- Mandate end-to-end testing, with rationale.
- Mention internal APIs as endpoints for testing.
- Warn against pitfalls of unit-testing.

It's more text than I would have liked, perhaps in later work we'll be
able to say it with less.

This change integrates feedback from the mailing list
'Guidance on Unit Tests' thread:
https://mail.gnome.org/archives/buildstream-list/2018-November/msg00045.html

13eb7ed2

by Angelos Evripiotis at 2018-12-13T18:01:56Z

Merge branch 'aevri/contributing_e2e_tests' into 'master'

contributing: more clarity on testing

See merge request BuildStream/buildstream!973

629a6e52

by Chandan Singh at 2018-12-14T19:34:20Z

Introduce new "source" command group

Following the message thread
https://mail.gnome.org/archives/buildstream-list/2018-November/msg00106.html,
implement a new command group called `source`. Move existing `track`,
`fetch`, and the recently added `source-checkout` commands under this
group.

For `track` and `fetch`, this is a BREAKING change, as the old commands
have been marked as obsolete. Using them will result in an error message
that refers people to use the new versions, like `bst source fetch`
instead of old `bst fetch`. `source-checkout` will now become
`source checkout` (the dash has turned into a space), and is not a
breaking change as it was added in the current development cycle.

Note that the functionality to hide commands from help output was added
only recently in Click, so the minimum version of Click that we now
require is 7.0.

Summary of changes:

* _frontend/cli.py: Add `source` command group, mark previous versions
  as obsolete and hide them from the help output.
* _frontend/complete.py: Fix completion for hidden commands.
* setup.py: Bump Click minimum version to 7.0.
* tests: Update to cope with the new command names.

Fixes #814.

f894c0a8

by Chandan Singh at 2018-12-14T19:34:20Z

NEWS: Add entry for new source command group

Also, change the existing entry about `source-checkout` to be
`source checkout` instead. And, while we are there, move it near the
other announcements about the source command group.

b23bec55

by Chandan Singh at 2018-12-14T20:07:13Z

Merge branch 'chandan/source-subgroup' into 'master'

Introduce new "source" command group

Closes #814

See merge request BuildStream/buildstream!1003

d2105909

by Raoul Hidalgo Charman at 2018-12-18T11:13:56Z

Add remote execution instance option

This is used when sending execution requests, to specify which instance of the
execution server to use.

Partial fix for #627.

a3bbec23

by Jim MacArthur at 2018-12-18T11:13:56Z

sandboxremote: Add server/storage config with defaults

Adds instance name support for the remote execution storage service.

89219f61

by Jim MacArthur at 2018-12-18T11:13:56Z

_cascache.py: Add instance names to GRPC calls

Adds the 'instance_name' parameter, which may be None, to most GRPC
calls in the CASCache object. ByteStream requests already have
instance_name supplied in the resource name, so do not need the
parameter.

Closes #627.

3dc20963

by Jim MacArthur at 2018-12-18T11:13:56Z

Documentation: Include instance-name in remote execution documentation

644d8b28

by Jim MacArthur at 2018-12-18T11:45:30Z

Merge branch 'raoul/627-RE-instance-config' into 'master'

Remote-execution instance configuration support

Closes #627

See merge request BuildStream/buildstream!952

e29aea36

by William Salmon at 2018-12-19T13:23:19Z

Basic options for shell --build to use buildtrees

Fixes issue #740

898a23a5

by Will Salmon at 2018-12-19T14:35:41Z

Merge branch 'willsalmon/shellBuildTrees' into 'master'

Shell --build has optional buildtrees

Closes #740

See merge request BuildStream/buildstream!986

a2f1d879
by Javier Jardón at 2018-12-19T15:36:11Z
```
README.rst: Add license badge
```

aae5e4b3

by Javier Jardón at 2018-12-19T16:09:48Z

Merge branch 'jjardon/license_badge' into 'master'

README.rst: Add license badge

See merge request BuildStream/buildstream!1014

2b767fe8

by Jürg Billeter at 2018-12-20T10:06:11Z

Move fetch logic from FetchQueue to Element

The queue shouldn't need to know about individual sources. This is in
line with _track() and _get_consistency().

7a102144

by Jürg Billeter at 2018-12-20T10:06:11Z

element.py: Do not call fetch() for cached sources

b325989e

by Jürg Billeter at 2018-12-20T10:07:20Z

tests/sources: Test that fetch() is not called for cached sources

77d8ad45

by Jürg Billeter at 2018-12-20T10:42:39Z

Merge branch 'juerg/fetch' into 'master'

Do not call fetch() for cached sources

See merge request BuildStream/buildstream!992

9835b7f1

by Chandan Singh at 2018-12-20T12:50:02Z

Expose base class for Git source plugins

Add a `_GitSourceBase` class to act as the base class for deriving
source plugins that work with Git. The first user of this base class is
the `git` source plugin that's in core at the moment.

The contents of this base class is almost identical to the existing `GitSource`
class. The two notable differences are:

- the private methods now have leading underscores, as some were missing
  it originally
- the name of the class

Note that we are exposing a private member here as we expect it to move to a
separate package soon. See the following discussion for more details:
https://gitlab.com/BuildStream/buildstream/issues/739#note_124819869

Fixes #739.

14da6955

by Chandan Singh at 2018-12-20T13:34:10Z

Merge branch 'chandan/abstract-git-source' into 'master'

Expose base class for Git source plugins

Closes #739

See merge request BuildStream/buildstream!1019

7368f569

by Angelos Evripiotis at 2018-12-20T13:59:23Z

_project::_find_project_dir: fix error message

Now that we are also looking for WORKSPACE_PROJECT_FILE, we should add
it to the error message. While we're there, also mention the directory
we are looking in - this could help scripters be clear about what went
wrong.

The new error message looks like this:

    Error loading project: None of ['project.conf', '.bstproject.yaml']
    found in '/src/temp/blah' or any of its parent directories

46efc91d

by Angelos Evripiotis at 2018-12-20T13:59:23Z

BREAK: remove auto-init behaviour

In the event that the project could not be found, stop BuildStream from
asking if the user would like to create a new project. Exit with error
instead, and give a hint to the user in case they're new.

As proposed on the mailing list here:
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00082.html

The new interaction looks like this:

    $ bst show nonsuch.bst
    No project found. You can create a new project like so:

        bst init

    Error loading project: None of ['project.conf', '.bstproject.yaml']
    found in '/src/temp/blah' or any of its parent directories

Fixes #826

e0c575c4

by Angelos Evripiotis at 2018-12-20T14:37:38Z

Merge branch 'aevri/rm-autoinit' into 'master'

BREAK: remove auto-init behaviour

Closes #826

See merge request BuildStream/buildstream!1015

c05d8b4f

by Chandan Singh at 2018-12-20T15:53:12Z

tests/frontend/logging.py: Fix regex Deprecation Warning

Use raw strings for regex searches, which is the preferred way to do
regular expressions in Python.

Without this patch, currently we get the following warnings:

```
tests/frontend/logging.py:44
  /builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:44: DeprecationWarning: invalid escape sequence \[
    m = re.search("\[\d\d:\d\d:\d\d\]\[\]\[\] SUCCESS Checking sources", result.stderr)

tests/frontend/logging.py:80
  /builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:80: DeprecationWarning: invalid escape sequence \d
    m = re.search("\d\d:\d\d:\d\d,\d\d:\d\d:\d\d.\d{6},\d\d:\d\d:\d\d,,,SUCCESS,Checking sources", result.stderr)
```

e8055a56

by Chandan Singh at 2018-12-20T15:53:12Z

buildstream/utils.py: Fix regex Deprecation Warning

Specify flags at the start of the _expression_ as per the recommendation
of the standard library.

Without this patch, we currently get the following warning:

```
tests/examples/junctions.py::test_open_cross_junction_workspace
  /builds/BuildStream/buildstream/dist/buildstream/buildstream/utils.py:213: DeprecationWarning: Flags not at the start of the _expression_ '\\/[^/]*\\Z(?ms)'
    regexer = re.compile(_expression_)
```

cd4889af

by Chandan Singh at 2018-12-20T16:34:08Z

Merge branch 'chandan/fix-warning' into 'master'

Fix Deprecation warnings from regex module

See merge request BuildStream/buildstream!1010

ac995236

by Tom Pollard at 2018-12-20T17:18:02Z

_stream.py: fix _buildtree_pull_required logic

c3153dea

by Tom Pollard at 2018-12-21T10:10:58Z

Merge branch 'tpollard/fixbuildtreereq' into 'master'

_stream.py: fix _buildtree_pull_required logic

See merge request BuildStream/buildstream!1021

caf49669