[Notes] [Git][BuildStream/buildstream][richardmaw/artifact-subcommands]

richardmaw-codethink pushed to branch richardmaw/artifact-subcommands at BuildStream / buildstream

Commits:

b6528441

by richardmaw-codethink at 2018-12-12T18:00:59Z

Merge branch 'richardmaw/artifact-log' into 'master'

Add artifact log command

See merge request BuildStream/buildstream!920

d03bf316

by Benjamin Schubert at 2018-12-13T10:24:11Z

Mock storage space checks for tests.

Fix #530

- Extract free space computation in a function for easier
mocking
- Mock space computation during cache quota tests
- Mock cache size during cachque quota tests
- Add two more tests when the configuration would require
to much storage space

a116f576

by Tristan Van Berkom at 2018-12-13T10:58:46Z

Merge branch 'BenjaminSchubert/fix-quota-tests' into 'master'

Mock storage space checks for tests.

Closes #530

See merge request BuildStream/buildstream!702

180fa774

by Benjamin Schubert at 2018-12-13T12:05:15Z

element.py: fix unbounded variable in nested python exception

As per https://docs.python.org/3.7/reference/compound_stmts.html#except
variables used in the "except" clause are erased at the end of the
block.

We had a conflict with a nested exception using the same variable as
before.

This renames the nested variable to avoid the clash

224aa4c2

by Benjamin Schubert at 2018-12-13T12:34:41Z

Merge branch 'bschubert/fix-unbound-variable-exception' into 'master'

element.py: fix unbounded variable in nested python exception

See merge request BuildStream/buildstream!1006

053beb66
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
manual plugin: Support virtual directories
```
29ab271c
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
makemaker plugin: Support virtual directories
```
ba955cf0
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
pip element plugin: Support virtual directories
```
6010b5a4
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
modulebuild plugin: Support virtual directories
```
3a6d27a4
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
distutils plugin: Support virtual directories
```

4c0e602c

by Tristan Van Berkom at 2018-12-13T14:23:19Z

BuildElement: Don't enable batching of prepare and assemble by default

Some external plugins depend derive from BuildElement and are broken
by BuildElement enabling this batching by default.

Instead, enable it in all of the individual build element plugin
implementations.

This fixes issue #800

60ddeeb9

by Tristan Van Berkom at 2018-12-13T14:58:28Z

Merge branch 'tristan/dont-batch-prepare-assemble-by-default' into 'master'

Dont batch prepare assemble by default

Closes #800

See merge request BuildStream/buildstream!1009

bf72cc42

by Angelos Evripiotis at 2018-12-13T17:31:17Z

contributing: mandate end-to-end tests, allow others

Update the section on adding tests, to fulfill these broad points:

- Mandate end-to-end testing, with rationale.
- Mention internal APIs as endpoints for testing.
- Warn against pitfalls of unit-testing.

It's more text than I would have liked, perhaps in later work we'll be
able to say it with less.

This change integrates feedback from the mailing list
'Guidance on Unit Tests' thread:
https://mail.gnome.org/archives/buildstream-list/2018-November/msg00045.html

13eb7ed2

by Angelos Evripiotis at 2018-12-13T18:01:56Z

Merge branch 'aevri/contributing_e2e_tests' into 'master'

contributing: more clarity on testing

See merge request BuildStream/buildstream!973

629a6e52

by Chandan Singh at 2018-12-14T19:34:20Z

Introduce new "source" command group

Following the message thread
https://mail.gnome.org/archives/buildstream-list/2018-November/msg00106.html,
implement a new command group called `source`. Move existing `track`,
`fetch`, and the recently added `source-checkout` commands under this
group.

For `track` and `fetch`, this is a BREAKING change, as the old commands
have been marked as obsolete. Using them will result in an error message
that refers people to use the new versions, like `bst source fetch`
instead of old `bst fetch`. `source-checkout` will now become
`source checkout` (the dash has turned into a space), and is not a
breaking change as it was added in the current development cycle.

Note that the functionality to hide commands from help output was added
only recently in Click, so the minimum version of Click that we now
require is 7.0.

Summary of changes:

* _frontend/cli.py: Add `source` command group, mark previous versions
  as obsolete and hide them from the help output.
* _frontend/complete.py: Fix completion for hidden commands.
* setup.py: Bump Click minimum version to 7.0.
* tests: Update to cope with the new command names.

Fixes #814.

f894c0a8

by Chandan Singh at 2018-12-14T19:34:20Z

NEWS: Add entry for new source command group

Also, change the existing entry about `source-checkout` to be
`source checkout` instead. And, while we are there, move it near the
other announcements about the source command group.

b23bec55

by Chandan Singh at 2018-12-14T20:07:13Z

Merge branch 'chandan/source-subgroup' into 'master'

Introduce new "source" command group

Closes #814

See merge request BuildStream/buildstream!1003

d2105909

by Raoul Hidalgo Charman at 2018-12-18T11:13:56Z

Add remote execution instance option

This is used when sending execution requests, to specify which instance of the
execution server to use.

Partial fix for #627.

a3bbec23

by Jim MacArthur at 2018-12-18T11:13:56Z

sandboxremote: Add server/storage config with defaults

Adds instance name support for the remote execution storage service.

89219f61

by Jim MacArthur at 2018-12-18T11:13:56Z

_cascache.py: Add instance names to GRPC calls

Adds the 'instance_name' parameter, which may be None, to most GRPC
calls in the CASCache object. ByteStream requests already have
instance_name supplied in the resource name, so do not need the
parameter.

Closes #627.

3dc20963

by Jim MacArthur at 2018-12-18T11:13:56Z

Documentation: Include instance-name in remote execution documentation

644d8b28

by Jim MacArthur at 2018-12-18T11:45:30Z

Merge branch 'raoul/627-RE-instance-config' into 'master'

Remote-execution instance configuration support

Closes #627

See merge request BuildStream/buildstream!952

e29aea36

by William Salmon at 2018-12-19T13:23:19Z

Basic options for shell --build to use buildtrees

Fixes issue #740

898a23a5

by Will Salmon at 2018-12-19T14:35:41Z

Merge branch 'willsalmon/shellBuildTrees' into 'master'

Shell --build has optional buildtrees

Closes #740

See merge request BuildStream/buildstream!986

a2f1d879
by Javier Jardón at 2018-12-19T15:36:11Z
```
README.rst: Add license badge
```

aae5e4b3

by Javier Jardón at 2018-12-19T16:09:48Z

Merge branch 'jjardon/license_badge' into 'master'

README.rst: Add license badge

See merge request BuildStream/buildstream!1014

2b767fe8

by Jürg Billeter at 2018-12-20T10:06:11Z

Move fetch logic from FetchQueue to Element

The queue shouldn't need to know about individual sources. This is in
line with _track() and _get_consistency().

7a102144

by Jürg Billeter at 2018-12-20T10:06:11Z

element.py: Do not call fetch() for cached sources

b325989e

by Jürg Billeter at 2018-12-20T10:07:20Z

tests/sources: Test that fetch() is not called for cached sources

77d8ad45

by Jürg Billeter at 2018-12-20T10:42:39Z

Merge branch 'juerg/fetch' into 'master'

Do not call fetch() for cached sources

See merge request BuildStream/buildstream!992

9835b7f1

by Chandan Singh at 2018-12-20T12:50:02Z

Expose base class for Git source plugins

Add a `_GitSourceBase` class to act as the base class for deriving
source plugins that work with Git. The first user of this base class is
the `git` source plugin that's in core at the moment.

The contents of this base class is almost identical to the existing `GitSource`
class. The two notable differences are:

- the private methods now have leading underscores, as some were missing
  it originally
- the name of the class

Note that we are exposing a private member here as we expect it to move to a
separate package soon. See the following discussion for more details:
https://gitlab.com/BuildStream/buildstream/issues/739#note_124819869

Fixes #739.

14da6955

by Chandan Singh at 2018-12-20T13:34:10Z

Merge branch 'chandan/abstract-git-source' into 'master'

Expose base class for Git source plugins

Closes #739

See merge request BuildStream/buildstream!1019

7368f569

by Angelos Evripiotis at 2018-12-20T13:59:23Z

_project::_find_project_dir: fix error message

Now that we are also looking for WORKSPACE_PROJECT_FILE, we should add
it to the error message. While we're there, also mention the directory
we are looking in - this could help scripters be clear about what went
wrong.

The new error message looks like this:

    Error loading project: None of ['project.conf', '.bstproject.yaml']
    found in '/src/temp/blah' or any of its parent directories

46efc91d

by Angelos Evripiotis at 2018-12-20T13:59:23Z

BREAK: remove auto-init behaviour

In the event that the project could not be found, stop BuildStream from
asking if the user would like to create a new project. Exit with error
instead, and give a hint to the user in case they're new.

As proposed on the mailing list here:
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00082.html

The new interaction looks like this:

    $ bst show nonsuch.bst
    No project found. You can create a new project like so:

        bst init

    Error loading project: None of ['project.conf', '.bstproject.yaml']
    found in '/src/temp/blah' or any of its parent directories

Fixes #826

e0c575c4

by Angelos Evripiotis at 2018-12-20T14:37:38Z

Merge branch 'aevri/rm-autoinit' into 'master'

BREAK: remove auto-init behaviour

Closes #826

See merge request BuildStream/buildstream!1015

c05d8b4f

by Chandan Singh at 2018-12-20T15:53:12Z

tests/frontend/logging.py: Fix regex Deprecation Warning

Use raw strings for regex searches, which is the preferred way to do
regular expressions in Python.

Without this patch, currently we get the following warnings:

```
tests/frontend/logging.py:44
  /builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:44: DeprecationWarning: invalid escape sequence \[
    m = re.search("\[\d\d:\d\d:\d\d\]\[\]\[\] SUCCESS Checking sources", result.stderr)

tests/frontend/logging.py:80
  /builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:80: DeprecationWarning: invalid escape sequence \d
    m = re.search("\d\d:\d\d:\d\d,\d\d:\d\d:\d\d.\d{6},\d\d:\d\d:\d\d,,,SUCCESS,Checking sources", result.stderr)
```

e8055a56

by Chandan Singh at 2018-12-20T15:53:12Z

buildstream/utils.py: Fix regex Deprecation Warning

Specify flags at the start of the _expression_ as per the recommendation
of the standard library.

Without this patch, we currently get the following warning:

```
tests/examples/junctions.py::test_open_cross_junction_workspace
  /builds/BuildStream/buildstream/dist/buildstream/buildstream/utils.py:213: DeprecationWarning: Flags not at the start of the _expression_ '\\/[^/]*\\Z(?ms)'
    regexer = re.compile(_expression_)
```

cd4889af

by Chandan Singh at 2018-12-20T16:34:08Z

Merge branch 'chandan/fix-warning' into 'master'

Fix Deprecation warnings from regex module

See merge request BuildStream/buildstream!1010

ac995236

by Tom Pollard at 2018-12-20T17:18:02Z

_stream.py: fix _buildtree_pull_required logic

c3153dea

by Tom Pollard at 2018-12-21T10:10:58Z

Merge branch 'tpollard/fixbuildtreereq' into 'master'

_stream.py: fix _buildtree_pull_required logic

See merge request BuildStream/buildstream!1021

caf49669