[Notes] [Git][BuildStream/buildstream][chandan/junction-dependency-forma

Chandan Singh pushed to branch chandan/junction-dependency-format at BuildStream / buildstream

Commits:

b1d2f001

by Tristan Van Berkom at 2018-12-06T14:18:31Z

source.py: Add new delegate method validate_cache()

This is guaranteed to be called only once for a given session once the
sources are known to be Consistency.CACHED, if source tracking is enabled
in the session for this source, then this will only be called if the
sources become cached after tracking completes.

ee7fc47f

by Tristan Van Berkom at 2018-12-06T14:18:31Z

git source plugin: Implementing submodule warnings

 o Unlisted submodule warning

   Now the git plugin will issue a configurable warning if a submodule
   exists and is used (checking out the submodule is not disabled),
   but is not specified in the source configuration.

 o Invalid submodule warning

   Now the git source plugin will issue a warning if the configuration
   specified a submodule which does not exist in the underlying git repository.

As a side effect, this patch also changes the flow control of the git
plugin such that submodules which are explicitly set to not be
checked out, are also not fetched but instead ignored completely.

5fc9a1da

by Tristan Van Berkom at 2018-12-06T14:18:31Z

git source plugin: Emmit the ref-not-in-track warning from validate_cache()

Now that we have Source.validate_cache(), this is a better place to emmit
the ref-not-in-track warning, since it will be emmitted at the earliest
opportunity and not only at Source.stage() or Source.init_workspace().

This also allows us to remove the `self.tracked` local state, and cleanup
some convoluted calling paths, removing some unnecessary parameters from
the usual codepaths and making the plugin overall more readable.

50cb2706

by Tristan Van Berkom at 2018-12-06T14:18:31Z

tests/testutils/repo/git.py: Adding remove_path() helper

A function for removing paths from a git repo, can also
be used to remove submodules.

f139ca8f

by Tristan Van Berkom at 2018-12-06T14:18:31Z

tests/sources/git.py: Testing the git:unlisted-submodule warning

 o Test that it is not triggered in show before fetch, because we
   don't know about the unlisted submodules yet

 o Test that it is triggered by a fetch command

 o Test that it is triggered by `show` after having completed a
   fetch command, since now we have the repository and know about
   the unlisted submodule

 o Test all of this under warning or error conditions (parameterized
   for fatal-warnings)

afe1aa9f

by Tristan Van Berkom at 2018-12-06T14:18:31Z

tests/sources/git.py: Testing the git:invalid-submodule warning

   o Test that it is not triggered in show before fetch, because we
     don't know the submodules yet so we cannot know if they are
     valid or not.

   o Test that it is triggered by a fetch command

   o Test that it is triggered by `show` after having completed a
     fetch command, since now we have the repository and know which
     specified submodules are invalid

   o Test all of this under warning or error conditions (parameterized
     for fatal-warnings)

57300201

by Tristan Van Berkom at 2018-12-06T14:18:31Z

tests/sources/git.py: Test unlisted submodules warning appearing after track

a0297625

by Tristan Van Berkom at 2018-12-06T14:18:31Z

tests/sources/git.py: Test invalid submodules warning appearing after track

60e96781

by Tristan Van Berkom at 2018-12-06T15:14:37Z

Merge branch 'tristan/submodule-warnings' into 'master'

Implement submodule warnings

See merge request BuildStream/buildstream!996

7fe55143

by Jonathan Maw at 2018-12-06T15:48:12Z

Fix bst source-checkout not working with open workspaces

2a6879a5

by Jonathan Maw at 2018-12-06T17:46:33Z

Merge branch 'jonathan/source-checkout-workspace' into 'master'

Fix bst source-checkout not working with open workspaces

Closes #807

See merge request BuildStream/buildstream!997

e61e6e50

by Tristan Van Berkom at 2018-12-07T07:57:13Z

_yaml.py: Added `allow_none` parameter to _yaml.node_get()

This allows specifying whether or not the code believes a None
value is acceptable if explicitly expressed to be None in the YAML.

A previous patch 3ba544b80f9f268be8ffe62fc8589b30212ec4a2 changed
the behavior to accept None for all values across the board, along
with allowing explicitly setting the `default_value` to `None` which
caused the code to be much more readable (this was the main motivation
of the patch, but it had the side effect of allowing None for everything).

In the majority of cases we load YAML however, it either has a value or
it is not specified in the YAML, and None is hardly ever acceptable
to be explicitly specified (it may be in the case you want to override
something with nothing in the YAML, but it is the edge case).

This had the side effect of causing unexpected crashes where the
YAML specifies an empty dictionary for instance.

Instead of forcing the caller to handle a possible None value, give
the choice to the caller if they really want to accept a None value,
and raise the LoadError otherwise.

This fixes issue #803, which is a crash due to receival of an
unexpected None value through _yaml.node_get()

504ecb3e

by Tristan Van Berkom at 2018-12-07T07:57:56Z

plugin.py: Added new `allow_none` parameter to Plugin.node_get_member(), defaulting to False.

Allow plugins to conveniently leverage the new `allow_none` parameter,
and have Plugin.node_get_member() raise a LoadError in the usual
case where None is not an acceptable value to be explicitly set in
the user provided YAML.

642ae4e8

by Tristan Van Berkom at 2018-12-07T07:57:56Z

tests/format/project.py: Added new regression test for empty dependency dictionaries

This adds a regression test for issue #803

2a0676c3

by Tristan Van Berkom at 2018-12-07T08:33:14Z

Merge branch 'tristan/yaml-optionally-allow-none' into 'master'

Only optionally allow None values in user provided YAML

Closes #803

See merge request BuildStream/buildstream!999

56c2c2cc

by Javier Jardón at 2018-12-09T16:59:26Z

buildstream/data/projectconfig.yaml: Remove default strip-binaries

They are too specific to be included by default

Recommendation is if you are building in Linux is to use the
ones begin used in the freedesktop-sdk project, for example

See #645

dcb6b5e8

by Javier Jardón at 2018-12-09T16:59:26Z

NEWS: Update for removal of default strip-commands

ae0afce5

by Javier Jardón at 2018-12-09T17:08:16Z

buildstream/buildelement.py: Document where to find strip commands for Linux

873b618c

by Javier Jardón at 2018-12-10T09:49:52Z

Merge branch 'jjardon/strip-binaries-removal' into 'master'

Remove default strip-commands

Closes #645

See merge request BuildStream/buildstream!987

f2fcc2f6

by Valentin David at 2018-12-11T12:12:08Z

Force updating tags when fetching git repository

When using aliases there are multiple remotes used in the cache
repository. When fetching, tags are not updated if the were previously
fetched from a different remote. Commits that not in a branch and only
tagged do not get fetched if the tag is not fetched.

Fixes #812

717c10d1

by Valentin David at 2018-12-11T12:44:56Z

Merge branch 'valentindavid/git_force_fetch_tags' into 'master'

Force updating tags when fetching git repository

Closes #812

See merge request BuildStream/buildstream!1000

7892287a

by Jonathan Maw at 2018-12-11T12:56:32Z

utils.py: Add a helper for searching upwards for files

i.e. with a given directory and filename, check parent directories until
either a directory with the filename is found, or you reach the root of
the filesystem.

This is a part of #222

67c7a58d

by Jonathan Maw at 2018-12-11T12:56:32Z

Create and store data inside projects when opening workspaces

Changes to _context.py:
* Context has been extended to contain a WorkspaceProjectCache, as there
  are times when we want to use it before a Workspaces can be initialised
  (looking up a WorkspaceProject to find the directory that the project is
  in)

Changes to _stream.py:
* Removed staging the elements from workspace_open() and workspace_reset()

Changes in _workspaces.py:
* A new WorkspaceProject contains all the information needed to refer back
  to a project from its workspace (currently this is the project path and
  the element used to create this workspace)
* This is stored within a new WorkspaceProjectCache object, which keeps
  WorkspaceProjects around so they don't need to be loaded from disk
  repeatedly.
* Workspaces has been extended to contain the WorkspaceProjectCache, and
  will use it when opening and closing workspaces.
* Workspaces.create_workspace has been extended to handle the staging of
  the element into the workspace, in addition to creating the equivalent
  WorkspaceProject file.

This is a part of #222

64836b18

by Jonathan Maw at 2018-12-11T12:56:32Z

_project.py: Rename _ensure_project_dir to _find_project_dir

This is a part of #222

7deeb2c3

by Jonathan Maw at 2018-12-11T12:56:32Z

cli.py: Use utils' search upwards helper when searching for project.conf

This is a part of #222

496f0ab7

by Jonathan Maw at 2018-12-11T12:56:32Z

_project.py: Find project from workspace if outside of a project

This is a part of #222

921f2bcb

by Jonathan Maw at 2018-12-11T12:56:32Z

_workspaces.py: Do not include .bstproject.yaml in the cache key

This is a part of #222

f145a3e4

by Jonathan Maw at 2018-12-11T12:56:32Z

cli: Interactively warn if the user is trying to close the workspace they're using to load the project

This involves changes in:
* _stream.py:
  * Add the helper Stream.workspace_is_required()
* userconfig.yaml:
  * Add a default value for prompt.really-workspace-close-project-inaccessible
* _context.py:
  * Load the prompt 'really-workspace-close-project-inaccessible' from
    user config.
* cli.py:
  * If buildstream is invoked interactively, prompt the user to confirm
    that they want to close the workspace they're using to load this
    project.

This is a part of #222

494cb7c6

by Jonathan Maw at 2018-12-11T14:12:54Z

tests: Test bst commands from an external workspace

This is a part of #222

fd1c5c5a

by Jonathan Maw at 2018-12-11T14:12:54Z

NEWS: Add an entry for being able to run commands from a workspace

This is a part of #222

4219a6b4

by Jonathan Maw at 2018-12-11T16:00:55Z

Merge branch 'jonathan/workspace-fragment-create' into 'master'

Support invoking buildstream from a workspace outside a project

See merge request BuildStream/buildstream!924

e564251b

by Jonathan Maw at 2018-12-11T16:15:19Z

_stream.py: Fix ugly error when opening a workspace using a relative path

7cf83343

by Jonathan Maw at 2018-12-11T16:15:19Z

_context.py: Store the directory buildstream was invoked on

a1dee91e

by Jonathan Maw at 2018-12-11T16:15:19Z

Make specifying elements optional in bst commands

Known issues:
* `bst shell` works, but `bst shell COMMANDS...` doesn't, because click
  has no way of separating optional args from variable-length args.
* `bst checkout` and `bst source-checkout`'s usage strings mark LOCATION
  as an optional argument. Because click gets confused if there's an
  optional argument before a mandatory argument, I had to mark LOCATION
  as optional internally.
* `bst workspace open` makes no sense with element being optional, so
  I skipped it.
* `bst workspace close` will probably need to be revisited when multiple
  projects can own one workspace.
* `bst workspace reset` will happily delete the directory you're
  currently in, requiring you to `cd $PWD` to see the contents of your
  directory.
  I could exclude the top-level directory of the workspace being
  deleted, but it is entirely valid to run workspace commands from deeper
  in the workspace.

This is a part of #222

9b192015

by Jonathan Maw at 2018-12-11T16:15:19Z

tests: Add tests for guessing element names

This is a part of #222

e340371f

by Jonathan Maw at 2018-12-11T16:15:19Z

Add NEWS for guessing the element in commands

This is a part of #222

bbf81c3d

by Jonathan Maw at 2018-12-11T16:15:19Z

doc: Point out that element is optional when commands are run from the workspace

This is a part of #222

a5a53ddd

by Jonathan Maw at 2018-12-12T13:44:09Z

Merge branch 'jonathan/workspace-fragment-guess-element' into 'master'

Guess the element when running commands in a workspace

See merge request BuildStream/buildstream!954

d55b9e39

by Phil Dawson at 2018-12-12T13:55:19Z

Add --tar option to source-checkout command

This commit is part of the work towards #672

733aab53

by Phil Dawson at 2018-12-12T13:55:19Z

Add --force / -f option to source-checkout command

c2efeba0

by Phil Dawson at 2018-12-12T14:43:40Z

Add option to source-checkout command to generate build scripts

85c61894

by Phil Dawson at 2018-12-12T14:43:40Z

Remove source bundle command

This is part of the work towards #672

a322d5c0

by Phil Dawson at 2018-12-12T14:43:40Z

NEWS: Add entry about the removal of source-bundle command

ec909605

by Phil Dawson at 2018-12-12T15:45:42Z

Merge branch 'phil/source-checkout-options' into 'master'

Retire bst source bundle command

Closes #672

See merge request BuildStream/buildstream!959

3697a611

by Richard Maw at 2018-12-12T16:31:38Z

cli: Add support for auto-completing artifact ref names

b3dceb16
by Richard Maw at 2018-12-12T16:32:41Z
```
cli: Add artifact command group
```
ba08a0cd
by Richard Maw at 2018-12-12T16:32:41Z
```
cli: Add artifact log command
```

70fb9554

by Richard Maw at 2018-12-12T16:32:41Z

tests: Add tests for artifact group commands

f773e746
by Richard Maw at 2018-12-12T16:33:02Z
```
NEWS: Mention `bst artifact log`
```

b6528441

by richardmaw-codethink at 2018-12-12T18:00:59Z

Merge branch 'richardmaw/artifact-log' into 'master'

Add artifact log command

See merge request BuildStream/buildstream!920

d03bf316

by Benjamin Schubert at 2018-12-13T10:24:11Z

Mock storage space checks for tests.

Fix #530

- Extract free space computation in a function for easier
mocking
- Mock space computation during cache quota tests
- Mock cache size during cachque quota tests
- Add two more tests when the configuration would require
to much storage space

a116f576

by Tristan Van Berkom at 2018-12-13T10:58:46Z

Merge branch 'BenjaminSchubert/fix-quota-tests' into 'master'

Mock storage space checks for tests.

Closes #530

See merge request BuildStream/buildstream!702

180fa774

by Benjamin Schubert at 2018-12-13T12:05:15Z

element.py: fix unbounded variable in nested python exception

As per https://docs.python.org/3.7/reference/compound_stmts.html#except
variables used in the "except" clause are erased at the end of the
block.

We had a conflict with a nested exception using the same variable as
before.

This renames the nested variable to avoid the clash

224aa4c2

by Benjamin Schubert at 2018-12-13T12:34:41Z

Merge branch 'bschubert/fix-unbound-variable-exception' into 'master'

element.py: fix unbounded variable in nested python exception

See merge request BuildStream/buildstream!1006

053beb66
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
manual plugin: Support virtual directories
```
29ab271c
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
makemaker plugin: Support virtual directories
```
ba955cf0
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
pip element plugin: Support virtual directories
```
6010b5a4
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
modulebuild plugin: Support virtual directories
```
3a6d27a4
by Tristan Van Berkom at 2018-12-13T14:23:19Z
```
distutils plugin: Support virtual directories
```

4c0e602c

by Tristan Van Berkom at 2018-12-13T14:23:19Z

BuildElement: Don't enable batching of prepare and assemble by default

Some external plugins depend derive from BuildElement and are broken
by BuildElement enabling this batching by default.

Instead, enable it in all of the individual build element plugin
implementations.

This fixes issue #800

60ddeeb9

by Tristan Van Berkom at 2018-12-13T14:58:28Z

Merge branch 'tristan/dont-batch-prepare-assemble-by-default' into 'master'

Dont batch prepare assemble by default

Closes #800

See merge request BuildStream/buildstream!1009

bf72cc42

by Angelos Evripiotis at 2018-12-13T17:31:17Z

contributing: mandate end-to-end tests, allow others

Update the section on adding tests, to fulfill these broad points:

- Mandate end-to-end testing, with rationale.
- Mention internal APIs as endpoints for testing.
- Warn against pitfalls of unit-testing.

It's more text than I would have liked, perhaps in later work we'll be
able to say it with less.

This change integrates feedback from the mailing list
'Guidance on Unit Tests' thread:
https://mail.gnome.org/archives/buildstream-list/2018-November/msg00045.html

13eb7ed2

by Angelos Evripiotis at 2018-12-13T18:01:56Z

Merge branch 'aevri/contributing_e2e_tests' into 'master'

contributing: more clarity on testing

See merge request BuildStream/buildstream!973

629a6e52

by Chandan Singh at 2018-12-14T19:34:20Z

Introduce new "source" command group

Following the message thread
https://mail.gnome.org/archives/buildstream-list/2018-November/msg00106.html,
implement a new command group called `source`. Move existing `track`,
`fetch`, and the recently added `source-checkout` commands under this
group.

For `track` and `fetch`, this is a BREAKING change, as the old commands
have been marked as obsolete. Using them will result in an error message
that refers people to use the new versions, like `bst source fetch`
instead of old `bst fetch`. `source-checkout` will now become
`source checkout` (the dash has turned into a space), and is not a
breaking change as it was added in the current development cycle.

Note that the functionality to hide commands from help output was added
only recently in Click, so the minimum version of Click that we now
require is 7.0.

Summary of changes:

* _frontend/cli.py: Add `source` command group, mark previous versions
  as obsolete and hide them from the help output.
* _frontend/complete.py: Fix completion for hidden commands.
* setup.py: Bump Click minimum version to 7.0.
* tests: Update to cope with the new command names.

Fixes #814.

f894c0a8

by Chandan Singh at 2018-12-14T19:34:20Z

NEWS: Add entry for new source command group

Also, change the existing entry about `source-checkout` to be
`source checkout` instead. And, while we are there, move it near the
other announcements about the source command group.

b23bec55

by Chandan Singh at 2018-12-14T20:07:13Z

Merge branch 'chandan/source-subgroup' into 'master'

Introduce new "source" command group

Closes #814

See merge request BuildStream/buildstream!1003

d2105909

by Raoul Hidalgo Charman at 2018-12-18T11:13:56Z

Add remote execution instance option

This is used when sending execution requests, to specify which instance of the
execution server to use.

Partial fix for #627.

a3bbec23

by Jim MacArthur at 2018-12-18T11:13:56Z

sandboxremote: Add server/storage config with defaults

Adds instance name support for the remote execution storage service.

89219f61

by Jim MacArthur at 2018-12-18T11:13:56Z

_cascache.py: Add instance names to GRPC calls

Adds the 'instance_name' parameter, which may be None, to most GRPC
calls in the CASCache object. ByteStream requests already have
instance_name supplied in the resource name, so do not need the
parameter.

Closes #627.

3dc20963

by Jim MacArthur at 2018-12-18T11:13:56Z

Documentation: Include instance-name in remote execution documentation

644d8b28

by Jim MacArthur at 2018-12-18T11:45:30Z

Merge branch 'raoul/627-RE-instance-config' into 'master'

Remote-execution instance configuration support

Closes #627

See merge request BuildStream/buildstream!952

e29aea36

by William Salmon at 2018-12-19T13:23:19Z

Basic options for shell --build to use buildtrees

Fixes issue #740

898a23a5

by Will Salmon at 2018-12-19T14:35:41Z

Merge branch 'willsalmon/shellBuildTrees' into 'master'

Shell --build has optional buildtrees

Closes #740

See merge request BuildStream/buildstream!986

a2f1d879
by Javier Jardón at 2018-12-19T15:36:11Z
```
README.rst: Add license badge
```

aae5e4b3

by Javier Jardón at 2018-12-19T16:09:48Z

Merge branch 'jjardon/license_badge' into 'master'

README.rst: Add license badge

See merge request BuildStream/buildstream!1014

2b767fe8

by Jürg Billeter at 2018-12-20T10:06:11Z

Move fetch logic from FetchQueue to Element

The queue shouldn't need to know about individual sources. This is in
line with _track() and _get_consistency().

7a102144

by Jürg Billeter at 2018-12-20T10:06:11Z

element.py: Do not call fetch() for cached sources

b325989e

by Jürg Billeter at 2018-12-20T10:07:20Z

tests/sources: Test that fetch() is not called for cached sources

77d8ad45

by Jürg Billeter at 2018-12-20T10:42:39Z

Merge branch 'juerg/fetch' into 'master'

Do not call fetch() for cached sources

See merge request BuildStream/buildstream!992

9835b7f1

by Chandan Singh at 2018-12-20T12:50:02Z

Expose base class for Git source plugins

Add a `_GitSourceBase` class to act as the base class for deriving
source plugins that work with Git. The first user of this base class is
the `git` source plugin that's in core at the moment.

The contents of this base class is almost identical to the existing `GitSource`
class. The two notable differences are:

- the private methods now have leading underscores, as some were missing
  it originally
- the name of the class

Note that we are exposing a private member here as we expect it to move to a
separate package soon. See the following discussion for more details:
https://gitlab.com/BuildStream/buildstream/issues/739#note_124819869

Fixes #739.

14da6955

by Chandan Singh at 2018-12-20T13:34:10Z

Merge branch 'chandan/abstract-git-source' into 'master'

Expose base class for Git source plugins

Closes #739

See merge request BuildStream/buildstream!1019

7368f569

by Angelos Evripiotis at 2018-12-20T13:59:23Z

_project::_find_project_dir: fix error message

Now that we are also looking for WORKSPACE_PROJECT_FILE, we should add
it to the error message. While we're there, also mention the directory
we are looking in - this could help scripters be clear about what went
wrong.

The new error message looks like this:

    Error loading project: None of ['project.conf', '.bstproject.yaml']
    found in '/src/temp/blah' or any of its parent directories

46efc91d

by Angelos Evripiotis at 2018-12-20T13:59:23Z

BREAK: remove auto-init behaviour

In the event that the project could not be found, stop BuildStream from
asking if the user would like to create a new project. Exit with error
instead, and give a hint to the user in case they're new.

As proposed on the mailing list here:
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00082.html

The new interaction looks like this:

    $ bst show nonsuch.bst
    No project found. You can create a new project like so:

        bst init

    Error loading project: None of ['project.conf', '.bstproject.yaml']
    found in '/src/temp/blah' or any of its parent directories

Fixes #826

e0c575c4

by Angelos Evripiotis at 2018-12-20T14:37:38Z

Merge branch 'aevri/rm-autoinit' into 'master'

BREAK: remove auto-init behaviour

Closes #826

See merge request BuildStream/buildstream!1015

c05d8b4f

by Chandan Singh at 2018-12-20T15:53:12Z

tests/frontend/logging.py: Fix regex Deprecation Warning

Use raw strings for regex searches, which is the preferred way to do
regular expressions in Python.

Without this patch, currently we get the following warnings:

```
tests/frontend/logging.py:44
  /builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:44: DeprecationWarning: invalid escape sequence \[
    m = re.search("\[\d\d:\d\d:\d\d\]\[\]\[\] SUCCESS Checking sources", result.stderr)

tests/frontend/logging.py:80
  /builds/BuildStream/buildstream/dist/buildstream/tests/frontend/logging.py:80: DeprecationWarning: invalid escape sequence \d
    m = re.search("\d\d:\d\d:\d\d,\d\d:\d\d:\d\d.\d{6},\d\d:\d\d:\d\d,,,SUCCESS,Checking sources", result.stderr)
```

e8055a56

by Chandan Singh at 2018-12-20T15:53:12Z

buildstream/utils.py: Fix regex Deprecation Warning

Specify flags at the start of the _expression_ as per the recommendation
of the standard library.

Without this patch, we currently get the following warning:

```
tests/examples/junctions.py::test_open_cross_junction_workspace
  /builds/BuildStream/buildstream/dist/buildstream/buildstream/utils.py:213: DeprecationWarning: Flags not at the start of the _expression_ '\\/[^/]*\\Z(?ms)'
    regexer = re.compile(_expression_)
```

cd4889af

by Chandan Singh at 2018-12-20T16:34:08Z

Merge branch 'chandan/fix-warning' into 'master'

Fix Deprecation warnings from regex module

See merge request BuildStream/buildstream!1010

ac995236

by Tom Pollard at 2018-12-20T17:18:02Z

_stream.py: fix _buildtree_pull_required logic

c3153dea

by Tom Pollard at 2018-12-21T10:10:58Z

Merge branch 'tpollard/fixbuildtreereq' into 'master'

_stream.py: fix _buildtree_pull_required logic

See merge request BuildStream/buildstream!1021

caf49669