[Notes] [Git][BuildStream/buildstream][jjardon/juerg/git-track-tags] 761

Javier Jardón pushed to branch jjardon/juerg/git-track-tags at BuildStream / buildstream

Commits:

e0eb51b1

by Tristan Van Berkom at 2018-03-02T07:11:54Z

_frontend/cli.py: Better help output for the `--option` main option.

Added metavar, making it more clear that you specify:

   bst --option OPTION VALUE

60e59a04

by Tristan Van Berkom at 2018-03-02T07:11:54Z

_frontend/cli.py: Adjust help output for `bst shell`

Dont specify how we fallback to launching a shell with `sh -i`,
since this is now configurable in project.conf.

5993c16c

by Tristan Van Berkom at 2018-03-02T08:37:17Z

NEWS: Adding missing entries for recently added features

85912562

by Tristan Van Berkom at 2018-03-02T11:43:10Z

_sandboxbwrap.py: Allow device nodes to be bind mounted

a3f1a8ef

by Tristan Van Berkom at 2018-03-02T11:51:23Z

_sandboxbwrap.py: Force cleanup when in interactive mode

When running a sandbox in interactive mode (implies `bst shell` was launched
or an interactive shell for debugging), dont bail out when cleaning up
directories which are not empty.

We treat this as a bug, if any of the base sandbox directories (/dev,
/tmp or /proc) are not empty when tearing down the sandbox, because
it would indicate something is wrong with bwrap.

When in interactive mode however, the user/project may have mounted
additional directories inside these base directories; for which we
need to create intermediate directories for the mount.

Instead of keeping track of every intermediate directory, just force
remove in interactive mode, as this is safe.

Ideally, we should fix upstream bwrap to cleanup the debris it creates
when exiting.

12ca9607

by Tristan Van Berkom at 2018-03-04T11:56:07Z

Enhanced bst shell configuration and cli options

Some changes to the host-files configuration:

  o Dont require `host-files` to not be directories

    We need to specify directories to mount from `project.conf` after all.

  o Added possibility of specifying optional mounts, to avoid
    meaningless warnings where optional files don't exist on
    the host

Added --mount CLI option to `bst shell`

This allows users to explicitly mount whatever they want into the
sandbox environment for `bst shell`.

This closes issue #274

bc0c4b71

by Tristan Van Berkom at 2018-03-04T12:14:19Z

tests/integration/shell.py: Updated shell tests

  o Added new test for `bst shell --mount`

  o Removed the test that a directory is refused in `host-files`,
    this is no longer a requirement

  o Updated names of the `host-files` mount specification members

  o Test the new `optional` feature of the mount specifications

83370f55

by Tristan Van Berkom at 2018-03-04T12:15:04Z

projectconf.rst: Amending documentation for `host-files`

This is now enhanced and an API change was introduced.

2e13d61e

by Tristan Van Berkom at 2018-03-06T07:00:03Z

NEWS: Adding entry for new `--mount` options of `bst shell`

06ac03d4

by Tristan Van Berkom at 2018-03-06T09:28:26Z

integration tests: Refactoring for dynamic project configurations

  o CliIntegration.run() now takes a `project_config` option, this
    will be composited on top of the existing, substituted project.conf

  o Removing gnomesdk alias from integration tests project.conf

  o Using `sysroot` alias instead of `gnome7` alias

  o Make base-alpine.bst use the `sysroot` alias now

  o Removed `create_project_config()` from shell.py tests, now
    use the new built-in feature for this which is cleaner.

c0be9295

by Tristan Van Berkom at 2018-03-06T09:28:26Z

_project.py: Support host environment expansion in `host-files` paths

This allows one to mount paths based on host specific environment
variables, this can be useful for mounting things under XDG_RUNTIME_DIR,
which is needed for host pulseaudio connections.

This is another part of addressing #223

eef26583

by Tristan Van Berkom at 2018-03-06T09:28:26Z

tests/integration/shell.py: Adding tests for env var expansions in `host-files`

ea74f326

by Tristan Van Berkom at 2018-03-06T10:31:13Z

Implement explicit environment assignments for `bst shell`

This introduces a new `environment` section of the project `shell`
configuration to set explicit env vars when running a non-isolated shell.

This supports host environment variable expansion.

This is a part of addressing #223

7ed0351a

by Tristan Van Berkom at 2018-03-06T10:33:22Z

tests/integration/shell.py: Testing new shell environment configuration

f8a71cab

by Tristan Van Berkom at 2018-03-06T11:22:02Z

doc/source/projectconf.rst: Documenting new `environment` shell configuration

And also ammend the documentation for `host-files` to note that
the paths support host side environment variable expansion.

Documenting both `host-files` and `environment` to require format version 4.

be6b5564

by Tristan Van Berkom at 2018-03-06T12:20:27Z

Removing all traces of `environment-inherit` shell configuration.

This is made redundant by the more complete `environment` configuration,
so lets quickly remove the former in this new format version 4.

1f63d214

by Tristan Van Berkom at 2018-03-06T12:41:36Z

NEWS: Adding a few missing entries for the release

ec5ee795
by Tristan Van Berkom at 2018-03-06T12:42:41Z
```
man: Updated man pages for the release
```

5a0f56af

by Tristan Van Berkom at 2018-03-06T13:12:10Z

doc/source/projectconfig.rst: Document `sources` overrides as since format version 1

This was added slightly before format version 1, but was done
without a version increment. Previous format versions cannot
be relied upon for use of this feature.

df9f696f

by James Ennis at 2018-03-07T08:42:49Z

_pipeline.py: Remove undefined variable source_index

141484fc

by Sam Thursfield at 2018-03-07T09:07:55Z

HACKING.rst: Mention benchmarking and profiling tools

This adds a reference to the benchmarking tool generated as part of
https://gitlab.com/BuildStream/buildstream/issues/205.

It also documents recommended strategies for profiling, which fixes
https://gitlab.com/BuildStream/buildstream/issues/206.

7fdddf30

by Tristan Van Berkom at 2018-03-07T11:44:37Z

Silence messages from Source cache interrogation

Source interrogation usually involves calling out to host tools
to quickly check if a given ref exists. This has however regressed
over time when running `bst build --track`.

This patch adds a new context manager to silence the messages,
and silences messages while calling `Source.get_consistency()`

Fixes #280

9f6ced45

by Jim MacArthur at 2018-03-07T12:36:19Z

status.py: Restore brackets to time in job display area

10fa1b4e

by Tristan Van Berkom at 2018-03-10T09:53:36Z

element.py: Report errors using `self.get_kind()`

We should always show the user the "kind" string when referring
to a plugin type, not the type name with `type(self).__name__`.

5e299889

by Tristan Van Berkom at 2018-03-13T07:57:15Z

_frontend/widget.py: Never abbreviate frontend messages

Messages which come from the frontend do not have any backing log
file where the remainder of the message can be reviewed. In the
case of lengthly detailed messages originating from the frontend,
we dont allow any abbreviation of the message.

9be5a16c

by Jim MacArthur at 2018-03-13T12:03:13Z

Remove other references to SequenceID from the code and tests

widget.py: Remove SequenceID from the list of widgets.
tests/frontend/logging.py: Remove %{sequence} check.

ad8bd6f9

by Jim MacArthur at 2018-03-13T12:03:13Z

Revert "Add sequence ID and widget for it."

This reverts commit 14234ccce3e7eec1153e810fab75afb2e90a2b22.

92b04688

by Jim MacArthur at 2018-03-14T10:51:56Z

doc/source/formatintro.rst: Better explanation of defaults

Users shouldn't see any difference between values hardcoded in Python (if
there are any) and the values in projectconfig.yaml, which were previously
items 0 and 1 respectively. Combined these both into 1 and made it clear that
not all of project.conf is applied right away.

76818202
by James Ennis at 2018-03-14T14:10:26Z
```
Add pylint to setup.cfg and setup.py
```

793780c6

by James Ennis at 2018-03-14T14:10:26Z

Generate pylint configuration file (pristine file):

NOTE: pylint was installed using pip: `pip3 install pylint` and
.pylintrc is the pylint configuration file which was generated with
`pylint --generate-rcfile` from a fresh fedora-27 docker image.

7e79c4af

by James Ennis at 2018-03-14T14:10:26Z

.pylintrc: Amend which warning messages are disabled

NOTE: We have categorised the warning messages into two groups.
Those that are of no use to us in the buildstream project and
those that should be enabled at some point.

6be20757

by James Ennis at 2018-03-14T14:10:26Z

.pylintrc: Alterations to the configuration file

388c1570
by Tristan Maat at 2018-03-14T14:10:26Z
```
fuse.py: skip file for linting
```
6eac2eda
by Tristan Maat at 2018-03-14T14:10:26Z
```
Add pylint to CI
```

ef0efd70

by James Ennis at 2018-03-14T14:10:26Z

pylint - dealt with anomalous-backslash-in-string warning

1ec691c7

by James Ennis at 2018-03-14T14:10:26Z

pylint - dealt with bad-continuation warning

cdd17bf3

by James Ennis at 2018-03-14T14:10:26Z

pylint - dealt with broad-except and bare-except warnings

a4a302bb

by James Ennis at 2018-03-14T14:10:26Z

pylint - dealt with consider-merging-isinstance warnings

1ba6167b

by James Ennis at 2018-03-14T14:10:26Z

pylint - dealt with dangerous-default-value warning

05fa346f

by James Ennis at 2018-03-14T14:10:26Z

pylint - dealt with global-statement warning

4741d137

by James Ennis at 2018-03-14T14:10:26Z

pylint - dealt with len-as-condition warning

0394d67d

by James Ennis at 2018-03-14T14:10:26Z

pylint - disabled no-member, bad-exception-context and catching-non-exception warnings

81833add

by James Ennis at 2018-03-14T14:10:26Z

pylint - dealt with superfluous-parens warning

b11b48ea

by James Ennis at 2018-03-14T14:10:26Z

pylint - dealt with unidiomatic-typecheck warning

9b4f54e1
by James Ennis at 2018-03-14T17:31:06Z
```
pylint - dealt with import warnings
```

432ddffd

by James Ennis at 2018-03-14T17:31:06Z

pylint - dealt with unnecessary-pass warning

4cfa0bdf

by James Ennis at 2018-03-14T17:31:06Z

pylint - dealt with redefined-argument-from-local warning

dfa45f04

by James Ennis at 2018-03-14T17:31:06Z

pylint - dealt with redefined-outer-name and redefined-built in warnings

3df8fb9f

by James Ennis at 2018-03-14T17:31:06Z

pushreceive.py: byteorder functions are now more sensibly named

ccec158a

by James Ennis at 2018-03-14T17:31:06Z

pylint - dealt with simplifiable-if-statement warning

16ff3374

by James Ennis at 2018-03-14T17:31:06Z

pylint - dealt with whitespace issues and disabled pep8 warnings

4b644aba

by Jürg Billeter at 2018-03-15T15:48:07Z

buildqueue.py: Do not mark assembly complete on build failure

This fixes internal error when build is retried.

Fixes #299

3f98a55d
by Jürg Billeter at 2018-03-15T15:50:08Z
```
element.py: Do not allow unplanned build jobs
```

a92f9ddb

by Jürg Billeter at 2018-03-15T17:43:05Z

_sandboxchroot.py: Ensure the cwd exists

This matches SandboxBwrap.

3861de9b

by Jürg Billeter at 2018-03-15T17:43:05Z

scriptelement.py: Mark install-root

This matches BuildElement.

9543d078

by Tristan Van Berkom at 2018-03-16T08:21:15Z

_project.py and docs: Move defaults into the defaults yaml file

Over time, the _project.py module has regressed into expressing
some defaults only hard coded into the python file instead of
properly exposing their default in the base configuration file
in data/projectconfig.yaml, where the default values can be observed
by users.

This patch rectifies that, and also restructures the relevant
surrounding documentation a bit.

c187ab76

by Tristan Van Berkom at 2018-03-16T08:39:37Z

doc/source/format.rst: Fix misnomer in documentation

The provided example shows the `tar` source ref being
expressed with a `sha256sum` key name, lets make the example
correct here and call it `ref`.

47ec4f52

by James Ennis at 2018-03-16T11:44:45Z

.pylintrc: Ensure pylint ignores gi.repository module as it dynamically adds classes

9cb30d35
by James Ennis at 2018-03-16T11:44:45Z
```
Make pylint and pep8 tests run by default
```
73993644
by Javier Jardón at 2018-03-17T09:25:26Z
```
docs/source/install.rst: Arch -> Arch Linux
```

cf778d36

by Javier Jardón at 2018-03-17T09:25:26Z

docs/source/install.rst: fix terminal box for ArchLinux

ca6dd66f

by Javier Jardón at 2018-03-17T09:29:05Z

source/install.rst: Divide in two sections

- installing from distro packages
- installing from source

5d7ee178

by Rafael Fontenelle at 2018-03-17T09:29:05Z

source/install.rst: Add instructions to install Arch packages

6b2f1727

by Tristan Maat at 2018-03-19T14:56:45Z

.gitlab-ci.yml: Fix lack of setuptools_scm causing CI failure

024a37c5

by Tristan Maat at 2018-03-19T14:56:50Z

.gitlab-ci.yml: Remove superfluous pytest-pylint install

810f6c33

by Tristan Van Berkom at 2018-03-20T07:55:56Z

Deprecate `--track-save` option from `bst build`

It makes no sense to support this now that we are adding project.refs,
instead of removing the option completely, we document it as
deprecated and print a warning.

  o _frontend/cli.py: Mark the option as deprecated, warn if it's used

  o _scheduler/trackqueue.py: Remove save optionality

  o _pipeline.py: Remove save optionality

  o tests/frontend/buildtrack.py: Stop testing for no-save functionality,

    This test was actually broken, and only save functionality was being
    tested. This was due to using a list [True] or [False] in the
    @pytest.mark.parameterize() statement, both of which are truthy values.

7777d390

by Tristan Van Berkom at 2018-03-20T07:55:56Z

source.py: Document ref accessors to mandate support for None values.

This is technically an API break, but will be transparant for the
vast majority of the current hand full of source implementations
which exist at this time. This is a lesser evil than bloating the
API with new methods.

1fbedea8

by Tristan Van Berkom at 2018-03-20T07:55:56Z

_downloadablefilesource.py: Support None values in ref accessors.

a39927a8

by Tristan Van Berkom at 2018-03-20T08:46:41Z

_projectrefs.py: Adding the ProjectRefs refs management object

This object manages the project.refs file in a project directory.

e2392ce7

by Tristan Van Berkom at 2018-03-20T08:46:41Z

_project.py: Load project.refs while loading the project.

This also bumps the format version to signify a new format
where storing refs centralized in project.refs is now supported.

19cad981

by Tristan Van Berkom at 2018-03-20T08:46:42Z

Fix #248 - Support project.refs in the core.

This adds a new Source.load_ref() API which is technically optional to
implement, projects which make use of a project.refs file must only
use source plugins which implement the new load_ref() method.

  * source.py: Added load_ref() API to load a ref from a specified node.

    This also adds _load_ref() and _save_ref() wrappers which handle
    the logistics of when to load and save a ref to which location.

    This also fixes _set_ref() to apply the ref to the node unconditionally,
    this must be done independantly of whether the ref actually changed.

  o Modifications to the loading process such that Source now can have
    access to the element name and source index.

  o _pipeline.py: Delegate abstract loading of source refs to Source._load_ref()

    - Print a summarized warning about redundant source references

    - Assert that one cannot track cross-junction elements without project.refs.

  o _scheduler/trackqueue.py: Delegate saving refs to Source._save_ref()

93e29955

by Tristan Van Berkom at 2018-03-20T08:46:42Z

Source plugins: Implement load_ref() in all source plugins

6b63828c

by Tristan Van Berkom at 2018-03-20T09:29:14Z

tests/frontend/buildtrack.py: Modified to test saving with project.refs

Test that we get the same functionality from the complex build + track
test regardless of whether we are saving refs inline or to a project.refs file.

50400e12

by Tristan Van Berkom at 2018-03-20T09:29:14Z

tests/pipeline/load.py: Testing source without load_ref() implementation

Test that we have no error when loading a project that doesnt use
project.refs, and that we get the expected graceful failure when
attempting to use project.refs with a Source which doesnt support it.

83a18f98

by Tristan Van Berkom at 2018-03-20T09:29:14Z

tests/format/project.py: Test that conditional statements are supported in project.refs

0f6b75dc

by Tristan Van Berkom at 2018-03-20T09:29:14Z

testutils/repo/git.py: Added branch() method

In order to test tracking on multiple branches, lets use
the Git repo scaffolding.

150c7441

by Tristan Van Berkom at 2018-03-20T09:29:14Z

tests/frontend/track.py: Testing that this works with project.refs

Only added condition to the simplest case here, the other cases
are mostly testing that track commands get the correct selection
of elements when using `--deps all` and `--except` arguments.

2bcae91f

by Tristan Van Berkom at 2018-03-20T09:29:14Z

tests/frontend/track.py: Test tracking with optionality

Tests that `bst track` sets the ref in the expected node if the
node containing the ref is conditionalized with a project option.

This tests both the regular inline behavior, and also the project.refs behavior.

da95c639

by Tristan Van Berkom at 2018-03-20T09:29:14Z

tests/frontend/track.py: Testing behavior of cross junction tracking

  o Test that this fails gracefully when the toplevel project uses
    inline ref-storage

  o Test that we successfully track cross junctioned elements when
    the project uses project.refs

208250cb

by Tristan Van Berkom at 2018-03-20T09:29:14Z

Documenting project.refs

  o Adding explanation of the `ref-storage` option in project.conf

  o Adjusting some adjacent relevant documentation

  o Adding a new projectrefs.rst section for the project.refs
    format itself, and linking it from the main authoring page.

3c3e9f79

by Tristan Van Berkom at 2018-03-20T09:29:14Z

NEWS: Adding news entry for new ref-storage options (project.refs)

Also announce --track-save deprecation

a3cb8199

by Tristan Van Berkom at 2018-03-20T09:56:40Z

source.py and loading: Refactoring of origin handles

The Source object previously stored the __origin_node,
__origin_toplevel and __origin_filename, this is from a time
when we did not hold on to the plugin's Provenance object
explicitly.

Since this information comes from the same place, let's just
use Plugin._get_provenance() to derive these values, instead
of redundantly carrying them along separately.

e8e88e5c

by Jonathan Maw at 2018-03-20T13:18:00Z

filter: Reword docstring to explain what the element does

The docstring included instructions on what it *should* be used for,
which is unnecessary.

Further, it was vague on what the element actually did. The new
docstring should be more explicit on what it does.

54827aa7

by Javier Jardón at 2018-03-20T15:35:07Z

Change theme to sphinx_rtd_theme

The sphinx_rtd_theme is a sphinx theme designed to look modern and be mobile-friendly
http://sphinx-rtd-theme.readthedocs.io/en/latest/index.html

d97e9b95

by Javier Jardón at 2018-03-20T15:35:07Z

source/index.rst: Separate general and reference documentation

Move artifacts from Installing section to general documentation

This change is needed to make the TOC work correctly with the
new theme

e2564d21

by Javier Jardón at 2018-03-20T16:31:53Z

source/install.rst: put Debian version under the same subsection

9cf75154

by Javier Jardón at 2018-03-20T16:31:53Z

source/conf.py: Keep documentation version in sync

91f47004
by Javier Jardón at 2018-03-20T16:45:34Z
```
source/conf.py: Update copyright year to 2018
```
6a5e8446
by James Ennis at 2018-03-20T17:05:02Z
```
index.rst: Fixed typo in About BuildStream
```

c36c545d

by Tristan Van Berkom at 2018-03-21T13:53:18Z

source.py: Source._save_ref() should return saved state

So we can report proper processed status from tracking queues

66ff0245

by Tristan Van Berkom at 2018-03-21T13:53:48Z

_scheduler/trackqueue.py: Fixed to report proper processed status.

This was a regression from the project.refs branch recently landed.

f9edceed

by Tristan Van Berkom at 2018-03-21T14:23:28Z

.gitlab-ci.yml: Pin the sphinx version to 1.7.1

The new version today 1.7.2 causes the following build error:

    Warning, treated as error:
    /builds/BuildStream/buildstream/dist/buildstream/doc/source/buildstream.sandbox.rst:document isn't included in any toctree

98b0d656

by Tristan Van Berkom at 2018-03-21T14:38:07Z

_message.py, _frontend/widget.py: Reviving MessageType.ERROR

We need to distinguish between:

  o Errors which occurred in a task, and are related to a log file
    and an elapsed time.

  o Errors which occurred in the main process and are not related
    to any timed activity

28bff099

by Tristan Van Berkom at 2018-03-21T14:38:07Z

_scheduler/queue.py: Handle errors in task post processing gracefully.

When errors occur after having processed an element, leaving these
unhandled results in a stack trace that is partially covered up by
the status area, and a hang is caused because we fail to update the
queues properly with the failed element.

This greatly improves the situation for issue #197

96c5c427

by Tristan Van Berkom at 2018-03-21T14:38:07Z

tests/frontend/track.py: Test that we fail gracefully for post tracking errors

This tests that we handle errors from Source.get_consistency() in the
post tracking state updates gracefully, one test added for a handled
failure, and another test added for an unhandled/unexpected exception.

cf43f8cf

by Tristan Van Berkom at 2018-03-21T14:38:07Z

tests/frontend/fetch.py: Test failure modes of Source.get_consistency()

Added a test that handled errors are reported at load time as expected.

Added another test that we get the expected exception. This needs to
be fixed, test contains FIXME: comment explaining that we could be
doing much better here.

This should be fixed in the context of issue #197

21fe51fb

by Tristan Van Berkom at 2018-03-21T14:38:07Z

_context.py: Fix Context._silence() context manager.

This needs to yield in a try: block, otherwise messages
remain silenced after an exception passes through this
context manager.

41274b2d

by Tristan Van Berkom at 2018-03-21T14:38:07Z

_scheduler/trackqueue.py: Remove unsafe calls to push/pop message depth

These messages have been silenced more reliably already
in Source._update_state(), using the Context._silence() context manager.

d246403e

by Tristan Van Berkom at 2018-03-21T15:58:19Z

git.py source plugin: Ignore inconsistent submodules.

If a submodule is listed in either the buildstream git source
definition, or in the .gitmodules file; but is not a valid submodule
which was added with `git submodule add`, we now emit a warning
and avoid exploding in the user's face.

This fixes issue #299

0f62dd05

by Tristan Van Berkom at 2018-03-21T15:58:40Z

tests/testutils/repo/git.py: Add new add_file() convenience

To write tests which add and commit a file.

54ba7324

by Tristan Van Berkom at 2018-03-21T15:58:40Z

tests/sources/git.py: Test that we ignore inconsistent submodules.

Test this at `bst track` time, when encountering a new ref which
adds a .gitmodules file but does not actually add the submodule,
we check that the expected warning is in the stderr and that BuildStream
does not error out for this.

dd67fecd

by Tristan Van Berkom at 2018-03-22T08:27:38Z

_artifactcache/ostreecache.py: Turn an ArtifactError() into an assert statement

This should be an assertion as it will trigger a proper BUG
message with a usable stack trace.

An incident of this ArtifactError() was reported in issue #305.

9766da6e
by Javier Jardón at 2018-03-22T08:40:24Z
```
Add HACKING document to official docs
```

92509174

by Javier Jardón at 2018-03-22T11:12:29Z

.gitlab-ci.yml: No need to install python2 for docs job

This also make the job faster: from ~1min to ~30s

a52cc0dd

by Javier Jardón at 2018-03-22T12:15:55Z

source/conf.py: change copyright/author to "BuildStream Team"

d1f25425

by Richard Maw at 2018-03-22T15:15:39Z

project.py: Add workspace.yml format versioning and version 1

357ac70e

by Richard Maw at 2018-03-22T15:41:34Z

frontend/workspace.py: Add tests for loading different versions

7d92ef0f

by Phillip Smyth at 2018-03-23T12:01:20Z

Created deb Source plugin for Issue #10

added tests

f4d3892e

by Jim MacArthur at 2018-03-23T14:37:38Z

Add 'sandbox' configuration key and build-uid/build-gid elements

This only affects SandboxBWrap at the moment.

buildstream/_loader.py: Add Symbol.SANDBOX and allow it in validation
buildstream/_metaelement.py: Add 'sandbox' variable and store it in the object
buildstream/_project.py: Add 'sandbox' configuration key and load it from
    project.conf.
buildstream/data/projectconfig.yaml: Default build-uid/build-gid values of 0
    for 'sandbox'.
buildstream/element.py: Add __extract_sandbox_config to find the final sandbox
    configuration. Pass this to the sandbox constructor.
buildstream/sandbox/_sandboxbwrap.py: If sandbox configuration was supplied,
    use it for uid and gid instead of the default 0.
buildstream/sandbox/_sandboxchroot.py: Throw exception if non-0 uid/gid were
    supplied.
buildstream/sandbox/__init__.py: Import SandboxConfig.
buildstream/sandbox/_private.py: New file, containing SandboxConfig. Made private
to avoid documentation for this class.

7d61e8c1

by Jim MacArthur at 2018-03-23T14:37:38Z

Add simple tests of builder UID.

tests/integration/build-uid: Add tests.
tests/integration/project/elements/build-uid/build-uid-default.bst:
    A test for the default uid/gid of 0 in the sandbox.
tests/integration/project/elements/build-uid/build-uid.bst:
    Tests for nonzero uid/gid values supplied in the element.
tests/integration/project/elements/build-uid/build-uid-1023.bst:
    Tests for nonzero uid/gid values supplied in project.conf.

cd7d0641
by Jim MacArthur at 2018-03-23T14:37:38Z
```
Add documentation on sandbox option
```
e302baf5
by Jim MacArthur at 2018-03-23T14:37:38Z
```
Increase BST_FORMAT_VERSION to 6
```
3ea2297b
by Jim MacArthur at 2018-03-23T14:37:38Z
```
NEWS: Add mention of the new sandbox option
```

83620b38

by Tristan Van Berkom at 2018-03-25T08:25:54Z

sandbox/_config.py: Replace docstring with comment.

For consistency, we document private things with comments,
and public things with docstrings.

c9b5692e

by Tristan Van Berkom at 2018-03-25T08:30:55Z

Consider new sandbox configuration in cache keys.

This adds SandboxConfig.get_unique_key() to delegate
the cache key contribution to the SandboxConfig.

Further, this moves over the OS and Machine Architecture
parts of cache key calculation into SandboxConfig.get_unique_key(),
removing the comment which speaks of later delegating this
part of cache key calculation to sandboxes.

Cache key calculation algorithm is backward compatible,
and so the artifact version remains unchanged.

094d878a

by Tristan Van Berkom at 2018-03-25T09:09:41Z

tests/cachekey: Added test to exercise new sandbox contributions to cache key

d8ec3ecb
by Jürg Billeter at 2018-03-25T09:20:07Z
```
tests/testutils/site.py: Add HAVE_ARPY
```

fa25a6a2

by Jürg Billeter at 2018-03-25T09:20:17Z

tests/sources/deb.py: Skip tests if arpy is not available

Fixes #317

c9701cdf
by Jürg Billeter at 2018-03-25T10:12:02Z
```
Remove unused imports
```
3845c3e0
by Jürg Billeter at 2018-03-25T10:12:02Z
```
.pylintrc: Enable unused-import checker
```

a27d6620

by Tristan Van Berkom at 2018-03-25T10:24:37Z

NEWS: Added news entry for newly added `deb` source

65654ef4
by Tristan Van Berkom at 2018-03-25T10:24:37Z
```
Docs: Adding deb source plugin to the docs index
```

e91dc833

by Tristan Van Berkom at 2018-03-25T10:28:48Z

man: Regenerating man pages for release

This needed an update at least for the deprecation
of `bst build --track-save`

b0e6a45e

by Phillip Smyth at 2018-03-26T15:27:47Z

Generate unique subdirs for built elements

Based on issue 89 (https://gitlab.com/BuildStream/buildstream/issues/89)
Ensuring that elements are staged into unique subdirs while building.
This patch supports that by doing the following:

* Modify project config to add 2 new variables ("project-name" and "element-name")
* Changed the default install-root from "/buildstream/install" to "/buildstream-install"
* Update the tests to accommodate these changes
* Update the expected cache keys in the tests

This fixes #89

b606fd14
by Tristan Maat at 2018-03-27T09:58:18Z
```
_ostree.py: Add diff_dirs function
```
fb332267
by Tristan Maat at 2018-03-27T09:58:18Z
```
_artifactcache: Add diff() interface
```

f761140f

by Tristan Maat at 2018-03-27T13:32:52Z

Make workspaces use objects instead of pipeline helper methods

6d5a7217
by Tristan Maat at 2018-03-27T13:32:54Z
```
Record last successful workspace build key
```
dc9a4d39
by Tristan Maat at 2018-03-27T13:32:54Z
```
Allow stage_artifact to update mtimes
```

18896a9e

by Tristan Maat at 2018-03-27T13:32:54Z

element.py: Update mtimes of modified dependency files (#216)

1d9ea917

by Tristan Maat at 2018-03-27T13:32:54Z

Add list of running files to workspace metadata

989bb3fe

by Tristan Maat at 2018-03-27T13:32:54Z

element.py: Disallow incremental builds for caches that can't diff

a7ae93ce

by Tristan Maat at 2018-03-27T13:32:55Z

integration/workspace.py: Add updated dependency tests

be5a8638

by Tristan Maat at 2018-03-27T13:32:55Z

frontend/workspace.py: Adjust tests to new workspace format

1e2100d9

by James Ennis at 2018-03-27T18:05:14Z

install.rst: Inform the user to restart their terminal once their PATH is adjusted

0cb0be7a

by James Ennis at 2018-03-27T18:05:14Z

install.rst: Change ~ for $HOME just incase it expands to characters bash does not like

9de1c626

by Javier Jardón at 2018-03-28T02:19:13Z

source/main_core.rst: Use toctree instead normal list

So index appear in left menu

This is a part of the fix for #246

500bd0d2

by Javier Jardón at 2018-03-28T02:19:43Z

source/main_using.rst: Use toctree instead normal list

So index appear in left menu

This is a part of the fix for #246

a32d15e3

by Javier Jardón at 2018-03-28T02:19:47Z

source/main_authoring.rst: Use toctree instead normal list

So index appear in left menu

This is a part of the fix for #246

bb75d4a5

by Javier Jardón at 2018-03-28T02:19:51Z

docs: Remove orphan tag from all pages

This is not needed anymore as all documented are part of the index

This is a part of the fix for #246

6e2a342d

by Tristan Van Berkom at 2018-03-28T03:44:02Z

_project.py: Move format version assertion before any validation

This fixes #314

95510ad6

by Tristan Maat at 2018-03-28T09:25:48Z

.gitlab-ci.yml: Use debian-8 for tests to test python3.4

2d7b721b

by Tristan Maat at 2018-03-28T12:16:10Z

_workspaces.py: Fix fallout from using python3.4

118bea08

by James Ennis at 2018-03-28T16:42:50Z

element.py: Make artifact lowercase in time_activity()

NOTE: artifact was also made lowercase in the function
get_pushed_elements() in runcli.py

9377212d

by Javier Jardón at 2018-03-29T10:45:58Z

Add basic resources page

with info about git repo, IRC, mailing list and big tracker

fbb0ca28

by Javier Jardón at 2018-03-29T10:45:58Z

source/index.rst: Add resources page to "contributing" section

b9546d4d

by Javier Jardón at 2018-03-29T11:25:08Z

HACKING.rst: Be specific about the only pip packages required are the python3 ones

63d99463

by Tristan Van Berkom at 2018-04-01T11:50:59Z

Include initialization time in the total session time of the build log

This patch refactors the frontend App object and touches a few internals.

  o The scheduler now takes a start time given to it at instantiation time,
    instead of considering the start time to commence in Scheduler.run()

  o The App.initialize() method has been swapped out for an
    App.initialized() context manager.

    This context manager now takes care of the main start/fail/success
    messages. For convenience and ensured consistency, this context manager
    is now responsible for:

      o Printing the startup heading
      o Printing the end of session summary
      o Exiting with a consistent error status in case of errors

  o The Pipeline() object no longer prints the start/fail/success messages,
    as they are now taken care of by the App()

  o The cli.py frontend code is much simplified by using the new context
    manager, also enforcing consistency in how we handle and report errors.

This fixes issue #186

aa8410d8

by Tristan Van Berkom at 2018-04-01T12:59:01Z

_workspaces.py: Dont unconditionally create workspace local state file at startup.

Fixes issue #257

23cea695

by Tristan Van Berkom at 2018-04-01T18:57:01Z

_frontend/widget.py: Add "Strict Build Plan" setting to session heading

Looks like this quite interesting attribute was not getting logged.

999d168a

by Tristan Van Berkom at 2018-04-01T19:02:02Z

_frontend/widget.py: Change heading separator to use '=' instead of '~'

Seems that using `~` conflicts with some markdown when trying to
paste build logs in, e.g. gitlab issues.

00ad7629

by Tristan Van Berkom at 2018-04-02T07:01:54Z

_frontend/app.py: Renamed from main.py

Since the CLI and the App object was split up, we should name
the file after the object which it implements.

6c9c2d42

by Tristan Van Berkom at 2018-04-02T07:53:45Z

_frontend/app.py: Split up initialization into two parts.

Created partial_init() function to do a partial initialization
which does not require loading the Pipeline, which is more expensive.

This is useful for some bst commands which dont operate on a pipeline.

8b5742dd

by Tristan Van Berkom at 2018-04-02T10:53:59Z

_exceptions.py: Adding new AppError exception and error domain

19e31adb

by Tristan Van Berkom at 2018-04-02T10:55:57Z

Refactoring of highlevel workspace code

Move all workspace related code out of Pipeline() and into the
frontend App() object.

Some changes in transition here include:

  o Workspaces() object methods for looking up and deleting workspaces
    now take an element name instead of an element.

  o Share code for partial App() initialization between the
    `workspace close` and `workspace list` commands

  o No longer require that an element exist in the project
    in order to close a workspace

This fixes issue #249

202dd7cd

by Tristan Van Berkom at 2018-04-02T11:00:06Z

source.py: Remove unused method Source._del_workspace()

This was never particularly useful, there is no circumstances
under which a workspace needs to be deleted, and a cache key
invalidated, in the course of a session.

A workspace is deleted only atomically as a part of `bst workspace close`,
which does not even load a pipeline anymore, so the pipeline state need not
be adjusted in this case.

22c6ef17

by Tristan Van Berkom at 2018-04-02T11:04:59Z

tests/frontend/workspace.py: Fix inaccurate comments

af719007

by Tristan Van Berkom at 2018-04-02T11:11:11Z

tests/frontend/workspace.py: Test that we can remove a workspace for a nonexisting element

Guard against regressions of issue #249

18b60dbb

by Tristan Van Berkom at 2018-04-02T11:17:54Z

tests/frontend/workspace.py: Reduce unneeded coverage here.

Here we are overly testing the same functionality which does
not require source specific support, e.g. lets not test resetting
of workspaces for every kind of repo, since we already test opening
and closing of workspaces for every repo kind, this is redundant
and slowing down tests.

af08b16e

by Tristan Van Berkom at 2018-04-03T13:49:10Z

HACKING.rst: Documenting naming policy for private symbols

This is a part of issue #285

70c73b93

by Tristan Van Berkom at 2018-04-03T13:49:10Z

_workspaces.py: Adhere to policy on private symbols

This is a part of issue #285

500f4f33

by Tristan Van Berkom at 2018-04-03T13:49:10Z

_context.py: Adhere to policy on private symbols

And adjust all surrounding sources for changed symbols.

This is a part of issue #285

01db4072

by Tristan Van Berkom at 2018-04-03T13:49:11Z

_exceptions.py: Adhere to policy on private symbols

And adjust all surrounding sources for changed symbols.

Also, added new LoadErrorReason.UNSUPPORTED_PLUGIN, required
for changes in how the project will report format version errors
for plugins at creation time

This is a part of issue #285

af34d716

by Tristan Van Berkom at 2018-04-03T13:49:11Z

_project.py: Adhere to policy on private symbols

And adjust all surrounding sources for changed symbols.

Additional details:

  o Added Project.get_shell_config() to report the shell configuration,
    instead of making those members all public

  o Moved assertions about project specified format versions required
    of plugins out of Plugin.__init__, and into Project.create_element()
    and Project.create_source(), so we can keep more things private

This is a part of issue #285

5c33a984

by Tristan Van Berkom at 2018-04-03T14:48:56Z

_pipeline.py: Adhere to policy on private symbols

This is a part of issue #285

6c14a05a

by Tristan Van Berkom at 2018-04-04T04:57:50Z

_plugincontext.py: Adhere to policy on private symbols

This is a part of issue #285

aff532d4

by Tristan Van Berkom at 2018-04-04T05:11:50Z

_variables.py: Adhere to policy on private symbols

This is a part of issue #285

355b2263

by Tristan Van Berkom at 2018-04-04T06:17:26Z

_artifactcache modules: Adhere to policy on private symbols

Additionally:

  o This shares more code by creating ArtifactCache.get_artifact_fullname(),
    which is used for the extract directory relative path in both backends,
    and for the ostree "ref" in the ostree backend.

  o Further, this removes some redundant documentation in derived abstract
    methods, and clarifies in both backends which methods are abstract methods,
    we should only be documenting abstract method semantics in one place, where
    they are defined.

This is a part of issue #285

71f07656

by Tristan Van Berkom at 2018-04-04T09:16:57Z

Fix strict setting to be consistent throughout the pipeline.

I originally changed the configuration of strict mode to be on
a per project basis in the user configuration, because the user should
be able to set their preference on a per project basis.

I however made the mistake to make the strict mode be considered
on a per project basis within a single pipeline, this commit corrects
the behavior such that when you are building a project with junctions,
only the toplevel project is considered when deciding strict mode
for the whole pipeline.

9012a5ed

by Tristan Van Berkom at 2018-04-05T06:16:15Z

element.py: Fix _update_state() to consider whether an element is to be built properly

This was previously using self._buildable() to determine whether
an element is built locally and not to be downloaded, which misses
out on elements which have already been built, like open workspaces
which can only have their cache key calculated after the build.

This fixes issue #316

daffe498

by Tristan Van Berkom at 2018-04-05T06:16:15Z

tests/frontend/workspace.py: Exercise the build test in strict and non-strict mode

This test fails without the previous patch fixing issue #316,
so this should guard against regressions of building and caching
workspace builds in non-strict mode.

638344e4

by Tristan Van Berkom at 2018-04-05T06:16:15Z

element.py: Updating comment for _assemble_done()

This was misinforming that it is only called in a subprocess,
it also needs to be called in the main process, also point out
that updating element state is a side effect of this.

cc43127e

by Tristan Van Berkom at 2018-04-05T06:16:15Z

_scheduler/buildqueue.py: Dont call Element._update_state() here

This is implied by Element._assemble_done()

8c8b1c54

by Tristan Van Berkom at 2018-04-05T06:35:20Z

_options package: Adhere to policy on private symbols

This also adds a couple of methods to OptionPool to avoid
exposing some internals to other parts of the codebase, and
calls those new methods from _project.py and _frontend/widget.py

This is a part of issue #285

b8c68284

by Tristan Van Berkom at 2018-04-05T06:35:20Z

_platform package: Adhere to policy on private symbols

This is a part of issue #285

ab63b536

by Tristan Van Berkom at 2018-04-05T08:13:35Z

_frontend/widget.py: Defend against empty string message detail

It appears we have some cases of empty detail strings that are
not None, this was resulting in an IndexError when trying to strip
the trailing newline from the message.detail.splitlines() result.

207b986f

by Tristan Van Berkom at 2018-04-05T08:32:42Z

_frontend/widget.py: Print the workspace directory for %{workspace-dirs}

Instead of printing the list of workspace directory for each source.

This fixes some fallout from the changes to make workspaces element-wide
instead of being on a per-source basis, which was merged as merge request !257
as a part of issue #209.

a4eb8bd2

by Tristan Van Berkom at 2018-04-05T08:35:03Z

element.py: Removing Element._workspace_dirs()

This private method is now unused and pointless since we
made workspaces an element-wide concept as a part of issue #209.

34210d1e

by Tristan Van Berkom at 2018-04-05T09:54:24Z

_platform/linux.py: Fix fallout from context private symbols refactor

This was still calling an outdated version of Context._message(),
which is now Context.message(), causing stack traces on platforms
without support for user namespaces.

This was fallout from commit 500f4f330fde8b6001a2f8d8921bd5b8acb79960

22fe6b74

by Tristan Van Berkom at 2018-04-05T13:11:58Z

doc/source/projectrefs.rst: Ammended documentation for project.refs

This was missing the main `projects` toplevel key

67338ef7

by Tristan Van Berkom at 2018-04-05T13:13:49Z

_frontend/app.py: More consistent error reporting

Use App.print_error() in all error exit circumstances, this
will automatically consider any detail strings reported in BstErrors.

1280b449

by Tristan Van Berkom at 2018-04-05T13:14:32Z

_exceptions.py: Added new LoadErrorReasons for junction load failures

81ec3635

by Tristan Van Berkom at 2018-04-05T13:14:32Z

_exceptions.py: Allow detail strings in LoadError exceptions

857751fb

by Tristan Van Berkom at 2018-04-05T13:14:32Z

_loader.py: Some fixes in how we load sources for junctions

  o Ensure that we call Source._load_ref(), and consider project.refs this way

  o Ensure that we report a warning in the case that project.refs is in use
    and the loaded junction source has a redundant ref which is going to
    be ignored

  o Report more distinct machine readable errors for failures to load
    junction element sources

  o Handle Consistency.INCONSISTENT and Consistency.RESOLVED separately:

    - The user should be told something different depending on whether
      they need to fetch or whether they need to track.

    - It is never possible to automatically fetch the source in the
      case that the source has no ref to begin with.

This also adjusts the test/loader/junctions.py test to expect the new error

50af604e

by Tristan Van Berkom at 2018-04-05T13:14:32Z

tests/frontend: Share the configure_project() function

Make buildtrack.py and track.py share the same configure_project() helper.

abb9ef22

by Tristan Van Berkom at 2018-04-05T13:14:32Z

tests/frontend/show.py: Test behaviors of showing junctioned elements

  o Test error conditions for showing an unfetched junctioned project

  o Test error conditions for showing an untracked junctioned project

Both tests check both modes of ref-storage.

This adds a new shared helper function `generate_junction`

24ab5ce7

by Tristan Van Berkom at 2018-04-05T13:14:32Z

tests/frontend/fetch.py: Added tests for automatically fetching the junctions

  o This tests that a Consistency.RESOLVED junction will automatically be fetched
    when `bst fetch` is run on a pipeline which refers to a junction.

  o Further, it tests that a Consistency.INCONSISTENT junction will bail
    out with the expected error message

Again testing with both ref-storage modes

ee73a87b

by Tristan Van Berkom at 2018-04-05T13:14:32Z

tests/frontend/track.py: Added junction related tests

  o Test that we bail out with the expected errors when the
    junction element in question is Consistency.INCONSISTENT

  o Test that tracking the junction itself, causes a subsequent
    show of the pipeline to not bail out anymore (tests that
    tracking works and persists for a junction element).

Again these use both ref-storage modes.

8704363a

by Tristan Van Berkom at 2018-04-05T13:27:16Z

tests/frontend/buildcheckout.py: Added junction related tests

  o Test that we get the expected error if the junction element
    the pipeline refers to is inconsistent

  o Test that the junction element is automatically fetched as
    a part of the build phase

Both tests using both ref-storage modes

09e52016

by Tristan Van Berkom at 2018-04-06T07:08:47Z

_exceptions.py: Added LoadErrorReason.INVALID_SYMBOL_NAME

1b4307f1

by Tristan Van Berkom at 2018-04-06T08:20:51Z

_yaml.py: Added assert_symbol_name() helper function.

This function asserts that a loaded symbol name is a valid one,
and raises an appropriate and consistent LoadError if an invalid
symbol name is encountered.

60f2a320

by Tristan Van Berkom at 2018-04-06T08:36:13Z

_project.py: Assert that the loaded project name is a valid symbol name

Fixes issue #339

337075bb

by Tristan Van Berkom at 2018-04-06T08:36:42Z

_options/optionpool.py: Assert valid symbol names for option names.

These must be alphanumeric and may not contain dashes.

550cb369

by Tristan Van Berkom at 2018-04-06T08:36:42Z

_options/option.py: Assert valid symbol names for variable exports

420114a0

by Tristan Van Berkom at 2018-04-06T08:36:42Z

tests/format/project.py: Converted to use CLI test harness

Moved the old style project format tests into the new CLI based
directory in tests/format/ and converted to use the CLI fixture
for these tests.

50b694a3

by Tristan Van Berkom at 2018-04-06T08:36:42Z

tests/project/plugins.py: Removed this old style test

This is sufficiently covered by other tests in tests/format/project.py

c2500610

by Tristan Van Berkom at 2018-04-06T08:36:42Z

tests/format/project.py: Added tests for loading of invalid project names

a2bc2a1c

by Tristan Van Berkom at 2018-04-06T08:36:42Z

tests/format/options.py: Added tests for invalid option name symbols

21050433

by Tristan Van Berkom at 2018-04-06T08:51:16Z

tests/format/options.py: Added tests for invalid variable names in options

06b1920b

by Tristan Van Berkom at 2018-04-06T08:51:16Z

doc/source/projectconf.rst: Properly document the project name.

This is a part of issue #339

23edfe5d

by Tristan Van Berkom at 2018-04-06T08:51:16Z

doc/source/projectconf.rst: Document restrictions on option and option variable names.

c55dad4b

by Tristan Van Berkom at 2018-04-06T11:45:13Z

data/userconfig.yaml: Change default logging to print full shas at init and bst show

This may be obnoxious for `bst show` purposes, which will now also print
the full artifact cache keys instead of nicely abbreviated ones, but it is a
more appropriate default for the heading of a build session.

This fixes issue #343

74c60178

by Chandan Singh at 2018-04-07T09:42:25Z

_frontend/cli.py: Add option to close multiple workspaces

At present, it is only possible to close workspaces for elements one at
a time. This can become slightly tedious process when you have multiple
workspaces open and you want to close all of them, maybe because you
just finished working on a set of related elements.

Instead of accepting a single element, accept a list of elements as
argument for `bst workspace close`. Additionally, add `-a`/`--all`
option to close all workspaces.

Fixes #337 - Add option to close all workspaces.

2ca11fde

by Tristan Van Berkom at 2018-04-07T10:02:13Z

_frontend/cli.py: Remove --no-checkout option for `bst workspace reset`

This option in `bst workspace reset` is just pointless.

3ba544b8

by Tristan Maat at 2018-04-07T10:05:10Z

Allow 'None' as a default_value for _yaml.node_get

60dbf19f

by Tristan Van Berkom at 2018-04-07T10:30:47Z

Complete the work started to allow None as default in _yaml.node_get()

This completes the work which was started in commit
3ba544b80f9f268be8ffe62fc8589b30212ec4a2 which only went half
way towards updating all the source code to be consistent and use
the new semantic which allows None values.

59064042

by Tristan Van Berkom at 2018-04-08T08:23:50Z

_project.py: Remove BST_WORKSPACE_FORMAT_VERSION

This is now defined in _workspaces.py

9be2dc5a

by Tristan Van Berkom at 2018-04-08T08:25:21Z

_versions.py: Added new file just to hold basic symbolic versions

This is where we're going to store BST_CORE_ARTIFACT_VERSION
and BST_FORMAT_VERSION from now on.

This was a little bit weirdly defined before, and now we need to
at least have BST_FORMAT_VERSION available from the frontend/cli
without onerous imports (we need to import it there, but dont want
to slow down load time for bash completion runs).

b7ec278d

by Tristan Van Berkom at 2018-04-08T10:59:31Z

_yaml.py: Allow None for provenance argument in _yaml.assert_symbol_name()

Since we now also use this to validate user input on the command line.

5de99191

by Tristan Van Berkom at 2018-04-08T12:56:33Z

_frontend/cli.py, _frontend/app.py: Implemented new `bst init` command.

This comes with an interactive mode unless the project name is specified
on the command line.

This fixes issue #342

337b7a31

by Tristan Van Berkom at 2018-04-08T12:57:01Z

tests/frontend/init.py: New test to test edge cases for new `bst init` command.

96af6dd5

by Tristan Van Berkom at 2018-04-08T12:57:01Z

More specific exceptions when a project.conf is missing.

Since we want to react and start an interactive session when the
project.conf is missing, we need a more specific error to catch.

1f57e598

by Tristan Van Berkom at 2018-04-08T12:57:01Z

_frontend/app.py: Automatically launch interactive `bst init` when project.conf is absent

When running a command where a project.conf does not exist, ask the
user if they would like to interactively create a project in the
said directory.

This is a part of issue #342

6d7f86fe

by Tristan Van Berkom at 2018-04-08T13:06:34Z

NEWS: Adding news entry for new `bst init` command

e0a8b9e5

by Tristan Van Berkom at 2018-04-08T13:55:03Z

_versions.py: Fixed copyright year on newly added file.

Oops.

8bfc7371

by Jim MacArthur at 2018-04-09T09:49:27Z

_ostree.py: Mention the remote URL when we fail to fetch remote refs

68cee1c7

by Tristan Van Berkom at 2018-04-09T10:25:36Z

tests/frontend/workspace.py: Fixed test_build to make the right assertion

This was removing a file from the workspace, building, checking out,
and then asserting that the file is *still gone* in the workspace,
ignoring the same file in the checkout.

66cf08fa

by Tristan Van Berkom at 2018-04-09T10:25:36Z

element.py: Cache source consistency logic and refactor workspace logic into Element

Workspaces are now element wide, so consistency edge cases must
be handled at the element level instead of the source level.

26f7f0b8

by Tristan Van Berkom at 2018-04-09T10:25:36Z

element.py: Factored out pointless extra Element._workspaced() method

62346702

by Tristan Van Berkom at 2018-04-09T10:25:36Z

element.py: Manage scheduled tracking state in Element

This removes the scheduled state of tracking from Sources, as
this is really an element wide thing.

To be consistent with assembly, now this comes with:

  o Element._schedule_tracking()
  o Element._tracking_done()
  o Element.__tracking_scheduled
  o Element.__tracking_done

Updated the TrackQueue() to call Element._tracking_done() similarly
to how we have BuildQueue() call Element._assemble_done().

0fbb550e

by Tristan Van Berkom at 2018-04-09T10:25:36Z

element.py, source.py: Removing knowledge of assemble scheduling state from source

And considering the workspace related edge cases in Element instead of Source.

67d22d76

by Tristan Van Berkom at 2018-04-09T10:25:36Z

element.py, source.py: Element consumes the rest of workspace logic.

And Source no longer has any __workspace handle.

9e74988e

by Tristan Van Berkom at 2018-04-09T11:24:34Z

_workspaces.py: Fix cache key calculation regression

This was regressed in the refactor done in commit f761140f

b8b41724

by Tristan Van Berkom at 2018-04-09T11:24:34Z

tests/frontend/workspace.py: Test that we detect modifications made to a workspace

This was previously only working for added or removed files and
broken for modified files.

8da99651

by Tristan Van Berkom at 2018-04-10T08:01:51Z

element.py: Removed useless __workspace pointer

I added this in the last refactor and forgot to remove it.

be6b353d

by Tristan Van Berkom at 2018-04-10T12:01:55Z

_workspaces.py: Remove knowledge of Elements completely

This makes workspaces more cleanly separated from everything else.

  o Removed some methods from Workspace()

  o Added Element._open_workspace() to initialize workspaces on
    sources

  o Cleanup some code in the pipeline

  o Have the App() call Element._open_workspace() instead of Workspace.open()

5a36d04e

by Tristan Van Berkom at 2018-04-10T12:02:15Z

_workspaces.py: Add generic serialization/deserialization

Also streamline the instantiation code paths to layer the yaml
parsing on top of simple dictionary deserialization.

888f0b9f

by Tristan Van Berkom at 2018-04-10T12:02:15Z

_workspaces.py: Added Workspaces.update_workspace()

This takes a serialized workspace dictionary as understood
by Workspace.from_dict() and created by Workspace.to_dict()

Further, this adds a Workspace.differs() method to compare
the state of two workspace instances.

0b5809c4

by Tristan Van Berkom at 2018-04-10T12:11:54Z

utils.py: Added _is_main_process()

A helper function to identify if we are running in the main process
or not.

a9cbb0f7

by Tristan Van Berkom at 2018-04-10T12:11:54Z

Serialize workspace modifications in the main process.

This patch makes the child process send any workspace state back
to the main process and saves it there directly before processing
the result of a build job, such that workspace modifications in
child tasks work transparently and no special care needs to be taken
to save them except when doing so in the main process.

Also this removes the line where we update workspace data at staging
time.

This fixes issue #352

a867f293

by Tristan Van Berkom at 2018-04-10T12:11:54Z

_workspaces.py: Assert that configuration is only saved in the main process.

37d97414

by Jim MacArthur at 2018-04-10T12:50:53Z

_frontend/widget.py: Correct log line if logdir is empty

cb386651

by Tristan Maat at 2018-04-11T06:49:22Z

HACKING.rst: Add integration and pytest notes

d46fbbcb

by Tristan Van Berkom at 2018-04-11T09:42:56Z

element.py: Preserve workspace state in failed build sandboxes.

As reported in regression #346, since we started mounting the workspaces
we now fail to debug build failures for workspaces, the shell command
no longer sees the workspaces files to debug.

As the failed sysroot represents a very particular build failure, it
is incorrect to attempt to remount a workspace again after a failed
build, as the workspace files may have changed since the build.

Instead, in the case of a build failure on a workspaced element, we
copy the files from the workspace in place (while automatically preserving
the mtimes and everything, as utils.safe_copy() does).

When we implement failed build artifacts, this case will have to be
handled in a very similar way, too.

This fixes issue #346

a1564c4a

by Tristan Van Berkom at 2018-04-11T09:42:56Z

tests/integration/shell.py: Test that workspaced files are visible in a bst shell

Test both in regular `bst shell --build`, and on a failed build sysroot
with `bst shell --sysroot`, the latter being the semantic used when debugging
a failed build.

This guards against regressions of issue #346

5217359f

by Javier Jardón at 2018-04-11T13:58:10Z

doc/source/install.rst: Update list of base requirements

274e1831

by Javier Jardón at 2018-04-11T13:58:10Z

source/install.rst: Update ArchLinux installation instructions

6ad88ecb

by Tristan Van Berkom at 2018-04-11T14:10:54Z

element.py: Restructure artifact metadata

 o Store the artifact.yaml into 4 separate files now
 o Only load on demand what is needed at a given time
 o Add local caching for what was loaded, ensuring we never redundantly load metadata

This breaks artifact format and will require an artifact revision

This should improve performance for issue #294

c94f1264

by Tristan Van Berkom at 2018-04-11T14:10:55Z

element.py, source.py: Cleanup how Source cache keys are calculated.

Recently after a refactor we kept the Source adding workspace keys
to the source keys because, now clean this up to have the workspace
key added directly in the Element cache key calculation.

This breaks cache keys.

acfe2494

by Tristan Van Berkom at 2018-04-11T14:10:55Z

buildstream/_versions.py: Increment BST_CORE_ARTIFACT_VERSION

Now that we've restructured the artifact metadata into
separate files, we need to rev the artifact version.

be10c324

by Tristan Van Berkom at 2018-04-12T12:18:13Z

_frontend/status.py: Fixed status widgets to show full element names.

95d1dafc

by Tristan Van Berkom at 2018-04-12T12:37:24Z

_frontend/widget.py: Fixed regression in logfile abbreviations.

Added required size_request() method to new MessageOrLogFile() widget.

This came up in merge request !377

fece8cc8

by Tristan Van Berkom at 2018-04-12T12:39:06Z

_frontend/cli.py, _pipeline.py: Add options for cross junction tracking.

This patch makes cross junction tracking disabled by default, which
was the initial intention when landing project.refs but never got around
to doing this (intended to get addressing of junctioned elements via
command line sorted first, but didnt happen).

This adds the following options to enable cross-junction tracking:

  o bst build -J / --track-cross-junctions

  o bst fetch -J / --track-cross-junctions

  o bst track -J / --cross-junctions

This also fixes `bst fetch --track` which had a bug, it was avoiding
to track and fetch elements which are in a `cached` consistency state,
which is wrong when `--track` is specified.

This also updates some test cases which were broken by
this change.

This fixes issue #354

337fc965

by Tristan Van Berkom at 2018-04-12T12:39:06Z

NEWS: Added news entry detailing that cross-junction tracking is disabled by default

e3e72616

by Richard Maw at 2018-04-12T13:25:25Z

mount: Wrap yields in context managers with try-finally

The terminator context will only clean up on a signal,
so if another exception causes context manager cleanup
then unmount won't be called unless it's part of another context manager
or is wrapped in a try block's except or finally.

Everywhere else's unmounts are handled by delegating to another context manager
but these were what needed to be fixed.

The change in buildstream/_fuse/mount.py would cause lockups
since the build worker process still having a subprocess
blocked its termination from completing
which in turn caused the pipeline manager process to block indefinitely on it.

4aa482ae

by Tristan Van Berkom at 2018-04-12T15:01:36Z

_pipeline.py: Fix inconsistent element error to print element fullnames.

39c4cd17

by Tristan Van Berkom at 2018-04-12T15:01:36Z

Fix disaster while making cross junction tracking optional.

This disaster was introduced in fece8cc81e8d8412e32c6667682a33e7d2f9dafe

When doing `bst build --track`, we were:

  o first scheduling every element to be tracked

  o later filtering out the cross junction elements

  o finally asserting consistency state, which would trigger
    an error because we previously scheduled for tracking.

Fixed this by moving all code which resolves elements to track
into Pipeline.initialize(), and removing special element list handling
from the individual build/fetch/track commands.

522d6538

by Tristan Van Berkom at 2018-04-12T15:01:36Z

_scheduler/trackqueue.py: Mark skipped elements as tracked.

This ensures their state will be updated, it is especially important
for sourceless elements which are scheduled to be tracked, like stack elements.

This fixes a regression from my previous refactor or Sources
which added Element._tracking_done()

7c9713ad

by Richard Maw at 2018-04-12T18:03:09Z

frontend: capture errors for later use

This will be used to provide an error summary before the pipeline summary.

30af44e8

by Richard Maw at 2018-04-12T18:03:09Z

frontend: Use textwrap.indent to indent

The intent is more obvious by using an existing module.

7b0a9656

by Richard Maw at 2018-04-12T18:03:09Z

frontend: Summarise build log output in --no-verbose mode

bea028a8

by Richard Maw at 2018-04-12T18:03:09Z

frontend: Control indentations of multi-line values

fc6d192d

by Richard Maw at 2018-04-12T18:03:09Z

frontend: Redisplay failure messages in result summary

It's more convenient than having to search back through all the output
to find out what the cause of failure was.

eb522a4f

by Richard Maw at 2018-04-12T18:03:09Z

tests: Check that output includes log files of failed builds

d94d47c5
by Richard Maw at 2018-04-12T18:03:09Z
```
NEWS: Mention failed job summary
```

f1567355

by Javier Jardón at 2018-04-12T22:09:00Z

.gitlab-ci.yml: Create template for linux-tests job

9bc2bb04

by Javier Jardón at 2018-04-12T22:09:00Z

.gitlab-ci.yml: Run tests in Fedora:27 as well

1af6f442

by Javier Jardón at 2018-04-12T22:09:00Z

.gitlab-ci.yml: Change jobs from <distro>-tests to tests-<distro>

So they show in order nicely in the gitlab CI UI

8378f5dc
by Javier Jardón at 2018-04-12T22:15:33Z
```
.gitlab-ci.yml: Fix coverage job
```

63b7c821

by Tristan Van Berkom at 2018-04-13T06:01:37Z

.gitlab-ci.yml: Show which coverage reports are getting combined in CI

This extra line shows which reports will be combined, which helps us
to more quickly debug problems in the coverage job of the .gitlab-ci.yml

3b8dd43e

by Tristan Van Berkom at 2018-04-13T10:32:53Z

plugin.py: Documentation improvements.

Adding "Abstract Methods" section indicating what implementors
have to implement.

Also fixed some inconsistencies, and moved the abstract methods
to the top of the class.

1ea74544

by Tristan Van Berkom at 2018-04-13T10:32:53Z

element.py: Documentation improvements

Added heading section describing the abstract methods, and
moving the abstract methods to the top of the class.

03bead6c

by Tristan Van Berkom at 2018-04-13T10:32:53Z

source.py: Enhanced documentation

Added new Abstract Methods section here, and rearranged sources
such that abstract methods are on top.

596264d1

by Tristan Van Berkom at 2018-04-13T10:36:27Z

buildelement.py: Documentation improvements

Document here in terms of what the BuildElement does to implement
the abstract Element methods.

299df233

by Tristan Maat at 2018-04-13T11:51:17Z

Add element.prepare method

This is one of the tasks of #209

ccb6acc1

by Tristan Maat at 2018-04-13T11:51:17Z

integration/workspace.py: Test that configure commands run only once

d1b14086

by Tristan Maat at 2018-04-13T11:51:17Z

tests/frontend/workspace.py: Automatically update version

5632d291

by Tristan Maat at 2018-04-13T11:51:17Z

tests/frontend/workspace.py: Test new workspace format version

2cec9c27
by Tristan Maat at 2018-04-13T11:54:10Z
```
Update plugin documentation
```
cb702588
by Tristan Maat at 2018-04-13T12:02:51Z
```
Add NEWS entry
```

a4927ad3

by Tristan Van Berkom at 2018-04-14T08:32:09Z

tests/frontend/show.py: Removing some unused variables

ae8543c3

by Tristan Van Berkom at 2018-04-14T08:32:09Z

tests/format/variables.py: Ported from old style tests

188d819d

by Tristan Van Berkom at 2018-04-14T08:32:09Z

buildelement.py: Adhere to policy on private symbols

This is a part of issue #285

d45e12f7

by Tristan Van Berkom at 2018-04-14T09:13:27Z

source.py: Adhere to policy on private symbols

This is a part of issue #285

3ef81d7f

by Tristan Van Berkom at 2018-04-14T09:14:12Z

buildelement.py: Use Element.get_variable() instead of poking at private thing

This is the only place where Element._get_variables() was used, which we'll
now happily remove - also Element implementations in general should not
have to use private API.

7db09e2c

by Tristan Van Berkom at 2018-04-14T10:19:51Z

element.py: Adhere to policy on private symbols

This involves a lot of reordering of function in the Element class.

This is a part of issue #285

476526fc

by Tristan Van Berkom at 2018-04-14T10:23:00Z

element.py: Rename _get_full_display_key() -> _get_display_key()

The extra "full" was a bit redundant here, the Element does not
export any "less than full" display key accessor, so lets use
a more simple name for this.

1094d62b

by Tristan Van Berkom at 2018-04-14T10:45:09Z

_artifactcache/ostreecache.py: Extracting a missing artifact is a bug

Raise an assertion here, not an ArtifactError.

d603051e

by Tristan Van Berkom at 2018-04-14T10:45:40Z

_artifactcache/tarcache.py: Extracting a missing artifact is a bug

Raise an assertion here, not an ArtifactError.

d41940f5

by Abderrahim Kitouni at 2018-04-14T13:53:36Z

plugins/elements/cmake.yaml: allow using ninja instead of make (#279)

This uses the build tool mode of cmake to have a single command that can call
either make or ninja.

I've also modified the tests to take the new commands into account

f6f4738c

by Tristan Van Berkom at 2018-04-15T10:04:07Z

_frontend/app.py: Adhere to policy on private symbols

This is a part of issue #285

06da7a02

by Tristan Van Berkom at 2018-04-15T11:10:01Z

_frontend/profile.py: Adhere to policy on private symbols

This is a part of issue #285

219c7d29

by Tristan Van Berkom at 2018-04-15T11:10:01Z

_frontend/status.py: Adhere to policy on private symbols

This is a part of issue #285

e769beca

by Tristan Van Berkom at 2018-04-15T11:10:01Z

_frontend/widget.py: Adhere to policy on private symbols

This is a part of issue #285

f2edff31

by Tristan Van Berkom at 2018-04-15T11:48:44Z

buildstream/plugin.py, doc/source/projectconf.rst: Changed policy for core plugin format revisioning

The policy for format revisioning of core plugins maintained in BuildStream
now changes such that we revision all plugins under the same core format version.

This means that format extensions in plugins should be revisioned in
the global _versions.BST_FORMAT_VERSION along with the core format, this
simplifies parameterization of project.conf for usage of core plugins.

This did not really require any code support as our plugins have never
been revisioned before.

43e09ccf

by Tristan Van Berkom at 2018-04-15T11:50:01Z

_versions.py: Bump BST_FORMAT_VERSION for new cmake format extension

With commit d41940f516498f827967b8e5d311ce6accb88f56, the cmake plugin
now supports a new `generator` option which allows one to specify whether
to use `Unix Makefiles` (default) or `Ninja` (optional).

This requires a format version bump if people want to be sure that
the feature is supported.

075b9819

by Tristan Van Berkom at 2018-04-15T11:52:56Z

NEWS: Adding news entry for cmake plugin enhancement.

938ec1f4

by Tristan Van Berkom at 2018-04-16T06:15:30Z

doc/source/format.rst: Some enhancements in how we document dependencies

 * Added link target in the project.conf documentation for the element-path

 * List elements in examples without the leading `elements/` path, this
   is misleading and predates the existence of the element-path

 * Specify "element path relative" instead of "project relative", this
   was also outdated. Use these opportunities to link back to the element
   path project.conf docs section

 * Document dependency attributes differently

   - Use a list of attributes

   - Link back to the first example when speaking of shorthand, instead
     of duplication the example in two places

   - Enhanced description of how junction attributes are used

   - Note that ordering in dependency declaration lists is not meaningful

     This is an incorporation of Javier Jardón's patch for issue #358

This fixes issue #358.

5cedd894

by Tristan Van Berkom at 2018-04-16T06:50:56Z

element.py: Document new Element.prepare() method as available since 1.2

7c2a43d1

by Tristan Van Berkom at 2018-04-16T06:53:17Z

element.py: Document new BST_FORBID_SOURCES and BST_FORBID_RDEPENDS as available since 1.2

Also use any() for the check on the generators, instead of expanding the
whole list exhaustively and checking if the list is not empty.

ca2331c3

by Tristan Van Berkom at 2018-04-16T07:45:04Z

source.py, element.py, _pipeline.py: Streamling preflighting.

Instead of having the pipeline preflight all sources separately
from elements, have the element preflight it's sources.

This is in order to simplify the shared code path for the pipeline
and the loader to use for instantiating elements.

Also updated tests to expect the new ElementError and SourceError
instead of the PipelineError which was raised for preflighting before.

cd90fbde

by Tristan Van Berkom at 2018-04-16T09:26:05Z

Clean up element/source instantiation code paths.

This removes the extra `kind` parameter from all of the related
codepaths, it is redundant since the `kind` attribute is already
stored on the MetaElement and MetaSource objects.

b4f0a52a

by Tristan Van Berkom at 2018-04-16T10:59:47Z

element.py, _pipeline.py: Moved instantiation codepath to Element class methods.

This will allow the instantiation codepath to be shared by the Loader
which also needs to instantiate elements for junctions.

d02a1b4e

by Tristan Van Berkom at 2018-04-16T11:34:40Z

_loader.py: Use shared element instantiation codepaths for junction loading.

The consequently fixes issue #292, and you can now use workspaces
to work on junctioned projects.

e48f2f82

by Tristan Van Berkom at 2018-04-16T12:14:37Z

tests/frontend/buildcheckout.py: Added regression tests for workspaced junctions

This guards against regressions of issue #292

827a1294

by Tristan Van Berkom at 2018-04-16T12:52:04Z

_pipeline.py: Stop printing warning when a build plan does not use some existing workspaces

Fixes issue #360

eb0ad4d4

by Tristan Van Berkom at 2018-04-16T13:30:54Z

.gitlab-ci.yml: Perform some python code analysis with radon

This adds a new job in the prepare stage which can be viewed in
the gitlab CI, and also produces an artifacts with the json metrics.

a33ac81e
by Tristan Van Berkom at 2018-04-16T13:58:26Z
```
.gitlab-ci.yml: Add coverage regex
```
86587bf9
by Javier Jardón at 2018-04-16T13:58:26Z
```
README.rst: Add coverage badge
```

9cf8d35f

by Tristan Van Berkom at 2018-04-17T07:02:09Z

_frontend: Fix error messaging for closing workspaces

This fixes issue #182 again

c9b3aad7

by Tristan Van Berkom at 2018-04-17T07:24:40Z

_frontend: Earlier assertion for `bst workspace open` when directory is not empty

This just raises the error to the user that the checkout directory
where they intend to create a workspace in is not empty, a little
bit earlier, without bothering to initialize the pipeline first.

548c03f5
by devcurmudgeon at 2018-04-17T07:47:16Z
```
Update README.rst with more detail
```

4fc2da9a

by Tristan Van Berkom at 2018-04-17T08:31:48Z

element.py: Fix force opening of workspaces.

This was broken because we let the Source objects initialize workspaces
directly in the target workspace directory, where files might already
exist.

Instead, initialize the workspace in a tempdir and hardlink the files
from there.

This fixes issue #364

098ee3cf

by Tristan Van Berkom at 2018-04-17T08:31:48Z

tests/frontend/workspace.py: Added tests for force opening workspaces

Guard against regressions of issue #364

226880ee

by Javier Jardón at 2018-04-17T08:49:59Z

README.rst: Point to new location of x86image example

880e1913
by Paul Sherwood at 2018-04-17T12:36:46Z
```
Fix typo, add inspiration projects
```

6c23e337

by Chandan Singh at 2018-04-17T13:01:22Z

refactor: Do not assume there is only one workspace target in app.py

While working with workspaces in _frontend/app.py, we currently assume
that there is only workspace in the pippeline. This limits our ability
to support multiple workspaces in a given command.

Remove this assumption from _frontend/app.py which will allow for the
possibility of supporting multiple workspaces from _frontend/cli.py in
future. This commit does not change the behavior of these commands
though as the target is instead passed as an argument.

b264430d

by Chandan Singh at 2018-04-17T13:01:22Z

_frontend/cli.py: Add option to reset multiple workspaces

!357 added support for closing multiple workspaces. Similarly, also
allow `bst workspace reset` to work on multiple workspaces, with `--all`
as a helper to reset all open workspaces.

78a86f60

by Chandan Singh at 2018-04-17T14:53:17Z

NEWS: workspace close/reset support multiple elements

fb6687b6

by Tristan Van Berkom at 2018-04-17T15:55:01Z

_loader package: Creating a new sub package to hold the loader related things.

5f3da731

by Tristan Van Berkom at 2018-04-17T15:55:01Z

_loader package: Splitting up the loader code into separate pieces

This also refactors the loader code to honor private symbol
naming policy.

This is a part of issue #285

1159bbc8

by Tristan Van Berkom at 2018-04-17T15:55:01Z

_context.py: Removing unused _platform private variable

ea7d6ed2

by Tristan Van Berkom at 2018-04-17T15:55:01Z

_context.py: Make `fetch_subprojects` public.

This is actually used by the Loader and is not private.

This is a part of #285

8027b30d
by James Ennis at 2018-04-17T16:52:27Z
```
index.rst: remove captions
```

f0b350f1

by James Ennis at 2018-04-17T16:52:27Z

main_install.rst: New file with install, docker and artifacts

We now have a main installing page with three sections:
1. How to install buildstream onto your machine.
2. How to run buildstream inside docker image.
3. How to install an artifact server

4a49cedb

by James Ennis at 2018-04-17T16:52:27Z

main_authoring.rst: move to using buildstream

5d2f6ec6
by James Ennis at 2018-04-17T16:52:27Z
```
Add a general documentation section
```

02119f19

by James Ennis at 2018-04-17T16:52:27Z

index.rst: Remove unnecessary toctree blocks

58f8e6bc
by James Ennis at 2018-04-17T16:52:27Z
```
Consistent titling
```

516daa5c

by James Ennis at 2018-04-17T16:52:27Z

invoking.rst: renamed to commands.rst (anchors also changed)

5b5c87bf

by James Ennis at 2018-04-17T16:52:27Z

config.rst: Renamed to user_config.rst and anchors changed

e4cea3da

by James Ennis at 2018-04-17T16:52:27Z

core_framework.rst: New file containing content that was in main_core.rst

9a3f008e

by James Ennis at 2018-04-17T16:52:27Z

main_core.rst: Change title and moved authoring, sandboxing and cachekeys documentation here

974bd246

by Tristan Van Berkom at 2018-04-18T03:44:50Z

_scheduler: Move job.shutdown() calling responsibility inside of Job.

This was being called by Queue if Job succeeds, or inside Job if
Job fails and retries, lets just keep this inside of Job.

835e5475

by Tristan Van Berkom at 2018-04-18T04:49:40Z

_scheduler: Slightly changing the Job / Queue API contracts

For the Job:

   o The action_cb, complete_cb and max_retries are now passed
     into the constructor, and the Job.spawn() API has no arguments.

   o The complete_cb() signature has changed to take a success
     boolean instead of a returncode integer, and also now receives
     the result directly.

For the Queue:

   o Adapted to new Job API contract changes

   o Changed the Queue.done() method to now take a success boolean
     instead of a returncode integer

Updated Queue implementations to handle the success boolean instead
of the returncode integer.

6a4949c2

by Tristan Van Berkom at 2018-04-18T06:04:32Z

_scheduler/job.py: Adhere to policy on private symbols

This also removes an illegal peek into element._Element__environment
and uses the public Element.get_environment() accessor instead, removing
the need to also sanitize the returned dictionary.

Further, this reduces the amount of data passed between processes
in asyncio callbacks.

This is a part of issue #285

494fdd65

by Tristan Van Berkom at 2018-04-18T06:18:43Z

_scheduler: Adding Queue.dequeue_ready() to avoid peeking into private queue data

And micro-optimizing the scheduler use a generator _expression_ with any()
to check the result instead of creating a list.

3b447405

by Tristan Van Berkom at 2018-04-18T06:44:48Z

_scheduler/queue.py: Adhere to policy on private symbols

And change the API contract a bit more, now the Queue object
takes the scheduler in the constructor and the Queue.attach()
method is removed.

This is a part of issue #285

853c54eb

by Tristan Van Berkom at 2018-04-18T09:38:40Z

_scheduler/job.py: Don't mess with scheduler loop when forking

Instead of unregistering the mainloop handling of SIGTSTP, block
it (along with SIGTERM) while forking, clear the signal handlers
and then unblock them in the child process; let the parent handle
SIGTSTP after unblocking in the main process.

268340a5

by Tristan Van Berkom at 2018-04-18T11:02:25Z

_scheduler/scheduler.py: Adhere to policy on private symbols

Also added Scheduler.jobs_suspended() context manager to
use in the App instead of exposing Scheduler.suspend_jobs()
and Scheduler.resume_jobs() along with signal callback connect
and disconnect APIs, this keeps things more private in the
scheduler.

This is a part of issue #285

e18b7e5b

by Tristan Van Berkom at 2018-04-18T11:02:25Z

_scheduler/fetchqueue.py: Adhere to policy on private symbols

This is a part of issue #285

ee7e89d2

by Tristan Van Berkom at 2018-04-18T14:13:21Z

HACKING.rst: Clarify policy that related issues must be mentioned in commit messages.

This was somehow less than obvious.

fd9ee860

by James Ennis at 2018-04-18T14:16:02Z

HACKING.rst: Include Documentation Formatting Policy

149d0859
by James Ennis at 2018-04-18T14:16:02Z
```
HACKING.rst: move location of useful links
```

39a7c1f9

by Tristan Van Berkom at 2018-04-18T14:16:55Z

HACKING.rst: Fixing profiling section to follow rst formatting policy

Two empty lines before a section title, no empty line after the section title

ae1d5093
by Javier Jardón at 2018-04-18T14:19:00Z
```
README.rst: Add pipeline status badge
```

499c6f9d

by Richard Maw at 2018-04-18T14:50:04Z

frontend: Log which elements resolved new keys

It can be difficult to pick out important information in all the messages,
so it's convenient to have classes of them collected into one place.

One such class of important information is when the cache key of an element changes,
such as when element's source is tracked or cached from a workspace,
since it can be important to know what the cache key is for that artifact.

Fixes issue #252

a850b8ed
by Richard Maw at 2018-04-18T14:50:33Z
```
Update NEWS
```
cd75869f
by Javier Jardón at 2018-04-18T15:25:10Z
```
Add about page
```

899c3ce0

by Javier Jardón at 2018-04-18T15:25:10Z

doc/source/index.rst: Replace welcome paragraph

64069c8c

by Tristan Van Berkom at 2018-04-19T06:11:47Z

_pipeline.py: Added PipelineSelection

Part of a slow, ongoing refactor to unmangle the pipeline into
something which just creates pipelines of elements for a centerpiece
to process.

This also renames Pipeline.deps_elements() -> Pipeline.get_selection()

67437803

by Tristan Van Berkom at 2018-04-19T06:11:47Z

_pipeline.py, _frontend/app.py: Added track_selection initialization argument

This informs the pipeline what PipelineSelection mode to use
for constructing the list of elements to track.

d73d75c4

by Tristan Van Berkom at 2018-04-19T06:11:47Z

_frontend/cli.py: Set the track_selection argument to the deps parameter

Tell the pipeline to only track either 'none' or 'all'
PipelineSelection modes.

This fixes issue #367

670487be

by Tristan Van Berkom at 2018-04-19T06:11:47Z

tests/frontend/track.py: Added test for `bst track --deps none`

This guards against regressions of issue #367

6fbc81e7

by Tristan Van Berkom at 2018-04-19T06:38:09Z

buildstream/_artifactcache/pushreceive.py: Fixing unused variables

480876af
by Tristan Van Berkom at 2018-04-19T06:38:09Z
```
buildstream/_context.py: Fixing unused variables
```

6004943e

by Tristan Van Berkom at 2018-04-19T06:38:09Z

buildstream/_frontend/app.py: Fixing unused variables

dc30af8a

by Tristan Van Berkom at 2018-04-19T06:38:09Z

buildstream/_frontend/status.py: Fixing unused variables

ec2648cb

by Tristan Van Berkom at 2018-04-19T06:38:09Z

buildstream/_loader/loader.py: Fixing unused variables

cb3a2149

by Tristan Van Berkom at 2018-04-19T06:38:09Z

buildstream/_options/optionpool.py: Fixing unused variables

a8c31ff3
by Tristan Van Berkom at 2018-04-19T06:38:09Z
```
buildstream/element.py: Fixing unused variables
```
f5b99559
by Tristan Van Berkom at 2018-04-19T06:38:09Z
```
buildstream/plugin.py: Fixing unused variables
```

849d66e0

by Tristan Van Berkom at 2018-04-19T06:38:09Z

buildstream/plugins/sources/_downloadablefilesource.py: Fixing unused variables

0a718d86

by Tristan Van Berkom at 2018-04-19T06:38:09Z

buildstream/sandbox/_mount.py: Fixing unused variables

0830323d
by Tristan Van Berkom at 2018-04-19T06:38:09Z
```
buildstream/utils.py: Fixing unused variables
```
c1c09e36
by Tristan Van Berkom at 2018-04-19T06:38:20Z
```
.pylintrc: Enable linting for unused variables
```
78bd7f56
by devcurmudgeon at 2018-04-19T11:11:09Z
```
Add example formats
```

61889ed5

by Tristan Van Berkom at 2018-04-20T05:32:05Z

plugins: Disable attribute-defined-outside-init pylint error

Plugins set their attributes in configure(), because the
constructor is not public API.

671a7412

by Tristan Van Berkom at 2018-04-20T05:34:04Z

_context.py: Properly declare and document log_message_format in the constructor

be5b26c3

by Tristan Van Berkom at 2018-04-20T05:35:43Z

_pipeline.py: Declare tracking related members in the constructor

1fcfd000

by Tristan Van Berkom at 2018-04-20T05:37:39Z

_artifactcache/pushreceive.py: Fixed ProcessWithPipes to declare attributes in constructor

2282af27

by Tristan Van Berkom at 2018-04-20T05:40:46Z

_scheduler/job.py: Disable attribute-defined-outside-init lint error

This is in a derived Process from the standard lib, we add this
here because we are unsure about the constructor signature.

5f378fea

by Tristan Van Berkom at 2018-04-20T05:41:04Z

.pylintrc: Enable attribute-defined-outside-init messages

This will give us an error if an instance attribute is ever
declared outside of the constructor, which usually constitutes
either a bug or a violation of the coding standard, rendering
code more difficult to read.

1b7b51ac

by milloni at 2018-04-20T10:50:06Z

setup.py: Assert Bubblewrap version

The required version is 0.1.2 based on the fact that we're using
--hostname and --remount-ro.

7854fc50

by milloni at 2018-04-20T10:52:38Z

doc/source/install.rst: Mention bwrap version requirements

aab0229b

by Tristan Maat at 2018-04-20T11:00:46Z

deb.py: Make pylint stop complaining about arpy

0213102c

by Tristan Maat at 2018-04-20T11:03:30Z

HACKING.rst: Add a note about arpy being required for docs

1598d35e

by Tristan Van Berkom at 2018-04-23T06:59:16Z

doc/source/index.rst: Make main documentation title sober again.

Drop the happy welcome phrase with exclamation point.

f7713400

by Tristan Van Berkom at 2018-04-23T07:37:02Z

tests/pipeline/preflight-error: Fixing typo in error string formatting

3199d357

by Tristan Van Berkom at 2018-04-23T08:19:58Z

Overwrite sys.excepthook to handle exceptions in the main application.

This uses the existing messaging system to report errors in the main
application as a BUG type. It requires the use of a global_app variable
which isn't ideal; this may be replaced in future. This partially addresses
issue #197.

Theoretically, an exception could occur before Scheduler.loop is set up,
hence the check for it when terminating all jobs.

NOTE: This was originally submitted by Jim MacArthur, and manually
      reapplied after some refactoring took place.

4dd72d6c

by Javier Jardón at 2018-04-23T11:13:35Z

Revert "doc/source/install.rst: Mention bwrap version requirements"

Seems this was not propperly rebased; libostree and bubblewrap requirements
are already there.
Also the paragraph about docker was removed in
commit f0b350f158ed14c97613dc22b83ed89759b8d255

This reverts commit 7854fc5037204474d16618c657a1cd1ca12c888e.

c939d5f9

by Javier Jardón at 2018-04-23T11:13:35Z

doc/source/install.rst: minimum bubblewrap version is 0.1.2

2fe1010a

by Tristan Van Berkom at 2018-04-23T13:00:28Z

doc/source/install.rst: Removing blank line which appeared below title

Sections have 2 blank lines before them, and no blank line after them.

a18fcf0c

by Jim MacArthur at 2018-04-23T13:03:27Z

Add artifact cache receive profiling domain

_artifactcache/pushreceive.py: Wrap OSTreeReceiver.do_run in a
  profiling domain.
_profile.py: Add 'ARTIFACT_RECEIVE' domain.

1cd3728c

by Jim MacArthur at 2018-04-23T13:03:27Z

HACKING.rst: Add a section on BST_PROFILE and the artifact cache.

15789951

by Tristan Van Berkom at 2018-04-23T13:04:17Z

HACKING.rst: Fixing formatting of titles.

Titles have 2 blank lines before them, and no blank lines after them.

75ebcf56

by Tristan Maat at 2018-04-23T13:10:18Z

bst-here: Add '-p' flag to upgrade buildstream (Issue #291)

4759e2ad

by Sam Thursfield at 2018-04-23T13:37:06Z

plugins/elements/compose.py: Use sets instead of lists where appropriate

There is no significant order for the lists of added, removed and
modified files, so use an unordered set() to store the data.

d423732f

by Sam Thursfield at 2018-04-23T13:37:06Z

plugins/elements/compose.py: Only calculate snapshot when needed

This is an optimisation for the case where the compose element doesn't
do any splitting, and also brings 2 related code fragments closer to
each other.

cb10e2bd

by Sam Thursfield at 2018-04-23T13:37:06Z

plugins/elements/compose.py: Avoid losing files inside directory symlinks

The logic for determining which files were removed by integration
commands was broken when dealing with files staged within symlink
directories.

This rather weird scenario is only possible because of the way
BuildStream layers artifacts. If artifact 1 contains a symlink from
`/sbin` to `/usr/sbin`, and artifact 2 is staged on top and contains
a file `/sbin/init`, then the resulting filesystem contains a file at
`/usr/sbin/init`.

The manifest used by the compose element is generated from the contents
of the individual artifacts, so it lists the original paths such as
`/sbin/init`, but would would not contain `/usr/sbin/init` as nothing
has processed the symlinks.

The path `/sbin/init` is valid inside the composed tree, but filesystem
traversals that don't follow symlinks will not report that path in their
results. The compose plugin would look for `/sbin/init` in the results
of `utils.list_relative_paths()`, find it missing, and would act as if
some integration command had removed the file. This meant it would not
end up in the results.

To fix this, I have inverted the logic that processes the results of the
integration commands. We now work through every path in the manifest
and check it against the results of the integration commands, rather
than the other way around, and if any path from the manifest doesn't
appear in the snapshot we assume that it has staged in a different
location due to symlinks.

See: https://gitlab.com/BuildStream/buildstream/issues/270

3ec1cab1

by Sam Thursfield at 2018-04-23T13:37:06Z

Add compose-symlinks testcase

This is a regression test for the issue raised here:
https://gitlab.com/BuildStream/buildstream/issues/270

caa52411

by Sam Thursfield at 2018-04-23T13:37:06Z

tests/testutils/integration.py: Avoid inconsistent symlink handling

The output of walk_dir() seemed to be inconsistent in how it traversed
symlinks. Presumably this is to do with differences in how the filesystem
return files. If we do an in-place sort of the list of files and directories
that we get, os.walk() will honour that order which should make the output
stable.

84862708

by Javier Jardón at 2018-04-24T01:45:08Z

doc/source/install.rst: Update install instruction for Debian strech

dabaf601

by Javier Jardón at 2018-04-24T01:45:08Z

doc/source/install.rst: Update build instructions for Debian Testing/Sid

fbcf4ad4

by Javier Jardón at 2018-04-24T01:45:08Z

doc/source/install.rst: Update build instructions for Fedora

4be5f1f6

by Javier Jardón at 2018-04-24T01:45:08Z

doc/source/install.rst: Add fuse to the list of dependencies

See #358

07ac1bfa

by Javier Jardón at 2018-04-24T01:45:08Z

doc/source/install.rst: Add requirements to use the bst cache

See #358

b6a64313

by Javier Jardón at 2018-04-25T08:22:53Z

Move curent before_script: commands to the only jobs where they are actually needed

ae927b81

by Javier Jardón at 2018-04-25T08:29:55Z

.gitlab-ci.yml: Add job to analyze project code quality with Code Climate CLI

See #431

60029b31
by Javier Jardón at 2018-04-25T08:29:55Z
```
.gitlab-ci.yml: Move radon job to post stage
```

38ac4992

by Tristan Van Berkom at 2018-04-25T13:29:05Z

_projectrefs.py: Adhere to policy on private symbols

A late fix for issue #285

7caecc68

by Tristan Van Berkom at 2018-04-25T14:26:42Z

_loader: Give the loaded MetaSource objects the kind of the owning element

This is needed so that Sources can derive whether they belong
to a junction or not, which is needed for separating where
junction refs are stored.

719a3fc3

by Tristan Van Berkom at 2018-04-25T14:26:42Z

Load and save junctioned source refs from/to junction.refs

  o _projectrefs.py: Additional constructor option to choose the base name

  o _project.py: Load two ProjectRefs objects, one for the junctions

  o source.py: Load and save junctioned source refs with the appropriate ProjectRefs object

  o tests: Updated some tests to expect junctions to be stored in junction.refs

This fixes issue #361

52da8751

by Tristan Van Berkom at 2018-04-25T14:32:47Z

doc: Clarify that junction sources are stored separately from project.refs

54633e92

by Javier Jardón at 2018-04-25T16:16:18Z

.gitlab-ci.yml: Run test in current Debian stable (stretch)

4be462d3

by Tristan Van Berkom at 2018-04-26T05:17:44Z

Use versioneer instead of setuptools_scm

Using setuptools_scm had a couple of bad problems:

  o Unexpected versioning semantics, setuptools_scm would
    increment the micro version by itself in the case that
    we derive a version number from something that is not a tag,
    making the assumption that we are "leading up to" the next
    micro version.

    People mostly dont expect this.

  o When installing in developer mode, i.e. with `pip3 install --user -e .`,
    then we were always picking the generated version at install time
    and never again dynamically resolving it.

    Many of our users install this way and update through git, so it's
    important that we report more precise versions all the time.

This commit needs to make a series of changes at the same time:

  o Adds versioneer.py to the toplevel, this is used by setup.py
    for various activities.

    This is modified only to inform the linter to skip

  o Adds buildstream/_version.py, which is generated by versioneer
    and gives us the machinery to automatically derive the correct version

    This is modified only to inform the linter to skip

  o Adds a .gitattributes file which informs git to substitute
    the buildstream/_version.py file, this is just to ensure that
    the versioning output would work if ever we used `git archive`
    to release a tarball.

  o Modifies setup.py and setup.cfg for versioneer

  o Modifies utils.py and _frontend/cli.py such as to avoid importing
    the derived version when running bash completion mode, we dont
    derive the version at completion time because this can result
    in running a subprocess (when running in developer install mode)
    and is an undesirable overhead.

  o Updates tests/frontend/version.py to expect changed version output

16e719a9
by Tristan Van Berkom at 2018-04-26T05:23:45Z
```
NEWS: Added news entry for versioning fixes
```

b5c59295

by Tristan Van Berkom at 2018-04-26T05:47:17Z

element.py: Added BST_FORBID_BDEPENDS to compliment BST_FORBID_RDEPENDS

14aa2590

by Tristan Van Berkom at 2018-04-26T05:47:17Z

plugins/elements/junction.py: Forbid dependencies

Move this logic into the junction element itself, instead
of special case erroring for this in the loader.

edfcd0f6

by Tristan Van Berkom at 2018-04-26T05:47:17Z

_loader.py: Removing special case error for junctions depending on elements

Leave this error to be handled by preflight.

Updated test case to expect the new ElementError instead of a LoadError

0b35585b

by Tristan Van Berkom at 2018-04-26T05:55:17Z

_versions.py: Bumping format version for changes project.refs behavior

If we want to depend on being able to revision junction.refs and
project.refs separately, then we are better off just considering the
project.refs feature as available since the new version 8.

This will not harm projects which depended on it since version 5 instead.

78ad3b77

by Tristan Van Berkom at 2018-04-26T05:57:57Z

doc/source/projectconf.rst: Documenting ref-storage as available since format version 8

d3a991a5

by Tristan Van Berkom at 2018-04-26T06:11:43Z

setup.py: Stop installing test cases.

As recommended by Valentin David in issue #332.

24028499
by Tristan Van Berkom at 2018-04-26T06:40:13Z
```
man: Updated man pages for 1.1.3 release
```

7bb59282

by Tristan Van Berkom at 2018-04-26T06:42:27Z

README.rst: Removed 'covers many programming languages' from the About BuildStream

This is rather innacurate, you can build software in any programming languange
given that you have provided the required resources for this in the runtime
you are using.

06ae434a

by Tristan Van Berkom at 2018-04-26T06:43:52Z

README.rst: Fixed formatting

No empty lines after a section title, 2 empty lines before a section title.

d1c3d705

by Jürg Billeter at 2018-04-30T09:03:43Z

_artifactcache/pushreceive.py: Fix commit object check in send_info()

Fixes #325.

6f1d1552

by Jürg Billeter at 2018-04-30T09:03:43Z

_artifactcache/pushreceive.py: Do not return True for skipped push

cb9f318b

by Jürg Billeter at 2018-04-30T09:06:10Z

element.py: Print info message when artifact was actually pushed

This matches the pull code path.

0916d81e

by Jürg Billeter at 2018-04-30T09:06:10Z

tests/testutils/runcli.py: Improve detection of pulled/pushed elements

f00a435c

by Phillip Smyth at 2018-05-01T11:34:19Z

_artifactcache/pushreceive.py: Modified incorrect error message

changed encode_message push exception from
'Command must by GLib.Variant'
to
'Command must be PushCommand'

eceb7c98

by Jürg Billeter at 2018-05-03T14:06:55Z

element.py: Fix buildable check in non-strict mode

Ensure that the strong cache key of each build dependency is available
before an element is built. Otherwise the strong cache key of the
element cannot be calculated and caching the artifact produces an
AssertionError.

In non-strict mode an element's strong cache key may not be available
yet even though an artifact is available in the local cache. This can
happen if the pull job is still pending as the remote cache may have an
artifact that matches the strict cache key, which is preferred over a
locally cached artifact with a weak cache key match.

Fixes #383.

494f18fe

by James Ennis at 2018-05-04T09:24:41Z

commands.rst: Split up commands to easily allow for further description

An introduction for this section was also added

451b05fc

by Valentin David at 2018-05-06T06:47:17Z

Fix provenance in error message for undefined variables.

745d0647

by Valentin David at 2018-05-06T06:47:17Z

tests/format/variables.py: Add a test for undefined variable error message

53bff7d6

by James Ennis at 2018-05-07T10:25:52Z

app.py: Ensure an elements sub-dir is created upon project initialisation

2e252e9d
by Chandan Singh at 2018-05-07T11:21:32Z
```
bst-here: Fix typo in help text
```

6ffca067

by Chandan Singh at 2018-05-07T11:21:32Z

bst-here: Allow users to specify Docker image

Currently, the name of the Docker image is hardcoded in bst-here script.
This makes it harder for users to override it, which may be needed for
various reasons (custom plugins, private Docker registries etc.)

Add `-i` option to allow users to specify the base image. When `-i` is
not provided, default to `BST_HERE_IMAGE` environment variable if it's
set and otherwise to the current image -
`buildstream/buildstream-fedora`.

Also, re-order the command-line options in help text and source code in
alphabetical order to maintain sanity as the number of options is slowly
growing.

Part of #378.

49176039

by Chandan Singh at 2018-05-07T11:21:32Z

bst-here: Allow users to specify additional volumes to mount

One may want to mount additional volumes to preserve certain directories
or to share some data between the host and the container. Allow users to
do so by providing a `-v` option that passes its arguments to
corresponding `-v`/`--volume` option for `docker run` command.

Part of #378.

e6049105

by Javier Jardón at 2018-05-07T11:40:54Z

.gitlab-ci.yml: Remove redundant declaration of dependencies:

"artifacts from all previous stages are passed by default."
https://docs.gitlab.com/ee/ci/yaml/#dependencies

03823d12

by milloni at 2018-05-07T13:51:16Z

_site.py: Add check_bwrap_version() function

Lazily parse the version of bwrap the first time the function is called.
On subsequent calls, used cached version info.

See: #373

f8952d6b

by milloni at 2018-05-07T13:51:16Z

Replace bwrap checks with calls to check_bwrap_version

See: #373

bbb894bd

by Tristan Van Berkom at 2018-05-07T18:59:38Z

tests/format/variables.py: Removed testing of stderr

We have machine readable errors for this purpose, and the
strings happen to change causing tests to break if we test the specific
UI (reported error strings are UI).

2390c814

by Tristan Van Berkom at 2018-05-07T18:59:38Z

_stream.py: New Stream object, main calling interface for BuildStream core

This is the first part of the pipeline refactor, at this stage
all calling interfaces remain the same, except that invocation
of the scheduler has been moved from Pipline to Stream.

11dffaef

by Tristan Van Berkom at 2018-05-07T18:59:38Z

_stream.py: Add StreamError exception

Use Stream error for Stream errors.

76c260b0

by Tristan Van Berkom at 2018-05-07T18:59:38Z

_stream.py: Absorb workspace functionality from App.

c9f67f1e

by Tristan Van Berkom at 2018-05-07T18:59:38Z

_frontend: Early logging initialization

  o This makes logging independent from the Pipeline()

  o Removed size_request Widget() method, add context to Widget() initializer

  o Make the Status() widget derive anything it needs through the Context()

541cd760

by Tristan Van Berkom at 2018-05-07T18:59:38Z

_stream.py, _frontend: Call Stream() APIs with CLI arguments directly.

This shifts the whole responsibility of interpreting command line
targets etc to the Stream() object itself. With this commit, the
Pipeline() truly becomes slaved to the Stream().

c81fb2da

by Tristan Van Berkom at 2018-05-07T18:59:38Z

Moving 'fetch_subprojects' configuration to stream <--> loader

This was previously decided in CLI, but knowledge of what to initialize
has been moved to Stream().

Now there is no more point to store this configuration in the Context,
we just have the Stream() decide it when asking the Pipeline() to
invoke the Loader().

a542c818

by Tristan Van Berkom at 2018-05-07T18:59:38Z

_frontend/app.py: Remove partially_initialized()

There is no more need for this distinction.

b8e15706

by Tristan Van Berkom at 2018-05-07T18:59:38Z

_artifactcache: Added ArtifactCache.setup_remotes()

This removes some additional initialization code from Pipeline().

Some symbols have changed here, the initialization is now called
from Stream(), and a test case was also adjusted for this.

5bdc0a79

by Tristan Van Berkom at 2018-05-07T18:59:38Z

_stream.py, _pipeline.py: Refactoring of the pipeline itself

Here the pipeline becomes essentially stateless, some dangling
state remains to be factored out because of frontend accesses
which will be changed in a later commit.

Essentially, the Pipeline.load() method no longer has any knowledge
of the specific purposes of the loaded targets, and now takes
a list of target groups and returns a corresponding list of element
groups.

The Stream() business logic methods now use other pipeline helper
methods to create and filter lists from the loaded target elements.

The Stream() also finally absorbs the Scheduler frontend facing
APIs. However Queues are still exposed on the Stream object for
logging purposes and through callbacks such that the frontend can
retry elements.

1ec80edc

by Tristan Van Berkom at 2018-05-07T18:59:38Z

_frontend/cli.py: Fetch all elements when tracking a build plan

It makes not sense to type `bst fetch --track --deps all <targets>`,
because tracking will inevitably modify the build plan.

Stream initialization will not cope with this either, instead of
silently doing something which does not make any sense, we add an
assertion that this should not happen.

Unfortunately since `plan` is the default deps type for `bst fetch`,
this is likely to happen so it's important to correct.

This patch adds a warning in the case tracking of the build plan
elements is requested, and converts the request to track all elements
instead.

32c30766

by Tristan Van Berkom at 2018-05-10T06:59:47Z

_frontend/app.py: Fixed ordering of initialization sequence.

We need the logging handler setup before creating the Stream(),
as the Stream() will expect logging to be already functional.

8985da07

by Valentin David at 2018-05-10T11:33:12Z

buildstream/plugins/sources/_downloadablefilesource.py: Store etag along with cache.

Fixes #377.

edf3e029

by Tristan Van Berkom at 2018-05-10T12:20:06Z

Add soft reset functionality for workspaces

Add `--soft` option to `bst workspace reset` which would allow uses to
reset workspace-related state without affecting its contents. This will
be useful in case when an user wants to re-run configure-commands for a
workspaced element.

Patch originally by Chandan Singh, rebased against recent refactor.

Fixes #375.

b2b365fb

by Chandan Singh at 2018-05-11T01:01:46Z

_project.py: Remove unused internal function _extract_plugin_paths()

For reference, this method was added in
commit c41f1093d83eb32606f81005227542699f756b0a, but since
commit 4912ed5f8922b47c1a16c3752eb92a3f1dea76b2 it has been unused.

e1420437

by Jürg Billeter at 2018-05-11T05:57:37Z

_artifactcache/ostreecache.py: Fix global remotes

Counter-intuitively, Python list += mutates the list. Use explicit
copy() and extend() instead of += to avoid adding project-specific
remotes to the global remote list.

6d71826e

by Jürg Billeter at 2018-05-11T05:57:37Z

_pipeline.py: Remove unused ignore_cache parameter from _Planner

37e5362e

by Jürg Billeter at 2018-05-11T05:57:37Z

_pipeline.py: Allow pulling strict artifacts in non-strict mode

Remotely cached artifacts matching the strict cache key take precedence
over locally cached artifacts matching only the weak cache key. However,
locally cached artifacts were excluded from the build plan, which means
that BuildStream never even checked whether the strict artifact is
available in the remote artifact cache.

This changes planning to keep cached elements in the build plan in
non-strict mode if a remote artifact cache is used.

df17f602

by Jürg Billeter at 2018-05-11T05:57:37Z

tests: Do not rely on 'downloadable' status

This is in preparation for moving away from summary files.

da4af798

by Jürg Billeter at 2018-05-11T05:57:37Z

_frontend/cli.py: Remove bst show --downloadable option

This is in preparation for moving away from summary files.

4e512529

by Jürg Billeter at 2018-05-11T05:57:37Z

_stream.py: Remove unused downloadable parameter from load_selection()

db503a42

by Jürg Billeter at 2018-05-11T05:57:37Z

_artifactcache: Add bool return value to pull()

Calling pull() for a potentially unavailable artifact is no longer
considered an error.

e2c150f3

by Jürg Billeter at 2018-05-11T05:57:37Z

_artifactcache: Add element paramater to has_fetch_remotes()

17fd0b24

by Jürg Billeter at 2018-05-11T06:01:12Z

element.py: Do not query remote artifact caches ahead of pull/push

095ce5a5

by Jürg Billeter at 2018-05-11T06:01:12Z

_artifactcache/artifactcache.py: Remove unused methods

31ae2f55

by Jürg Billeter at 2018-05-11T06:01:12Z

_artifactcache/ostreecache.py: Do not use summary file

6ba62c57

by Jürg Billeter at 2018-05-11T06:01:12Z

doc/source/artifacts.rst: Drop section about summary file updates

Summary files are no longer used.

b78df1e2

by Jürg Billeter at 2018-05-11T06:01:12Z

_ostree.py: Remove unused function list_remote_refs()

270da060

by Jürg Billeter at 2018-05-11T06:01:12Z

Do not pull/fetch/build elements that are not required

For `bst build --deps plan`, do not process elements in pull/fetch/build
queues until they are requested by a reverse dependency.

3d9491dd

by Jürg Billeter at 2018-05-11T08:19:54Z

_stream.py: Do not force `PipelineSelection.ALL` for build with tracking

With the dynamic plan using `_set_required()` we can avoid
pulling/fetching/building unneeded elements even if elements are being
tracked.

01a02a9a

by Tristan Van Berkom at 2018-05-11T08:45:09Z

_workspaces.py: Use file mtime for workspace cache keys instead of checksumming

The rationale for this is basically:

  o A workspace is completely local

  o A cache key for a workspace is used to determine if the
    workspace has changed or not

  o Within reason, one should not expect buildstream to detect
    a change without a file's mtime having been modified

On my machine, it takes about 1 minute to checksum 11GB of WebKit checkout,
that is annoyingly a long time. With this change, it takes only 5 seconds.

Not perfect, but it's a start.

This is related to issues #294, #295 and #392

b9dea666

by Chandan Singh at 2018-05-11T11:29:45Z

bst-init: Ensure --element-path is respected by the command

Previously "elements" was hard-coded as the path for the elements
directory whereas it was supposed to be configurable via the
`element_path` option to `init_project()`. This led to incorrect
behavior when `bst init` was run `--element-path` option.

Also, extend tests to test the creation of elements directory.

Fixes #398.

cb24a1d8

by Tristan Van Berkom at 2018-05-16T06:18:07Z

doc/source/projectconf.rst: Added link anchors for project option types

cb076de8

by Tristan Van Berkom at 2018-05-16T06:18:07Z

doc/source/projectconf.rst: Added link anchor for source aliases

bb8bd1cf

by albfan at 2018-05-16T07:51:31Z

doc: Add first example, building on a flatpak SDK

This adds:
  o A ToC area for adding examples
  o The instructive example page for the first example
  o The example project under doc/examples
  o The corresponding integration test in tests/examples

b7e75da3

by Tristan Van Berkom at 2018-05-16T07:51:31Z

tests/examples/flatpak-autotools.py: Workaround setuptools bug

Until the setuptools bug is fixed, symlinks are not included
in source distributions - this works around the problem until which
point we can use a setuptools version without the bug.

b98791cb

by Tristan Van Berkom at 2018-05-16T07:51:31Z

doc/source/examples.rst: Rearranged ToC for examples

We want one example per file, not a huge file full of different examples.

e6f59b63

by Tristan Van Berkom at 2018-05-16T07:51:31Z

doc/examples/flatpak-autotools/README.md: Removing

This is redundant with doc/source/examples_flatpak_base.rst, lets
please only document things once.

c205991e

by Tristan Van Berkom at 2018-05-16T07:51:31Z

doc/source/examples_flatpak_autotools.rst: Enhanced example

This has now changed to:

  * Have explanations of the project.conf and each element
    in the example, explaining what they are for

  * Have links into the reference for the specific features
    this example uses, such that the reader can get familiar
    with the reference manual from example windows

e023beb5

by Tristan Van Berkom at 2018-05-17T05:46:46Z

_frontend: Notify failures when on linux

Opens the door to platform specific frontend features,
and implements a platform specific "notify" method.

This is based on Valentin David's patch on merge request 447,
and this fixes issue #385.

84a57b2f

by Tristan Van Berkom at 2018-05-17T10:56:04Z

_frontend/linuxapp.py: Fixed crash in previous commit

Commit e023beb56b180316a58f409d48f158e9dc1ece95 introduced
a crash with an invalid access to os.environ, this fixes it.

314c25d8

by Chandan Singh at 2018-05-17T13:21:09Z

element.py: Fix consistency of workspaced elements when ref is missing

Fixes #393.

ec16561f
by Jonathan Maw at 2018-05-17T15:45:50Z
```
tests: Add workspace tests for filters
```

8316a1fe

by Jonathan Maw at 2018-05-17T15:45:50Z

tests: Add track tests for the filter element

38dc2dda

by Jonathan Maw at 2018-05-17T15:45:50Z

element.py: Add _get_source_element() method

6e2e976c

by Jonathan Maw at 2018-05-17T15:45:50Z

filter.py: Redirect _get_source_element() calls to its build depend

We do this because trying to open/close/reset a workspace on a filter
element doesn't make sense, but is probably meant for the element that
the filter element filters.

c09be483

by Jonathan Maw at 2018-05-17T15:45:50Z

Redirect elements when performing tracking and workspace commands

i.e. when an element is specified, it may be replaced with its source element.

206a7ee8
by Jonathan Maw at 2018-05-21T09:59:45Z
```
NEWS: Add item for element redirection
```
77da5945
by toscalix at 2018-05-22T14:19:19Z
```
bst bug template
```
3749e44e
by toscalix at 2018-05-22T14:21:31Z
```
bst merge request template
```
b69e6482
by toscalix at 2018-05-22T14:25:22Z
```
correction :
```
4d92c106
by toscalix at 2018-05-22T14:27:29Z
```
correction :
```

a06778aa

by James Ennis at 2018-05-23T14:45:48Z

junction.py: Include note regarding explicit tracking

9b261f56

by toscalix at 2018-05-24T19:07:24Z

bst_task.md: default template also as option

a4cd9d7c

by toscalix at 2018-05-24T19:11:27Z

Merge branch 'master' of https://gitlab.com/BuildStream/buildstream

bst_task.md : default template also as option.

This is the default template. It is neccessary to include it in this directory to have it as option which is required in special cases.

This fix issue https://gitlab.com/BuildStream/nosoftware/alignment/issues/12

c0de75e2
by albfan at 2018-05-25T14:35:18Z
```
.gitignore: Ignore autogenerated docs
```

4708ee6b

by Antoine Wacheux at 2018-05-31T12:57:51Z

_artifactcache/ostreecache.py: Fix artifact cache initialization result tuple

In case of failure, the tuple contained 4 elements instead of 3, causing
BuildStream to crash.

065f5ac7
by Tristan Maat at 2018-06-05T08:47:33Z
```
_context.py: Normalize user-defined paths
```

279dcb40

by Tristan Maat at 2018-06-05T08:47:33Z

tests/context/context.py: Test setting XDG_CACHE_HOME

fe24fc3b

by Gökçen Nurlu at 2018-06-05T16:23:22Z

Add test to verify cleanup behaviour after bwrap

This adds `sandbox-bwrap/test-cleanup.bst` to test the cleanup behaviour in
`sandbox/_sandboxbwrap.py`: There already exists a non-empty `/tmp` folder and
BuildStream should not try to remove it after a command is executed with bwrap.

75fe8037

by Gökçen Nurlu at 2018-06-05T16:23:22Z

_sandboxbwrap.py: Fix post-bwrap cleanup behaviour

The cleanup was supposed not to remove folders (`/dev`, `/tmp`, `/proc`) if
they already existed before bwrap but it did the opposite: it tried to remove
them if they existed before, and didn't remove them if they were created during
bwrap. This was caused by a `not` clause, and this removes it.

Fixes #379

8d88b52a

by Phillip Smyth at 2018-06-05T18:00:03Z

utils.py: Correcting a typo in safe_remove's comment

1d694b28

by Chandan Singh at 2018-06-05T19:34:21Z

_artifactcache/pushreceive.py: Add Click type for CLI argument 'repo'

The CLI for `bst-artifact-receive` expects a `repo` argument, which is
supposed to be a directory, but Click currently expects it to be just
any string. This results in stack traces like the one below when the
argument provided is not a directory:

        $ ~/.local/bin/bst-artifact-receive --pull-url http://foo foobaz
        Traceback (most recent call last):
          File "/root/.local/bin/bst-artifact-receive", line 8, in <module>
            sys.exit(receive_main())
          ...
          File "/src/buildstream/buildstream/_artifactcache/pushreceive.py", line 581, in __init__
            self.repo.open(None)
        GLib.Error: g-io-error-quark: /src/buildstream/43fref: opendir(/src/buildstream/foobaz): No such file or directory (1)

Add types for this argument such that it throws better error messages
when it receives bad arguments. With the Click types added, it will
instead fail with messages like these:

        $ ~/.local/bin/bst-artifact-receive --pull-url http://foo foobaz
        Usage: bst-artifact-receive [OPTIONS] REPO

        Error: Invalid value for "repo": Directory "foobaz" does not exist.

        $ ~/.local/bin/bst-artifact-receive --pull-url http://foo setup.py
        Usage: bst-artifact-receive [OPTIONS] REPO

        Error: Invalid value for "repo": Directory "setup.py" is a file.

Fixes #409.

7f9216b3

by Tristan Van Berkom at 2018-06-06T17:20:24Z

_frontend/app.py: Remove unused variable at global scope

5dc8ab98
by Ed Baunton at 2018-06-06T17:56:59Z
```
Add a kind for Make
```

b36db258

by Tristan Van Berkom at 2018-06-06T18:50:54Z

doc/source/authoring.rst: Adding `make` element to the docs

5ddec56f

by Tristan Van Berkom at 2018-06-06T18:54:03Z

_versions.py: Bump format version to 9

For addition of new `make` plugin

17fa7ba4

by Tristan Van Berkom at 2018-06-06T18:54:27Z

make plugin: Ammended documentation to note it's since version.

0f2fd8c8
by Tristan Van Berkom at 2018-06-06T18:55:56Z
```
NEWS: Mentioning new `make` build element
```
55dc3b27
by Tristan Van Berkom at 2018-06-06T18:59:00Z
```
make build element: Add support for PREFIX
```

7df95654

by Chandan Singh at 2018-06-06T21:27:30Z

_loader/loader.py: Report element-path when failing to load elements

It can be confusing, especially to new BuildStream users, that the CLI
expects targets to be specified relative to element-path and not the
current directory. Previously, the CLI would give a generic message
stating that the file could not be found but it was not obvious that it
was looking in the `element-path` directory.

Explicitly print the element-path in the summary. Also, try to check if
the specified element exists in the elements directory and print a hint
to use the element-path relative paths if that's the case.

Fixes #396.

This is is also related to #341. This commit aims to tackle that issue
by trying to educate users about element-path.

a5ff465d

by Chandan Singh at 2018-06-06T22:10:49Z

_project.py: Allow running bst commands from subdirectories of project root

When initializing the project, BuildStream will continue searching for
project.conf in parent directories in case it is not found in the
current directory.

Fixes #368.

69e59443

by Chandan Singh at 2018-06-06T22:10:49Z

_frontend/cli.py: Try to autocomplete element paths when running from a subdirectory

The previous commit added support for running bst commands form
subdirectories of the project root. Make autocomplete also work in a
similar way.

e36001f9
by Ed Baunton at 2018-06-07T13:43:27Z
```
Make `bst help` work
```
b25c31ea
by Ed Baunton at 2018-06-07T17:34:27Z
```
Implement bst help
```

8234e9c7

by Tristan Van Berkom at 2018-06-07T20:14:11Z

_frontend/cli.py: Allow specifying commands in `bst help`

  o This supports deeply nested commands as well as shallow commands

  o Automated support for bash completions included

9c65b908

by Tristan Van Berkom at 2018-06-07T20:14:11Z

tests/completions: Added test case for custom `bst help` completions

4632cb03
by Tristan Maat at 2018-06-07T20:35:51Z
```
_ostree.py: Reintroduce remove()
```

60a29f68

by Tristan Maat at 2018-06-07T20:35:51Z

_ostree.py: Introduce _list_all_refs() and list_artifacts()

The unused list_remote_refs() function has also been removed as
part of this commit.

b7191fda
by James Ennis at 2018-06-07T20:35:51Z
```
pushreceive.py: Ensure there is a repopath
```

58adbe0b

by James Ennis at 2018-06-07T20:41:03Z

pushreceive.py: Remove LRP artifacts from cache, introduce clean_up_cache()

This fixes #136

8a0c0a9a

by James Ennis at 2018-06-07T20:41:33Z

element_generators.py: Add a utility function to create arbitrary size elements

New file consisting of functions that allow us to generate elements
on the fly
__init__.py now includes create_element_size()

ed0df6b5

by James Ennis at 2018-06-07T20:41:33Z

push.py: Add tests to test remote cache expiry

The tests include a new environment variable: BST_CACHE_QUOTA.
This variable is used in receive_main() in pushreceive.py.
Test names: test_artifact_expires, test_large_artifact_is_not_cached_remotely
test_recently_pulled_artifact_does_not_expire: marked as xfail until
we implement LRU expiry in remote share

b8a52cbd

by James Ennis at 2018-06-07T20:41:33Z

pushreceive.py: Add comments to code and name variables more sensibly

b823a616

by James Ennis at 2018-06-07T20:41:33Z

pushreceive.py: Ensure huge artifacts are not pushed

a231d41b
by James Ennis at 2018-06-07T20:41:33Z
```
pushreceive.py: Abstract the buffer_
```

de194dad

by Tristan Van Berkom at 2018-06-07T20:41:33Z

_artifactcache/pushreceive.py: Cleanup reported error when receiving oversized artifacts

This user facing string was redundantly declared in two places, only
the message when catching the error was ever printed.

8f2bf4e6

by Tristan Van Berkom at 2018-06-08T19:48:11Z

autotools plugin: Dont regenerate existing configure scripts

This closes #256

af10c1ba

by Valentin David at 2018-06-08T21:07:22Z

Interpret names as colon separated junction path in loader.

'a.bst:b.bst' gets interpreted as 'b.bst' from junction 'a.bst'.

Part of #359.

ccec163b

by Valentin David at 2018-06-08T21:07:22Z

Reword uses of project in Workspaces.

Make it clear we expect the top-level project here as we use it to
resolve paths relative to project directory.

130bfbb8

by Valentin David at 2018-06-08T21:07:22Z

Handle cross junction elements in workspaces.

Workspaces are now index by colon separated junction path. This
now allows to create workspaces for elements in external projects.

Workspaces are owned by context instead of root project. However
it is initialized once top-level project is registered as we need
to resolve paths relatively to this top-level project.

Part of #359.

acde3ba8

by Valentin David at 2018-06-08T21:07:22Z

Allow tracking dependencies within sub-projects.

--track-cross-junctions now concerns crossing junctions rather than
forbidding elements in sub-project to be tracked.

Part of #359.

71b78794

by Tristan Van Berkom at 2018-06-09T19:48:42Z

doc/source/index.rst: Use only one level depth ToC on main page

The main page has too much information on it otherwise, we want
a friendly, not overwhelming first page to our docs.

055b77e8

by Tristan Van Berkom at 2018-06-09T21:14:10Z

doc: Added plugins as ToC elements instead of orphaned links

  o Now the page titles are declared in plugins, allowing for
    a more descriptive ToC

  o Makefile and plugin.rsttemplate updated to not produce the title,
    to no longer use `:orphan:` for plugin pages, and to ignore any
    private modules in the plugin directories.

  o Interestingly, now the docs will fail to build if you add
    a new plugin and forget to add it to the documentation.

5f794829

by Tristan Van Berkom at 2018-06-09T21:23:23Z

doc/source/main_core.rst: Use a hidden toctree to hide away some things

Sphinx generates some library style module index, we just include
it in a hidden toctree and avoid using it altogether.

96c219d0

by Tristan Van Berkom at 2018-06-09T21:36:07Z

doc/source/core_framework.rst: Use a toctree instead of links

1e98ecb2

by Tristan Van Berkom at 2018-06-09T22:00:24Z

doc/source/index.rst: Add resources directly at the bottom of the main page

This is nice to have on the main page, and is only a few links, dont
like having a whole toplevel ToC entry for this.

77dd1091

by Tristan Van Berkom at 2018-06-09T22:08:49Z

doc: Split out cache keys and sandboxing writings into a sub section

53653d29

by Tristan Van Berkom at 2018-06-09T22:09:50Z

doc/source/authoring.rst: Removed unneeded link anchor from here

52412db4

by Tristan Van Berkom at 2018-06-09T22:18:28Z

.gitlab-ci.yml: Stop depending on specific version of sphinx

This branch fixes #312 by using a hidden toctree to include
the buildstream package and reducing the amount of allowed :orphan:
pages.

8e6f1a9b

by Tristan Van Berkom at 2018-06-09T23:44:25Z

doc: Stop generating the modules index

We still have a few unused artifacts in the docs generation,
this is just one less.

99fb2d53

by Tristan Van Berkom at 2018-06-09T23:45:41Z

doc/examples/flatpak-autotools: Use consistent project name

This was named autotools-flatpak, changed to flatpak-autotools.

710e1fd4

by Tristan Van Berkom at 2018-06-09T23:48:09Z

doc/source/examples_flatpak_autotools.rst: Added some links to the example project on gitlab.

4af8112d

by Tristan Van Berkom at 2018-06-10T00:11:07Z

doc/source/examples.rst: Use a flat ToC here

Here we're really listing examples, a ToC with depth is
not great here.

d0dc9e64

by Tristan Van Berkom at 2018-06-10T00:21:47Z

doc/source/main_using.rst: Reversed ToC here

Seems that the "commands" is taking a lot of space such that
we can't see the other sections here easily, that is alright
if "commands" remains at the end.

ea775fa3
by Tristan Van Berkom at 2018-06-10T01:12:37Z
```
doc: Removed some unused link targets
```

9f77cb66

by Tristan Maat at 2018-06-11T18:16:33Z

Handle missing tags in git repositories correctly

Fixes issue #380

e112a88f

by Tristan Van Berkom at 2018-06-12T21:38:25Z

doc/bst2html.py: Added bst2html.py

This baby runs bst and captures the output with colors enabled
and then generates some html we can include in documentation.

These can be generated in CI continuously and used in the documentation.

52f70a0d

by Tristan Van Berkom at 2018-06-12T21:38:26Z

doc/Makefile: Run bst2html.py to generate .html files from .run files

If you need an example output of bst to put in the documentation,
just add a .run file to the doc/examples directory and it will result
in a similarly named .html file in doc/source/examples.

050cf2eb

by Tristan Van Berkom at 2018-06-12T21:38:26Z

doc/sessions: Added run files describing scenarios to build

The Makefile uses these to run some scenarios which are
later included by the documentation directly.

c900d42e

by Tristan Van Berkom at 2018-06-12T21:38:26Z

doc/source/examples_flatpak_autotools.rst: Using generated examples

Show the commands at work in this example.

4dfc8607

by Tristan Van Berkom at 2018-06-12T21:38:26Z

doc/source/examples.rst: Restructuring individual examples into an examples subdirectory

19dddf82

by Tristan Van Berkom at 2018-06-12T21:38:26Z

doc/source/sessions: Committing generated session files.

This is only to make it easier for people who just want to
build docs locally and not regenerate the session files.

The session snapshot html files are always generated in CI every time.

f3b19f2c

by Tristan Van Berkom at 2018-06-12T21:38:26Z

HACKING.rst: Added guidelines for adding examples

And some instructions about generating `bst` output for documentation

d9da078d

by Tristan Van Berkom at 2018-06-12T21:38:26Z

.gitlab-ci.yml: Speed up docs build by setting up BST_SOURCE_CACHE

Use the gitlab cache for caching sources needed for building projects
in order to collect sample output for documentation pages.

53bc7272

by Tristan Van Berkom at 2018-06-13T02:41:15Z

doc: Don't allow setting a height on the generated session html

This was a bad idea and doesnt play well with mobile UIs, better
off to just include the whole thing even if it's long, and let
the backing page handle vertical scrolling.

9d7296eb

by Tristan Van Berkom at 2018-06-13T04:24:27Z

doc: Overhaul of page names and titles

 o Giving main pages simple word titles

   This makes the main page:

   * About
   * Installing
   * Using
   * Reference
   * Contributing

 o Now named all rst files with their parent page name as a prefix.

 o Also changed some titles to make overall consistent titles.

ba018efe

by Chandan Singh at 2018-06-13T20:24:15Z

contrib/bst-here: Fix command/arg in usage instructions

As we can only accept arguments when a command is specified, make it
clear in the usage instructions.

fa73abe4

by Chandan Singh at 2018-06-13T20:24:15Z

contrib/bst-here: Remove redundant create_volume_if_not_exists() method

This method doesn't really do anything as this is already the default
behavior of `docker volume create` so remove it and always call
`docker volume create` directly. This command will always print the name
of the volume on STDOUT which is not very interesting so silence that.
(If it errors out for some reason, that output will go to STDERR so the
user should still be able to see that.)

6da05577

by Tristan Van Berkom at 2018-06-14T03:12:53Z

doc: Improve the new console output html generator

Before we were creating one description file for each output,
making it easier to declare a make rule for it - but the result
was that we would have to build things more and it takes a
long time.

Instead, now we have session files which describe a series of commands
to run in a session, and each command optionally produces an output file.

5ba5415e

by Tristan Van Berkom at 2018-06-14T22:22:03Z

doc/Makefile: Fixed loop over session files

This list needs to not be quoted.

a01a5cc8

by Tristan Van Berkom at 2018-06-16T17:27:49Z

doc/bst2html.py: Add support for `fake-output` when running commands.

When specifying a fake-output string, we don't really run the command
or assume it was a `bst` command, and we pretend that `fake-output`
was the output of the command.

Specifying an empty string explicitly enables the behavior too
for faking a command that has no stdout/stderr.

This also adds the "remove-files" hack allowing the session scripts
to remove files before executing commands (kind of unsure if we're
gonna keep this...)

b592a80b

by Tristan Van Berkom at 2018-06-16T19:44:56Z

doc/bst2html.py, doc/Makefile: Added --force option

If --force is not specified, then we'll skip session files in
the case that all of the outputs exist.

Now setting BST_FORCE_SESSION_REBUILD when building the docs
will cause the session files to be rebuilt regardless of whether
they exist or not.

The .gitlab-ci.yml was also changed to use this and force rebuilds.

1b88e2c6

by Tristan Van Berkom at 2018-06-16T20:14:04Z

doc: Adding part 1 of the getting started tutorial

  o doc/Makefile: Added new directory to collect rst files from

  o doc/examples/first-project: Added the "first-project" example
    project.

  o doc/source/sessions/first-project-*.html: Added the generated
    snippets

  o doc/source/using_tutorial.rst: Added the new main tutorial page

  o doc/source/tutorial/first-project.rst: Added part 1 of the tutorial here

  o tests/examples/first-project.py: Added test for the example project

This is largely based on an example by Javier Jardón, which was
submitted at https://gitlab.com/BuildStream/buildstream/merge_requests/323

Fixes #103

6fa3f922

by Tristan Van Berkom at 2018-06-16T20:14:04Z

doc/Makefile: Stage stored sessions from session-stored

Separate the revisioned provisional session html files such
that the git tree does not become dirty as a result of a
documentation build process - which messes up the docs version
number and the version number printed in some command line output.

3f560378

by Tristan Van Berkom at 2018-06-18T00:58:20Z

doc/source/tutorial/first-project.rst: Adding links

Linking out to the relevant invoking pages for the command line
reference, and adding a link anchor here for use by the next chapter.

73b93002

by Tristan Van Berkom at 2018-06-18T01:37:22Z

doc: Adding part 2 of the getting started tutorial

  o doc/examples/running-commands: New example project of a `manual` build element

  o doc/sessions/running-commands.run: New session file to capture bst output

  o doc/source/sessions-stored: Added new generated sessions

  o doc/source/tutorial/running-commands.rst: New tutorial entry describing how
                                              commands are run in the sandbox

  o tests/examples/running-commands.py: Test case validating the tutorial's assertions

72fbaa1c

by Tristan Van Berkom at 2018-06-18T19:01:14Z

doc/source/tutorial/running-commands.rst: Adding summary

And adding some link anchors needed by the incomming chapter.

b4105e8d

by Tristan Van Berkom at 2018-06-18T19:17:52Z

doc: Adding part 3 of the getting started tutorial: autotools element

This part of the tutorial uses a lot of the work from Phil Dawson
and James Ennis, and uses their example submitted on merge request
499 as a basis to introduce the user to yaml composition and variable
resolution.

This is a part of issue #103

9b81f958

by Tristan Van Berkom at 2018-06-18T19:18:19Z

doc/source/using_tutorial.rst: Use a numbered ToC for the tutorial

48b4cb5c

by Tristan Van Berkom at 2018-06-19T00:19:09Z

doc/examples/autotools/project.conf: Correcting project name here

5b6441fa

by Tristan Van Berkom at 2018-06-19T02:41:50Z

doc: Use consistent titles when referring to files in an example project

Use the following form across the board:

  ``elements/foo.bst``
  ~~~~~~~~~~~~~~~~~~~~

  .. literalinclude:: ../path/to/foo.bst
     :language: yaml

Always use an example project relative path, too.

bb223156

by Tristan Van Berkom at 2018-06-19T02:47:46Z

doc/source/examples/flatpak-autotools.rst: Reordering of titles.

Place the titles of literally included `bst` files directly before
the includes, and moved all related text to start below the included
file for each section.

4016bec1
by Gökçen Nurlu at 2018-06-19T04:41:24Z
```
Remove shebangs from python files

Fixes #424
```

4f168b9b

by Daniel Playle at 2018-06-20T22:05:19Z

Restrict version of pylint

pylint >2 is not compatible with pytest_pylint in its current form. As
such, allowing any version of pylint for testing results in a failure.

This commit restricts down the allowable versions of pylint to those
that are both compatible with pytest_pylint, and also offer the feature
set that we require.

See https://gitlab.com/BuildStream/buildstream/issues/427 for further
details.

49fb3636

by Phil Dawson at 2018-06-21T23:32:46Z

MANIFEST.in: Fix typo when including BuildStream.doap

Correct includsion of buildstream.doap which does not exist
to 'include BuildStream.doap'.

This commit resolves issue #430

411dd100

by Patrick Martin at 2018-06-22T15:41:34Z

doc/source/index.rst: fix tiny, but impactful typo

Fixes #435

d9a433a7

by Javier Jardón at 2018-06-22T17:03:44Z

BuildStream depends on host tools for Source plugins

b5d91794

by Javier Jardón at 2018-06-22T17:03:44Z

Source plugin tar depends on host's lzip

Issue #353

7d97c6d3

by Javier Jardón at 2018-06-22T17:03:44Z

Source plugin bzr depends on host's bzr

Issue #353

3b1e869b

by Javier Jardón at 2018-06-22T17:03:44Z

Source plugin git depends on host's git

Issue #353

be92cc8d

by Javier Jardón at 2018-06-22T17:03:44Z

Source plugin patch depends on host's patch

Issue #353

e90098e0

by Javier Jardón at 2018-06-22T17:03:44Z

Source plugin deb depends on host's arpy python package

Issue #353

4f1d5a1c

by Tristan Van Berkom at 2018-06-22T17:05:39Z

doc/source/core_plugins.rst, doc/source/install_main.rst: Fixing link anchor name

In core_plugins.rst, we are already using _plugins, _plugins_build_elements,
so lets call this one _plugins_sources to be consistent.

10d21ff0
by Valentin David at 2018-06-23T01:17:40Z
```
Fix element check for BST_FORBID_BDEPENDS
```

6641bcd6

by Tristan Van Berkom at 2018-06-25T07:04:58Z

doc/source/main_using.rst: Updating headline for this page.

The purpose of this page has changed with time, better to clarify
this in the headline of the page.

6d4cbe0c

by Tristan Van Berkom at 2018-06-25T07:04:58Z

doc/source/format_declaring.rst: Improving linkage to other parts of the manual

Adding a couple of important links to relevant material.

4b499ba3

by Tristan Van Berkom at 2018-06-25T07:04:58Z

doc/source/tutorial/autotools.rst: Consistent wording of "chapter"

We've been calling these tutorials "chapters", let's not start
also calling them "sections".

1d84bb80

by Tristan Van Berkom at 2018-06-25T07:04:58Z

doc/source/tutorial/running-commands.rst: Added tip about using `--` in `bst shell`

a25d683e

by Tristan Van Berkom at 2018-06-25T07:04:58Z

doc: Adding part 4 of the getting started tutorial: integration commands

This new section talks about how integration commands work
and shows them at work.

91d87e3c

by Tristan Van Berkom at 2018-06-26T23:40:28Z

_scheduler/job.py: Added long comment

This explains the nature of a complicated asyncio callback,
which I've looked up on multiple occasions and is not available
in python online docs (only in the source can you follow it).

463698ec

by Francisco Redondo Marchena at 2018-06-28T14:18:10Z

Add error message when running commands on directories

Issue #446

48848f4e

by James Ennis at 2018-06-29T08:45:12Z

loader.py: Check whether the dir specified is also a <dir>.bst file

  - Slight modification to the wording of existing error message.
  - Closes #446

aaf52d03

by Tristan Maat at 2018-06-29T15:56:20Z

cli.py: Make buildstream check element paths instead of click

This is to avoid inconsistencies when dealing with paths inside an
elements directory

a85aaee6
by Tiago Gomes at 2018-07-02T14:45:04Z
```
Fix documentation typo
```

9067e269

by Tiago Gomes at 2018-07-02T14:45:04Z

Provide better error message on missing commands

Before running a command in the sandbox, check its existence and fail
early if it does not.

This fixes issue #289.

c5fe9465

by Jürg Billeter at 2018-07-08T12:33:46Z

_signals.py: Guard against use in non-main thread

3d1e23d6

by Jürg Billeter at 2018-07-09T13:44:49Z

tests: Remove unneeded ArtifactShare.update_summary() method

The OSTree summary file is no longer used.

5f4b9377

by Jürg Billeter at 2018-07-09T13:44:49Z

_artifactcache/artifactcache.py: Update remote init error message

initialize_remotes() no longer fetches remote refs.

9defbb62

by Jürg Billeter at 2018-07-09T13:44:49Z

tests/frontend/push.py: Add missing skip annotation for non-Linux

bc5a40e3
by Phillip Smyth at 2018-07-11T11:06:40Z
```
Move _list_dir_contents to __init__.py
```

ddeac63a

by Jonathan Maw at 2018-07-16T12:19:14Z

tests: Fix filter tests not checking whether files should be missing.

They weren't actually catching it if you checked-out the entire depended
element, instead of just the specified split domains

4c6512d6

by Jonathan Maw at 2018-07-16T16:12:08Z

Only redirect elements for workspace and track operations

It was redirecting for checkout operations

fa2eaba2
by Jürg Billeter at 2018-07-17T05:56:40Z
```
tests: Use context manager for ArtifactShare
```

ba4581f8