[Notes] [Git][BuildStream/buildstream][valentindavid/331

Valentin David pushed to branch valentindavid/331_include at BuildStream / buildstream

Commits:

7e9282c9

by Phillip Smyth at 2018-07-17T12:53:55Z

buildstream/_frontend/cli.py: Added a `--deps` flag to `bst checkout`

buildstream/_stream.py: Added deps param to _prepare_sandbox function call
buildstream/element.py: Added deps param and logic to _prepare_sandbox function

2797d1fd

by Phillip Smyth at 2018-07-17T12:53:55Z

tests/frontend/buildcheckout.py: Added tests for "Add a `--deps` flag to `bst checkout`"

70a83ef8
by William Salmon at 2018-07-17T14:04:16Z
```
Adding a helpful link to the example
```

e4f8fb07

by devcurmudgeon at 2018-07-17T15:07:05Z

Fix 'main install' to be explicit that it is for Linux distros only

87c103d8

by Jürg Billeter at 2018-07-17T15:07:56Z

_frontend/cli.py: Fix help text for `bst checkout --deps`

4c09b686

by Jürg Billeter at 2018-07-17T17:31:32Z

_artifactcache/cascache.py: Fix prune()

_reachable_refs_dir() failed to add directory digests to the set of
reachable objects.

adedd867
by Tristan Maat at 2018-07-18T05:11:03Z
```
Bump required python version to 3.5
```

b8543f8a

by Tristan Maat at 2018-07-18T05:11:03Z

_exceptions.py: Add `detail` to ArtifactErrors

fc79ad2c

by Tristan Maat at 2018-07-18T05:11:03Z

utils.py: Allow `list_relative_paths` to list directories

4f9da15d

by Tristan Maat at 2018-07-18T05:11:03Z

_scheduler/*queue.py: Move queues to a subdirectory

24925634
by Tristan Maat at 2018-07-18T05:45:59Z
```
Make Jobs abstract and element-independent
```
33984448
by Tristan Maat at 2018-07-18T05:45:59Z
```
Add cache_quota to user config
```

1ec5c7b1

by Tristan Maat at 2018-07-18T05:45:59Z

Make elements keep track of their built artifact size

7229d2e5

by Tristan Maat at 2018-07-18T05:45:59Z

Compute the artifact cache size after each build/pull

dc17de38

by Tristan Maat at 2018-07-18T06:07:24Z

Clean the artifact cache when we hit the cache quota

When the cache quota is hit, we will remove any elements not required
for the current build, until our cache is only filled halfway.

6ff12e5e

by Tristan Maat at 2018-07-18T06:07:24Z

tests/artifactcache/expiry.py: Add expiry tests

308d59be
by Tristan Maat at 2018-07-18T06:07:24Z
```
NEWS: Add news about cache expiry
```

a950a985

by Tristan Van Berkom at 2018-07-18T09:59:03Z

_context.py: Adding apis for message recording and task logging.

Added the new Context.recorded_messages() context manager, this
causes messages to be logged in a dedicated log file before being
propagated to the log handler, and also provides an open log handle
for the capturing and logging of stdout/stderr from child processes.

This comes with the additional accessors:

   Context.get_log_handle()
   Context.get_log_filename()

d835c37f

by Tristan Van Berkom at 2018-07-18T09:59:03Z

Refactor: Use new logging mechanism from Context

  o element.py / plugin.py: Removed supporting logging code, and derive
                            the log handle from Context.

  o _scheduler/scheduler.py, _scheduler/queues/queue.py: Adapt to new Job initializer API for the logfile

  o _scheduler/jobs/job.py: Run job activities within the new context manager
                            which turns on logging for a job. Also removed a lot
			    of custom logging abstract methods which are unneeded.

  o _scheduler/jobs: Job implementations need not implement so much custom logging.

26b9f6fa

by Tristan Van Berkom at 2018-07-18T11:52:43Z

_stream.py: Clarifying ambiguous FIXME comment about required artifacts

3814905a

by Daniel Playle at 2018-07-18T12:08:24Z

Reorder app.py imports

When testing locally on some systems, it appears that pylint fails on
the order of imports for `buildstream/_frontend/app.py`. This commit
changes the order of these imports.

This commit addresses issue !453.

48f66a3c

by Chandan Singh at 2018-07-18T18:03:31Z

.gitlab-ci.yml: Use testsuite images for running tests

Instead of re-using the user-facing `buildstream/buildstream-fedora`
image for tests (that comes with BuildStream pre-installed), use
`buildstream/testsuite-fedora` image that is supposed to be used
exclusively for these tests.

Similarly, use the dedicated `buildstream/testsuite-debian` image
instead of `buildstream/buildstream-debian`.

Fixes #458.

For related discussion, see
https://gitlab.com/BuildStream/buildstream-docker-images/issues/8.

1016dcb4

by Josh Smith at 2018-07-19T12:52:21Z

scheduler.py: Correct some anomalies within the docstrings

709b9bca

by Francisco Redondo Marchena at 2018-07-19T14:03:12Z

source-bundle: Enable --except option

Before this option was listed in help but was ignored when
creating the source-bundle.

Issue #468

96d07153

by Jordan Petridis at 2018-07-19T23:54:39Z

Docs: Update the required build packages for fedora based systems.

`ruamel.yaml` seems to require `Python.h` header file to build.

`python3-devel` is what provides it for Fedora.

93a77c1d

by Javier Jardón at 2018-07-19T23:58:28Z

Docs: Update the required build packages for the rest of the systems

Also remove the recomendation to install psutil as we need to build
other python modules anyway (like ruamel)

Completes 96d07153b7817cdaeda57dd163eed52b2b1b31e8

57e0d1c5

by William Salmon at 2018-07-20T00:03:06Z

Added dependency to the Docs

These dependencies are needed to use the plugins that are installed
as part of this guild.

As the dependencies added are not dependencies of the core package
they have been added to a separate section.

The lzip package is needed to build gnome.
https://wiki.gnome.org/Newcomers/BuildSystemComponent

1b6688eb

by Javier Jardón at 2018-07-20T00:07:23Z

install_linux_distro.rst: document plugins installed by default

fc63b972

by Javier Jardón at 2018-07-20T00:12:50Z

install_linux_distro.rst: Document plugins deps for the rest of the systems

790fb40b

by Tiago Gomes at 2018-07-20T09:07:01Z

Update checkout man page for new --deps option

9c1f024b

by Tiago Gomes at 2018-07-20T09:07:01Z

Add support for creating a tarball on bst checkout

One of the tests added is configured to be skipped for now, as dumping
binary data is causing a bad descriptor exception when using the pytest
capture module.

Closes #263.

0ddbf0e0

by Tristan Van Berkom at 2018-07-20T11:52:00Z

setup.py: Specify minimum required version of pytest-cov plugin

This causes the new artifact tests to pass (unless you happened
to already have a recent enough version of pytest-cov, in which case
you didn't notice the breakage).

6158ee5a

by Sam Thursfield at 2018-07-20T13:17:14Z

Fix crash when --debug is passed

I hit the following backtrace running `bst --debug push` and `bst
--debug build`:

    pid:16736 id:000[--:--:--][][] START   Push
    pid:16736 id:000[--:--:--][][] START   Loading pipeline
    pid:16736 id:000[00:00:00][][] SUCCESS Loading pipeline
    pid:16736 id:000[--:--:--][][] START   Resolving pipeline
    pid:16736 id:000[--:--:--][][] BUG     'MesonElement' object has no attribute '_Element__cache_key'

        Traceback (most recent call last):
          File "/home/sam/.local/bin/bst", line 8, in <module>
            sys.exit(cli())
          File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
            return self.main(*args, **kwargs)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/cli.py", line 162, in override_main
            standalone_mode=standalone_mode, **extra)
          File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 697, in main
            rv = self.invoke(ctx)
          File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
            return _process_result(sub_ctx.command.invoke(sub_ctx))
          File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
            return ctx.invoke(self.callback, **ctx.params)
          File "/home/sam/.local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
            return callback(*args, **kwargs)
          File "/home/sam/.local/lib/python3.6/site-packages/click/decorators.py", line 27, in new_func
            return f(get_current_context().obj, *args, **kwargs)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/cli.py", line 471, in push
            app.stream.push(elements, selection=deps, remote=remote)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_stream.py", line 341, in push
            fetch_subprojects=True)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_stream.py", line 814, in _load
            fetch_subprojects=fetch_subprojects)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_pipeline.py", line 119, in load
            for meta in meta_elements
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_pipeline.py", line 119, in <listcomp>
            for meta in meta_elements
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 894, in _new_from_meta
            element = project.create_element(artifacts, meta)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_project.py", line 184, in create_element
            element = self._element_factory.create(self._context, self, artifacts, meta)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_elementfactory.py", line 57, in create
            return element_type(context, project, artifacts, meta, default_config)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 196, in __init__
            super().__init__(meta.name, context, project, meta.provenance, "element")
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 171, in __init__
            self.debug("Created: {}".format(self))
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 407, in debug
            self.__message(MessageType.DEBUG, brief, detail=detail)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/plugin.py", line 652, in __message
            self.__context.message(message)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_context.py", line 338, in message
            self._message_handler(message, context=self)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/app.py", line 665, in _message_handler
            text = self.logger.render(message)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 585, in render
            return self._render(message)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 620, in _render
            text += widget.render(message)
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/_frontend/widget.py", line 228, in render
            _, key, missing = plugin._get_display_key()
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 1138, in _get_display_key
            cache_key = self._get_cache_key()
          File "/home/sam/.local/lib/python3.6/site-packages/buildstream/element.py", line 991, in _get_cache_key
            return self.__cache_key
        AttributeError: 'MesonElement' object has no attribute '_Element__cache_key'

The issue here is that when `--debug` is enabled, Plugin.__init__()
tries to log a message like this:

        pid:16929 id:001[--:--:--][????????][ main:tracker.bst                   ] DEBUG   Created: meson element at tracker.bst [line 1 column 0]

The log formatter is trying to get the cache key of the element in
order to show it in the log, but the attribute hasn't yet been set
because we've not got to the Element() constructor.

f484ab4f

by Phillip Smyth at 2018-07-20T14:29:44Z

Updating .bzr plugin to omit the .bzr dir

Fixes issue #455

b662f175

by Tristan Van Berkom at 2018-07-20T14:29:44Z

git source plugin: Omit the .git directory completely.

This will break builds which use `git describe` to determine
their version number until a more elegant solution is implemented,
and will unblock work on caching of build trees in the artifacts.

Fixes issue #455

80effb5c

by Tristan Van Berkom at 2018-07-22T08:30:40Z

HACKING.rst: No longer requiring issue numbers in commit messages.

Instead, we are enabling the merge commit feature in gitlab and
hopefully reducing some of the friction in review this way.

c7124c3d

by Tristan Van Berkom at 2018-07-22T09:46:05Z

HACKING.rst: Added convention for naming of branches in merge requests.

282b1092

by Tristan Van Berkom at 2018-07-23T08:47:36Z

plugins/sources/git.py: Use --shared instead of --hardlinks

This improves staging performance by avoiding making any
copies of the objects in the cloned repository in the staging
area before removing the .git directory.

Fixes #488

fe0df616

by Tristan Van Berkom at 2018-07-23T09:20:44Z

Merge branch 'tristan/git-stage-with-shared-clone' into 'master'

plugins/sources/git.py: Use --shared instead of --hardlinks

Closes #488

See merge request BuildStream/buildstream!557

6559db4c

by Josh Smith at 2018-07-23T09:22:39Z

element.py: Make Element._get_brief_display_key() available to Internal API

This change will allow for more detailed info messages throughout
the CASCache.

6b82e476

by Josh Smith at 2018-07-23T09:22:39Z

cascache.py: Add push info detailing which remote is being pushed to

Work towards fixing issue #275

1d36df25

by Josh Smith at 2018-07-23T09:22:39Z

cascache.py: Add pull info detailing which remote is being pulled from

Work towards fixing issue #275

364706f3

by Tristan Van Berkom at 2018-07-23T09:57:02Z

Merge branch 'Qinusty/275' into 'master'

Indicate where artifacts are going to and coming from in the log

Closes #275

See merge request BuildStream/buildstream!553

e70237a5

by Phillip Smyth at 2018-07-23T13:26:47Z

Adding caching build trees

buildstream/element.py: Adding build tree to cache
buildstream/_versions.py: Bumping BST_CORE_ARTIFACT_VERSION

335afb5b

by Tristan Van Berkom at 2018-07-23T14:07:07Z

Merge branch 'caching_build_trees_limited' into 'master'

Adding caching build trees (limited)

See merge request BuildStream/buildstream!560

60290f31

by Javier Jardón at 2018-07-23T15:29:37Z

doc/source/install_linux_distro.rst: buildstream doesn't depend on ostree or pygobject anymore

de162e43

by Javier Jardón at 2018-07-24T11:58:16Z

Merge branch 'jjardon/ostree_repo' into 'master'

doc/source/install_linux_distro.rst: buildstream doesn't hard depend on ostree or pygobject anymore

Closes #492

See merge request BuildStream/buildstream!558

6106d657

by Josh Smith at 2018-07-25T10:50:14Z

cascache.py: Remove misleading info message

This addresses issue #501.

4a637d63

by Jonathan Maw at 2018-07-25T12:05:19Z

Merge branch 'Qinusty/501' into 'master'

Remove misleading info message

See merge request BuildStream/buildstream!567

a0ad985f

by Phil Dawson at 2018-07-25T13:03:23Z

tests/frontend/show.py: Add test case for maximum recursion depth being exceeded

Add test to ensure gracefull handling of exception thrown while loading
a pipeline due the python's max recursion depth being exceeded.

This is part of the work for issue #203

95b4eae4

by Phil Dawson at 2018-07-25T13:03:23Z

app.py: Handle exception thrown when recursion limit is exceeded

Because the RecursionError exception was introduced in Python 3.5, until we
drop support for for Python 3.4, we must use RuntimeError.

889bf238

by Jim MacArthur at 2018-07-25T13:43:21Z

Merge branch 'phil/203-BuildStream-crashes-when-dependency-tree-too-deep' into 'master'

Phil/203 BuildStream crashes when dependency tree too deep

See merge request BuildStream/buildstream!512

54fee6c8

by Phillip Smyth at 2018-07-25T14:15:36Z

_stream.py: Added functionality for workspace open -f

tests/frontend/workspace.py: Added tests

f62b6cb7

by Jonathan Maw at 2018-07-25T15:01:33Z

Merge branch 'bst_workspace_open_force_does_nothing' into 'master'

_stream.py: Added functionality for workspace open -f

See merge request BuildStream/buildstream!549

c049d021

by Ed Baunton at 2018-07-25T19:43:51Z

Add remote source plugin

Add a plugin that supports downloading files verbatim from a source with
an optional overridable filename and destination directory. Bumps bst
format version to 10.

Fixes #163

bd1196ef

by Ed Baunton at 2018-07-26T13:22:52Z

Provide explicit reason for SourceError exception on path

When the user provides a path for the filename parameter, provide a reason

9a46c16f

by Ed Baunton at 2018-07-26T14:06:52Z

Merge branch 'edbaunton/remote-source' into 'master'

Add remote source plugin

Closes #163

See merge request BuildStream/buildstream!541

7ceacfea

by William Salmon at 2018-07-26T14:32:49Z

Documentation typos and fixes

Daniel went through the Doc's etc and made a list of suggestions.
These are the result of the most easily patched suggestions and
one or two thing that I noticed as well.

8c9fd9e4

by Javier Jardón at 2018-07-26T15:32:16Z

Merge branch 'willsalmon/documentation_form_notes' into 'master'

Documentation typos and fixes

See merge request BuildStream/buildstream!569

9ebd7fe1

by Josh Smith at 2018-07-27T04:41:26Z

_exceptions.py: Modify BstError API to allow optional retry

job.py: Changes to the logic surrounding retry attempts and child process return codes
element.py, source.py: ElementError and SourceError also implement this change.

These exceptions now have an optional parameter of temporary which defaults to false. This will potentially break
backwards compatibility where exceptions were previously raised and a retry was intended.

To trigger a retry, one must now raise their SourceError or ElementError with temporary=True.

This aims to fix #397.

ca224425

by Josh Smith at 2018-07-27T04:41:26Z

_downloadablefilesource.py: Update network errors to be temporary

This follows the change in 67ecd97a05279a3b7570ad59f05bf0a5973ef04c.

0a7cfcdf

by Josh Smith at 2018-07-27T04:41:26Z

tests/sources: deb, tar and zip all test for retry behaviour

This is following 5d6418a06a3a0262df5f6b9e6e1a36578515fd4a where
DownloadableFileSource was modified to raise 'temporary'
errors when relating to file downloading.

9b2bc78f

by Josh Smith at 2018-07-27T04:41:26Z

cascache.py: Update cache push/pull errors to be temporary

Further work needs to be done for the current grpc exceptions which are reraised.

4986527f

by Josh Smith at 2018-07-27T04:41:26Z

plugin.py: Extend Plugin.call() API

Plugin.call() now takes fail_temporarily as an optional parameter,
when supplied it will cause subsequent failures to trigger temporary
errors as opposed to permanent errors.

This also extends Plugin.check_output() which makes use of Plugin.call()

a9a1048e

by Josh Smith at 2018-07-27T04:41:26Z

git.py: Update git plugin to fail temporarily on clone failures

The git plugin will now make use of the fail_temporarily parameter
to Plugin.call(), allowing failures to trigger a retry.

aa071cb6

by Tristan Van Berkom at 2018-07-27T06:13:53Z

Merge branch 'tristan/temporary-failures' into 'master'

Temorary failures

Closes #397

See merge request BuildStream/buildstream!574

4d86e7c5

by Tristan Van Berkom at 2018-07-27T06:43:48Z

element.py, source.py, plugin.py: Adjusting API version since annotations.

Since we backported the temporary/permanent failures patch for #397
into the `bst-1.2` branch, we need to adjust the since versions in master
down to 1.2.

fb222ba1

by Josh Smith at 2018-07-27T08:50:45Z

cascache.py: Raise ArtifactError on grpc error

This aims to fix issue #502.

48916b8a

by Tristan Maat at 2018-07-27T10:22:41Z

Merge branch 'Qinusty/502' into 'master'

cascache.py: Raise ArtifactError on grpc error

Closes #502

See merge request BuildStream/buildstream!576

23e080b9
by James Ennis at 2018-07-27T11:13:50Z
```
README.md: Minor corrections
```

196cfffc

by James Ennis at 2018-07-27T11:13:50Z

README.md: Direct users towards the tutorial before existing projects:

  This patch also removes displaying the projects that live within
  the buildstream-examples repository, as we wish to deprecate this
  repo and migrate the two examples.
  - #438 and #439

  In addition to this, both examples do not 'build' in their current
  state.

2b93574e

by James Ennis at 2018-07-27T11:13:50Z

main_install.rst: Add note to refer Windows and macOS users to docker

63e2320e
by Josh Smith at 2018-07-27T11:16:55Z
```
.gitignore: Ignore .pytest_cache/
```

6ea97b17

by Javier Jardón at 2018-07-27T11:54:20Z

Merge branch 'jennis/doc_improvements' into 'master'

Refer readers to our tutorial before referring them to existing bst projects

See merge request BuildStream/buildstream!578

8b46e874
by Jonathan Maw at 2018-07-27T12:24:56Z
```
project: Parse and store mirrors
```

a2e9c62a