[Notes] [Git][BuildStream/buildstream][coldtom/autotools-libtool] 80 com

Javier Jardón pushed to branch coldtom/autotools-libtool at BuildStream / buildstream

Commits:

6047a575

by Tristan Van Berkom at 2018-08-28T13:41:36Z

tests/frontend/workspaces.py: Removing some redundant tests

Removed redundant tests from recently merged !740, this new
test does not need to run for every different source kind.

7535fda8

by Jürg Billeter at 2018-08-28T14:25:35Z

Merge branch 'tristan/remove-redundant-test' into 'master'

tests/frontend/workspaces.py: Removing some redundant tests

See merge request BuildStream/buildstream!741

10eeb503
by Javier Jardón at 2018-08-28T22:22:00Z
```
doc/source/conf.py: Fix E201, E202 warnings
```
d3c32ca2
by Javier Jardón at 2018-08-28T22:22:00Z
```
doc/source/conf.py: Fix E402 warning
```
3ae5fd05
by Javier Jardón at 2018-08-28T22:22:00Z
```
Fix E305 warnings
```
c6fb5ba7
by Javier Jardón at 2018-08-28T22:22:00Z
```
Fix E741 warnings
```

c5eca59d

by Tristan Van Berkom at 2018-08-29T10:45:33Z

Merge branch 'jjardon/pycodestyle_fixes' into 'master'

Some pycodestyle (PEP8) fixes

See merge request BuildStream/buildstream!746

63c6ee72

by Tristan Van Berkom at 2018-08-29T10:46:30Z

doc: Adding badges.py release badge generator

This adds a step to the docs generation Makefile to generate
release.svg and snapshot.svg badges, modelled after the gitlab
badges.

This also adds the generated badges directory in docs to .gitignore

2d527052

by Tristan Van Berkom at 2018-08-29T10:46:30Z

README.rst: Adding the release and snapshot badges to the readme

This is a part of #528

29e7eea8

by Tristan Van Berkom at 2018-08-29T10:46:30Z

doc: Adding the release badges to the install page and the semantic versioning page

This is a part of #528

5d508779

by Tristan Van Berkom at 2018-08-29T11:13:53Z

Merge branch 'tristan/docs-version-badge' into 'master'

Release badges

See merge request BuildStream/buildstream!742

323a5403

by Chandan Singh at 2018-08-29T12:13:28Z

setup.py: Add more metadata useful for PyPI

- setup.py: List useful links using `project_urls`

- setup.py: Specify minumum python vresion using `python_requires`

  `python_requires` is the new way of specifying requirements for python
  version, as documented @
  https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires.

  This will also make this requirement appear nicely on the PyPI project
  page.

- setup.py: Add project classifiers

  PyPI will use these classifiers to categorize projects while searching
  or browsing. Full list of classifiers can be found at
  https://pypi.org/classifiers.

5fc7b5ac

by Tristan Van Berkom at 2018-08-29T12:37:08Z

Merge branch 'chandan/setup.py-metadata' into 'master'

setup.py: Add more metadata useful for PyPI

See merge request BuildStream/buildstream!718

42ad937d

by Josh Smith at 2018-08-29T12:38:15Z

Prevent ValueError on URLs without an alias

Fixes #597

aa3a33b3

by Tristan Van Berkom at 2018-08-29T13:03:58Z

Merge branch 'Qinusty/597-non-alias-url-fix' into 'master'

Fix ValueError traceback for URL without alias or URI scheme

Closes #597

See merge request BuildStream/buildstream!700

2a2a79de

by Valentin David at 2018-08-29T14:11:48Z

Disable round-tripping when element is not modified

2df7607c

by Tristan Van Berkom at 2018-08-29T14:43:55Z

Merge branch 'valentindavid/roundtripping_only_when_modified' into 'master'

Disable round-tripping when element is not modified

See merge request BuildStream/buildstream!748

c146dde5
by Benjamin Schubert at 2018-08-29T16:00:24Z
```
Log not-found objects in the cache as SKIPPED
```

19e87afb

by Benjamin Schubert at 2018-08-29T16:01:12Z

Add tests that not-found objects in cache are shown as SKIPPED

9cc5817f

by Qinusty at 2018-08-29T16:25:48Z

Merge branch 'bschubert/log-missed-cache' into 'master'

Log not-found objects in the cache as SKIPPED

See merge request BuildStream/buildstream!729

3c8646e3

by Josh Smith at 2018-08-29T16:34:22Z

Add cyclic check within variable resolution

This aims to address #600, this will raise an exception when a resolved
variable contains a reference to the variable.

d41c42d6

by Josh Smith at 2018-08-29T16:34:22Z

Add tests for cyclic variables check

Note: This modifies the docker containers used for testing to supply the
pytest-timeout package.

c7b72c30

by Qinusty at 2018-08-29T16:58:48Z

Merge branch 'Qinusty/600-recursive-variables' into 'master'

Add cyclic check within variable resolution

See merge request BuildStream/buildstream!712

d28e1353

by Tristan Van Berkom at 2018-08-30T06:02:41Z

tests/plugins/filter.py: Don't run redundant tests

There is no reason that the filter element codepaths can behave
differently depending on the Source implementation used in the test,
as the Source implementation does not have any filter specific
virtual methods.

Removing the redundant tests and just performing these tests with the git source.

e77ae07a

by Tristan Van Berkom at 2018-08-30T06:36:41Z

Merge branch 'tristan/reduce-filter-tests' into 'master'

tests/plugins/filter.py: Don't run redundant tests

See merge request BuildStream/buildstream!753

9ea40e2c

by Tristan Van Berkom at 2018-08-30T06:38:13Z

tests/frontend/mirror.py: Reenable test_mirror_fetch_upstream_absent[ostree]

This test was skipped because of issue #538, but #538 was fixed
and the test was still not reenabled.

8aec1102

by Tristan Van Berkom at 2018-08-30T07:06:17Z

Merge branch 'tristan/538-reenable-ostree-test' into 'master'

tests/frontend/mirror.py: Reenable test_mirror_fetch_upstream_absent[ostree]

See merge request BuildStream/buildstream!755

fcc17c82

by Tristan Van Berkom at 2018-08-30T08:44:34Z

source.py: Stylistic changes in Source.__do_fetch()

Added some comments to make the flow easier to follow, and
removed an annoying 'success' variabled in favor of a for / else
loop statement.

60ad044a

by Tristan Van Berkom at 2018-08-30T08:44:34Z

source.py: Move Source.mark_download_url() to the public methods.

This was sitting in the section for abstract methods, but this
is most definitely not an abstract method to be implemented by
Sources.

c339f917

by Tristan Van Berkom at 2018-08-30T08:44:34Z

plugins/source/git.py: Fix formatting of url in tracking

This was displaying the aliased URL which is pretty useless,
use the translated URL for the timed activity.

26565f11

by Tristan Van Berkom at 2018-08-30T09:12:28Z

Merge branch 'tristan/source-mirroring-changes' into 'master'

Minor code changes revolving around source mirroring

See merge request BuildStream/buildstream!758

a38d9163

by Valentin David at 2018-08-30T10:26:57Z

Report processing errors from tracking

Failures to write files when tracking were not reported.

Fixes #533.

02138181

by Tristan Van Berkom at 2018-08-30T11:36:00Z

Merge branch 'valentindavid/post_tracking_errors' into 'master'

Report processing errors from tracking

Closes #533

See merge request BuildStream/buildstream!747

d23d4f3e

by Tristan Van Berkom at 2018-08-30T12:02:27Z

_artifactcache/artifactcache.py: Write the cache_size file atomically

This is causing issues while the size file is being read and written
simultaneously.

The proper fix will be to read/add/save the file atomically and that
will require locking, but this fix is a good stop gap to existing crashes.

bb6aa5a5

by Tristan Van Berkom at 2018-08-30T12:27:55Z

Merge branch 'tristan/atomic-cache-size-file' into 'master'

_artifactcache/artifactcache.py: Write the cache_size file atomically

See merge request BuildStream/buildstream!762

680e4fe1

by knownexus at 2018-08-30T13:37:55Z

Replacing string 'bzr' with value from host tools

8cd719eb

by Phillip Smyth at 2018-08-30T15:47:06Z

Merge branch 'bzr_fix' into 'master'

Replacing string 'bzr' with value from host tools

See merge request BuildStream/buildstream!764

18d0bfb4

by Tom Pollard at 2018-08-30T19:14:33Z

buildstream/_project.py: Report if project.conf is missing name

Explicitly check that project.conf contains a name. This resolves
the issue of the provenance check from _yaml.py incorrectly reporting
the offending file as the default_config_node projectconfig.yaml
when no name key exists in the pre_config_node dict.

95121148

by Tiago Gomes at 2018-08-30T20:52:29Z

Merge branch 'tpollard/591' into 'master'

buildstream/_project.py: Report if project.conf is missing name

See merge request BuildStream/buildstream!680

775d3fca

by Tristan Van Berkom at 2018-08-31T06:05:32Z

_project.py: Adding FIXME comment to address #591 properly

0682aa1c

by Josh Smith at 2018-08-31T10:58:12Z

job.py: Modify retry messages to be FAIL

This adjusts the message handler for the child processes to no longer
override the message type.

This also removes the ability for unhandled non BstError exceptions to
retry.

68eff9ba

by Josh Smith at 2018-08-31T10:58:20Z

tests: Modify tests to ensure retry FAILURE

9b18e1bf

by Tristan Van Berkom at 2018-08-31T11:22:51Z

Merge branch 'Qinusty/retries-should-fail' into 'master'

Retries log as failures

See merge request BuildStream/buildstream!766

e2d8fedf

by Josh Smith at 2018-08-31T15:53:43Z

scheduler.py: Move cache_size logs into folder

This prevents the cache_size.pid.log files from cluttering the root
log directory.

6805a2ab

by Tristan Van Berkom at 2018-09-01T08:24:05Z

Merge branch 'Qinusty/cache-size-directory' into 'master'

Move cache_size.pid.log files into a subdirectory of logs

See merge request BuildStream/buildstream!769

f8b06acc

by Tristan Van Berkom at 2018-09-02T09:33:39Z

source.py: Document Source.get_source_fetchers() to return an iterable

Also highlight the fact that the plugin can rely on the fetcher's
fetch() method getting called before consuming the next item in the
list, which is the magick behavior that the git plugin relies on.

This is a part of #620

559b6fbc

by Tristan Van Berkom at 2018-09-02T09:34:10Z

source.py: Fixing docs link formatting to be consistent.

43ad22d7

by Tristan Van Berkom at 2018-09-02T09:34:10Z

source.py: Documenting that marking download URLs is mandatory

A download URL must be interpreted by the core at `Plugin.configure()`
time, even if only employed later on.

This is a part of #620

16462e9c

by Tristan Van Berkom at 2018-09-02T09:36:49Z

plugin.py: Added _configure() and _get_configuring() private APIs

Keeps track of whether the plugin is currently being configured.

Adjusted Element and Source classes to call _configure() in place
of calling configure() directly.

This is a part of #620

1f9c4147

by Tristan Van Berkom at 2018-09-02T09:37:12Z

source.py: Added `primary` argument to URL marking APIs

The Source must now mention whether the marked or translated
URL is "primary" or not. Even when a Source may have multiple
URLs, the auxilliary URLs are derived from the primary one, not
only is this true for git, but it is mandated by our tracking
API which assumes there is a primary URL.

This adjusts the `git` source and the test `fetch_source.py` source
to behave properly and advertize it's primary URL properly.

This is a part of #620

f9b2f1a9

by Tristan Van Berkom at 2018-09-02T09:37:21Z

source.py: Track marked URLs and assert they are marked during Plugin.configure()

This cannot test for unaliased URLs, as those can be discovered later
on outside of user provided element configuration; at least we
assert that if an alias was used, we have seen it at load time.

This will cause a BUG to occur for a plugin which marks an aliased
URL (or attempts to translate one) outside of `Plugin.configure()`,
if that URL was not previously seen.

This is a part of #620

88460cd2

by Tristan Van Berkom at 2018-09-03T06:23:38Z

Merge branch 'tristan/source-fetcher-changes' into 'master'

Source fetcher changes

See merge request BuildStream/buildstream!772

2339f0c4

by Tom Pollard at 2018-09-03T09:44:36Z

plugins/git.py: Warn if ref is not in given track

Add a helper function assert_ref_in_track to git.py GitMirror()
which is used when staging & initing the source workspace. It
checks the element's ref exists in the track (branch/tag) if it
has been specified, raising a warning if necessary. The warning makes
use of the warning token 'REF_NOT_IN_TRACK' from the configurable
CoreWarnings. If the element has been tracked with bst, it is assumed
that the value of ref exists in the track as it was generated from it
& as such is not asserted.

c96fec5d

by Tom Pollard at 2018-09-03T09:44:36Z

tests/sources/git.py: Add tests for REF_NOT_IN_TRACK

Add tests that cover assert_ref_in_track & the configurable
CoreWarnings REF_NOT_IN_TRACK warnings token.

6a0cdedf

by Tom Pollard at 2018-09-03T10:16:39Z

Merge branch 'tpollard/483' into 'master'

plugins/git.py: Warn if ref is not in given track

See merge request BuildStream/buildstream!564

9b327eb6

by Ben Brewer at 2018-09-04T14:41:23Z

Improve documentation for artifact cache installation

Remove ambiguity about systemd service files being separate.
Add URL for more information about systemd service files.
Add note about public keys being mandatory for self-signed certs.
Make cert/key file naming consistent throughout document.

3e67e64a

by Javier Jardón at 2018-09-04T16:43:13Z

Merge branch 'benbrewer/install-artifacts-doc-improvements' into 'master'

Improve documentation for artifact cache installation

See merge request BuildStream/buildstream!777

3409609e

by Daniel Silverstone at 2018-09-04T16:55:51Z

jobs.py: Reduce FD leaks from queues and process objects

The garbage collector can take too long to get around to cleaning
up the Queue and Process instances in completed Job instances.  As
such, FDs tend to leak and in very large projects this can result
in running out of FDs before a build, fetch, track, or other process
can complete.  This patch reduces the chance of that by only creating
the queue when it's needed, and forcing the queue and process instances
to be deleted when the parent is finished with them.

Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>

7b32e1ec

by Tristan Maat at 2018-09-04T17:20:55Z

Merge branch 'danielsilverstone-ct/maybe-reduce-fd-leaks' into 'master'

jobs.py: Reduce FD leaks from queues and process objects

See merge request BuildStream/buildstream!778

936bb93a

by Jim MacArthur at 2018-09-07T12:57:28Z

cascache.py: Preparation for remote execution

Refactor the push() and pull() implementations so that API additions
needed for remote-execution is made easier.

https://gitlab.com/BuildStream/buildstream/issues/454

b0f46545

by Martin Blanchard at 2018-09-07T12:57:29Z

cascache.py: Introduce new push helpers

Add push_directory() and push_message() helpers along with a
verify_digest_pushed().

https://gitlab.com/BuildStream/buildstream/issues/454

50bf3139

by Martin Blanchard at 2018-09-07T12:57:29Z

cascache.py: Introduce new pull helpers

Add a pull_tree() helper.

https://gitlab.com/BuildStream/buildstream/issues/454

853e0af7

by Martin Blanchard at 2018-09-07T12:57:29Z

_casbaseddirectory.py: Add a method for hash recalculation

https://gitlab.com/BuildStream/buildstream/issues/454

e7659c29