[Notes] [Git][BuildStream/buildstream][jennis/add_new_profile

James Ennis pushed to branch jennis/add_new_profile_topic at BuildStream / buildstream

Commits:

f874295f

by Tristan Van Berkom at 2019-01-18T15:59:28Z

bzr source plugin: Use lock files to avoid corrupting the source cache

This patch by itself fixes #868

42a2fe3c

by Tristan Van Berkom at 2019-01-18T15:59:28Z

bzr source plugins: Remove all of the atomic backup directory business logic

Follow up of last commit which uses exclusive locking to
protect bzr operations instead.

a8713ed2

by Tristan Van Berkom at 2019-01-18T15:59:28Z

testutils/runcli.py: Added cli.get_element_states()

With get_element_state(), you need to invoke BuildStream once
for every element state you want to observe in a pipeline.

The new get_element_states() reports a dictionary with
the element state hashed by element name and is better to use
if you have more than one element to observe the state of.

a895cb2a

by Tristan Van Berkom at 2019-01-18T15:59:28Z

tests/frontend/track.py: Use test_track_recurse() to stress test Sources

This causes multiple source instances to interact with the same
backing data store at the same time, increasing the likelyhood
of triggering issues around concurrent access.

This more reliably triggers issue #868

4236bcc7

by Tristan Van Berkom at 2019-01-18T16:35:23Z

Merge branch 'tristan/fix-bzr-race' into 'master'

Fix bzr race conditions

Closes #868

See merge request BuildStream/buildstream!1083

951a8df1
by Angelos Evripiotis at 2019-01-18T18:02:22Z
```
cli.py: add a hint about '--' to 'bst shell' help
```

9911023f

by Angelos Evripiotis at 2019-01-18T18:56:21Z

Merge branch 'aevri/shell_separator_hint' into 'master'

cli.py: add a hint about '--' to 'bst shell' help

See merge request BuildStream/buildstream!1078

c9ce89d2

by Tristan Van Berkom at 2019-01-18T19:36:26Z

_cas/cascache.py: Cleanup directories when removing refs

With out this, empty directories in the refs/heads directory just
grow unconditionally.

c536ab6a

by Tristan Van Berkom at 2019-01-18T19:36:26Z

tests/artifactcache/expiry.py: Test refs directory cleanup

Enhance the test which checks removal of the extract directories
with an additional check that the ref directories are cleaned up
when removing artifacts.

99699ffc

by Tristan Van Berkom at 2019-01-18T19:36:26Z

utils.py: Added _tempnamedfile()

When used in a child process in BuildStream, this should be
used instead of tempfile.NamedTemporaryFile() directly, otherwise
we fail to cleanup the file on SIGTERM.

8ce483d4

by Tristan Van Berkom at 2019-01-18T19:36:26Z

_cas/cascache.py: Use utils._tempdir() and utils._tempnamedfile()

The direct usage of tempfile.TemporaryDirectory() and
tempfile.NamedTemporaryFile() here causes leakage of any temporary
data when the process operating on temporary data is terminated
with SIGTERM.

Using the utilities ensures that trash is not left behind in
~/.cache/buildstream/artifacts/tmp when the user terminates
BuildStream with ^C.

73c7252d

by Tristan Van Berkom at 2019-01-18T20:57:42Z

Merge branch 'tristan/cas-cleanup-improve' into 'master'

CASCache cleanup improvements

See merge request BuildStream/buildstream!1087

adfb9291

by Tristan Van Berkom at 2019-01-18T21:23:05Z

tests/testutils/python_repo.py: Use subprocess to run sdist

The current approach using setuptools.sandbox.run_setup() was
causing a spurious (but highly frequent) failure where setuptools
gets mixed up with it's manipulation of sys.modules and hits a
RuntimeError as a dictionary changes size while being iterated over.

For instance: https://gitlab.com/BuildStream/buildstream/-/jobs/147967307

Since this already happens in an isolated virtual environment created
by tox, we should not need additional sandboxing here from setuptools,
and launching this as a subprocess will be safer.

d114a6bd

by Tristan Van Berkom at 2019-01-18T22:07:22Z

Merge branch 'tristan/fix-pip-source-test' into 'master'

tests/testutils/python_repo.py: Use subprocess to run sdist

See merge request BuildStream/buildstream!1090

bf591ade

by Chandan Singh at 2019-01-18T22:09:04Z

Derive import plugin from Element instead of BuildElement

The `import` element is not really a build element. The main purpose of
the `BuildElement` class is to allow users to run `build-commands`,
`install-commands` etc. But, `import` does not run such commands.
Moreover, we already override all the methods provided by
`BuildElement`. So it only makes it confusing to have it derived from
`BuildElement` class when it is not a build element.

So, derive it from the base `Element` class instead.

2233a532

by Chandan Singh at 2019-01-18T22:46:41Z

Merge branch 'chandan/import-is-not-buildelement' into 'master'

Derive import plugin from Element instead of BuildElement

See merge request BuildStream/buildstream!1089

aa3411f9

by Tristan Maat at 2019-01-18T22:47:46Z

testutils/runcli.py: Allow removing artifacts from arbitrary dirs

`remove_artifact_from_cache` used a hard-coded path to remove
artifacts, which wasn't sufficient for integration tests.

fff882fe

by Tristan Maat at 2019-01-18T22:47:46Z

widget.py: Avoid "showing 0 lines" messages when there are no lines

This happened when bst is invoked with --message-lines 0 or
--error-lines 0, and was arguably a little too verbose (the user
explicitly asked us not to show them any lines, after all).

Fixes #779

e230dedb

by Tristan Van Berkom at 2019-01-18T23:23:35Z

Merge branch 'tlater/message-lines' into 'master'

Avoid "showing 0 lines" messages when we're asked to show no lines

Closes #779

See merge request BuildStream/buildstream!1031

43797617

by Richard Maw at 2019-01-21T10:41:37Z

_gitsourcebase.py: Fetch with explicit refspecs

Old versions of git lack --force and --tags,
but the same effect can be had by specifying refspecs.

ce8dab0f

by Richard Maw at 2019-01-21T10:41:37Z

tests/frontend/workspace.py: Skip test_open_multi_unwritable when root

The test assumes that a directory with write permission removed isn't writable,
this isn't the case if the process running the tests has CAP_DAC_OVERRIDE
which is common when running as root.

50165081

by Richard Maw at 2019-01-21T10:41:37Z

tests/sources/git.py: Skip tests that assume too new a git

test_track_invalid_submodule depends on being able to remove a submodule
by `git rm $submoduledir`, but old versions of git don't update .gitmodules
so BuildStream still thinks there's a submodule present.

For expediency the test is skipped rather than changed to manually remove
the entry from .gitmodules if git hasn't done it,
since in the common case git is new enough to do that itself.

test_git_describe expects --first-parent to find another tag,
but `bst track` will gracefully degrade if the option doesn't work
so a different history will be retained with old versions of git.

It's of marginal benefit to add additional cruft
to test for different output on old versions of git that won't persist forever.

8677a256

by Benjamin Schubert at 2019-01-21T10:41:37Z

Add a Centos runner

Centos is apparently different enough from fedora when running tests.

33782865

by Valentin David at 2019-01-21T11:47:18Z

Merge branch 'richardmaw/centos-oldgit-test-fixes' into 'master'

Fix CentOS

Closes #833

See merge request BuildStream/buildstream!1085

c00b3782

by Chandan Singh at 2019-01-21T19:37:53Z

CONTRIBUTING.rst: Fix formatting of link to pip docs

The link explaining python's requirements files was formatted
incorrectly (missing underscore at the end). This resulted in the text
being rendered literally, as opposed to being converted into a
hyperlink.

638ceb8a

by Chandan Singh at 2019-01-21T20:16:38Z

Merge branch 'chandan/fix-contrib-link' into 'master'

CONTRIBUTING.rst: Fix formatting of link to pip docs

See merge request BuildStream/buildstream!1094

383142e3

by James Ennis at 2019-01-22T12:32:43Z

using_commands.rst: Split out top level commands and subcommand groups

ac135333

by James Ennis at 2019-01-22T12:32:43Z

using_commands.rst: Add the source checkout command

a3fc350f

by James Ennis at 2019-01-22T12:32:43Z

Move push and pull to the new artifact subcommand group

This commit also ensures that if we try to use the 'old' commands,
BuildStream will fail and instruct the user to use the new command.

9eefe863

by James Ennis at 2019-01-22T12:32:43Z

cli: Add artifact checkout subcommand

'artifact checkout' has slightly different behaviour from 'checkout',
that is, either '--directory' or '--tar' are now required options.
This is a step towards allowing checkout to take multiple args.

fbd15939

by James Ennis at 2019-01-22T12:32:43Z

Mark 'old' checkout command as obsolete

This commit marks 'bst checkout' as a 'hidden' command. If used,
the user will be prompted to use the new 'bst artifact checkout'
command.

All tests which used 'bst checkout' have been modified to use
the new artifact sub-command.

This partially solves #822.

9e8034e9

by James Ennis at 2019-01-22T12:32:43Z

Add NEWS entry describing the deprecation of checkout, pull and push

1bccf1d2

by James Ennis at 2019-01-22T12:38:40Z

using_commands.rst: Split out the artifact subcommands

d9072371

by James Ennis at 2019-01-22T12:38:40Z

using_commands.rst: Add artifact log subcommand

077d5a96

by James Ennis at 2019-01-22T12:41:29Z

Change bst checkout/pull/push references to bst artifact checkout/pull/push

ea2fbe4d

by James Ennis at 2019-01-22T13:02:14Z

man: Regenerate all of our man pages

Due to an upstream click_man issue:
https://github.com/click-contrib/click-man/issues/10
generating the man pages with our current setup.py only generates
a man page for bst-artifact-server, our first entry point.
I then had to remove this entry point from setup.py and regenerate to
obtain man pages for the commands in cli.py

9c2a6b93

by James Ennis at 2019-01-22T15:08:17Z

Merge branch 'jennis/migrate_pull_push_commands' into 'master'

Move push/pull/checkout to the artifact subcommand group

See merge request BuildStream/buildstream!1045

35377522

by James Ennis at 2019-01-22T15:28:13Z

cli.py: Add an obsoletion note to our deprecated commands.

Click 7.0 allows us to declare commands as 'hidden'.
However, sphinx-click and click-man still generate these hidden
commands in the documentation and man pages, respectively.

This is a stop gap solution until the upstream issues (tracked in
issues #879 and #881) have been addressed.

76148785

by James Ennis at 2019-01-22T16:15:20Z

Merge branch 'jennis/add_obsoletion_note' into 'master'

Add an obsoletion note to our deprecated commands.

See merge request BuildStream/buildstream!1097

1443c542

by Tristan Van Berkom at 2019-01-22T16:26:56Z

_scheduler: Fix dont display a failure for terminated jobs

This fixes a recent regression introduced in c2fc2a5ea

6e5c9987

by Tristan Van Berkom at 2019-01-22T17:14:50Z

Merge branch 'tristan/fix-terminated-status' into 'master'

_scheduler: Fix dont display a failure for terminated jobs

See merge request BuildStream/buildstream!1096

3315b9a1

by Tristan Van Berkom at 2019-01-22T18:46:24Z

tests/integration/pullbuildtrees.py: Fix the non-integration case.

This test has one test case which is marked as an integration test,
and the other is not an integration test, but was using the integration
cli. The integration cli does not work correctly if not run in
integration mode.

This was causing an error locally in conftest.py when trying to
create a tmpdir inside a nonexisting integration cache directory.

56e857f4

by Tristan Van Berkom at 2019-01-22T19:59:57Z

Merge branch 'tristan/fix-pullbuildtrees-test' into 'master'

tests/integration/pullbuildtrees.py: Fix the non-integration case.

See merge request BuildStream/buildstream!1098

3895571a

by Chandan Singh at 2019-01-23T00:45:47Z

tox.ini: Unpin sphinx dependency

https://github.com/rtfd/sphinx_rtd_theme/pull/672 has been fixed
upstream, and the newer versions of `sphinx_rtd_theme` do not break
search functionality with Sphinx >= 1.8.

528e8ed7

by Chandan Singh at 2019-01-23T01:31:49Z

Merge branch 'chandan/unpin-sphinx' into 'master'

tox.ini: Unpin sphinx dependency

See merge request BuildStream/buildstream!1093

0098a900

by Tristan Van Berkom at 2019-01-23T13:57:53Z

tests: Migrated cache quota test into artifactcache/cache_size.py

Instead of sitting mysteriously alone in internals/utils.py

9be31641

by Tristan Van Berkom at 2019-01-23T13:57:53Z

_artifactcache.py: Raise ArtifactError() when quota size exceeds disk space.

This is not an error related to loading data, like a parse error
in the quota specification is, but a problem raised by the artifact
cache - this allows us to assert more specific machine readable
errors in test cases (instead of checking the string in stderr, which
this patch also fixes).

This also removes a typo from the error message in the said error.

  * tests/artifactcache/cache_size.py

    Updated test case to expect the artifact error, which consequently
    changes the test case to properly assert a machine readable error
    instead of asserting text in the stderr (which is the real, secret
    motivation behind this patch).

  * tests/artifactcache/expiry.py: Reworked test_invalid_cache_quota()

    Now expect the artifact error for the tests which check configurations
    which create caches too large to fit on the disk.

3a6e953f

by Tristan Van Berkom at 2019-01-24T01:59:13Z

Merge branch 'tristan/insufficient-storage-error' into 'master'

Tristan/insufficient storage error

See merge request BuildStream/buildstream!1102

68339b19

by Tristan Van Berkom at 2019-01-24T05:01:55Z

tests/testutils/runcli.py: Make get_element_states() take a list of targets

Instead of a single target, we can always provide a single target
in a list.

6ecc2b0a

by Tristan Van Berkom at 2019-01-24T05:01:55Z

tests/artifactcache/expiry.py: Refactored to use get_element_states()

1140aed5

by Tristan Van Berkom at 2019-01-24T05:01:55Z

tests/elements/filter.py: Refactored to use get_element_states()

b7ea8b74

by Tristan Van Berkom at 2019-01-24T05:01:55Z

tests/sources/remote.py: Refactored to use get_element_states()

c2c004f9

by Tristan Van Berkom at 2019-01-24T05:01:55Z

tests/frontend/pull.py: Refactored to use get_element_states()

21b2958b

by Tristan Van Berkom at 2019-01-24T05:01:55Z

tests/frontend/push.py: Refactored to use get_element_states()

83fcaa9f

by Tristan Van Berkom at 2019-01-24T05:01:56Z

tests/frontend/track.py: Refactored to use get_element_states()

46eb3018

by Tristan Van Berkom at 2019-01-24T05:01:56Z

tests/frontend/workspace.py: Refactored to use get_element_states()

341b131b

by Tristan Van Berkom at 2019-01-24T06:13:57Z

Merge branch 'tristan/test-element-states' into 'master'

Reduce number of calls to `bst show` in tests

See merge request BuildStream/buildstream!1103

ef2b4648

by Tristan Van Berkom at 2019-01-24T06:14:20Z

tests/frontend/track.py: test_track_error_cannot_write_file() fixup

This tests how BuildStream reacts when it fails to write the tracking
results to the element files or project.refs file, which is an operation
that plugins do not play a part in.

As such, removing the per repo kind parameterization from this test
as multiple runs are redundant here.

40c18174

by Tristan Van Berkom at 2019-01-24T07:00:50Z

Merge branch 'tristan/track-test-reduce' into 'master'

test_track_error_cannot_write_file() fixup

See merge request BuildStream/buildstream!1104

5b0bba85

by Jonathan Maw at 2019-01-24T10:15:06Z

Test that tracking in workspaces actually works

Previously, it merely tested that buildstream did not fall other,
rather than whether it did anything useful.

58d7d722

by Jonathan Maw at 2019-01-24T10:18:01Z

tests: Test that fetching an open workspace will fetch its dependencies

Previously, there was no way of detecting whether fetching happened, as
an element with an open workspace will not be fetched.

24bd8994

by Jürg Billeter at 2019-01-24T13:36:33Z

Merge branch 'jonathan/test-missing-workspace-guessing' into 'master'

Add tests to cover reinstated support for guessing targets

See merge request BuildStream/buildstream!1042

e03dd5fc

by Jürg Billeter at 2019-01-24T13:37:28Z

_frontend/cli.py: Guess element also for bst build --all

There is no reason to disallow guess_element() for bst build --all.

3850274b

by Phillip Smyth at 2019-01-24T13:37:28Z

projectconfig.yaml: Add key for default targets

_versions.py: Bump format version

ab72d384

by Jürg Billeter at 2019-01-24T13:37:28Z

_project.py: Add get_default_target() and get_default_targets() methods

_frontend/cli.py: Use new methods.

Based on patches by Phillip Smyth.

ee2d8434

by Jürg Billeter at 2019-01-24T13:37:28Z

_stream.py: Add ignore_junction_targets parameter

This filters out junctions from the list of targets.

2f175f0a

by Jürg Billeter at 2019-01-24T13:37:28Z

_frontend/cli.py: Ignore junctions in default targets where appropriate

Junctions cannot be built, pulled, or pushed. Specifying a junction on
the command line for these commands will result in an error. However,
junctions may be in the list of default targets, so they need to be
ignored for build, pull, and push commands.

5a351dee

by Jürg Billeter at 2019-01-24T13:38:50Z

format_project.rst: Add documentation for default targets

77414518

by Phillip Smyth at 2019-01-24T13:38:50Z

tests/frontend: Add default target tests for bst show and build

ce1c10ec

by Jürg Billeter at 2019-01-24T13:38:50Z

tests/frontend/fetch.py: Add default target test for bst source fetch

99b5c0af

by Jürg Billeter at 2019-01-24T13:38:50Z

tests/frontend/pull.py: Add default target test for bst push/pull

3642c8e7

by Jürg Billeter at 2019-01-24T13:38:50Z

tests/frontend/buildcheckout.py: Add default target test with junction

Test that `bst build` does not fail in a project where the list of
default targets includes a junction (junctions cannot be built).

50c5159f
by Phillip Smyth at 2019-01-24T13:38:50Z
```
NEWS: Add entry for default target feature
```

05587f22

by Jürg Billeter at 2019-01-24T15:10:08Z

Merge branch 'issue-638-validate-all-files' into 'master'

Add support for default targets

See merge request BuildStream/buildstream!925

3e36e363

by Tristan Van Berkom at 2019-01-24T16:55:24Z

_scheduler/resources.py: Dont error out in unregister_exclusive_interest()

Don't require the interest to be registered, just discard any interest,
this function just sets a bit in a mask, and is not intended to maintain
a balance like the reserve() function is.

ce01f87e

by Tristan Van Berkom at 2019-01-24T16:55:24Z

_scheduler/scheduler.py: Run cache size exclusively at startup

When running any session that has Queues which require Resource.CACHE,
check if our loaded estimated size exceeds the quota, and if so;
lock the Resource.CACHE resource exclusively right away and run
an exclusive initial cache size job.

This ensures we cleanup first before doing anything which might
add to the cache at startup time, if deemed needed.

This is a partial fix for issue #737

2479e8df

by Tristan Van Berkom at 2019-01-24T16:55:24Z

_frontend/widget.py: Render core messages more like other messages

In order to test when core activities occur by parsing the stderr
in tests, we should make the messages conform more.

At the same time, this restores alignment of columns in core
messages with the element processing related messages.

Also, _scheduler/scheduler.py is updated to make it's activity names
conform to the (current) 5 character limit for the sake of alignment.

The tests/frontend/logging.py test gets it's regexes updated for
the log lines it checks for in stderr.

fdb8ff65

by Tristan Van Berkom at 2019-01-24T16:55:24Z

tests/artifactcache/expiry.py: Test that expiry happens first

acd0bf22

by Tristan Van Berkom at 2019-01-24T18:01:41Z

Merge branch 'tristan/cache-management' into 'master'

Cache management fixes

See merge request BuildStream/buildstream!1091

41f03296