[Notes] [Git][BuildStream/buildstream][jennis/new_artifact

James Ennis pushed to branch jennis/new_artifact_subcommands at BuildStream / buildstream

Commits:

0b83d024

by Tristan Van Berkom at 2019-01-10T20:02:50Z

_pipeline.py: Fix the planner to retain the original order when depth sorting

The algorithm adds elements to a dictionary and then sorts the dictionary
by the discovered depths while recursing - using an OrderedDict is enough
to ensure a stable order.

This fixes `bst show --deps plan ...` reporting different results on
each invocation.

This fixes an aspect of #824

630e26f1

by Tristan Van Berkom at 2019-01-10T20:03:52Z

testutils/runcli.py: Added Result.get_start_order()

Added a function to report the list of elements which appeared in
a given queue in the order of their first appearance.

d1d7de57

by Tristan Van Berkom at 2019-01-11T04:13:16Z

tests/frontend/order.py: Adding tests ensuring stable element processing order

For each sample project and expected result order, this test ensures that:

  * bst show --deps plan: Always shows the expected build order.

  * bst source fetch: Always executes in the expected build order.

  * bst build: Always builds in the expected order (disabled).

The build part of the test is currently disabled as the scheduler
seems to be messing with the order even in a `--builders 1` scenario.

276b8d48

by Tristan Van Berkom at 2019-01-11T04:42:24Z

Merge branch 'tristan/element-processing-order' into 'master'

Make build plan element list stable

See merge request BuildStream/buildstream!1058

fc3dcec8

by Angelos Evripiotis at 2019-01-11T16:45:00Z

dev-requirements: pytest-cov==2.6.1 for '--no-cov'

Bump the version of pytest-cov, so that we pick up the fix for
the '--no-cov' option breaking in pytest 4.0:

    https://github.com/pytest-dev/pytest-cov/pull/230

This was the error you would get prior to this version:

    _pytest.warning_types.RemovedInPytest4Warning: config.warn has been
    deprecated, use warnings.warn instead

Although it says 'deprecated', it did result in
termination with stack trace.

f86b7ff3

by Angelos Evripiotis at 2019-01-11T16:45:00Z

contributing: '--last-failed' and '--no-cov' tips

To help new folks get to grips with pytest, add some tips on other
frequently used features.

d983f231

by Angelos Evripiotis at 2019-01-11T17:19:06Z

Merge branch 'aevri/tox_no_cov' into 'master'

pytest-cov==2.6.1, contributing: '--last-failed' and '--no-cov' tips

See merge request BuildStream/buildstream!1059

829a2b93

by Tristan Van Berkom at 2019-01-12T21:32:13Z

_scheduler/queues/queue.py: Put elements in the skip list, not jobs

Minor correction, looks like we're not observing this queue
otherwise we'd be seeing crashes.

993e30ae

by Tristan Van Berkom at 2019-01-12T22:02:55Z

Merge branch 'tristan/fix-skipped-elements' into 'master'

_scheduler/queues/queue.py: Put elements in the skip list, not jobs

See merge request BuildStream/buildstream!1063

32732e01

by Javier Jardón at 2019-01-14T09:04:01Z

.gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25

4f5f1184

by Valentin David at 2019-01-14T09:40:56Z

Merge branch 'jjardon/freedesktop-sdk_latest' into 'master'

.gitlab-ci.yml: Use latest freedesktop-sdk 18.08.25

Closes #858

See merge request BuildStream/buildstream!1064

bb80a2b8

by Chandan Singh at 2019-01-14T14:30:15Z

.gitlab-ci.yml: Use newer image for overning aarch tests

During recent reorganizing of `.gitlab-ci.yml` to work with `tox`, seems
like we missed to update the image used by the overnight aarch tests,
meaning that they currently fail due to `tox` being missing from them.

While these tests will be skipped on MRs usually, here is an example of
what the tests look like if they are actually run with this change:
https://gitlab.com/BuildStream/buildstream/-/jobs/145449561.

Fixes #859.

10b3ee62

by Chandan Singh at 2019-01-14T14:57:08Z

Merge branch 'chandan/fix-overnight-aarch' into 'master'

Chandan/fix overnight aarch

Closes #859

See merge request BuildStream/buildstream!1065

542cdaf0

by Jürg Billeter at 2019-01-14T18:46:57Z

_frontend/cli.py: Also check original args for --config in bash completion

The path of the config file generated by testutils for completion tests
is passed as regular argument, not via COMP_WORDS. Use that config file
in complete_artifact() to ensure the test uses the right artifact
directory.

ff666e76

by James Ennis at 2019-01-14T18:46:57Z

completions.py: Add a test for our artifact ref autocompletions

Since the artifact subcommand group has been created, we have been able
to tab complete both element names and artifact refs as arguments to
the artifact subcommands (e.g. bst artifact log), this commit adds a
test for this.

273b0f55

by Tristan Van Berkom at 2019-01-14T20:24:57Z

Merge branch 'jennis/add_artifacts_completion' into 'master'

completions.py: Add a test for our artifact ref autocompletions

See merge request BuildStream/buildstream!1054

2e3c2ea2

by Javier Jardón at 2019-01-14T22:26:04Z

.gitlab-ci.yml: Use latest bst-external

which include flatpak_repo plugin needed to build fdsdk

d60d2e31

by Javier Jardón at 2019-01-15T00:39:22Z

Merge branch 'jjardon/bst_external_0_9' into 'master'

.gitlab-ci.yml: Use latest bst-external

Closes #861

See merge request BuildStream/buildstream!1068

605836c1

by Chandan Singh at 2019-01-15T00:57:05Z

.gitlab-ci.yml: Add job to attempt to update dependencies

This will help us check if BuildStream is working with the latest
version of dependencies, as per our constraints. This job is allowed to
fail but its failure should signal that we need to add stricter
constraints in some of our `.in` requirements files.

006370af

by Chandan Singh at 2019-01-15T01:21:33Z

Merge branch 'chandan/reqs-update-test' into 'master'

.gitlab-ci.yml: Add job to attempt to update dependencies

See merge request BuildStream/buildstream!1038

4b544555

by Chandan Singh at 2019-01-15T21:28:40Z

.gitlab-ci.yml: Add tests for python 3.7

We already have tests for python 3.5 and 3.6 but not 3.7.

Fixes https://gitlab.com/BuildStream/buildstream/issues/838.

80fe0d9a

by Javier Jardón at 2019-01-15T22:48:22Z

Merge branch 'chandan/python37-tests' into 'master'

.gitlab-ci.yml: Add tests for python 3.7

Closes #838

See merge request BuildStream/buildstream!1074

c91784ab

by Chandan Singh at 2019-01-15T22:49:01Z

conftest.py: Don't use deprecated get_marker() function

Starting from `pytest` version 4.1.0, `Node.get_marker()` has been
removed, and hence our tests break when running with newer versions of
`pytest`. It was deprecated since a while back but it has recently been
removed completely. Use `get_closest_marker()` as a replacement that is
suggested in the changelog, and seems to work fine for our use case.

See https://github.com/pytest-dev/pytest/pull/4564 for more context on
the upstream issue.

One way of verifying this change is that this should fix the recently
added `tests-fedora-update-deps` job, that was failing before due to
this issue.

ecae4d73

by Javier Jardón at 2019-01-15T23:23:41Z

Merge branch 'chandan/fix-pytest-get-marker' into 'master'

conftest.py: Don't use deprecated get_marker() function

See merge request BuildStream/buildstream!1073

0eac4008

by Valentin David at 2019-01-16T10:04:57Z

buildstream/_gitsourcebase.py: Reduce git history for git describe.

Found during #833.

`git rev-list --boundary tag..HEAD` unfortunately gives boundaries
that are deeper than should when there is a merge commit between `tag`
and `HEAD`. The common ancestory of the two parents of the merge is
a boundary instead of the parent of the branch that is not traversed.

`--ancestry-path` fixes this issue by restricting `git` traversing
those branches.

a405e08f

by Valentin David at 2019-01-16T10:04:57Z

buildstream/_gitsourcebase.py: Fix case where HEAD is tagged

`git rev-list --boundary HEAD..HEAD` does not return any boundary.
So in this case we need to manually tag the HEAD as a boundary.

c0631d48

by Valentin David at 2019-01-16T10:33:52Z

Merge branch 'valentindavid/git-reduced-history' into 'master'

buildstream/_gitsourcebase.py: Reduce git history for git describe.

See merge request BuildStream/buildstream!1069

b608ac86

by James Ennis at 2019-01-16T11:05:44Z

_context.py: Doc fix: get_toplevel_project() returns object not list

97a3beb6

by James Ennis at 2019-01-16T11:05:44Z

_context.py: Add documentation to get_workspaces() command

d6587aa0

by James Ennis at 2019-01-16T11:33:21Z

Merge branch 'jennis/doc_fixes_in_context' into 'master'

Small documentation/comment fixes in context.py

See merge request BuildStream/buildstream!1072

2683f98a

by Raoul Hidalgo Charman at 2019-01-16T11:55:07Z

_cas: Rename artifactcache folder and move that to a root module

Other components will start to reply on cas modules, and not the artifact cache
modules so it should be organized to reflect this.

All relevant imports have been changed.

Part #802

d2cc4798

by Raoul Hidalgo Charman at 2019-01-16T11:55:07Z

casremote.py: Move remote CAS classes into its own file

Part of #802

76f67483

by Raoul Hidalgo Charman at 2019-01-16T11:55:07Z

cas: move remote only functions to CASRemote

List of methods moved
* Initialization check: made it a class method that is run in a subprocess, for
  when checking in the main buildstream process.
* fetch_blobs
* send_blobs
* verify_digest_on_remote
* push_method

Part of #802

6c428bc9

by Jürg Billeter at 2019-01-16T12:56:38Z

Merge branch 'raoul/cas-refactor' into 'master'

Cas refactor

See merge request BuildStream/buildstream!1071

2648b0bd
by Angelos Evripiotis at 2019-01-16T14:44:25Z
```
bst-docker-import: fix Chandan's name
```

780d0f3a

by Angelos Evripiotis at 2019-01-16T15:13:06Z

Merge branch 'chadnan' into 'master'

bst-docker-import: fix Chandan's name

See merge request BuildStream/buildstream!1076

4ca37dcd

by Tristan Van Berkom at 2019-01-16T15:47:21Z

source-determinism.py integration test: Dont use integration_cache fixture

This was just deadcode, it's not really used here.

6286d820

by Tristan Van Berkom at 2019-01-16T15:47:21Z

conftest.py: Use different artifact directory for integration tests

To ensure we can run integration tests in parallel, use a tempdir
for the artifact cache of each separate integration test run.

This makes it possible to run multiple full test runs including
integration tests in parallel under detox.

471af316

by Tristan Van Berkom at 2019-01-16T16:32:52Z

Merge branch 'tristan/detoxing-integration-tests' into 'master'

Make integration tests parallelizable

See merge request BuildStream/buildstream!1077

081e8426

by James Ennis at 2019-01-16T17:58:41Z

setup.py: Import server main from _cas/casserver.py

Since the CAS refactor, we have not been able to execute
bst-artifact-server. This commit ensures that we can.

This fix closes #867

e385660c

by James Ennis at 2019-01-16T18:33:20Z

Merge branch 'jennis/fix_bst_artifact_server' into 'master'

Ensure that we can execute `bst-artifact-server`

Closes #867

See merge request BuildStream/buildstream!1079

6991d6d6

by Tristan Van Berkom at 2019-01-16T19:28:14Z

_scheduler: Refactor of queues and resources.

This branch makes the following changes:

  * jobs/job.py: No longer stores any interested resource list

    Jobs are ephemeral again, they only ever exist while they
    are running.

  * queues/queue.py: Revert to only handling lists of elements

    Elements pass through the queues, Queue.harvest_jobs()
    replaces Queue.pop_ready_jobs() and now the Queue stops
    creating jobs as soon as there are not enough resources
    for the job.

    Also removed unused `prepare()` abstract method.

  * queues/buildqueue.py: Adapt the part where we launch a job

    This part needs to be reworked anyway, just touch it up for
    now so that it doesnt break with the surrounding changes.

  * jobs/{cachesize,cleanup}job.py: Expose uniform complete callback

    Allows the scheduler to manage resource deallocation for these
    two job completions as a custom thing, at the same phase
    that the Queues take care of their own resource deallocation.

  * resources.py: No longer has knowledge of the job

    Since jobs are ephemeral, they are not a suitable place
    to store the resource identifiers, these must be provided
    by the callers wherever needed.

    Now the main Resources object is owned by the Scheduler
    but shared with Queues, each take care of managing the
    resources of the jobs they create through the same
    resource API.

  * scheduler.py: Reverted to only creating jobs on demand

    This changes the flow of the scheduler such that whenever
    jobs complete, the queues are interrogated for as many
    jobs which can run at the moment but not more; and this
    completely removes the waiting list.

    For the internal cache management jobs, we handle this
    with a little state instead of having a waiting list
    and only launch when the resources permit it.

By abolishing the scheduler waiting list and creating jobs
on demand, we fix the order of element processing and consequently
fix issue #712.

04ba59bc

by Tristan Van Berkom at 2019-01-16T19:28:14Z

tests/frontend/order.py: Enable the test for build and fix the fetch tests

With the scheduler changes, fetch jobs get automatically skipped
so the output is changed, using a separate repo for each element
fixes the test such that every fetch job gets a job launched.

32bdded8

by Tristan Van Berkom at 2019-01-16T19:28:14Z

_artifactcache/artifactcache.py: Rephrase failure message

It was saying "There is not enough space to build the given element.",
this makes me think the error is associated to a specific element, but
this does not make sense to show up in a cleanup task.

Instead say "There is not enough space to complete the build.", which
should be more clear that even after cleaning up there is not enough
space.

17f6e5a8

by Tristan Van Berkom at 2019-01-16T20:10:30Z

Merge branch 'tristan/element-processing-order' into 'master'

Scheduler refactor, fix processing order

Closes #712

See merge request BuildStream/buildstream!1067

19adc9e5

by Tristan Van Berkom at 2019-01-16T20:48:10Z

element.py: Cleanup temporary staging directories on termination

Use utils._tempdir() which in turn uses _signals.terminator() for
this purpose.

This also changes utils._tempdir() to use utils._force_rmtree()
so that it is safe to use for cleaning up the staging directory.

This patch fixes orphaned temporary directories being left behind
in the artifact cache directory at forceful termination time.

5e9598c5

by Tristan Van Berkom at 2019-01-16T21:17:26Z

Merge branch 'tristan/cleanup-staging-tempdir' into 'master'

element.py: Cleanup temporary staging directories on termination

See merge request BuildStream/buildstream!1080

5ea2f59d
by Tristan Van Berkom at 2019-01-16T23:35:21Z
```
tests: Migrate completions test to tests/frontend
```

1eafc25c