[Notes] [Git][BuildStream/buildstream][aevri/shell_separator

Angelos Evripiotis pushed to branch aevri/shell_separator_hint at BuildStream / buildstream

Commits:

081e8426

by James Ennis at 2019-01-16T17:58:41Z

setup.py: Import server main from _cas/casserver.py

Since the CAS refactor, we have not been able to execute
bst-artifact-server. This commit ensures that we can.

This fix closes #867

e385660c

by James Ennis at 2019-01-16T18:33:20Z

Merge branch 'jennis/fix_bst_artifact_server' into 'master'

Ensure that we can execute `bst-artifact-server`

Closes #867

See merge request BuildStream/buildstream!1079

6991d6d6

by Tristan Van Berkom at 2019-01-16T19:28:14Z

_scheduler: Refactor of queues and resources.

This branch makes the following changes:

  * jobs/job.py: No longer stores any interested resource list

    Jobs are ephemeral again, they only ever exist while they
    are running.

  * queues/queue.py: Revert to only handling lists of elements

    Elements pass through the queues, Queue.harvest_jobs()
    replaces Queue.pop_ready_jobs() and now the Queue stops
    creating jobs as soon as there are not enough resources
    for the job.

    Also removed unused `prepare()` abstract method.

  * queues/buildqueue.py: Adapt the part where we launch a job

    This part needs to be reworked anyway, just touch it up for
    now so that it doesnt break with the surrounding changes.

  * jobs/{cachesize,cleanup}job.py: Expose uniform complete callback

    Allows the scheduler to manage resource deallocation for these
    two job completions as a custom thing, at the same phase
    that the Queues take care of their own resource deallocation.

  * resources.py: No longer has knowledge of the job

    Since jobs are ephemeral, they are not a suitable place
    to store the resource identifiers, these must be provided
    by the callers wherever needed.

    Now the main Resources object is owned by the Scheduler
    but shared with Queues, each take care of managing the
    resources of the jobs they create through the same
    resource API.

  * scheduler.py: Reverted to only creating jobs on demand

    This changes the flow of the scheduler such that whenever
    jobs complete, the queues are interrogated for as many
    jobs which can run at the moment but not more; and this
    completely removes the waiting list.

    For the internal cache management jobs, we handle this
    with a little state instead of having a waiting list
    and only launch when the resources permit it.

By abolishing the scheduler waiting list and creating jobs
on demand, we fix the order of element processing and consequently
fix issue #712.

04ba59bc

by Tristan Van Berkom at 2019-01-16T19:28:14Z

tests/frontend/order.py: Enable the test for build and fix the fetch tests

With the scheduler changes, fetch jobs get automatically skipped
so the output is changed, using a separate repo for each element
fixes the test such that every fetch job gets a job launched.

32bdded8

by Tristan Van Berkom at 2019-01-16T19:28:14Z

_artifactcache/artifactcache.py: Rephrase failure message

It was saying "There is not enough space to build the given element.",
this makes me think the error is associated to a specific element, but
this does not make sense to show up in a cleanup task.

Instead say "There is not enough space to complete the build.", which
should be more clear that even after cleaning up there is not enough
space.

17f6e5a8

by Tristan Van Berkom at 2019-01-16T20:10:30Z

Merge branch 'tristan/element-processing-order' into 'master'

Scheduler refactor, fix processing order

Closes #712

See merge request BuildStream/buildstream!1067

19adc9e5

by Tristan Van Berkom at 2019-01-16T20:48:10Z

element.py: Cleanup temporary staging directories on termination

Use utils._tempdir() which in turn uses _signals.terminator() for
this purpose.

This also changes utils._tempdir() to use utils._force_rmtree()
so that it is safe to use for cleaning up the staging directory.

This patch fixes orphaned temporary directories being left behind
in the artifact cache directory at forceful termination time.

5e9598c5

by Tristan Van Berkom at 2019-01-16T21:17:26Z

Merge branch 'tristan/cleanup-staging-tempdir' into 'master'

element.py: Cleanup temporary staging directories on termination

See merge request BuildStream/buildstream!1080

5ea2f59d
by Tristan Van Berkom at 2019-01-16T23:35:21Z
```
tests: Migrate completions test to tests/frontend
```

1eafc25c