-
0751fc8f
by Jürg Billeter
at 2019-01-04T11:27:58Z
Merge branch 'juerg/source-guess-target' into 'master'
_frontend/cli.py: Reinstate support for guessing targets
See merge request BuildStream/buildstream!1036
-
ebd965fe
by Chandan Singh
at 2019-01-04T15:21:35Z
Rename "tools" directory to "requirements"
This seems like a better name for the directory, as it more closely
describes the purpose of its contents.
-
bb712886
by Chandan Singh
at 2019-01-04T15:21:35Z
Add Makefile to update requirements files
-
b9792837
by Chandan Singh
at 2019-01-04T15:21:35Z
CONTRIBUTING.rst: Add instructions to update requirements files
Split the "The MANIFEST.in and setup.py" section in two:
"Managing data files" and "Updating BuildStream's Python dependencies".
Briefly explain the layout of `requirements` directory and add
instructions to use the Makefile added in the last commit.
-
45692335
by Tristan Van Berkom
at 2019-01-04T15:50:14Z
Merge branch 'chandan/update-requirements-one-liner' into 'master'
Add Makefile to update requirements files
See merge request BuildStream/buildstream!1035
-
f42dcb54
by Chandan Singh
at 2019-01-04T17:16:43Z
.gitlab-ci.yml: Remove prepare stage
As we now run tests using `tox`, we don't need to worry about manually
packing and unpacking BuildStream. So, we can remove the preapre stage
entirely.
Update `coverage` and nightly jobs to appropriately cope with this
change. Both these jobs now install all runtime dependencies from
requirements files.
-
ee29f207
by Chandan Singh
at 2019-01-04T17:41:02Z
Merge branch 'chandan/no-sdist' into 'master'
.gitlab-ci.yml: Remove prepare stage
See merge request BuildStream/buildstream!1037
-
f122cfe8
by Jürg Billeter
at 2019-01-06T17:57:33Z
_frontend/app.py: Use buildtree for interactive shell on build failure
Fixes: e29aea36 ("Basic options for shell --build to use buildtrees")
-
7dfb85b3
by Tristan Van Berkom
at 2019-01-06T18:31:25Z
Merge branch 'juerg/shell-buildtree' into 'master'
_frontend/app.py: Use buildtree for interactive shell on build failure
See merge request BuildStream/buildstream!1039
-
891fcb0e
by Tristan Van Berkom
at 2019-01-07T16:47:01Z
Fix stack traces discovered with ^C forceful termination.
* utils.py:_kill_process_tree(): Ignore NoSuchProcess errors
These are caused because we issue SIGTERM, and if the process
has not exited after a timeout, we kill it.
* _scheduler/jobs/job.py: Stop handling NoSuchProcess errors here
redundantly, they are already ignored.
It seems that we were ignoring it after sleeping when terminating
tasks from the scheduler... but we were not ignoring it when performing
the same pattern in the `Plugin.call()` -> `utils._call()` path, so
we were still getting these exceptions at termination time from host
tool processes launched by source plugins.
-
5de42d43
by Tristan Van Berkom
at 2019-01-07T18:00:37Z
Merge branch 'tristan/keyboard-interrupt-stack-trace' into 'master'
Fix stack traces discovered with ^C forceful termination.
See merge request BuildStream/buildstream!1043
-
059035b9
by Tristan Van Berkom
at 2019-01-07T18:02:00Z
_scheduler/scheduler.py: Make _schedule_jobs() private
This is not used anywhere outside of the Scheduler, currently
only the Scheduler itself is allowed to queue a job at this level.
If the highlevel business logic for automatic queueing of auxiliary
jobs moves to another location, we can make this public again.
-
b83d1b1f
by Tristan Van Berkom
at 2019-01-07T18:02:00Z
_scheduler/scheduler.py: Only run one cache size job at a time
When queuing the special cache management related cleanup and
cache size jobs, we now treat these jobs as special and do the
following:
* Avoid queueing a cleanup/cache_size job if one is already queued
We just drop redundantly queued jobs here.
* Ensure that jobs of this type only run one at a time
This could have been done with the Resources mechanics,
however as these special jobs have the same properties and
are basically owned by the Scheduler, it seemed more straight
forward to handle the behaviors of these special jobs together.
This fixes issue #753
-
16a8816f
by Tristan Van Berkom
at 2019-01-07T18:02:00Z
Scheduler: Introduced JobStatus instead of simple success boolean
This changes the deepest callback from when a Job completes to
propagate a JobStatus value instead of a simple boolean, and updates
all of the effected code paths which used to receive a boolean
to now handle the JobStatus values.
This further improves the situation for issue #753, as now we avoid
queueing cache size jobs for pull jobs which are skipped.
-
c2fc2a5e
by Tristan Van Berkom
at 2019-01-07T18:02:00Z
_scheduler/jobs/job.py: Removed 'skipped' property
This is redundant now that we report it through the JobStatus.
-
3e3984ad
by Tristan Van Berkom
at 2019-01-07T18:50:23Z
Merge branch 'tristan/one-cache-size-job' into 'master'
Only queue one cache size job
Closes #753
See merge request BuildStream/buildstream!1040
-
512c726e
by Tristan Van Berkom
at 2019-01-08T03:38:11Z
sandbox/sandbox.py: Fix regression of command logging
Since we added batch commands, the batch commands print the
text of the commands directly in the message text, but this is wrong.
The detail string is the appropriate place for text of unknown lengths
(the user can actually configure how many max lines of commands they
want to see in their log), the message text itself should be controlled
and brief enough to avoid text wrapping.
-
01171988
by Tristan Van Berkom
at 2019-01-08T04:20:14Z
Merge branch 'tristan/fix-command-status-messages' into 'master'
sandbox/sandbox.py: Fix regression of command logging
See merge request BuildStream/buildstream!1044
-
6c1d06d6
by Phil Dawson
at 2019-01-08T10:24:32Z
element.py: remove reference to source bundle command
This command has been replacved by the bst source checkout command
-
914ecb72
by Jürg Billeter
at 2019-01-08T10:54:02Z
Merge branch 'phil/remove-source-bundle-reference' into 'master'
element.py: remove documentation reference to source bundle command
See merge request BuildStream/buildstream!1041
-
81f7a964
by Tristan Van Berkom
at 2019-01-08T16:55:58Z
sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.
When code is faulty, in a plugin or in the core, we should
get a stack trace and a BUG message.
-
edcc43ed
by Tristan Van Berkom
at 2019-01-08T19:19:10Z
Merge branch 'tristan/sandbox-programming-error' into 'master'
sandbox/sandbox.py: Use assertions for programming errors instead of BstErrors.
See merge request BuildStream/buildstream!1046
-
17c5ca2d
by Valentin David
at 2019-01-09T10:45:23Z
Use relative path to project directory for remote execution certificates/keys
-
605f8d11
by Valentin David
at 2019-01-09T10:45:23Z
Add support for user remote execution configuration
Fixes #631.
-
65ea03ab
by Valentin David
at 2019-01-09T10:45:23Z
Add support for https channel to remote execution and actions servers
Fixes #780.
-
ba05c4fa
by Valentin David
at 2019-01-09T12:58:59Z
Merge branch 'valentindavid/remote_execution_configuration' into 'master'
Remote execution configuration: HTTPS and user configuration
Closes #780 and #631
See merge request BuildStream/buildstream!1030
-
eb0dbcfc
by Valentin David
at 2019-01-09T13:56:58Z
buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY
ScriptElement does not use Sandbox.get_directory. It works using it
with remote execution.
Fixes #850
-
f7681925
by Valentin David
at 2019-01-09T14:44:22Z
Merge branch 'valentindavid/script_virtual_directory' into 'master'
buildstream/plugins/elements/script.py: Mark script as BST_VIRTUAL_DIRECTORY
Closes #850
See merge request BuildStream/buildstream!1047
-
f87d1c93
by Jim MacArthur
at 2019-01-09T15:01:18Z
sandboxremote.py: Rename 'instance_name' option to 'instance-name'
It was 'instance-name' in the documentation.
-
ed8bc5bc
by Jim MacArthur
at 2019-01-09T15:30:28Z
Merge branch 'jmac/rename_instance_name' into 'master'
Rename 'instance_name' option to 'instance-name'
See merge request BuildStream/buildstream!1048
-
f29a0995
by Tristan Van Berkom
at 2019-01-09T19:51:14Z
Support running test environments in parallel with `detox`
This patch namespaces the test temp directory and the output
coverage report file with the name of the environment under test,
such that separately run tests do not access the same files.
When running tests without tox, directly through setup.py,
then the tmp directory will still be `./tmp`.
* .gitignore: Added new .coverage-reports/ directory
* .gitlab-ci.yml: Rely on tox to combine and report coverage, only
tell tox about the COVERAGE_PREFIX so that results can be namespaced
by CI job name.
This change also publishes the sources and final combined `.coverage`
file in an output gitlab artifact for inspection, and lists some missing
dependencies to the `coverage` job.
* tox.ini: Add comments and refactor main [testenv] section so that
other environments dont inherit too much unrelated cruft.
Generate the coverate reports in the respective {envtmpdir} so that
all per-process coverage files are prefixed with a full path, ensuring
that concurrent runs don't mix reports and addressing concerns
raised in #844.
Also implemented new `tox -e coverage` environment to combine
any found coverage and print a report.
* .coveragerc: Omit .tox/ directory from coverage stats
Fixes issue #844
-
d364ad02
by Tristan Van Berkom
at 2019-01-09T19:52:22Z
CONTRIBUTING.rst: Updated to mention new `coverage` tox environment.
Also point out that it is possible to run test environments in
parallel using the `detox` tool.
-
1e352434
by Tristan Van Berkom
at 2019-01-09T20:36:06Z
Merge branch 'tristan/detox-tests' into 'master'
Allow using detox, and fix/refactor collection of coverage reports
Closes #844
See merge request BuildStream/buildstream!1051
-
116c1070
by Tristan Van Berkom
at 2019-01-09T20:39:15Z
.coveragerc: Omit some things which are irrelevant to cover.
* Omit versioneer's _version.py
* Omit our __main__.py which is used only internally for generating
documentation
-
c8f3616d
by Tristan Van Berkom
at 2019-01-09T21:12:29Z
Merge branch 'tristan/refine-coverage' into 'master'
.coveragerc: Omit some things which are irrelevant to cover.
See merge request BuildStream/buildstream!1052
-
c87bb592
by Jürg Billeter
at 2019-01-10T09:43:37Z
_platform/platform.py: Add canonicalize_arch() method
-
26e33346
by Jürg Billeter
at 2019-01-10T12:50:15Z
_options/optionarch.py: Accept architecture aliases
Accept common architecture aliases for arch options instead of only
accepting the canonicalized, OS-independent architecture name. This
restores compatibility with existing projects and may simplify option
handling for projects that only target a single OS (and thus, do not
need OS-independent architecture names).
-
06deb4c4
by Jürg Billeter
at 2019-01-10T12:50:15Z
element.py: Accept architecture aliases for sandbox config
Accept common architecture aliases for the sandbox config for
consistency with arch options.
-
0c2a66b3
by Jürg Billeter
at 2019-01-10T13:06:02Z
tests/format/optionarch.py: Add tests for architecture aliases
-
f86bc760
by Jürg Billeter
at 2019-01-10T13:34:56Z
Merge branch 'juerg/arch' into 'master'
Accept architecture aliases
See merge request BuildStream/buildstream!1034
-
347eb34e
by Angelos Evripiotis
at 2019-01-10T14:38:23Z
contributing: fix 'oprtation' and some other typos
-
6bc27bc1
by Angelos Evripiotis
at 2019-01-10T15:07:46Z
Merge branch 'aevri/contrib-typos' into 'master'
contributing: fix 'oprtation' and some other typos
See merge request BuildStream/buildstream!1055
-
0b83d024
by Tristan Van Berkom
at 2019-01-10T20:02:50Z
_pipeline.py: Fix the planner to retain the original order when depth sorting
The algorithm adds elements to a dictionary and then sorts the dictionary
by the discovered depths while recursing - using an OrderedDict is enough
to ensure a stable order.
This fixes `bst show --deps plan ...` reporting different results on
each invocation.
This fixes an aspect of #824
-
630e26f1
by Tristan Van Berkom
at 2019-01-10T20:03:52Z
testutils/runcli.py: Added Result.get_start_order()
Added a function to report the list of elements which appeared in
a given queue in the order of their first appearance.
-
d1d7de57
by Tristan Van Berkom
at 2019-01-11T04:13:16Z
tests/frontend/order.py: Adding tests ensuring stable element processing order
For each sample project and expected result order, this test ensures that:
* bst show --deps plan: Always shows the expected build order.
* bst source fetch: Always executes in the expected build order.
* bst build: Always builds in the expected order (disabled).
The build part of the test is currently disabled as the scheduler
seems to be messing with the order even in a `--builders 1` scenario.
-
276b8d48
by Tristan Van Berkom
at 2019-01-11T04:42:24Z
Merge branch 'tristan/element-processing-order' into 'master'
Make build plan element list stable
See merge request BuildStream/buildstream!1058
-
a8d5d104
by Raoul Hidalgo Charman
at 2019-01-14T10:30:50Z
_cas: Rename artifactcache folder and move that to a root module
Other components will start to reply on cas modules, and not the artifact cache
modules so it should be organized to reflect this.
All relevant imports have been changed.
Part #802
-
35f68e2d
by Raoul Hidalgo Charman
at 2019-01-14T10:37:40Z
casremote.py: Move remote CAS classes into its own file
Part of #802
-
f9f76f0c
by Raoul Hidalgo Charman
at 2019-01-14T10:37:40Z
cas: move remote only functions to CASRemote
List of methods moved
* Initialization check: made it a class method that is run in a subprocess, for
when checking in the main buildstream process.
* fetch_blobs
* send_blobs
* verify_digest_on_remote
* push_method
Part of #802
-
54afbe28
by Raoul Hidalgo Charman
at 2019-01-14T10:40:38Z
artifactcache: Add tmpdir passed to CASRemote
This is a temporary directory within artifact cache.
Part of #802
-
002636ee
by Raoul Hidalgo Charman
at 2019-01-14T10:40:38Z
artifactcache: Move pull logic into CASRemote
Seperates the pull logic into a remote/local API, so that artifact cache
iterates over blob digests checks whether it has them, and then requests them
if not. The request command allows batching of blobs where appropriate.
Tests have been updated to ensure the correct tmpdir is set up in process
wrappers, else invalid cross link errors happen in the CI. Additional asserts
have been added to check that the temporary directories are cleared by the end
of a pull.
Part of #802
-
c13fc843
by Raoul Hidalgo Charman
at 2019-01-14T10:40:38Z
artifactcache: implement new push methods
Similar to the pull methods, this implements a yield_directory_digests methods
that iterates over blobs in the local CAS, with the upload_blob sending blobs
to a remote and batching them where appropriate.
Part of #802
