-
5c208698
by Jim MacArthur
at 2018-11-26T10:23:31Z
Docs: Add remote execution architecture documentation.
-
5af455a4
by Jim MacArthur
at 2018-11-26T10:55:22Z
Merge branch 'jmac/execution-architecture-docs' into 'master'
Docs: Add remote execution architecture documentation.
See merge request BuildStream/buildstream!923
-
4ca892e3
by Jim MacArthur
at 2018-11-27T09:30:13Z
tests: Avoid hangs due to exceptions in subprocesses
-
d09a18b8
by Jim MacArthur
at 2018-11-27T10:31:07Z
Merge branch 'jmac/fix-test-hangs-2' into 'master'
Avoid hanging artifact cache tests
See merge request BuildStream/buildstream!964
-
df60258a
by Jürg Billeter
at 2018-11-27T12:11:24Z
source.py: Add optional _get_local_path() method
-
42809933
by Jürg Billeter
at 2018-11-27T12:11:24Z
local.py: Implement _get_local_path()
-
eb5b10f7
by Jürg Billeter
at 2018-11-27T12:11:24Z
_loader.py: Use Source._get_local_path() for subprojects
This eliminates unneeded copies of the project directory for junctions
with a single local source.
-
b2ea208c
by Jürg Billeter
at 2018-11-27T12:50:46Z
Merge branch 'juerg/local-junctions' into 'master'
Optimization for local junctions
See merge request BuildStream/buildstream!290
-
223c0073
by Jürg Billeter
at 2018-11-27T13:41:09Z
tests/integration/manual.py: Add test for command logging
-
38d16c62
by Jürg Billeter
at 2018-11-27T13:41:09Z
sandbox: Deduplicate code to process cwd, env, and command arguments
-
85f5b574
by Jürg Billeter
at 2018-11-27T13:41:09Z
sandbox/sandbox.py: Add SandboxFlags.NONE
This improves readability.
-
bdd6925f
by Jürg Billeter
at 2018-11-27T13:41:09Z
element.py: Use SandboxFlags.NONE instead of 0
-
69005c76
by Jürg Billeter
at 2018-11-27T13:41:09Z
scriptelement.py: Use SandboxFlags.NONE instead of 0
-
f1767de2
by Jürg Billeter
at 2018-11-27T13:41:09Z
sandbox/sandbox.py: Add command batching API
This adds the batch() context manager.
-
87f7c99a
by Jürg Billeter
at 2018-11-27T13:41:09Z
element.py: Pass element object to sandbox
This allows the sandbox to output messages using the plugin id.
-
024d584d
by Jürg Billeter
at 2018-11-27T13:41:09Z
element.py: Cache artifact also on SandboxCommandError
-
bb7c3213
by Jürg Billeter
at 2018-11-27T13:41:09Z
element.py: Defer workspace.prepared setting in case of pending commands
-
04f05b1a
by Jürg Billeter
at 2018-11-27T13:41:09Z
element.py: Support command batching for integration commands
-
7827ac50
by Jürg Billeter
at 2018-11-27T13:44:49Z
element.py: Add batch_prepare_assemble() method
This allows batching of all sandbox commands aross prepare() and
assemble().
-
63db90b2
by Jürg Billeter
at 2018-11-27T13:44:49Z
buildelement.py: Support batching for integration and build commands
-
73dec86e
by Jürg Billeter
at 2018-11-27T13:44:49Z
scriptelement.py: Support batching for integration and build commands
-
79442e0a
by Jürg Billeter
at 2018-11-27T13:44:49Z
compose.py: Support batching for integration commands
-
6018b3bd
by Jürg Billeter
at 2018-11-27T13:44:49Z
sandbox/_sandboxremote.py: Implement command batching
Execute batched commands in a single shell script.
-
ec3d4486
by Jürg Billeter
at 2018-11-27T13:44:49Z
NEWS: Add entry for command batching
-
127d332f
by Jürg Billeter
at 2018-11-27T14:16:50Z
Merge branch 'juerg/command-batching' into 'master'
Command batching
Closes #675
See merge request BuildStream/buildstream!915
-
d5b5538d
by Jim MacArthur
at 2018-11-27T15:32:19Z
artifactcache: Move ArtifactCacheSpec code into CASRemoteSpec.
There is nothing in ArtifactCacheSpec that's actually specific to
artifacts, so I've made it a CAS class so we can use the same
spec for remote execution.
-
f224d4cb
by Jim MacArthur
at 2018-11-27T15:32:19Z
artifactcache.py, _context.py: Move CASCache object into context
Since the artifact cache and remote execution share the same
local CAS store, they should share the same CASCache object. Moving
this into context allows us to do this.
-
926c1446
by Jim MacArthur
at 2018-11-27T15:33:34Z
_sandboxremote: Remove references to artifact cache.
Uses the new config options introduced earlier to specify remote
exec storage and execution service.
-
d7cdb03e
by Jim MacArthur
at 2018-11-27T15:33:36Z
Documentation: Update to show new config options
-
95d8244f
by Jim MacArthur
at 2018-11-27T15:33:36Z
tests/sandboxes/remote-exec-config.py: New test.
Tests that we get a sensible error message when we specify incomplete
or wrong information in the configuration for remote execution.
-
7f2f62eb
by Jim MacArthur
at 2018-11-27T15:33:36Z
artifactcache.py: Remove unused verify_digest_pushed
-
c55ca3a5
by Jim MacArthur
at 2018-11-27T15:33:36Z
Docs: Correct URLs in artifact cache example
The port number must come after the hostname and before the path.
The path is ignored by CASCache anyway, so including it is misleading.
-
2a8a3b19
by Jim MacArthur
at 2018-11-27T18:17:03Z
Merge branch 'jmac/remote_execution_split' into 'master'
Split remote execution from artifact cache
Closes #750
See merge request BuildStream/buildstream!946
-
499c70fd
by Valentin David
at 2018-11-28T09:11:21Z
setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52
-
3513580c
by Valentin David
at 2018-11-28T14:02:32Z
Merge branch 'valentindavid/ruamel_0_15_41' into 'master'
setup.py: require ruamel.yaml >= 0.15.41 < 0.15.52
See merge request BuildStream/buildstream!975
-
26cdee08
by Valentin David
at 2018-11-28T14:29:52Z
Use f_bavail to query available space. Not f_bfree.
f_bfree space might not be usable. In practice we see failures in big
disks because f_bfree is over 2GB and f_bavail is 0. We get ENOSPC if
writing on disk then.
-
58ca298f
by Valentin David
at 2018-11-28T14:29:52Z
Make cache clients not fail when a blob is not available.
We plan to make cache incomplete. That is some blobs are missing. For
most of cases we will delete references when requested if they are
incomplete. But there will be corner cases where objects are removed
after the reference is requested.
-
227fa26d
by Valentin David
at 2018-11-28T14:29:52Z
Fix type error in RPC messages
-
5ef19a0b
by Valentin David
at 2018-11-28T14:29:52Z
Avoid copying temporary file when adding object to CAS in server.
The file is already a temporary file and does not need copy. ENOSPC
is thrown during that copy in issue #609.
Fixes #678.
-
8d2946ff
by Valentin David
at 2018-11-28T14:29:52Z
"Fallocate" object disk space to avoid getting NOSPC error when writing
This locks the temporary object file so that cleanup does not need to
be done for every write.
-
b587953a
by Valentin David
at 2018-11-28T14:29:52Z
Update mtimes of objects for requested references.
This also remove references when some objects are missing. This is in
preparation for the move from reference to object garbage collection.
-
a64f667d
by Valentin David
at 2018-11-28T14:29:52Z
Move cas server from ref-based to object-based garbage collection.
-
353b90dd
by Valentin David
at 2018-11-28T14:29:52Z
Cleanup cache in cas server more agressively
When there is less than 2GB left, it cleans up have 10GB available.
These values are configurable.
-
ba9afa98
by Valentin David
at 2018-11-28T14:29:52Z
Lock cache cleanup in cas server
Cleaning up in parallel might slow down the cleaning process
-
9a458402
by Valentin David
at 2018-11-29T08:47:40Z
Merge branch 'valentindavid/cache_server_fill_up' into 'master'
Fix cleanup of cache in server when disk is full
Closes #678
See merge request BuildStream/buildstream!830
-
47b7a9ba
by Jim MacArthur
at 2018-11-29T09:58:48Z
_sandboxremote.py: Remove unnecessary tests.
push_message and push_directory will both raise assertion failures if
they fail to send the digest to the server. Checking the digest
returned by each only tests that the content was hashed locally, which
cannot reasonably fail.
-
4a8d0565
by Jim MacArthur
at 2018-11-29T10:42:31Z
Merge branch 'jmac/no-verify-digests' into 'master'
_sandboxremote.py: Remove unnecessary tests.
Closes #786
See merge request BuildStream/buildstream!976
-
2e78e0d1
by Valentin David
at 2018-11-29T13:18:17Z
Add support for .netrc in remote/tar/zip plugins
Fixes #723.
-
a6144100
by Valentin David
at 2018-11-29T13:18:17Z
.gitlab-ci.yml: Update image versions where forgotten
-
f6c184f5
by Valentin David
at 2018-11-29T16:15:56Z
Merge branch 'valentindavid/netrc' into 'master'
Add support for .netrc in remote/tar/zip plugins
Closes #723
See merge request BuildStream/buildstream!908
-
1f31aefc
by Angelos Evripiotis
at 2018-11-30T09:34:38Z
man/: update with changes since Apr 2018
It seems that the regenerated man pages haven't been committed for some
time. Add them all in one lump now.
Had to manually strip the version number from the generated files.
Had to manually fix the NAME field of bst.1 to not be 'bst - None'.
Oddly had to invoke click-man directly in order to get the changes to
be applied, the usual setup.py route mentioned in contributing.rst
didn't work.
Eyeballed the result of invoking man on each of the commands, seems
good.
-
033a5ad9
by Jürg Billeter
at 2018-11-30T10:09:11Z
Merge branch 'aevri/update_man' into 'master'
man/: update with changes since Apr 2018
See merge request BuildStream/buildstream!981
-
1123b9a1
by Tom Pollard
at 2018-11-30T13:30:35Z
_stream.py: Ability to pull missing buildtrees outside of pull/build
Adds helper function _buildtree_pull_required() to determine if a
pullqueue should be constructed, for commands outside of bst pull
and build where it is determined that an element's buildtree
artifact is to be required given the respective semantics and
config. Utilised in push() to attempt to mitigate skipping the push
of partial elements without the user having to have preceded it with
an explicit pull.
cli.py: Add new behaviour to push command description
element.py: Move _cached_buildtree() to be non local private method,
use _KeyStrength types to reduce duplication.
tests/integration/pullbuildtrees.py also updated to cover this
use-case.
-
ad293ed3
by Jürg Billeter
at 2018-11-30T14:10:06Z
Merge branch 'tpollard/774' into 'master'
_stream.py: Ability to pull missing buildtrees outside of pull/build
Closes #774
See merge request BuildStream/buildstream!978
-
0d634c21
by Phillip Smyth
at 2018-11-30T14:16:36Z
plugin.py: Add BAD_ELEMENT_SUFFIX warning
-
d97ef822
by Phillip Smyth
at 2018-11-30T14:16:36Z
_project.py: Check for fatal_warnings earlier
-
59d442e5
by Phillip Smyth
at 2018-11-30T15:41:57Z
loader: Added file suffix check and helper function
_loader/loader.py: Added Suffix check and warning for named elements and add helper function
-
2d6ec07d
by Phillip Smyth
at 2018-11-30T15:41:57Z
cli.py: no autocomplete on files without .bst
-
ac0bc4c0
by Phillip Smyth
at 2018-11-30T16:50:35Z
Added tests for .bst suffix and completions
buildcheckout.py: Add mandatory .bst suffix tests
completions.py: Add test for fail on invalid suffix
Added required files for testing
integration/source-determinism.py: renamed test elements to end with .bst
loader/__init__.py: initialised context properly with load and a message handler
-
d4bc662a
by Phillip Smyth
at 2018-11-30T16:50:50Z
NEWS: Added update about the Mandatory .bst change
-
7cea464a
by Jürg Billeter
at 2018-11-30T17:21:37Z
Merge branch 'mandatory_suffix' into 'master'
Mandatory .bst suffix
See merge request BuildStream/buildstream!967
-
cc5ef68c
by Tristan Van Berkom
at 2018-12-03T08:41:42Z
MANIFEST.in: Include the sources for rebuilding the documentation session HTML
This was previously missing, causing our CI to forget to rebuild the sessions
when updating the documentation.
This fixes issue #778
-
87ba19d7
by Tristan Van Berkom
at 2018-12-03T09:19:54Z
Merge branch 'tristan/fix-doc-session-rebuilds' into 'master'
MANIFEST.in: Include the sources for rebuilding the documentation session HTML
Closes #778
See merge request BuildStream/buildstream!984
-
1b24148f
by Chandan Singh
at 2018-12-03T09:46:13Z
Ensure `--deps=none` option works for `bst checkout`
Currently, `bst checkout --deps none` command always produces empty
output. Fix this issue and add regression test for the same.
- element_enums.py: Add Scope.NONE.
- element.py: Ensure Scope.NONE works correctly in addition to
Scope.RUN/Scope.ALL in Element.dependencies() and Element.search().
- tests/frontend/buildcheckout.py: Fix tests for `--deps none`.
Fixes #670.
-
8ac1e757
by Chandan Singh
at 2018-12-03T09:46:14Z
Add `--deps build` option to `bst checkout`
As discussed in
https://mail.gnome.org/archives/buildstream-list/2018-September/msg00064.html,
add `--deps build` option to `bst checkout`. This will allow users to
checkout the all build dependencies of a given element using a single
command.
- _frontend/cli.py: Add `--deps build` option for `bst checkout`.
- element.py: Support `deps='build'` in Element._prepare_sandbox().
- tests/frontend/buildcheckout.py: Ensure `--deps build` works as
expected.
-
ac60f47a
by Benjamin Schubert
at 2018-12-03T09:46:14Z
Refactor and simplify _prepare_sandbox for elements
Before we would have a intricate logics with multiple arguments
that might get ignored.
This simplifies the design and introduces a bool `shell` instead
of having two different variables concerned about scope
-
b1430fb5
by Tristan Van Berkom
at 2018-12-03T11:01:54Z
Merge branch 'chandan/bst-checkout-build' into 'master'
Add `--deps build` option to `bst checkout`
Closes #670
See merge request BuildStream/buildstream!819
-
9022a778
by Javier Jardón
at 2018-12-03T11:14:55Z
buildstream/plugins/sources/git.py: warn about checkout by default
See #783
-
4462892e
by Tristan Van Berkom
at 2018-12-03T11:45:07Z
Merge branch 'jjardon/doc_git_submodules' into 'master'
git: warn about checkout submodules by default
See merge request BuildStream/buildstream!983
-
6fcfa452
by Jonathan Maw
at 2018-12-03T13:21:05Z
_yamlcache.py: Use a project's junction name if present
This is required because if there are projects with the same name
loaded, that have elements in the same path, with the same contents,
they are considered the same, despite actually belonging to different
Projects.
This fixes buildstream issue #795
-
d0f9c724
by James Ennis
at 2018-12-03T13:21:05Z
tests/loader/junctions.py: Test that we can build a junction that is referred differently, twice.
Test required to close #795 - We should be able to build a project which refers
to the same junction twice, but with two different filenames.
-
f7581026
by Jürg Billeter
at 2018-12-03T16:27:03Z
Merge branch 'jonathan/fix-identical-element' into 'master'
_yamlcache.py: Use a project's junction name if present
Closes #795
See merge request BuildStream/buildstream!980
-
98c15463
by Tristan Van Berkom
at 2018-12-05T07:04:56Z
_scheduler/queues: Don't call update state outside of error handling harness
Commit 3fa79d8da, part of an initiative for caching of the failed builds,
introduced a call to Element._update_state() after a job completes and before
entering the error handling harness intended for handling plugin raised errors.
Element._update_state() can result in triggering plugin code to
run, so this is incorrect, and causes raised errors to crash BuildStream
if they happen here.
After analyzing the code, it appears that this additional call to
Element._update_state() is unneeded, and was only added because the
state needs to be updated for a failure as well as a success.
Instead, we now have the BuildQueue call Element._assemble_done()
unconditionally, regardless of whether the build was successful
or not, which has the same effect and also reads better.
In addition, added a FIXME comment that we are still conditionally
updating the artifact cache size from BuildQueue.done() only
if the build is successful, which is incorrect because failed builds
also increase the local artifact cache size - to fix this we need
to communicate the added artifact size through Element._assemble()
regardless of whether the build succeeded or failed.
-
5d77b871
by Tristan Van Berkom
at 2018-12-05T07:37:54Z
Merge branch 'tristan/refactor-queues-update-state' into 'master'
_scheduler/queues/queue.py: Don't call update state outside of error handling harness
See merge request BuildStream/buildstream!988
-
82513109
by Valentin David
at 2018-12-05T09:47:52Z
git source plugin: Track git tags and save them to reproduce a minimum shallow repository
Instead of tag information being fetched which can change with time,
they are tracked and saved in the projects.refs/.bst. Then we re-tag
automatically the closest tag so that `git describe` works and is
reproducible.
This new feature is opt-in with the new `track-tags` configuration,
and must be used to fix modules which are broken by our new policy
of omitting the `.git/` repository when staging git sources.
This fixes issue #487
-
ec7b62c3
by Valentin David
at 2018-12-05T09:50:15Z
NEWS: Add news about broken 'git describe' by default.
-
761e7570
by Tristan Van Berkom
at 2018-12-05T10:23:49Z
Merge branch 'valentindavid/git_describe_tracking' into 'master'
Track of git tags and save them to reproduce minimum shallow repository
Closes #487
See merge request BuildStream/buildstream!906
-
2f5966bd
by Raoul Hidalgo Charman
at 2018-12-05T10:37:10Z
platform.py: Add mapping of uname to OS-independent values
-
1ad35fcd
by Raoul Hidalgo Charman
at 2018-12-05T10:44:02Z
_config.py: Use os and architecture settings
In element, platform asks for host os and architecture to default to when
SandboxConfig is initialised.
This changes element cache keys so those have been updated in the tests.
-
9f75c274
by Raoul Hidalgo Charman
at 2018-12-05T10:44:02Z
optionarch.py: update to use same arch names as SandboxConfig
Also update tests to be consistent with this
-
a825096c
by Raoul Hidalgo Charman
at 2018-12-05T10:44:02Z
_sandboxremote.py: Add os and arch to command uploaded
These are contained within the platform field.
-
8e09a81c
by Raoul Hidalgo Charman
at 2018-12-05T10:44:02Z
optionos.py: Add new option for OS configuration
-
defec112
by Raoul Hidalgo Charman
at 2018-12-05T10:44:02Z
optionos.py: Add tests for OptionOS
Same format as OptionArch tests, but with OS's.
-
e5d8c7d8
by Raoul Hidalgo Charman
at 2018-12-05T10:44:02Z
_platform: Add checks for os and architecture
-
b0603fb0
by Raoul Hidalgo Charman
at 2018-12-05T10:44:02Z
Sandbox: use linux32 for x86-32 builds on x86-64 machines
o _platform/linux.py: Add linux32 flag to send to sandbox bwrap when the
build arch is x86-32 and the machines arch is x86-64 or similarly with
aarch32 and aarch64.
o sandbox/_sandboxbwrap.py: Use flag passed to start bwrap command with
linux32 if set.
-
f4d76996
by Raoul Hidalgo Charman
at 2018-12-05T14:58:49Z
format_project.rst: Update documentation for changes to OptionArch
-
81163a79
by Raoul Hidalgo Charman
at 2018-12-05T14:58:53Z
format_declaring.rst: Update documentation for sandbox options
-
23c27580
by Raoul Hidalgo Charman
at 2018-12-05T14:58:53Z
format_project.rst: Add docs for OptionOS
-
eb1ed410
by Jürg Billeter
at 2018-12-05T15:28:52Z
Merge branch 'raoul/775-execution-environment-reqs' into 'master'
Execution environment reqs
Closes #775
See merge request BuildStream/buildstream!969
-
9ef1a8a2
by Raoul Hidalgo Charman
at 2018-12-05T15:32:22Z
utils.py: Add message digest function
Get cascache.py to use this
-
4a56bf2a
by Raoul Hidalgo Charman
at 2018-12-05T15:33:44Z
_sandboxremote.py: Add checks to action cache before attempting to push.
Stops unneccesary pushing of builds that have already been built, just checks
the action cache to begin with.
Fixes #628
-
3641b6f6
by Raoul Hidalgo Charman
at 2018-12-05T15:33:44Z
_sandboxremote.py: Add action-cache-service endpoint
The endpoint may not necessarily be the same as execution-service endpoint.
Part of #628
-
9d77351f
by Raoul Hidalgo Charman
at 2018-12-05T15:33:44Z
format_project.rst: Add docs for action-cache-service
Part of #628
-
a53d6d1f
by Raoul Hidalgo Charman
at 2018-12-05T16:36:00Z
Merge branch 'raoul/628-RE-flow-optimisation' into 'master'
Remote-execution client flow optimisation
Closes #628
See merge request BuildStream/buildstream!982
-
4688b2c9
by Tristan Van Berkom
at 2018-12-06T07:35:59Z
tests/sources/git.py: Refactor ref-not-in-track test to use parameterization
Instead of duplicating the whole test body, run it twice while
checking for a warning or an error depending on the parameter.
-
03028c7f
by Tristan Van Berkom
at 2018-12-06T07:35:59Z
types.py: Moving CoreWarnings to the types.py module
This also ensures it is exposed via the main buildstream __init__.py
file, fixes the imports of CoreWarnings, and adjusts the documentation
links to point to the right place.
-
3b7f8df8
by Tristan Van Berkom
at 2018-12-06T07:35:59Z
Source: Fix typo in documentation, and make it a link to `bst track`
Now the commands have documentation linkability so we can link to
them when referring to them anywhere in the docs. This reference
was previously referring to a bogus non-existent `build-stream track`,
maybe this dates way back to the days before we named the frontend `bst`.
-
e5c0f067
by Tristan Van Berkom
at 2018-12-06T07:35:59Z
git source plugin: Fixing documentation linking and typo
Fixes the configurable warning part of this to properly link
to the project.conf documentation describing what configurable
warnings actually are.
Also fix the plugin raise the actual configurable warning which
it advertizes that it raises; which is 'inconsistent-submodule',
not 'inconsistent-submodules'.
-
205c27d8
by Tristan Van Berkom
at 2018-12-06T07:35:59Z
git source plugin: Rename symbol for consistency and readability
Change 'INCONSISTENT_SUBMODULE' for 'WARN_INCONSISTENT_SUBMODULE' to
make things a bit more readable and comprehensive.
-
bea4d4f5
by Tristan Van Berkom
at 2018-12-06T09:37:50Z
Merge branch 'tristan/misc-cleanup' into 'master'
Misc cleanups
See merge request BuildStream/buildstream!993
-
b1d2f001
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
source.py: Add new delegate method validate_cache()
This is guaranteed to be called only once for a given session once the
sources are known to be Consistency.CACHED, if source tracking is enabled
in the session for this source, then this will only be called if the
sources become cached after tracking completes.
-
ee7fc47f
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
git source plugin: Implementing submodule warnings
o Unlisted submodule warning
Now the git plugin will issue a configurable warning if a submodule
exists and is used (checking out the submodule is not disabled),
but is not specified in the source configuration.
o Invalid submodule warning
Now the git source plugin will issue a warning if the configuration
specified a submodule which does not exist in the underlying git repository.
As a side effect, this patch also changes the flow control of the git
plugin such that submodules which are explicitly set to not be
checked out, are also not fetched but instead ignored completely.
-
5fc9a1da
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
git source plugin: Emmit the ref-not-in-track warning from validate_cache()
Now that we have Source.validate_cache(), this is a better place to emmit
the ref-not-in-track warning, since it will be emmitted at the earliest
opportunity and not only at Source.stage() or Source.init_workspace().
This also allows us to remove the `self.tracked` local state, and cleanup
some convoluted calling paths, removing some unnecessary parameters from
the usual codepaths and making the plugin overall more readable.
-
50cb2706
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
tests/testutils/repo/git.py: Adding remove_path() helper
A function for removing paths from a git repo, can also
be used to remove submodules.
-
f139ca8f
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
tests/sources/git.py: Testing the git:unlisted-submodule warning
o Test that it is not triggered in show before fetch, because we
don't know about the unlisted submodules yet
o Test that it is triggered by a fetch command
o Test that it is triggered by `show` after having completed a
fetch command, since now we have the repository and know about
the unlisted submodule
o Test all of this under warning or error conditions (parameterized
for fatal-warnings)
-
afe1aa9f
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
tests/sources/git.py: Testing the git:invalid-submodule warning
o Test that it is not triggered in show before fetch, because we
don't know the submodules yet so we cannot know if they are
valid or not.
o Test that it is triggered by a fetch command
o Test that it is triggered by `show` after having completed a
fetch command, since now we have the repository and know which
specified submodules are invalid
o Test all of this under warning or error conditions (parameterized
for fatal-warnings)
-
57300201
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
tests/sources/git.py: Test unlisted submodules warning appearing after track
-
a0297625
by Tristan Van Berkom
at 2018-12-06T14:18:31Z
tests/sources/git.py: Test invalid submodules warning appearing after track
-
60e96781
by Tristan Van Berkom
at 2018-12-06T15:14:37Z
Merge branch 'tristan/submodule-warnings' into 'master'
Implement submodule warnings
See merge request BuildStream/buildstream!996
-
7fe55143
by Jonathan Maw
at 2018-12-06T15:48:12Z
Fix bst source-checkout not working with open workspaces
-
2a6879a5
by Jonathan Maw
at 2018-12-06T17:46:33Z
Merge branch 'jonathan/source-checkout-workspace' into 'master'
Fix bst source-checkout not working with open workspaces
Closes #807
See merge request BuildStream/buildstream!997
-
e61e6e50
by Tristan Van Berkom
at 2018-12-07T07:57:13Z
_yaml.py: Added `allow_none` parameter to _yaml.node_get()
This allows specifying whether or not the code believes a None
value is acceptable if explicitly expressed to be None in the YAML.
A previous patch 3ba544b80f9f268be8ffe62fc8589b30212ec4a2 changed
the behavior to accept None for all values across the board, along
with allowing explicitly setting the `default_value` to `None` which
caused the code to be much more readable (this was the main motivation
of the patch, but it had the side effect of allowing None for everything).
In the majority of cases we load YAML however, it either has a value or
it is not specified in the YAML, and None is hardly ever acceptable
to be explicitly specified (it may be in the case you want to override
something with nothing in the YAML, but it is the edge case).
This had the side effect of causing unexpected crashes where the
YAML specifies an empty dictionary for instance.
Instead of forcing the caller to handle a possible None value, give
the choice to the caller if they really want to accept a None value,
and raise the LoadError otherwise.
This fixes issue #803, which is a crash due to receival of an
unexpected None value through _yaml.node_get()
-
504ecb3e
by Tristan Van Berkom
at 2018-12-07T07:57:56Z
plugin.py: Added new `allow_none` parameter to Plugin.node_get_member(), defaulting to False.
Allow plugins to conveniently leverage the new `allow_none` parameter,
and have Plugin.node_get_member() raise a LoadError in the usual
case where None is not an acceptable value to be explicitly set in
the user provided YAML.
-
642ae4e8
by Tristan Van Berkom
at 2018-12-07T07:57:56Z
tests/format/project.py: Added new regression test for empty dependency dictionaries
This adds a regression test for issue #803
-
2a0676c3
by Tristan Van Berkom
at 2018-12-07T08:33:14Z
Merge branch 'tristan/yaml-optionally-allow-none' into 'master'
Only optionally allow None values in user provided YAML
Closes #803
See merge request BuildStream/buildstream!999
-
56c2c2cc
by Javier Jardón
at 2018-12-09T16:59:26Z
buildstream/data/projectconfig.yaml: Remove default strip-binaries
They are too specific to be included by default
Recommendation is if you are building in Linux is to use the
ones begin used in the freedesktop-sdk project, for example
See #645
-
dcb6b5e8
by Javier Jardón
at 2018-12-09T16:59:26Z
NEWS: Update for removal of default strip-commands
-
ae0afce5
by Javier Jardón
at 2018-12-09T17:08:16Z
buildstream/buildelement.py: Document where to find strip commands for Linux
-
873b618c
by Javier Jardón
at 2018-12-10T09:49:52Z
Merge branch 'jjardon/strip-binaries-removal' into 'master'
Remove default strip-commands
Closes #645
See merge request BuildStream/buildstream!987
-
f2fcc2f6
by Valentin David
at 2018-12-11T12:12:08Z
Force updating tags when fetching git repository
When using aliases there are multiple remotes used in the cache
repository. When fetching, tags are not updated if the were previously
fetched from a different remote. Commits that not in a branch and only
tagged do not get fetched if the tag is not fetched.
Fixes #812
-
717c10d1
by Valentin David
at 2018-12-11T12:44:56Z
Merge branch 'valentindavid/git_force_fetch_tags' into 'master'
Force updating tags when fetching git repository
Closes #812
See merge request BuildStream/buildstream!1000
-
7892287a
by Jonathan Maw
at 2018-12-11T12:56:32Z
utils.py: Add a helper for searching upwards for files
i.e. with a given directory and filename, check parent directories until
either a directory with the filename is found, or you reach the root of
the filesystem.
This is a part of #222
-
67c7a58d
by Jonathan Maw
at 2018-12-11T12:56:32Z
Create and store data inside projects when opening workspaces
Changes to _context.py:
* Context has been extended to contain a WorkspaceProjectCache, as there
are times when we want to use it before a Workspaces can be initialised
(looking up a WorkspaceProject to find the directory that the project is
in)
Changes to _stream.py:
* Removed staging the elements from workspace_open() and workspace_reset()
Changes in _workspaces.py:
* A new WorkspaceProject contains all the information needed to refer back
to a project from its workspace (currently this is the project path and
the element used to create this workspace)
* This is stored within a new WorkspaceProjectCache object, which keeps
WorkspaceProjects around so they don't need to be loaded from disk
repeatedly.
* Workspaces has been extended to contain the WorkspaceProjectCache, and
will use it when opening and closing workspaces.
* Workspaces.create_workspace has been extended to handle the staging of
the element into the workspace, in addition to creating the equivalent
WorkspaceProject file.
This is a part of #222
-
64836b18
by Jonathan Maw
at 2018-12-11T12:56:32Z
_project.py: Rename _ensure_project_dir to _find_project_dir
This is a part of #222
-
7deeb2c3
by Jonathan Maw
at 2018-12-11T12:56:32Z
cli.py: Use utils' search upwards helper when searching for project.conf
This is a part of #222
-
496f0ab7
by Jonathan Maw
at 2018-12-11T12:56:32Z
_project.py: Find project from workspace if outside of a project
This is a part of #222
-
921f2bcb
by Jonathan Maw
at 2018-12-11T12:56:32Z
_workspaces.py: Do not include .bstproject.yaml in the cache key
This is a part of #222
-
f145a3e4
by Jonathan Maw
at 2018-12-11T12:56:32Z
cli: Interactively warn if the user is trying to close the workspace they're using to load the project
This involves changes in:
* _stream.py:
* Add the helper Stream.workspace_is_required()
* userconfig.yaml:
* Add a default value for prompt.really-workspace-close-project-inaccessible
* _context.py:
* Load the prompt 'really-workspace-close-project-inaccessible' from
user config.
* cli.py:
* If buildstream is invoked interactively, prompt the user to confirm
that they want to close the workspace they're using to load this
project.
This is a part of #222
-
494cb7c6
by Jonathan Maw
at 2018-12-11T14:12:54Z
tests: Test bst commands from an external workspace
This is a part of #222
-
fd1c5c5a
by Jonathan Maw
at 2018-12-11T14:12:54Z
NEWS: Add an entry for being able to run commands from a workspace
This is a part of #222
-
4219a6b4
by Jonathan Maw
at 2018-12-11T16:00:55Z
Merge branch 'jonathan/workspace-fragment-create' into 'master'
Support invoking buildstream from a workspace outside a project
See merge request BuildStream/buildstream!924
-
e564251b
by Jonathan Maw
at 2018-12-11T16:15:19Z
_stream.py: Fix ugly error when opening a workspace using a relative path
-
7cf83343
by Jonathan Maw
at 2018-12-11T16:15:19Z
_context.py: Store the directory buildstream was invoked on
-
a1dee91e
by Jonathan Maw
at 2018-12-11T16:15:19Z
Make specifying elements optional in bst commands
Known issues:
* `bst shell` works, but `bst shell COMMANDS...` doesn't, because click
has no way of separating optional args from variable-length args.
* `bst checkout` and `bst source-checkout`'s usage strings mark LOCATION
as an optional argument. Because click gets confused if there's an
optional argument before a mandatory argument, I had to mark LOCATION
as optional internally.
* `bst workspace open` makes no sense with element being optional, so
I skipped it.
* `bst workspace close` will probably need to be revisited when multiple
projects can own one workspace.
* `bst workspace reset` will happily delete the directory you're
currently in, requiring you to `cd $PWD` to see the contents of your
directory.
I could exclude the top-level directory of the workspace being
deleted, but it is entirely valid to run workspace commands from deeper
in the workspace.
This is a part of #222
-
9b192015
by Jonathan Maw
at 2018-12-11T16:15:19Z
tests: Add tests for guessing element names
This is a part of #222
-
e340371f
by Jonathan Maw
at 2018-12-11T16:15:19Z
Add NEWS for guessing the element in commands
This is a part of #222
-
bbf81c3d
by Jonathan Maw
at 2018-12-11T16:15:19Z
doc: Point out that element is optional when commands are run from the workspace
This is a part of #222
-
a5a53ddd
by Jonathan Maw
at 2018-12-12T13:44:09Z
Merge branch 'jonathan/workspace-fragment-guess-element' into 'master'
Guess the element when running commands in a workspace
See merge request BuildStream/buildstream!954
-
d55b9e39
by Phil Dawson
at 2018-12-12T13:55:19Z
Add --tar option to source-checkout command
This commit is part of the work towards #672
-
733aab53
by Phil Dawson
at 2018-12-12T13:55:19Z
Add --force / -f option to source-checkout command
-
c2efeba0
by Phil Dawson
at 2018-12-12T14:43:40Z
Add option to source-checkout command to generate build scripts
-
85c61894
by Phil Dawson
at 2018-12-12T14:43:40Z
Remove source bundle command
This is part of the work towards #672
-
a322d5c0
by Phil Dawson
at 2018-12-12T14:43:40Z
NEWS: Add entry about the removal of source-bundle command
-
ec909605
by Phil Dawson
at 2018-12-12T15:45:42Z
Merge branch 'phil/source-checkout-options' into 'master'
Retire bst source bundle command
Closes #672
See merge request BuildStream/buildstream!959
-
3697a611
by Richard Maw
at 2018-12-12T16:31:38Z
cli: Add support for auto-completing artifact ref names
-
b3dceb16
by Richard Maw
at 2018-12-12T16:32:41Z
cli: Add artifact command group
-
ba08a0cd
by Richard Maw
at 2018-12-12T16:32:41Z
cli: Add artifact log command
-
70fb9554
by Richard Maw
at 2018-12-12T16:32:41Z
tests: Add tests for artifact group commands
-
f773e746
by Richard Maw
at 2018-12-12T16:33:02Z
NEWS: Mention `bst artifact log`
-
b6528441
by richardmaw-codethink
at 2018-12-12T18:00:59Z
Merge branch 'richardmaw/artifact-log' into 'master'
Add artifact log command
See merge request BuildStream/buildstream!920
-
d03bf316
by Benjamin Schubert
at 2018-12-13T10:24:11Z
Mock storage space checks for tests.
Fix #530
- Extract free space computation in a function for easier
mocking
- Mock space computation during cache quota tests
- Mock cache size during cachque quota tests
- Add two more tests when the configuration would require
to much storage space
-
a116f576
by Tristan Van Berkom
at 2018-12-13T10:58:46Z
Merge branch 'BenjaminSchubert/fix-quota-tests' into 'master'
Mock storage space checks for tests.
Closes #530
See merge request BuildStream/buildstream!702
-
180fa774
by Benjamin Schubert
at 2018-12-13T12:05:15Z
element.py: fix unbounded variable in nested python exception
As per https://docs.python.org/3.7/reference/compound_stmts.html#except
variables used in the "except" clause are erased at the end of the
block.
We had a conflict with a nested exception using the same variable as
before.
This renames the nested variable to avoid the clash
-
224aa4c2
by Benjamin Schubert
at 2018-12-13T12:34:41Z
Merge branch 'bschubert/fix-unbound-variable-exception' into 'master'
element.py: fix unbounded variable in nested python exception
See merge request BuildStream/buildstream!1006
-
4d3c032a
by Thomas Coldrick
at 2018-12-13T13:45:50Z
bst fmt: Add basic functionality
Adds basic functionality for `bst fmt`, formatting the yaml files of
elements
TODO:
* Weird issue in yaml dumping - see #767
* Add a check option, which just checks if format correct
* Add a canonical order of top-level nodes on a per project basis
-
6b0a2e1a
by Thomas Coldrick
at 2018-12-13T13:45:50Z
bst fmt: Add tests for core functionality
-
778f1db9
by Thomas Coldrick
at 2018-12-13T13:45:50Z
bst-fmt: Allow greater control over order
Allow authors of plugins to utilise the Plugin.keyorder attribute to
define an order for the sub-headers in config. Modify `bst fmt` to
modify the yaml into that order. Add canonical keyorder for every plugin
already in core.
In terms of implementation, this adds a custom yaml dumper subclassed
from ruamel.yaml's RoundTripDumper, which will dump a dict in the order
defined in the keyorder attribute. The Plugin class is given a keyorder
attribute, which defines the order in which keys should be dumped. The
element and source classes add the common fields used in them, so as to
minimise work for plugin authors. Plugin authors should add their custom
keys at configure time.
This causes some stripping of comments, which seems to be due to a bug
in the ruamel.yaml RoundTripDumper, as that also strips the same
comments. It also will remove blank spaces, again probably due to
limitations in ruamel.
