-
5b0bba85
by Jonathan Maw
at 2019-01-24T10:15:06Z
Test that tracking in workspaces actually works
Previously, it merely tested that buildstream did not fall other,
rather than whether it did anything useful.
-
58d7d722
by Jonathan Maw
at 2019-01-24T10:18:01Z
tests: Test that fetching an open workspace will fetch its dependencies
Previously, there was no way of detecting whether fetching happened, as
an element with an open workspace will not be fetched.
-
24bd8994
by Jürg Billeter
at 2019-01-24T13:36:33Z
Merge branch 'jonathan/test-missing-workspace-guessing' into 'master'
Add tests to cover reinstated support for guessing targets
See merge request BuildStream/buildstream!1042
-
e03dd5fc
by Jürg Billeter
at 2019-01-24T13:37:28Z
_frontend/cli.py: Guess element also for bst build --all
There is no reason to disallow guess_element() for bst build --all.
-
3850274b
by Phillip Smyth
at 2019-01-24T13:37:28Z
projectconfig.yaml: Add key for default targets
_versions.py: Bump format version
-
ab72d384
by Jürg Billeter
at 2019-01-24T13:37:28Z
_project.py: Add get_default_target() and get_default_targets() methods
_frontend/cli.py: Use new methods.
Based on patches by Phillip Smyth.
-
ee2d8434
by Jürg Billeter
at 2019-01-24T13:37:28Z
_stream.py: Add ignore_junction_targets parameter
This filters out junctions from the list of targets.
-
2f175f0a
by Jürg Billeter
at 2019-01-24T13:37:28Z
_frontend/cli.py: Ignore junctions in default targets where appropriate
Junctions cannot be built, pulled, or pushed. Specifying a junction on
the command line for these commands will result in an error. However,
junctions may be in the list of default targets, so they need to be
ignored for build, pull, and push commands.
-
5a351dee
by Jürg Billeter
at 2019-01-24T13:38:50Z
format_project.rst: Add documentation for default targets
-
77414518
by Phillip Smyth
at 2019-01-24T13:38:50Z
tests/frontend: Add default target tests for bst show and build
-
ce1c10ec
by Jürg Billeter
at 2019-01-24T13:38:50Z
tests/frontend/fetch.py: Add default target test for bst source fetch
-
99b5c0af
by Jürg Billeter
at 2019-01-24T13:38:50Z
tests/frontend/pull.py: Add default target test for bst push/pull
-
3642c8e7
by Jürg Billeter
at 2019-01-24T13:38:50Z
tests/frontend/buildcheckout.py: Add default target test with junction
Test that `bst build` does not fail in a project where the list of
default targets includes a junction (junctions cannot be built).
-
50c5159f
by Phillip Smyth
at 2019-01-24T13:38:50Z
NEWS: Add entry for default target feature
-
05587f22
by Jürg Billeter
at 2019-01-24T15:10:08Z
Merge branch 'issue-638-validate-all-files' into 'master'
Add support for default targets
See merge request BuildStream/buildstream!925
-
3e36e363
by Tristan Van Berkom
at 2019-01-24T16:55:24Z
_scheduler/resources.py: Dont error out in unregister_exclusive_interest()
Don't require the interest to be registered, just discard any interest,
this function just sets a bit in a mask, and is not intended to maintain
a balance like the reserve() function is.
-
ce01f87e
by Tristan Van Berkom
at 2019-01-24T16:55:24Z
_scheduler/scheduler.py: Run cache size exclusively at startup
When running any session that has Queues which require Resource.CACHE,
check if our loaded estimated size exceeds the quota, and if so;
lock the Resource.CACHE resource exclusively right away and run
an exclusive initial cache size job.
This ensures we cleanup first before doing anything which might
add to the cache at startup time, if deemed needed.
This is a partial fix for issue #737
-
2479e8df
by Tristan Van Berkom
at 2019-01-24T16:55:24Z
_frontend/widget.py: Render core messages more like other messages
In order to test when core activities occur by parsing the stderr
in tests, we should make the messages conform more.
At the same time, this restores alignment of columns in core
messages with the element processing related messages.
Also, _scheduler/scheduler.py is updated to make it's activity names
conform to the (current) 5 character limit for the sake of alignment.
The tests/frontend/logging.py test gets it's regexes updated for
the log lines it checks for in stderr.
-
fdb8ff65
by Tristan Van Berkom
at 2019-01-24T16:55:24Z
tests/artifactcache/expiry.py: Test that expiry happens first
-
acd0bf22
by Tristan Van Berkom
at 2019-01-24T18:01:41Z
Merge branch 'tristan/cache-management' into 'master'
Cache management fixes
See merge request BuildStream/buildstream!1091
-
41f03296
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_frontend/app.py: Initialize logging before preflighting the artifact cache
The artifact cache emits messages, and we want to allow that in preflight.
-
24ca2f46
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
utils.py: Add _get_volume_size()
We can streamline this call to os.statvfs() in a few places.
-
9fd9fbb9
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Refactored to use utils._get_volume_size()
This will benefit from a better UtilError being raised, and
and turns the artifact cache's local function into a one liner.
The loop which finds the first existing directory in the
given path has been removed, being meaningless due to the
call to os.makedirs() in ArtifactCache.__init__().
The local function was renamed to _get_cache_volume_size() and
no longer takes any arguments, which is more suitable for the
function as it serves as a testing override surface for
unittest.mock().
The following test cases which use the function to override
the ArtifactCache behavior have been updated to use the new
overridable function name:
tests/artifactcache/cache_size.py
tests/artifactcache/expiry.py
-
7ee0c579
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Added ArtifactCacheUsage()
A simple object which creates a snapshot of current
usage statistics for easy reporting in the frontend.
-
353293b6
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_context.py: Added get_artifact_cache_usage()
A frontend facing API for obtaining usage statistics.
I would have put this on Stream instead, but the Context
seems to be the de facto place for looking up the artifact cache
in general so let's put it here.
-
51ed36de
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_frontend/widget.py: Added cache usage entry in the startup heading
-
5797238b
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_frontend/status.py: Added Cache size usage indicator to status bar
This also adds some comments around the main status bar heading
rendering function.
-
8074ebf4
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Add status messages in cache management operations
Added some useful status messages when:
* Calculating a new artifact cache usage size
* Starting a cleanup
* Finishing a cleanup
Also enhanced messaging about what was cleaned up so far when
aborting a cleanup.
-
49c11bc8
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Added client progress callback to ArtifactCache.clean()
-
3616e939
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_scheduler/jobs/job.py: Allow subclasses to message the frontend
-
bcd19266
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_scheduler/jobs/cleanupjob.py: Update cache size while processing
Updates the known cache size in the main process while the cleanup
process is ongoing, so that the status indicators update live
while the cleanup happens.
-
9c33107f
by Tristan Van Berkom
at 2019-01-24T18:02:08Z
_artifactcache.py: Correcting API documenting comment for remove()
This seems to have been copy/pasted from cascache, and
documents the function to possibly return None if defer_prune
was specified, but this function does not expose defer_prune.
-
a2140d74
by Tristan Van Berkom
at 2019-01-24T19:44:52Z
Merge branch 'tristan/cache-management-logging' into 'master'
Cache management logging enhancements
See merge request BuildStream/buildstream!1105
-
d5847462
by James Ennis
at 2019-01-25T16:35:21Z
_profile.py: Added a new profiling topic, scheduler
profile_start() and profile_end() calls have been incorporated into
Scheduler.run()
-
56a3954c
by James Ennis
at 2019-01-25T16:35:21Z
_profile.py: Added a new profiling topic, load-selection
profile_start() and profile_end() calls have been added to
Stream.load_selection()
-
116da6d7
by James Ennis
at 2019-01-25T16:35:21Z
_profile.py: Update copyright statement
-
137d31cd
by James Ennis
at 2019-01-25T17:59:22Z
Merge branch 'jennis/add_new_profile_topic' into 'master'
Add new 'scheduler' and 'load-selection' profiling topics
See merge request BuildStream/buildstream!1088
-
22b3a0c1
by Tristan Van Berkom
at 2019-01-25T18:35:36Z
_artifactcache.py: Don't require the quota to be available on disk.
Instead only rely on the headroom to be enough to protect against
out of space conditions. The headroom can become configurable as
a separate step is required.
The changes to achieve this are:
* Rename ArtifactCache.has_quota_exceeded() to ArtifactCache.full().
* ArtifactCache.full() now also reports True if the available
space on the artifact cache volume is smaller than the headroom.
This ensures jobs get triggered to cleanup the cache when
reaching the end of the disk.
* When loading the artifact quota, it is now only an error if
the quota exceeds the overall disk space, not if it does not
fit in the available space.
It is still a warning if the quota does not fit in the
available space on the artifact cache volume.
* Updated scheduler.py and buildqueue.py for the API rename
* tests: Updated the artifactcache/expiry.py test for its
expectations in this regard.
Added a new test to test an error when quota was specified to
exceed total disk space, and adjusted the existing tests to
expect a warning when the quota does not fit in the available
space.
This fixes issue #733 and #869.
-
27ca6cc7
by Tristan Van Berkom
at 2019-01-25T21:18:15Z
Merge branch 'tristan/cache-quota-max-only' into 'master'
_artifactcache.py: Don't require the quota to be available on disk.
Closes #869 and #733
See merge request BuildStream/buildstream!1106
-
89ba2abe
by Chandan Singh
at 2019-01-25T23:50:49Z
tox.ini: Add environment to update man pages
Previously, one had to manually install `click-man` package, and
remember to run the correct command. Now, we can simply run `tox -e man`
to update the man pages.
-
d7438688
by Chandan Singh
at 2019-01-25T23:50:49Z
man: Refresh man pages
Notable changes:
* New `source` and `artifact` command groups
* Man pages corresponding to obsolete commands, that were marked as
hidden in Click, have now been removed.
Fixes #881.
-
4edbbd27
by Chandan Singh
at 2019-01-25T23:50:49Z
setup.py, CONTRIBUTING.rst: Recommend using tox to generate man pages
Simplify our docs, by requesting users to run `tox -e man` to update man
pages, instead of manually installing `click-man` and running the
command manually.
-
a3e7aee8
by Chandan Singh
at 2019-01-25T23:50:49Z
setup.py: Do not error out when man directory is empty/missing
If the `man` directory is empty, then it won't be copied in the source
distribution, and `list_man_pages()` will throw an exception when trying
to list files in a non-existent directory. This prevents us from
installing the BuildStream package when the man pages are not there.
The most common use-case for this is when we want to re-generate the man
pages but want to install the package before re-generating them.
-
222753ac
by Tristan Van Berkom
at 2019-01-26T05:10:36Z
Merge branch 'chandan/toxic-man' into 'master'
Generate man pages using tox & update them
Closes #880 and #881
See merge request BuildStream/buildstream!1107
-
5df4105a
by Angelos Evripiotis
at 2019-01-28T10:13:40Z
news: fix 'osbolete' spelling
-
9c981eff
by Angelos Evripiotis
at 2019-01-28T10:17:57Z
Fixup refs to 'bst track'
Now that 'bst track' is obsolete, change guidance to refer to the
replacement 'bst source track' instead.
-
bef80291
by Angelos Evripiotis
at 2019-01-28T10:19:17Z
Fixup refs to 'bst fetch'
Now that 'bst fetch' is obsolete, change guidance to refer to the
replacement 'bst source fetch' instead.
-
564cb245
by Angelos Evripiotis
at 2019-01-28T11:36:16Z
Merge branch 'aevri/bst_track_guidance' into 'master'
Fixup refs to 'bst track' and 'bst fetch'
See merge request BuildStream/buildstream!1086
-
a3e2cdd2
by Tom Pollard
at 2019-01-28T12:14:40Z
_stream.py: Add use_artifact_config opt arg for load_selection()
use_artifact_config added as an optional default arg, allowing
for loading of given elements artifact remote config.
-
805baf7d
by Tom Pollard
at 2019-01-28T12:14:40Z
Download buildtrees on demand for bst shell --use-buildtree
Provide bst shell --use-buildtree the ability to attempt to
acquire missing buildtrees, given respective option, user
pull-buildtree context and remote availability.
_frontend/cli.py: Refactor logic for determining --use-buildtree
option with given opportunity to attempt pulling a non-local
buildtree. Element loaded with artifact_config to allow remote
querying.
_stream.py: With given user option and element state, construct
PullQueue to fetch remote buildtree. Continue or Error without
buildtree if cannot be attained.
tests/integration/build-tree.py: Update to support new usecases
-
38356932
by Tom Pollard
at 2019-01-28T13:44:37Z
Merge branch 'tpollard/829' into 'master'
Download buildtrees on demand for bst shell --use-buildtree
Closes #829
See merge request BuildStream/buildstream!1050
-
3590ca8c
by Abderrahim Kitouni
at 2019-01-28T14:59:19Z
requirements/requirements.in: require protobuf >= 3.6
This is needed since 0f2bc3754
-
39b952dc
by Abderrahim Kitouni
at 2019-01-28T15:19:04Z
requirements/requirements.in: require Click >= 7.0
This is needed since 629a6e524, and was lost in the conversion to requirements.in
-
80b36d0c
by Javier Jardón
at 2019-01-28T17:54:52Z
Merge branch 'abderrahim/protobuf-version' into 'master'
requirements/requirements.in: update minimum versions
Closes #884
See merge request BuildStream/buildstream!1114
-
a1ab48da
by Valentin David
at 2019-01-28T21:30:26Z
Fix crash when spawned job completes very fast
Job can complete before we return from `Job.span()` to
`Scheduler._spawn_job()`, so that `_active_jobs` would not yet contain
the job.
This would print a stack on the console and try to run a second time
the job which can have unexpected effects.
In order to reproduce the issue, in
`buildstream/_scheduler/jobs/job.py`, in `Job.spawn`,
add a call to `time.sleep()` right before call to
`asyncio.get_child_watcher()`.
This fixes issue #857.
-
2fcb4491
by Jürg Billeter
at 2019-01-28T22:36:22Z
Merge branch 'valentindavid/crash_in_scheduler_857' into 'master'
Fix crash when spawned job completes very fast
Closes #857
See merge request BuildStream/buildstream!1095
-
1c05a092
by Valentin David
at 2019-01-29T05:58:17Z
Fix type of error codes in CAS server
Fixes #882.
-
785da59c
by Jürg Billeter
at 2019-01-29T06:49:10Z
Merge branch 'valentindavid/wrong_type_in_status_code' into 'master'
Fix type of error codes in CAS server
Closes #882
See merge request BuildStream/buildstream!1099
-
ddef91ea
by Valentin David
at 2019-01-29T07:23:35Z
Make sure testing cache directory exists
Fixes #873
-
6a4c8611
by Jürg Billeter
at 2019-01-29T08:18:45Z
Merge branch 'valentindavid/make_cache_dir' into 'master'
Make sure testing cache directory exists
Closes #873
See merge request BuildStream/buildstream!1092
-
86c8e414
by Angelos Evripiotis
at 2019-01-29T10:39:29Z
BREAK:remove unconditional 'are you sure?' prompts
This is a breaking change, as it affects behaviour that people might be
relying on. An entry has been added to NEWS.
As proposed on the mailing list, this change removes the unconditional
prompts on:
o: bst workspace reset
o: bst workspace close --remove-dir
If interactive, these commands would always interrupt you with a prompt
like this:
This will remove all your changes, are you sure?
This seems like it may just save someone's work some time. It may also
condition folks to hit 'y' quickly without thinking.
This change also makes the non-interactive behaviour consistent with the
interactive behaviour in the default case. There is also the case of the
prompt configured by 'really-workspace-close-project-inaccessible',
which may be tackled in later work.
This change also removes the new config options to suppress those
prompts, and their associated news entry.
The relevant bit of the mailing list conversation is here:
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00106.html
The issue to make interactive and non-interactive behaviour consistent
is here:
https://gitlab.com/BuildStream/buildstream/issues/744
-
aae35e13
by Angelos Evripiotis
at 2019-01-29T12:08:07Z
Merge branch 'aevri/are_you_sure' into 'master'
BREAK:remove unconditional 'are you sure?' prompts
See merge request BuildStream/buildstream!1061
-
03111d39
by Dor Askayo
at 2019-01-30T10:35:06Z
filter.py: don't recurse when staging dependencies
Also bump the element's version so cached artifacts would be
invalidated.
Fixes #883
-
7256bb0c
by James Ennis
at 2019-01-30T11:34:04Z
Merge branch 'doraskayo/filter-indirect-deps-fix' into 'master'
filter.py: don't recurse when staging dependencies
Closes #883
See merge request BuildStream/buildstream!1110
-
36746730
by Chandan Singh
at 2019-01-31T10:50:05Z
tox.ini: Specify minimum version of click-man
`click-man` versions < 0.3.0 do not properly support multiple
entrypoints. Since this was added to `tox` after `0.3.0` was released,
`tox` should never be pulling older versions. But, let's add it here for
documentation purposes.
See
https://gitlab.com/BuildStream/buildstream/merge_requests/1107#note_135187046
for some background on this.
-
fa4a21ce
by Chandan Singh
at 2019-01-31T12:15:43Z
Merge branch 'chandan/min-version-click-man' into 'master'
tox.ini: Specify minimum version of click-man
See merge request BuildStream/buildstream!1120
-
dd791373
by Chandan Singh
at 2019-01-31T14:32:44Z
testutils/site.py: Support parsing more exotic git versions
We use output of `git --version` to determine if we can run some tests
that rely on features from newer git versions. Usually, we expect the
output to be like:
git version 2.17.2
On some platforms, like MacOS, there could be a suffix after the version
string, so that it looks something like:
git version 2.17.2 (Apple Git-113)
This causes things to fail like so:
ValueError: invalid literal for int() with base 10: '2 (Apple Git-113)\n'
Fix logic around `HAVE_OLD_GIT` such that we split the output of
`git --version` without limit on how many times we split. Previously we
used to split only twice so the suffixes like `(Apple Git-113)` are not
part of the parsed version.
-
96c0fbd6
by Chandan Singh
at 2019-01-31T15:39:19Z
Merge branch 'chandan/fix-git-version-mac' into 'master'
testutils/site.py: Support parsing more exotic git versions
See merge request BuildStream/buildstream!1118
-
d25e2795
by Benjamin Schubert
at 2019-01-31T17:06:23Z
Add LoadElement in dependency list for LoadElement idrectly
This removes the need for the 'Dependency' list to then be matched
with the corresponding LoadElement and will allow iterating the
graph more easily
-
2d0eebbf
by Benjamin Schubert
at 2019-01-31T17:06:23Z
Pass element directly to collect_element
This simplifies the loading
-
583bd97d
by Benjamin Schubert
at 2019-02-01T10:26:37Z
Merge branch 'bschubert/loader' into 'master'
Cleanup loader by linking LoadElements sooner
See merge request BuildStream/buildstream!1122
-
51cec3da
by Phil Dawson
at 2019-02-01T14:25:44Z
tests/cachekey: Test cache keys are independent of target elements
-
2b38aabe
by Phil Dawson
at 2019-02-01T15:33:00Z
Merge branch 'phil/cache-key-stability-test' into 'master'
tests/cachekey: Test cache keys are independent of target elements
See merge request BuildStream/buildstream!1123
-
dbb3d232
by James Ennis
at 2019-02-01T15:51:32Z
filter.py/filter.yaml: Documentation improvements
-
7e4205cb
by James Ennis
at 2019-02-01T15:51:32Z
filter.py: Add an example to the documentation
-
4109a34a
by James Ennis
at 2019-02-01T17:11:29Z
Merge branch 'jennis/filter-docs' into 'master'
Improve our filter documentation
Closes #278
See merge request BuildStream/buildstream!1112
-
c9345014
by James Ennis
at 2019-02-04T13:53:42Z
filter.py: Fail if declared domains do not exist in the parent element
This patch also uncovered the fact that our test_filter_deps_ok() test
has been inaccurate. Thus the element built in this test
(deps-permitted.bst) has been modified so that it build depends on the
input.bst element, as it should.
tests/filter.py: Ensure deps_ok test passes
-
3ab09651
by James Ennis
at 2019-02-04T14:47:43Z
Merge branch 'jennis/warn_for_nonexistent_domains' into 'master'
Fail when we explictly try to include/exclude non-existent domains in a filter element
See merge request BuildStream/buildstream!1117
-
f56471ef
by Raoul Hidalgo Charman
at 2019-02-04T16:40:40Z
rootcachedir: add new dir option that's default root to other dirs
Deprecates artifactdir and builddir.
Fixes #870
-
10c3ad6a
by Raoul Hidalgo Charman
at 2019-02-04T16:40:40Z
CASCache: Move cache check methods here
A lot of code that checks cache quota has been touched.
Part of #870
-
85fe3180
by Raoul Hidalgo Charman
at 2019-02-04T16:50:16Z
_basecache.py: Move artifactcache methods to base cache
ArtifactCache now derives from BaseCache, and so will future SourceCache.
Part of #440
-
d1f27064
by Raoul Hidalgo Charman
at 2019-02-04T16:57:50Z
sourcecache: WIP commit with sketch of class