-
55956762
by Richard Maw
at 2018-09-13T16:50:33Z
element: Remove __cached field
This can get out of sync with other two cache states,
and we can do without it.
-
fc7f83ac
by richardmaw-codethink
at 2018-09-13T17:14:31Z
Merge branch 'richardmaw/element-cache-state-simplify' into 'master'
Simplify element state by removing `__cached`
See merge request BuildStream/buildstream!784
-
233a7d83
by Richard Maw
at 2018-09-14T08:53:14Z
Handle sockets when copying trees into artifacts
We can't include a socket in a CAS tree, but it's mostly meaningless to do so
since there can't possibly be a process serving it.
-
f06f234a
by Richard Maw
at 2018-09-14T08:53:14Z
tests: cover builds that make sockets
-
f86ab8f6
by richardmaw-codethink
at 2018-09-14T09:46:07Z
Merge branch 'richardmaw/builddir-sockets' into 'master'
Fix: While caching build artifact: "Cannot extract [path to socket file] into staging-area. Unsupported type."
See merge request BuildStream/buildstream!783
-
e7427462
by Richard Maw
at 2018-09-14T10:28:17Z
tests: Use cli.configure for custom logging test
Overriding the config with a custom config file on the command-line
prevents it merging in the test-specific configuration
and can permit it to attempt to initialise the user's cache.
-
800a8403
by Richard Maw
at 2018-09-14T10:28:17Z
Tidy relative workspace tests
They were moving the whole tmpdir to move the project repository.
This moves the cache directories etc. as well, meaning cli.run can't find them.
This was worked around by setting configure=False,
but this has the side-effect of making use of the user's caches,
which it should not be doing for reproducibility reasons.
By changing the tempdir layout to have the project in a subdirectory
we can move the project around for the relative workspace tests
without losing track of the configured state directories,
so we can leave configure=True and avoid touching the user's caches.
-
d7152ef4
by richardmaw-codethink
at 2018-09-14T10:55:16Z
Merge branch 'richardmaw/test-config-fixes' into 'master'
Fix tests that attempt to access the home directory
See merge request BuildStream/buildstream!780
-
160bb0c6
by Tristan Van Berkom
at 2018-09-14T12:07:46Z
_artifactcache/artifactcache.py: Changes to mark_required_elements()
This was previously append_required_artifacts(), which presumed that
we knew at startup time what the cache keys of all elements in the
loaded pipeline would be.
This fixes unexpected deletions of required artifacts when
dynamic tracking is enabled with `bst build --track-all target.bst`
-
39125d24
by Tristan Van Berkom
at 2018-09-14T12:07:46Z
tests/artifactcache/expiry.py: Cleanup of test for required artifacts
This commit renames test_never_delete_dependencies() to
test_never_delete_required(), renders the test more readable by renaming
some elements and reordering some statements and makes the comments more
straight forward and accurate.
-
f60558a3
by Tristan Van Berkom
at 2018-09-14T12:07:46Z
tests/artifactcache/expiry.py: Assert the expected errors
These tests were not checking that we fail for the expected reasons.
Added `res.assert_task_error(ErrorDomain.ARTIFACT, 'cache-too-full')`
where we expect to fail because the cache is too full.
-
ce68fd27
by Tristan Van Berkom
at 2018-09-14T12:07:46Z
testutils/repo/git.py: Added modify_file() method
This allows one to modify a file in an existing git repo,
as opposed to adding a new one.
-
532ec1eb
by Tristan Van Berkom
at 2018-09-14T12:07:46Z
testutils/element_generators.py: Changing sized element functions
* create_element_size()
Now uses a git Repo object instead of a local source, and
returns the repo.
* update_element_size()
Added this function which can now resize the expected output
of an element generated with create_element_size(), useful
to allow testing sized elements with the tracking feature.
-
20b797cb
by Tristan Van Berkom
at 2018-09-14T12:07:46Z
tests/artifactcache/expiry.py: Added test_never_delete_required_track()
Same test as test_never_delete_required(), except that this test ensures
that we never delete required artifacts when their cache keys are
discovered dynamically during the build.
-
c2af0d51
by Tristan Van Berkom
at 2018-09-14T12:44:42Z
Merge branch 'tristan/fix-required-artifacts' into 'master'
Don't delete required artifacts when tracking is enabled
See merge request BuildStream/buildstream!793
-
924cdc75
by Tiago Gomes
at 2018-09-14T15:32:01Z
core: remove unused variable on Context
-
510ccbfd
by Tiago Gomes
at 2018-09-14T15:32:21Z
artifactcache: fix oversight
We want to check if some file is already cached here, not the parent
directory.
-
82d4e2ac
by Tiago Gomes
at 2018-09-14T15:32:21Z
scheduler: add missing copyright
Tristan Maat created the original file, so he is added as the author.
-
18b37aab
by Tiago Gomes
at 2018-09-14T15:34:10Z
artifactcache: improve _create_tree()
* Rename it to _commit_directory() because… it is what it does; and
also for symmetry with _fetch_directory().
* Rename digest to dir_digest to make it clear this is a digest for a
directory. A following commit will also reuse the same variable name
* Document method.
-
32fad24f
by Tiago Gomes
at 2018-09-14T15:38:41Z
artifactcache: improve _fetch_directory()
* Rename tree to dir_digest to make it clear this is a Digest object,
and not a Tree object.
* Add documentation
-
46cbd889
by Tiago Gomes
at 2018-09-14T15:38:43Z
artifactcache: rename get_quota_exceeded()
-
2fa92716
by Tiago Gomes
at 2018-09-14T15:38:43Z
scheduler: minor cleanups to CleanupJob
Remove unneeded cruft.
-
c285f244
by Tiago Gomes
at 2018-09-14T16:08:07Z
Merge branch 'tiagogomes/some-cleanups' into 'master'
Bunch of cleanups
See merge request BuildStream/buildstream!798
-
662d22a9
by Tristan Van Berkom
at 2018-09-16T08:54:00Z
buildstream/_frontend/complete.py: Added missing click license compliance
Somehow I missed this when originally forking the file from the click
library, now noticing that we should have followed what was written
in: https://github.com/pallets/click/blob/master/LICENSE
-
a344407b
by Richard Maw
at 2018-09-16T10:22:59Z
sandbox: deduplicate default environment logic
-
c8fab440
by Richard Maw
at 2018-09-16T10:22:59Z
Bump Artifact Version
Since we now set PWD in the environment of builds
existing builds may behave differently so must cache differently now.
-
eb9481da
by Richard Maw
at 2018-09-16T10:22:59Z
Cachekey update helper: Document ensuring . in PYTHONPATH
The current directory isn't always in the python module search path,
so we have to ensure it is for the script to work.
Strictly speaking, the user may already have a modified PYTHONPATH
at which point PYTHONPATH=".${PYTHONPATH+:$PYTHONPATH}" is necessary,
but it's probably premature to overcomplicate the documentation like that
before we discover it's a problem.
-
9b7b2a25
by Tristan Van Berkom
at 2018-09-16T10:50:36Z
Merge branch 'richardmaw/subprocess-PWD' into 'master'
Address post-merge review of Ensure PWD is set in process environment
See merge request BuildStream/buildstream!788
-
8db62230
by Adam Jones
at 2018-09-17T07:39:06Z
Add a section on how to raise an issue
-
6f0a3e98
by Adam Jones
at 2018-09-17T07:39:20Z
Rename HACKING.rst to CONTRIBUTING.rst
-
ade19151
by Adam Jones
at 2018-09-17T07:40:45Z
update HACKING references
-
b9c5ff7d
by Tristan Van Berkom
at 2018-09-17T08:09:12Z
Merge branch 'tristan/contributing-guide' into 'master'
Update contributing guide
See merge request BuildStream/buildstream!801
-
727f2faa
by Tristan Van Berkom
at 2018-09-18T07:43:07Z
_project.py: Fix option resolution in element & source overrides
This ensures that option expressions are resolved in the project
level overrides before attempting to composite them on the instantiated
elements. Seems this is a regression from introducing the include
directive.
This fixes issue #658
-
ffa0bb36
by Tristan Van Berkom
at 2018-09-18T07:47:44Z
tests/format/optionoverrides.py: Added test for options in element overrides
This is a regression test for issue #658
-
f2ae46f8
by Tristan Van Berkom
at 2018-09-18T08:14:23Z
Merge branch 'tristan/fix-override-options' into 'master'
Fix override options
Closes #658
See merge request BuildStream/buildstream!802
-
7b117e40
by Daniel Silverstone
at 2018-09-18T08:41:32Z
_artifactcache/artifactcache.py: Ensure no double-setup of remotes
Since ArtifactCache.setup_remotes() can be expensive and should only
happen once, this commit will assert() if it is called a second time
on an artifact cache instance.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
345f5f49
by Daniel Silverstone
at 2018-09-18T08:41:32Z
tests/artifactcache/pull.py: Do not double-initialize remotes
The initialization of remotes is done by ArtifactCache.setup_remotes()
and as such it was wrong for these tests to be calling
CASCache.initialize_remotes() a second time.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
e32221b6
by Daniel Silverstone
at 2018-09-18T08:45:50Z
sandbox/_sandboxremote.py: Acquire artifact cache via Platform
The SandboxRemote used to construct its own CASCache which was
considered dangerous. This patch replaces that with acquisition of
the cache via the Platform singleton, hopefully eliminating issues
from having more than one artifact cache object in a single process.
Signed-off-by: Daniel Silverstone <daniel silverstone codethink co uk>
-
b587579f
by Tristan Van Berkom
at 2018-09-18T09:53:26Z
Merge branch 'danielsilverstone-ct/fix-654' into 'master'
sandbox/_sandboxremote.py: Acquire cache via Platform
See merge request BuildStream/buildstream!797
-
30b41959
by Tristan Van Berkom
at 2018-09-18T09:56:45Z
_artifactcache/artifactcache.py: Error out gracefully when push remote is mal-specified
When configuring a push remote and specifying either the client-cert
or the client-key, then both must be specified. This ensures we
get an informative error instead of a stack trace and BUG.
Fixes issue #625
-
41e8dc81
by Tristan Van Berkom
at 2018-09-18T09:56:45Z
tests/artifactcache/config.py: Added test for invalid push remote configuration
Test that we get the expected error when configuring a client-cert
without client-key, or the inverse.
-
97071b6e
by Tristan Van Berkom
at 2018-09-18T10:16:43Z
Merge branch 'tristan/fix-artifact-config-crash' into 'master'
Fix artifact config crash
Closes #625
See merge request BuildStream/buildstream!804
-
081dcafa
by Richard Maw
at 2018-09-18T13:22:38Z
fuse: Report the correct device number for devices
This fixes all devices being mapped to the non-existant device 0,
which prevents being able to use even safe devices like /dev/null
through the hardlinks FUSE layer.
-
d0425608
by Richard Maw
at 2018-09-18T13:22:38Z
FUSE: Mount with -odev in chroot sandbox
This is needed to permit access to the device nodes added to /dev
on Linux when FUSE is used as root.
The chroot sandbox only works with all privileges,
so there's no explicit check for being root
or having the appropriate capabilities.
A check for whether it's running as root isn't needed on Linux with bubblewrap
because /dev or its devices are mounted on top of the FUSE layer,
so device nodes are accessed directly rather than through the FUSE layer.
-
8430fdc7
by Richard Maw
at 2018-09-18T13:22:38Z
tests: test that integration commands can use /dev
-
d8450166
by Tristan Van Berkom
at 2018-09-18T13:50:15Z
Merge branch 'richardmaw/fix-chroot-sandbox-devices' into 'master'
fix chroot sandbox devices
See merge request BuildStream/buildstream!781
-
ba4da3a4
by knownexus
at 2018-09-18T16:45:09Z
Incorrect error when malformed project.conf
bst build returns "missing project.conf" when a project.conf is invalid
This results in an existing project with malformed yaml being dismissed
and attempts to make a new project
Added new exception
-
89a16e17
by knownexus
at 2018-09-18T16:45:09Z
Adding test for Invalid Yaml
-
72b59021
by Phillip Smyth
at 2018-09-19T08:48:51Z
Merge branch 'issue-642-Invalid_project.conf_seen_as_missing' into 'master'
Incorrect error when malformed project.conf
Closes #642
See merge request BuildStream/buildstream!792
-
ca811a4d
by Josh Smith
at 2018-09-19T09:25:21Z
Rework Skipped usage
The SKIPPED message type is now used to indicate the end of a task which
was successful without having to perform the given task.
This overhauls the use of `Queue.done()` and therefore queues do not
need to provide a processed/skipped return value from `done()`. Instead
this is replaced with the action of raising a `SkipJob` exception from
within `Queue.process()`.
-
16fbc3e3
by Josh Smith
at 2018-09-19T09:25:21Z
tests.py: Test skip on push
Adds a test to ensure that BuildStream alerts the user of a skipped push
when the remote already has the artifact cached.
-
ab56f693
by Josh Smith
at 2018-09-19T09:25:21Z
element.py: Remove redundant timed_activity
This removes the timed_activity for an element _push action. This is
unnecessary as the job is already being timed elsewhere.
-
198fbfe8
by Josh Smith
at 2018-09-19T09:25:21Z
cascache.py: Move push/pull messaging to cascache
Pulled/Pushed messages will no longer be produced from within
element.py, instead they will be produced during CasCache push() and
pull() appropriately.
Message consistency has also been improved.
-
213f7794
by Josh Smith
at 2018-09-19T09:25:21Z
cascache.py: Modify messaging API calls
Both pulling and pushing INFO messages are now status messages.
Calls to the messaging API through `self.context.message()` have now
been switched to `element.info`.
-
dddd6025
by Qinusty
at 2018-09-19T10:59:48Z
Merge branch 'Qinusty/skipped-rework' into 'master'
Add SkipJob for indicating a skipped activity
See merge request BuildStream/buildstream!765
-
16d9c6e5
by Tristan Van Berkom
at 2018-09-19T11:08:33Z
_frontend/status.py: Completely remove the blessings dependency from BuildStream
This actually improves reliability of the status bar because we
now disable it completely in the case that not all of the terminal
escape sequences are supported on the given terminal.
This replaces the few functions we were using, to move the cursor
up one line, move it to the beginning of the line, and to clear a
line, with low level functions provided by the curses module in
the standard library.
This change makes it easier for downstream distro package maintainers
to package BuildStream, particularly on Fedora.
Asides from changing _frontend/status.py, this commit includes the
following changes:
* _frontend/app.py: Use python isatty() function to determine if
we are connected to a tty, instead of relying
on blessings.
* setup.py: Remove the dependency on blessings.
-
7dd74b01
by Tristan Van Berkom
at 2018-09-19T11:27:55Z
Merge branch 'tristan/remove-blessings' into 'master'
_frontend/status.py: Completely remove the blessings dependency from BuildStream
See merge request BuildStream/buildstream!808
-
335d5a45
by Adam Jones
at 2018-09-19T13:00:45Z
Fix rst links for guideline links
-
667dd4ca
by Javier Jardón
at 2018-09-19T13:25:36Z
Merge branch 'adamjones/contributing-links' into 'master'
Fix rst link formatting for guideline links
See merge request BuildStream/buildstream!811
-
68ef69e4
by Tristan Van Berkom
at 2018-09-21T05:20:46Z
source.py: Fix re-instantiation
This fixes an issue where the re-instantiated Source used with
Source mirroring enabled is not completely initialized.
Failing to load the ref from the project.refs file for instance,
will result in a crash at `fetch` time.
This fixes issue #666
-
662c729f
by Tristan Van Berkom
at 2018-09-21T05:59:30Z
Merge branch 'tristan/fix-source-reinstantiation' into 'master'
source.py: Fix re-instantiation
Closes #666
See merge request BuildStream/buildstream!817
-
461a0588
by Jim MacArthur
at 2018-09-21T10:53:11Z
element.py: Remove early warning and check directory is not None
The check for directory should have always been here. The early warning
test doesn't work for `bst checkout` because caches haven't been configured
at the point of the test, so I've had to remove it.
-
aa9caaac
by Jim MacArthur
at 2018-09-21T10:53:11Z
_sandboxremote.py: Ignore return value of push_directory
The return value is always upload_vdir.ref.
-
2aae68c7
by Jim MacArthur
at 2018-09-21T10:53:11Z
cascache.py: make push_remote raise exception if no push remotes exist
Also add docstring to method and remove return value since it was useless
-
ca1bb72c
by Jim MacArthur
at 2018-09-21T10:53:11Z
tests/artifactcache/push.py: Ignore return value of push_directory
-
55c93a82
by Jim MacArthur
at 2018-09-21T11:26:55Z
Merge branch 'jmac/remote_exec_checkout_fix' into 'master'
Remote exec: Remove early warning and check directory is not None
See merge request BuildStream/buildstream!800
-
e209beb0
by Chandan Singh
at 2018-09-21T13:10:08Z
_stream.py: Ensure source-bundle's source directory exists
Currently, `source-bundle` command is entirely broken as it tries to stage the
sources in a directory that doesn't exist. Fix it by ensuring that we create
the necessary directories before calling any methods that try to use those
directories.
This fix comes with a regression test to ensure that the basic use-case
of `source-bundle` continues to work in future.
Fixes https://gitlab.com/BuildStream/buildstream/issues/651.