[BuildStream] Update on making buildtree uploaded configurable

[Tom Pollard <tom pollard codethink co uk>]

Hi all,

Before breaking up for the year I pushed a WIP [1] for adding the
optional configuration for not uploading buildtree artifacts to a remote
cache. This work is being done against the given issue [2], where I've
attempted to tackle it given the approach which was somewhat agreed
upon. As it stands the configuration to 'push partial artifacts' is
added to the CASRemoteSpec, allowing different projects to define this
configuration for the same remote url as is currently supported for
generic pushing.

Handling this whilst still keeping the casserver 'dumb' in terms of its
knowledge of artifact structure has been somewhat cumbersome, an example
of this is the walking through refs, where previously *any* dangling
object would be considered an error. The recent proposal 'Artifact as a
Proto' [3] highlights this as an issue that would be of benefit. The WIP
has tests which demonstrate much of the functionality that I expect,
given a mixture of artifact states, both locally & remotely. Although
the issue is to cover making upload(push) configurable, this obviously
has a knock-on effect to download(pull) when a given buildstream task is
expecting the assumption that if a remote has an artifact then it will
have the buildtree cached too.

Any input or feedback to the WIP MR or original issue would be greatly
appreciated at this point, if anything to agree upon the implementation
approach.

Thanks,

Tom

[1] https://gitlab.com/BuildStream/buildstream/merge_requests/1024
[2] https://gitlab.com/BuildStream/buildstream/issues/566
[3]
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00148.html
-- 
https://www.codethink.co.uk/privacy.html