Re: [BuildStream] Coping with partial artifacts

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Thu, 2018-11-01 at 22:25 +0000, Sander Striker via BuildStream-list
wrote:
> Hi,
>
> I expect CAS and ArtifactCache (and ActionCache) to contain partial
> artifacts at some point due to different retention policies for
> different parts.  For example, build trees might expire much faster
> than the core artifact (the part that is staged when depended upon). 
> Similarly logs may expire as well.
>
> This is all policy based, and very much use case specific.  I do not
> wish for us to design or implement how partial expiry of artifacts
> might work, I merely want us to acknowledge that artifacts may become
> partial in the future.  And with that acknowledgement consider future
> proofing buildstream to gracefully deal with partial artifacts.

Since we have agreed to implement configurability of uploads of the
build trees[0], yes.

However for the rest, log files, etc; I think not - this is not a
generic thing and I consider it very unwise to go down a path where
arbitrary parts of an artifact might be missing.

BuildStream needs to know that an artifact exists or not, and rely on
this to make the assumption that all of the components of the artifact
exist.

Implementing configurability of the build trees on it's own is going to
already impose considerable complexity, we should not allow this to
grow any further.

Cheers,
    -Tristan

[0]: https://gitlab.com/BuildStream/buildstream/issues/566