[BuildStream] Artifact as a Proto plan

[Raoul Hidalgo Charman <raoul hidalgocharman codethink co uk>]

Hi everyone,

As artifact as a proto discussion has settled down (see [1] and [2]) as
well as some other further discussion, this email aims to be a summary
and plan for what future work needs to be done.

An artifact service will be implemented that replaces the current
generic reference storage, which will return artifact proto given a key
as described in [1], rather than returning a digest to a root directory.
On top of the message described in [1], this will have a Boolean which
describes whether a buildtree is to be expected for the artifact type,
not necessarily wether it will have one or not. This is important for
elements such as compose which will not have a buildtree, and should not
expected to have one even if a user requests build trees.

The service will implement the following RPCs, somewhat similar to to
the current reference service:
* GetArtifact: given a reference returns a single artifact proto, the
service first checking whether buildtrees and log files are present and
updating the proto.
* GetArtifacts: gives a list of references returns the artifact protos
that are available. This will have an additional Boolean option for
checking content as GetArtifact does, but without it will return
artifact protos without updating  them.
* UpdateArtifact: given a reference update to new artifact proto.

Various options will then want to be configured on the client side, as
to whether to download buildtrees and log files. We will also want to
eventually implement server side policies, for example a server may want
to reject anyone trying to push log files, or when to expire them.

Following on from this, in future we will want to have our own
capabilities service (similar to the REAPI capabilities service), that
allows clients to ask what an artifact service will do, such as allowing
log files and buildtrees. This would remove the need to have a status
RPC, as the current reference service does.

A separate source cache service will also be implemented for the source
cache along similar lines to this, with a proto containing relevant
source data.

Cheers,
Raoul

[1] -
https://mail.gnome.org/archives/buildstream-list/2018-December/msg00148.html
[2] -
https://mail.gnome.org/archives/buildstream-list/2019-January/msg00013.html