Re: [BuildStream] Artifact as a Proto plan

[Jürg Billeter <j bitron ch>]

Hi Raoul,

On Mon, 2019-02-11 at 17:35 +0000, Raoul Hidalgo Charman wrote:
> On 11/02/2019 16:35, Jürg Billeter wrote:
> > > * GetArtifacts: gives a list of references returns the artifact protos
> > > that are available. This will have an additional Boolean option for
> > > checking content as GetArtifact does, but without it will return
> > > artifact protos without updating  them.
> >
> > Do we really need/want this? What's the use case?
> >
> > Besides possible scaling concerns, this would make it impossible to use
> > a single artifact server in a team where not everyone should have
> > access to all artifacts. For sensitive content, we would likely anyway
> > recommend setting up separate artifact servers, however, there may be
> > scenarios where not listing artifacts may suffice as isolation.
>
> For a project build we can quickly see what artifacts are not available
> and what need to be rebuilt, rather than putting load on the server as
> it has to traverses all the and check content. Or do you mean the use of
> check content flag?

If the artifact is actually needed/used, traversing the artifact's
content is anyway required, so I don't see how this would help.

With large projects like the 'Debian' benchmark test project, we'll
reach >100k refs, and I don't think this approach scales.

Also, checking everything at startup is not necessarily desirable.
Besides blocking the start of the pipeline until the initial check is
complete, it would also not be efficient with multiple concurrent
clients building the same elements, see #179. We actually used to
follow this approach and moved to dynamic pulling, see !446.

> For isolation this would have to behave similarly to GetArtifact, so I'm
> not sure how it would differ? If a client without access tries to use
> GetArtifact for an artifact it doesn't have permission for the service
> should throw a not found error or similar.

As the cache keys are based on content hashes, only users that already
have the project (incl. source hashes or access to source repositories)
can construct the cache keys. I.e., you can't just fetch artifacts of
arbitrary projects from the server, if listing is not supported.

I don't know how important this will be in the real world. However, due
to the other concerns I don't think we actually need it, so I'd be in
favor of not supporting it at all, at least for the time being.

Cheers,
Jürg