Re: [BuildStream] Artifact cache authentication/metrics

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

First of all, sorry that your email went unanswered, I tried to catch
up with unanswered emails yesterday and this one fell off the grid.

On Sat, 2018-08-04 at 12:52 +0100, Adam Jones via BuildStream-list
wrote:
> Hi All,
>
> I was speaking with Edmund about configuring the buildstream cache and
> he raised some interesting questions about how well equipped the cache
> is for shared instances.
>
> The idea being that in the future people may want to share a cache
> between multiple projects/teams (at scale), but this would pose
> multiple problems:
>
> A) How will these different entities access the cache server?
> B) Can custom authentication methods be implemented? 
> C) Can the access rates be tracked and logged for each user?
>
> I explained that the docs does mention the use of a reverse-proxy for
> authentication, but this does not get any further elaboration:
> "Alternatively, you can set up a reverse proxy and handle
> authentication and authorization there."

Personally, I'm satisfied with this phrase for reference materials;
setting up a reverse proxy is something you can do with tech like
apache, it's straight forward for a sysadmin, and a sysadmin will
probably be picky about how they want it done.

I think that we currently provide enough docs to get off the ground; by
demonstrating at least one basic way to get off the ground and stating
that the http protocol is used (I don't know if this is stated clearly
or strongly enough in the docs though).

Beyond this, you get into networking mumbo jumbo that nobody
understands except sysadmins... or everyone understand except me, take
your pick :)

> And that currently to push the server you have to maintain a list of
> authorized certificates, under authorized.crt. But apart from that i am
> not sure how granular you can be.

Certificates are complex beasts; as I understand you can do all sorts
of elaborate things with them, like use a certificate to create a child
certificate, having the power to later revoke the child certificate (I
can imagine this being interesting for a participating organization to
grant permission and later revoke permission for it's members, without
requiring access to the artifact server host to update it).

Again, certs fall into that networking territory where I have developed
a natural aversion to any knowledge retention; maybe the above
functionalities requires a certificate authority service to work.

I am unsure if any of that kind of functionality is possible with the
way BuildStream itself uses the certificates to authenticate at the
endpoints.

Care to ring in here Jürg ?

Fancy certificate features aside, granularity can still be down to each
user; although in most setups as far as I can see, it is less
interesting to have individual developers pushing artifacts, and much
more interesting to have build servers build things, while developers
benefit from only having to download, and only having to build locally
the things which they want to test and experiment with.

The certs only authorize upload, and download never requires
authentication of any sort from the perspective of the http service
itself; I believe this is why we mention in the docs that a sysadmin
versed in the dark arts of networking, could set it up behind a reverse
proxy in such a way as to ensure only authenticated users can see the
`http` service at all.

Cheers,
    -Tristan