BuildStream CAS server testing

[Adam Jones <adam jones codethink co uk>]

Hi all,

We have been testing the new CAS server that Jürg has been
implementing, over at the freedesktop-sdk project.

I'd just like to publish some findings on the performance and how this
has helped our issues.

Since using the original ostree backed cache, we were experiencing
performance issues and timeout issues under heavy loads:

 * https://gitlab.com/freedesktop-sdk/freedesktop-sdk/issues/35
 * https://gitlab.com/freedesktop-sdk/freedesktop-sdk/issues/33

These issues were generally seen when we had 5 or more pipelines
triggered, due multiple MRs being submitted to the project, we also had
to upgrade the instance which hosted the cache to a 4 x x86_64 + 4GB of
RAM instance, just to deal with the traffic.

So to test the new CAS, we have set an instance up at Digital Ocean
with the spec: 1 x x86_64 + 1GB of RAM, with the expectation to
increase this as we go.

The initial testing shows that the issues we faced with timeout issues
have been mitigated and such a small server handled 5 pipelines, with 0
failures.

We have tested up to 20 pipelines, which is almost 100 builds, and we
began to see issues with timeouts on Gitlab and our Docker instance,
which manages the runners, but the CAS itself did not timeout!

So we are eagerly anticipating the release of the new server so we can
ramp up our CI and maybe reduce the price plan of our cache server.

Many thanks,

Adam Jones