ostree/builder improvements - moving towards devops/self-deployment

[Colin Walters <walters verbum org>]

Hi,

So the gnome-ostree builder has been cranking along now, and helping
ensure things build (in some configuration).  Thank you guys very
much for allowing my code to run inside the infrastructure, it's
been quite useful to me in both a developer capacity and a release-team
member.

However, while "does it build" has traditionally been the minimum bar
for a GNOME release, I'd like to raise that somewhat.  In order to do
so, we need automated testing.

In order to do automated testing, I need the ability to allocate and
deallocate virtual machines in an automated fashion.  You could call
this an "internal cloud".

Being able to allocate/deallocate VMs would also allow us to move
more towards a "devops" model, where the people doing development and
deployment are (as much as possible) the same.

One specific characteristic that distinguishes this "internal cloud"
from the way other GNOME machines are deployed is that they should not
have access to the backplane, or in general be treated as "privileged"
in any way.  So the developers have root on the machines, but root
doesn't allow you to affect any *other* machines.

A specific technology that could be used to implement this at first
would be granting specific ssh keys access to a libvirt shell on a bare
metal host (these ssh keys would be both people and services, e.g.
ostree.gnome.org would have its own ssh keypair).

A far more grand solution would be OpenStack, but to be honest while
a lot of companies are piling on to it, it's really just a pile of
Python scripts still.

In the meantime, I plan to prototype out a bit of OSTree-based automated
testing using AWS (mostly EC2), so I'm not blocking on this happening
in GNOME infrastructure first.

But I'd love to be able to join the two when we have initial versions of
both, so I'm hoping to start a discussion here first.  Thanks!