Re: gnome-continuous: speeding up "builddisks"

[Owen Taylor <otaylor redhat com>]

On Wed, 2016-08-31 at 16:17 +0200, Bastien Nocera wrote:
> >  

[...]

> On Wed, 2016-08-31 at 09:37 -0400, Owen Taylor wrote:
> >  It would be possible to enhance ostree to
> >     write ownership as xattrs, and then convert when copying into
> > the
> >     VM, or to even have ostree directly stream a tarfile to be
> > unpacked
> >     in the VM, but either would be significant work.
> >
> >     A secondary issue is that this only applies to the initial
> >     creation of the VM, and doesn't allow for optimizing updates.
>
> Do we need to optimise updates if the initial image creation is fast
> enough, or would the created image not be updatable if we went that
> way?

In the build.gnome.org, switching to always create new VMs with the
copy-in method would take the builddisks step from ~20m to ~8m. This is
a big improvement already, but updating existing images could
potentially be as fast as ~2m without guestfs.

The images created by the copy-in method would be just as updatable as
before. They also would be smaller - if I'm not mistaken, we accumulate
history in our download images, then just manually truncate it at some
point and accept the multi-hour builddisks from scratch.

One possibility is to do a bit of a hybrid - instead of having a
separate "bootstrap" image, we could build the images using the copy-in
method, then boot the images themselves to a special systemd target
to self-update them.

> >  2) Do the ostree operations inside a VM with the image mounted.
> >
> >     The issue here is that we need an operating system with ostree
> >     in it to do the deployment, causing a bootstrapping issue - we
> >     need an operating system to make an operating system. Using
> >     a fixed non-GNOME-continuous operating system image for this
> >     would be possible, though with some potential for ostree
> > version
> >     skew.
> >
> >     But the approach that I prototyped out is creating a
> >     "bootstrap" image by checking out the destination ostree,
> >     copying it into the VM at / instead of as a proper ostree
> >     deployment, then doing the necessary fixups to make that boot.
> >     (E.g., adding a fstab that mounts a tmpfs on /var.) This is a 
> >     bit hacky, but not too bad.
> >
> >     (Another approach would be to do this from the initramfs but
> >     getting ostree into the initramfs would require a gigantic
> >     pile of libraries.)
>
> ostree + glib + libsoup (and glib-networking, libproxy, I guess).
> That's probably not any worse than when we put Xorg in the initramfs
> to do boot splashes :)

No doubt :-) ldd ostree shows 37 libraries libarchive, gpgme, libxml2,
etc, which would end up about doubling the number of libraries in the
initrd.

[...]

> > Does this seem like something that is worth pursuing? Do people
> > have
> > any other ideas for improvements?
>
> Is the problem an I/O throughput problem, or a CPU throughput
> problem? If it is the former, maybe we could still use guestfs but
> with the image in RAM. If it's a CPU problem, we could possibly use a
> VM with a physical partition instead of an image, so that direct I/O
> kicks in (libguestfs uses a VM internally, but I'm not sure it can do
> direct I/O, especially on top of an image).

The problem is communication overhead - in particular, ostree does
many, many, many stat() calls and round-tripping those through the fuse
interface slows things to a crawl. The "copy-in" case still uses
libguestfs to do the final writing of the image, but by streaming a tar
file rather than through the FUSE interface.

- Owen