Idea: daily packs

[Owen Taylor <otaylor redhat com>]

I took advantage of having a repository that contained only a single
completely gnome-ostree build (-devel and -runtime) to compare two
things:

 * Downloading a tar.gz of the repository - took about 7 minutes to
   download 460 megabytes.

 * Pulling the majority of the repository using:

    ostree remote add gnome http://<myserver>/repo trees/gnomeos-3.6-i686-{runtime,devel}
    ostree-pull -related gnome

   Took 45-50 minutes to transfer ~1.2GB of data.

This is with pretty good connectivity to the server (wifi and cable modem
but 1MByte/sec and 30msec ping time.)

I was talking to Colin in the office yesterday and he said that he had
pretty much given up on packfiles because over time access to pack files
became very sparse - you would end up downloading a 25M pack file to get
one header file that was unchanged since last year.

But what if we basically just in a cron job created *one* giant pack file
that had all the objects referenced by the gnomeos-3.6-i686-{runtime,devel},
(leaving loose files around), then created another such file the next day
and deleted the last one?

That would make the initial checkout vastly faster and more efficient, without
having to give much thought to some sophisticated algorithm for clustering.

The one part of this idea that I haven't figured out is how you would keep
the next day's 'ostree --pull' from downloading the next huge tarball instead
of a few thousand smaller files. Without a smart server it seems hard to
optimize - maybe you make the gigantic pack explicitly a 'bootstrap pack'
and there would be index information so the client could match up what was
specified for an initial pull with the right bootstrap index. A bit ugly.
Or maybe you do it explicitly:

 ostree add gnome http://ostree.gnome.org/repo --bootstrap=gnomeos-3.6-i686-runtime-devel

- Owen

Note: The initial pull time isn't awful and even on slower network is probably only
a couple of hours - so not a first priority. But factor-of-6 is a pretty big speedup
to try to get at some point.

Note: I subsequently tried using 'ostree pack' to create a gigantic single file pack
and it did give the expected speedup to something similar to the .tar.gz download.