static deltas progress

[Colin Walters <walters verbum org>]

A status update: I've been working on "static deltas" this week{,end}, made a lot of progress, and it's been a lot of fun - an interesting problem space.

Here's some example output from a "large" gnome-continuous update.  This update

is a good sample for a "major version" update.  We have almost 7000 modified files,

which in the future would be a good target for rsync-style rolling checksums.

Anyways one thing to note here - the first static delta part packs together

1892 objects - that would normally be 1892 HTTP requests, down to one

HTTP request for a ~8MB file.  I suspect this is going to be a *massive*

speed win.

Another thing you should note here is support for "fallback" fetches.

This is where an object (say a debuginfo file) is so large that it

doesn't make sense to put it in a delta, we just fall back to individual HTTP

fetch for it.

This is an important optimization as it helps avoid excessive space

usage on the server.

$ rm repo/deltas -rf; ostree --repo=repo static-delta gnome-continuous/buildmaster/x86_64-runtime^

Generating static delta:

  From: 382f1a813d9fb3c37be50899884302fa3f0dba118a6b8b8e60b95c05ab5dbb66

  To: fc60582e57c4e191542f4f75803b79022e86a81c26224fda2868d6c956b23de3

modified: 6923 removed: 469 added: 1524

new reachable: metadata=1341 content=4371

fallback for c0453ccb2c1bab118bf4d566b60c7addd2cf69717252da2592b6234c42d5b6de.file

part 0 n:1892 compressed:7846837 uncompressed:28864233

part 1 n:574 compressed:13389181 uncompressed:31966816

part 2 n:323 compressed:14332005 uncompressed:30342279

part 3 n:622 compressed:6695393 uncompressed:31959437

part 4 n:572 compressed:7581129 uncompressed:31912274

part 5 n:448 compressed:10559161 uncompressed:28460875

part 6 n:528 compressed:6388117 uncompressed:31960123

part 7 n:643 compressed:6162769 uncompressed:31812155

part 8 n:109 compressed:978513 uncompressed:5680847

delta uncompressed=252959039 compressed=73933105 loose=97473683

More information in:

https://bugzilla.gnome.org/show_bug.cgi?id=721799