install-module / ftp.gnome.org / master.gnome.org

[Olav Vitters <olav vitters nl>]

Currently working on redesigning a new master.gnome.org[1][4].

As part of this, I'd like some quick thoughts on:
 - diff/delta files between versions
 - gzip/bz2/lzma compression; lzma only?

Note:
I know most in Europe/USA will have a fast internet connection. I'd like
to see if there is a real need for downloading tarballs[2]. I assume
most will get everything from their distribution, but I still want to
optimize the bandwidth if there is some need.


 ==  A. diff/delta files between versions ==

I've disabled the 'diff' files between versions as it:
 * Doesn't handle binary diffs
 * Didn't (although it could) handle translation changes

I saw a program called xdelta3, which allows you to do binary diffs. Is
something like binary diffs wanted? And if so, what program should be
used?

Things xdelta3 can do:
  18137401  gtk+-2.18.6.tar.bz2
  18304767  gtk+-2.18.7.tar.bz2
 => 167366 increase in size

Trying out the different xdelta3 options, I got:
    511056  gtk.6.7-2.xd3 (sane)
    411600  gtk.6.7-3.xd3 (sane)
    341020  gtk.6.7-4.xd3 (bzip2 type compression)
    340460  gtk.6.7-6.xd3 (increasing options to highest values possible)
    336837  gtk.6.7-7.xd3 (increasing options to highest values possible)

So in best case scenario, it uses twice the difference of the bz2
compressed tarball size. ~350KB will be a much quicker download than
~18MB.


 ==  B. What kind of compression should be used for tarballs? ==

Currently we use 'gz' and 'bzip2'. Last time it was asked (lots of years
ago), there was a mention that not everyone has 'bzip2'.

I'd like to:
 - 'Kill .gz'[3]
 - Introduce some kind of lzma type compression
   Pretty every distribution released over the last few years has
   support for this. Further, http://www.7-zip.org/ has support for this
   on Windows.
   Conversion might take a while on master.gnome.org, but seems
   manageable. Not sure if I can easily combine lzma and xdelta3.
 - Ideally only have one compression type on ftp.gnome.org
   We have enough disk space, but I still don't want to waste it if
   there is no need. Additionally, minimizing disk usage speeds up
   backups.


Any thoughts on this?




[1] Bugreport about the new master.gnome.org:
  https://bugzilla.gnome.org/show_bug.cgi?id=613033

[2] Statistics for ftp.gnome.org are available at:
  http://www.acc.umu.se/technical/statistics/ftp/gnome.html.en
Though I actually see window.gnome.org as the main user in there (maybe
we're downloading from ftp.gnome.org instead of using /ftp). Further, it
statistics seem to take everything with 'gnome' in the name (includes
some debian iso files).

[3] Compression comparison:
23473694  gtk+-2.12.7.tar.gz   (gzip -9)
16111869  gtk+-2.12.7.tar.bz2  (bzip2 -9)
11658942  gtk+-2.12.7.tar.lzma (lzma -6, used the default)
(ancient gtk+ version, assume new versions have similar compression
 ratios)

[4] Other than a VM, not much is ready. I still need to sort out a lot
of details.

-- 
Regards,
Olav