Re: Moving existing ostree users to a new branch - take 2

[Alexander Larsson <alexl redhat com>]

On Fri, 2017-05-12 at 13:49 -0400, Colin Walters wrote:
> On Fri, May 12, 2017, at 10:17 AM, Robert McQueen wrote:
> > As a side thought, we're a little worried about the summary file
> > getting
> > much larger over time as well. When you get up to 100s or 1000s of
> > Flatpaks we're talking a _lot_ of refs. A cool 10MB a day just to
> > refresh your app list (something you might very reasonably want to
> > do
> > daily, so you can implement some policy based off of that) isn't
> > too
> > great if that adds up to 300MB over a month - 60% of your data
> > plan.
>
> Yeah...the summary is a very, very basic design.  I didn't put a lot
> of
> thought into it.  It kind of more came out of the requirements for
> metalink <https://bugzilla.gnome.org/show_bug.cgi?id=729388>
> which I never ended up actually using.
>
> One thing I've thought about is basically making things recursive;
> imagine we have a special ref:
>
> ostree-metadata
> whose contents are a filesystem tree that in turn contain whatever
> data we want.  For example there might be a deltas/ subdirectory,
> a refs/ subdirectory, etc.

So, I worry a bit too about summary file scalability, and whenever I
think about making it more complex I end up with this idea too, so I
think that is the correct way to do it long term. However, i'd like to
point out that summary-as-a-ref implies gpg signing the summary :)

However, I've also thought a bit about short-term fixes, as I add a
bunch of extra metadata to flatpak summaries that make them even
larger. For instance, the flathub summary is currently 286kb for 40
apps (and some themes) with related branches (like debuginfo and
locales) times 4 different arches.

Fortunately, it turns out that the summary files compress really well,
since they have a lot of repeating text. Just a simple gzip of the
flathub summary makes it go from 286k to 99k. 

We could easily get this win in transfer size automatically, in a
backwards compatatible way by using gzip transfer encoding for the
summary file. I think we just need to set the right HTTP headers when
we request the summary file and then make sure the server supports
compression.

I filed this issue about this: 
 https://github.com/ostreedev/ostree/issues/802

I would love if someone looked into this.

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                            Red Hat, Inc 
       alexl redhat com            alexander larsson gmail com 
He's a sword-wielding small-town librarian gone bad. She's a manipulative 
nymphomaniac soap star from beyond the grave. They fight crime!