Re: Repo scalability issues and solutions

[Jonathan Dieter <jdieter gmail com>]

Apologies, this was help up in moderation, so I've joined the list

On Tue, 2020-08-25 at 14:38 +0200, Alexander Larsson via ostree-list
wrote:
<snip>
> Add deltas for incremental updates of summaries
> ===============================================
>
> This only helps with the network transfer, but OTOH, it does so
> extremely well. A simple bsdiff of the entire summary is probably the
> easiest way to do this.
>
> However, for this to work we need to be able to identify the summary
> version you have (and which is on the server), and to store multiple
> versions of if. The easiest way to do so is to store them by sha256,
> just like objects.  Then you have some top-level summary index file
> that list the sha256 of the current summary file. In fact, it
> probably
> will have a list of summary files (for the per-arch summaries), which
> is good because this single file will allow atomically updating all
> the sub-summary files in one change.
>
>
> Does the above make sense to everyone? Do we have any other ideas how
> we could do better? Do we have some important feature we would like
> in
> the new format?
>
> Note: While some of these changes apply to ostree, some apply just to
> flatpak. However, I want to synchronize the changes so that we only
> have to do a single format-change.

What about using zchunk?  It basically allows you to download just the
differences between a local older version of a file and a remote newer
version.  It's what we're using in Fedora for metadata for the last
couple of releases.

(Disclaimer: I wrote it, so I'm obviously biased)

Jonathan