Re: Repo scalability issues and solutions

["Colin Walters" <walters verbum org>]

Sorry about the delay here.

On Tue, Aug 25, 2020, at 8:38 AM, Alexander Larsson via ostree-list wrote:

> Does the above make sense to everyone? Do we have any other ideas how
> we could do better? Do we have some important feature we would like in
> the new format?
>
> Note: While some of these changes apply to ostree, some apply just to
> flatpak. However, I want to synchronize the changes so that we only
> have to do a single format-change.

Right, implicitly here you mean "ostree for operating systems" and I think that's a core tension here: what 
one wants for operating systems w/ostree is quite different in scale and mechanics from applications.

It's not clear to me actually that *all* of the logic around applications needs to live in ostree itself - 
for example, flatpak can define its own higher level metadata and tooling.  Now there are clearly tradeoffs 
here - for example, it might mean that local mirroring requires using some flatpak tooling on the server side 
too (but you're already building that right?).

Some of your proposed changes make total sense, but I think given the possibility to "greenfield" things here 
more it's probably worth thinking about higher level and more long-term changes.  We don't want to accumulate 
too many formats because the pull code is already incredibly complicated and under-documented.  Data formats 
are hard.

One random idea I had last time this was discussed was using ostree for metadata too; basically the metadata 
for a repo would be in a special `ostree-metadata` ref and it would have a filesystem layout like 
`/deltas/{00,01,...}/list` where the `list` object would be a gvariant list.  A lot like exploding our a{sv}s 
into an ostree filesystem.  I don't recall where we ended up with that.  It'd definitely involve more round 
trips but I have trouble thinking of any solution that doesn't.