Re: Moving existing ostree users to a new branch - take 2

[Colin Walters <walters verbum org>]

On Fri, May 12, 2017, at 10:17 AM, Robert McQueen wrote:
> This live TLS requirement is also not great for Endless where we're
> looking at caching ostrees (with the refs that point to the latest
> OS/flatpak/etc) and either sharing those again over LAN, USB, etc.

No one is trying to impose a "requirement" keep in mind.  I'm more
saying that if one *is* connected, the guarantees provided by TLS
(particularly pinned) are quite good and the tooling around TLS
is very extensive and well tested.

For the fully disconnected case - I'd like to brainstorm things
like the "commit ref binding" approach first than driving more
logic into the summary.

> As a side thought, we're a little worried about the summary file getting
> much larger over time as well. When you get up to 100s or 1000s of
> Flatpaks we're talking a _lot_ of refs. A cool 10MB a day just to
> refresh your app list (something you might very reasonably want to do
> daily, so you can implement some policy based off of that) isn't too
> great if that adds up to 300MB over a month - 60% of your data plan.

Yeah...the summary is a very, very basic design.  I didn't put a lot of
thought into it.  It kind of more came out of the requirements for
metalink <https://bugzilla.gnome.org/show_bug.cgi?id=729388>
which I never ended up actually using.

One thing I've thought about is basically making things recursive;
imagine we have a special ref:

ostree-metadata
whose contents are a filesystem tree that in turn contain whatever
data we want.  For example there might be a deltas/ subdirectory,
a refs/ subdirectory, etc.

It'd make the pull code even more complicated, but would have a lot
of nice properties in that one could simply sign ostree-metadata's commit,
and have dedup of shared data (e.g. if the deltas don't change, clients
don't need to redownload that object)

(On the client side we'd actually need to store this metadata by prefixing
 with the remote name or something, otherwise they'd collide locally)

I think https://ipfs.io/ does something like this with metadata.

Or, perhaps the simplest thing is to add support for inline signatures
and compression as a first step.

But if I was ordering things, I'd at least parallelize work on "commit ref binding"
with "simple summary tweaks", and again possibly prefer the first if
we can agree that it lets us back away from signed summaries.