On Fri, 2013-08-09 at 17:24 +0200, Sjoerd Simons wrote:
Right, our current thinking with an index (which could be a seperate
file or part of the commit metadata a{sv})), was that it would give you
the list of objects needed for a specific commit and their sizing
information. Once you have that you can work how much data you'll need
to retrieve and store.
Either should be fairly easy to implement; the cost would be duplicating
that information across each commit. Maybe it's a command line option
to say whether or not commits should have it.
See also the entry in TODO:
* Rethink commit objects
In particular, allowing commit objects to have non-checksummed data
would be very helpful for this case, because then we could add (or
remove) metadata like sizing data after the fact.
The way I was thinking about breaking the commit format is having a
magic header in the front - if we see that, it's a "version 2" commit.
Something like this, where the first metadata is not checksummed
commit = magic metadata commit-content
magic = just make up some magic bits, like 0xFF OSTREE
metadata = "a(ss)a(say)"
commit-content = metadata "ay" "ay"
Where the latter ay/ay pair are the required dirmeta and tree objects.
Then pretty much everything else that's in the current commit object
like the parent goes in the checksummed metadata as
ostree.parent, timestamp is ostree.timestamp etc.
The reason I chose a(ss)a(say) for metadata is to allow both string and
binary values, without using a{sv} due to:
https://bugzilla.gnome.org/show_bug.cgi?id=673012
The sizing information could go in the non-checksummed binary a(say)
part as ostree.objectsizes, and the "ay" there is a(ayt). Or maybe for
non-checksummed content we can just use a{sv} for metadata.
Sounds cool, that would definitely be good to have. But support for the fallback case where static deltas aren't available would still be good to have.
A basic form of static delta would just fall back to fetching each individual object; then we could reuse it too as a carrier for metadata. I may prioritize landing just that basic framework; once it's there it's possible to improve the format and compiler over time.
Exactly, having a tool that can say: Hey, There is a new update available, you'll have to download 45MB of data to get it. Oh btw, It will take 70MB on disk, so you have enough free space available. Shall i download?
Ah, you need uncompressed size too of course; I've added a note about that to the static deltas plan. I can't easily push my branch because this stupid hotel is blocking outbound ssh, but I've attached my current README-deltas.md file in case it's useful. So...a lot of options. But if you don't want to dive into reworking commit objects or static deltas, I'd certainly be OK with carrying a proof-of-concept patch that added a separate .sizes file or something for now if that's easier. But for more production upgrades, I think the static deltas plan is a lot more of a comprehensive fix.
Attachment:
README-deltas.md
Description: application/genesis-rom