Re: rsyncing repositories

[Phil Wise <phil advancedtelematic com>]

On 21.11.2016 11:23, Alexander Larsson wrote:
> On tor, 2016-11-17 at 11:35 +0100, Phil Wise wrote:
> > > This is not *quite* safe in the same sense that a regular rsync is,
> > > but its *almost* safe and could be made so with a bit more
> > > scripting.
> >
> > We've been working on a tool called 'garage-push'[1] for
> > incrementally
> > pushing OSTree repositories to a remote server.
> >
> > It performs uploads in parallel, but maintains the invariant that all
> > the dependencies of an object are present on the server before the
> > object itself is uploaded.
> >
> > This invariant means it can perform incremental uploads and avoid
> > querying the server for parts of the tree that are unchanged, since
> > once
> > it knows that one node of the filesystem tree is present it can
> > assume
> > that all the child nodes are also present on the server.
>
> That is pretty cool. 
>
> How does this handle the summary file? Do you run "ostree summary" on
> the server, or do you have a full local copy of the repo that you run
> it on and then upload?
>
> Given a server with a custom API like e.g. S3 which has a way to
> enumerate files/object you could reimplement "ostree summary" for the
> remote, which would give you a full "ostree push-and-update" operation.

At the moment it doesn't handle summary files at all :(  Our plan for
static deltas is to generate them on the (smart) server and treat them
as an optimization between the ostree client and server.  I haven't
looked into this at all yet, though.

This would be harder in the 'dumb server' case, although I wonder if
fetching entire files would be good enough for many people, especially
if the entire server infrastructure was just 'use S3' (or Azure Blob
store...)

Regards,

Phil
-- 
Phil Wise, ATS Advanced Telematic Systems GmbH
Kantstrasse 162, 10623 Berlin
Managing Directors: Dirk Pöschl, Armin G. Schmidt
Register Court: HRB 151501 B, Amtsgericht Charlottenburg