Re: rsyncing repositories

[Phil Wise <phil advancedtelematic com>]

> This is not *quite* safe in the same sense that a regular rsync is,
> but its *almost* safe and could be made so with a bit more scripting.

We've been working on a tool called 'garage-push'[1] for incrementally
pushing OSTree repositories to a remote server.

It performs uploads in parallel, but maintains the invariant that all
the dependencies of an object are present on the server before the
object itself is uploaded.

This invariant means it can perform incremental uploads and avoid
querying the server for parts of the tree that are unchanged, since once
it knows that one node of the filesystem tree is present it can assume
that all the child nodes are also present on the server.

In our case the server is custom and uses a different authentication
scheme to AWS S3, but it wouldn't be too hard to add support for AWS
Authorization strings as an alternative to the 'TreehubServer' we have
in there today.

Best Regards,

Phil

[1] https://github.com/advancedtelematic/sota-tools

On 17.11.2016 11:15, Alexander Larsson wrote:
> On ons, 2016-11-02 at 12:54 -0600, Dan Nicholson wrote:
> > Back from vacation now, so I'm starting to think about this again.
>
> So, this is not exactly related to rsync, but I've recently realized
> that amazon S3 is an excellent host for ostree repositories. It doesn't
> support rsync, but there is a tool (aws s3 sync) that is very similar.
> I sync my repo with:
>
>   aws s3 sync /my/local/repo s3://my-bucket/the-repo)
>
> This is not *quite* safe in the same sense that a regular rsync is, but
> its *almost* safe and could be made so with a bit more scripting.
>
> The interesting thing though is that S3 *does* support uploads, so we
> *could* make an "ostree push" operation that works for it. Is anyone
> interested in looking at this?

-- 
Phil Wise, ATS Advanced Telematic Systems GmbH
Kantstrasse 162, 10623 Berlin
Managing Directors: Dirk Pöschl, Armin G. Schmidt
Register Court: HRB 151501 B, Amtsgericht Charlottenburg