Re: mirroring OSTree repositories using Pulp

[Dan Nicholson <dbn endlessos org>]

On Mon, Jul 19, 2021 at 9:38 AM Dennis Kliban via ostree-list
<ostree-list gnome org> wrote:
> I am working on adding OSTree repository management to Pulp (https://pulpproject.org). Pulp users mirror 
> content from the internet or they upload content that they create.
>
> Are there public raw OSTree repositories that I could mirror as a proof of concept?

I don't think you'll really find bare OSTree repos publicly since
they're of very little use on a server. They really only make sense on
a deployment since they're dependent on filesystem attributes that
won't typically transfer over HTTP.

> Are there public archive OSTree repositories?

Sure. https://dl.flathub.org/repo, but that's a monster. We have
several at Endless. https://ostree.endlessm.com/ostree/eos-sdk is
pretty reasonably sized. RedHat should have several from the CoreOS
team.

> It seems to me that users would benefit the most from being able to mirror raw repositories that their 
> clients could consume. Is there a benefit to mirroring archive repositories?

If the intention is to mirror something that will be added as an
ostree remote, then you definitely want archive repos. That is almost
certainly the way any server is setup and IIRC you can't even pull
from bare repos. If you're trying to do something like a backup of a
live system that uses ostree, then a bare/bare-user/bare-user-only
repo would make sense.

The reason archive repos are used on servers is because all the file
metadata is stored in the ostree object itself instead of via
filesystem attributes. I.e., it doesn't matter who the owner of the
objects are or what mode they are on the system. The metadata gets
transferred along with the contents from a the object in an archive
repo. Plus the objects are pre-compressed in an archive repo, cutting
down on transfer costs.

Either way, it's pretty straightforward to play around with a repo and
see what's in it.

ostree --repo=repo init --mode=archive
mkdir files
touch files/a files/b
ostree --repo=repo commit --branch test --gpg-sign <somegpgkeyid> files
ostree --repo=repo static-delta generate --empty test
ostree --repo=repo summary --update --gpg-sign <somegpgkeyid>

Then poke around in the repo directory. That should have most
everything you'd ever encounter. One thing to note is that recently
flatpak has started using some repository extensions, so there are a
couple things in the flathub repo that you might not find in a pure
ostree repo.

--
Dan