Re: Adding Avahi and USB drive support to libostree

[Colin Walters <walters verbum org>]

On Wed, Mar 15, 2017, at 12:55 PM, Philip Withnall wrote:
> Hi list,
>
> I’ve been thinking about the best way to upstream the support for
> grabbing OSTree updates from the local network and from USB drives
> which Endless OS has ...

That's pretty cool!

> I’ve been thinking today about whether we should instead be treating
> them like mirrors of the canonical upstream remote. I *think* we could
> potentially already implement this by having some external process
> generate a local mirrorlist file for each remote, 

Currently, the mirrorlist code *does* fetch one-by-one in
order, but I'm not sure I want to make that ABI.  I guess though we could
easily have some logic like "try file:// URIs first" even if in the
future we add parallelization or "fastestmirror" type logic.

> and update it just
> before polling for updates. However, this is a bit ugly, prone to
> synchronisation problems between the update processes, 

Because it involves mutating the remote on disk?  One approach
we could take would be add support for e.g. `/run/ostree/remotes.d/foo.d`
where eos-updater would have the logic for "okay, a USB device was plugged
in", then:

cat > /run/ostree/remotes.d/endlessos.conf.d/usb.conf << EOF
[remote "endless"]
prependurls=file:///mnt/usb/repo;
EOF

Where "prependurls" is prepended to the main `url=` line from the remote.

> and would mean
> having a daemon in ostree.git (which I know people are against).

To be clear I'm not personally against it strictly - we already have a somewhat
messy situation with the "ostree-host" portions living in the same repo, which
means downstreams already need to subpackage, see e.g.:
https://bugzilla.redhat.com/show_bug.cgi?id=1331369

But keep in mind we have an `ostreedev/` github org - we can easily have
`ostreedev/ostreed` in the future.

> Instead, I’m wondering whether libostree could gain explicit support
> for finding mirrors using Avahi and monitoring mounts, similarly to how
> it can currently resolve metalink and mirrorlist files.

> So for Avahi support, I’d define a DNS-SD record format, and some
> repository configuration which would enable it.

This part is *clearly* a daemon, right?  The enablement of this feature
seems like it'd be a systemd unit, not a flag in the repo.

(Or well, I guess it depends whether we're talking about the client
 or server logic)

> The format would need
> to scale to multiple repositories (for example, if a machine used
> OSTree for the OS, and flatpak for apps). Any machines on the local
> network which advertise repositories matching the local repository’s
> canonical remote URI, 

Maybe instead we should have a UUID field in repositories in the repo/config?
We could easily change `ostree init` to generate one.
(Although it'd be prone to being copied if people used `rsync`)

> would be added as HTTP mirrors. This would
> require some server support (at the minimum, an Avahi .service file and
> a running OSTree simple HTTP server).

This is the part that is a daemon.

> For USB drive support, I’d define some repository configuration which
> enables it and specifies which well-known directory to look for in each
> mount (for example, check if $mount/.updates-in-here is an OSTree
> repository for each $mount point when we do a pull; ‘.updates-in-here’
> would be specified in the repository configuration), and adds each of
> them as a mirror.

Circling back to the top then, my instinct in the short term is to change
libostree to make it less awkward for daemons to inject configuration
into the remote, and add some baked in logic like "try file:// URIs first".

> In both cases, requiring GPG verification of the remote would be
> essential in any normal situation. All the mirrored content would have
> to be signed with the same GPG key as the canonical upstream.

Right, we'd get that for free with the "drop-in remote" model, since
we wouldn't be overriding the gpgkey.