OSTree updates/pulls from LAN and pendrive

[Krzesimir Nowak <krzesimir kinvolk io>]

Hi,

I'm tasked with writing/updating a software that updates the OSTree based system from a bit different sources than the usual URL from the "remote" configuration. The possible sources are a computer on the LAN or just a pendrive.

Updating from LAN involves writing a server that publishes itself and then just serves the files from the OSTree repository. It would have a lot in common with ostree's trivial httpd, but with additional complication that the server's OSTree repo is usually of a "bare" type, not "archive-z2".

Updating from pendrive means some detection that pendrive with an update was plugged in and then act accordingly.

When investigating, I got a couple points I'd like to address in ostree or to get the advice about it.

1. Would an API (likely a single function) for making it easier to write a server serving what ostree pull would expect be welcome in libostree? It would likely take an OstreeRepo and a path relative to OstreeRepo's repodir and return a GInputStream via an output parameter (likely a GFileInputStream for content objects in archive-z2 repos or for non-content objects or deltas or summary files and a gzipping GConverterInputStream for content objects in bare repos). It would also return an error if path was outside of the repo directory (like ../../etc/passwd) or it was a directory or it was not not found. Something like:

gboolean
ostree_repo_fetch_path (OstreeRepo *repo, const char *path, GInputStream **in, GError **error);

2. Could we add another option to ostree_pull_with_options that overrides the URL used for fetching objects? Currently the remote_name_or_baseurl can be either a remote name or a file:// URL. When updating from LAN, we would like to pretend that the objects came from some specific remote despite using a different URL. That would basically mean using the override instead of ostree_repo_remote_get_url when remote_name_or_baseurl is not a file.

3. We want to provide updates from a local filesystem. Sounds at first as a solved problem - ostree pull-local (or the call to the ostree_repo_pull_with_options function with a "file://" URL). But there are some things to consider:

- whether the update is easy to produce - means it would be best if we could just use some ostree library functions without messing in the repo directory ourselves or inventing too much of our own metadata.

- whether the update is easy to consume - look above, but on the client side. Ideally, the call to ostree_repo_pull_with_options should suffice.

- whether the update works for any previous version - it would be nice to have a single update that works on several computers with different older versions of the system.

- whether it really fits the limitations of a typical pendrive - it would be best for the update to be small as possible and contained in only a few directories/files.

We had some ideas for it, like:

- a completely self-contained inline static delta from empty to latest version (basically the same thing as xdg-app bundle)

  - easy to produce - totally, ostree_repo_static_delta_generate with "from" being NULL, the "min-fallback-size" option being 0 and the "inline" option being TRUE.

  - easy to consume - eh, not really sure about it, it is just a call to the ostree_repo_static_delta_offline_execute, but I guess xdg-app does not allow updating installed apps/remotes from bundles for a reason. It's likely because it is hard to tell whether the bundle is really an update, I'm not sure if there is some information telling us about it that we can trust.

  - works for any version - dunno, maybe? The bundle has all the files, so how old the previous version is likely irrelevant

  - fits the limits - it is just a single file so it's good, but it can be huge, the whole OS is put into it.

- a "truncated" or "crippled" OSTree repo, which is basically a repository with a probably empty "objects" directory and with no-fallback-objects inline static delta files in the "deltas" directory.

  - easy to produce - nay, we need the objects to be able to generate the delta, so my idea was to have a notion of a source and target repositories in ostree_repo_static_delta_generate, so it would generate the delta files based on the source repo, but put them in the target repo. The target repo would be the one on the pendrive.

  - easy to consume - likely, the aim is to have an OSTree repo there, so it would be pullable

  - works for any version - it wouldn't work for versions that are older than the "oldest" delta file.

  - fits the limits - likely, we would probably fill whole space to get as much delta files as possible, so we could update most of the old systems. And it shouldn't be too many files, either.

- a shallow OSTree archive-z2 repo that has only objects reachable from the only commit it has

  - easy to produce - hm, maybe? Create a new archive-z2 repo and write all the objects reachable from the toplevel commit of the source repo?

  - easy to consume - sure, it's pullable

  - works for any version - I guess it has the same problem as xdg-app bundle here

  - fits the limits - same as xdg-bundle - likely huge (but still maybe smaller due to gzipping) and there are a lot of files on top of that

- something completely custom, rather a last resort option. Likely a list of inline static delta files together with some key file describing them

  - easy to produce - would need a call to ostree_repo_static_delta_generate several times, so not so?

  - easy to consume - we would need to reinvent parts of the ostree_repo_pull I guess

  - works for any version - has the same problem as the "truncated" repo

  - fits the limits - same as the "truncated" repo I think.

Thanks,

Krzesimir Nowak

--

Kinvolk GmbH | Graunstr. 43, 13355 Berlin
Geschäftsführer/Directors: Alban Crequy, Chris Kühl, Iago López Galeiras
Registergericht/Court of registration: Amtsgericht Charlottenburg
Registernummer/Registration number: HRB 171414 B
Ust-ID-Nummer/VAT ID number: DE302207000