Updating the OS data that ostree doesn't manage

[Daniel Drake <drake endlessm com>]

Hi,

For building OS products, ostree provides a great solution for
managing the OS filesystem and letting us update it later.

However, in order to actually make the OS usable, the deployment of
the ostree onto a filesystem needs to be combined with a few other
steps, which also involve writing stuff to disk: creating partitions
and filesystems, installing the bootloader, etc. This is typically
done by some kind of image builder or installation app, which will
handle those aspects immediately before or after deploying the ostree.
I'll refer to this as "installation time" here.

It's easy to handle these actions at installation time,
straightforward enough that you can do the whole thing fairly
comfortably in shell script. But what happens if you want to change
any of these aspects later? For example, tweak the swap partition,
adjust the filesystem flags, update the bootloader, etc. While ostree
lets us update the main filesystem content on existing installations
with ease, other solutions are needed for post-install updates to
these other bits.

I checked a fresh install of Silverblue and actually it seems to be
impressively "pure" in that it doesn't do too much in addition to
ostree deploy other than the completely essential stuff, but there are
a handful of changes made like writing /etc/default/grub, locale,
timezone.
And I note that Silverblue already faces at least one pain point
originating from this area:
https://fedoraproject.org/wiki/Common_F31_bugs#On_Fedora_Silverblue.2FIoT.2C_the_GRUB_menu_shows_duplicate_entries

I imagine other system integrators that use ostree may perform a
higher degree of file system modifications at installation time, as
there is a class of changes that make sense as "sensible defaults" but
probably don't want to be hardcoded in the ostree - e.g. installing
some default flatpak remotes, preinstalling some flatpaks, setting up
some site-specific networking config into /etc. Any changes made along
those lines may need to be updated later. I collectively refer to all
of these disk changes mentioned so far as "extra configuration".

The problem space is even wider for Endless, where we have multiple
products built around a single ostree; the difference in extra
configuration is one of the key distinguishing factors between each
product. And the inability to update extra configuration after
installation time has become a growing pain over the years. Through
maintaining a fairly broad product over the years we've accumulated
many details to tweak on existing installs, big and small, such as:
 - Adding collection ID to existing ostree/flatpak repos
 - Adding flathub remotes
 - Moving stuff from the core OS into flatpaks, requires the flatpak
to be auto-installed on OS update to avoid loss of functionality
 - Tweaking swap setup based on newer learnings
 - Fixing permissions of stuff in /var

After a lot of brainstorming, I am experimenting with a framework for
a solution here for Endless, and I'd like to share a brief description
below in case it is useful for anyone else facing similar challenges,
or if anyone else would like to add some thoughts to the pool.

My idea is to combine the ostree delivery with extra data that defines
and describes the extra configuration to be applied to the product,
and to view the extra configuration as a requirement for OS
installation & upgrade. The extra configuration will be used both at
installation time in order to make all of the essential and base
changes to a blank disk where the OS is being installed (installing
bootloader, setting up flatpak refs, etc), and is also used upon OS
upgrade when the new ostree is deployed, in order to apply the latest
extra configuration which may include changes.

It's an implementation detail but in this case I am experimenting with
Ansible playbooks as the extra configuration format. Ansible makes it
nice to express changes along the lines of "make this specific change
if it hasn't already been made"; if you use Ansible well then your
playbooks have the nice property of being valid both for first-time
setup at installation time and also for upgrading those details on
existing installations later.

In the case of something like Silverblue where it seems like all users
fundamentally run the same single software product, the extra
configuration data could be shipped in the ostree itself.
In the Endless case though, with multiple products built from a single
ostree, we plan to
ship the product-specific extra configuration data separately, but in
a way that each bundle of such data is considered to be specific to
and fully compatible with a certain ostree commit (I'm avoiding all
difficulties around trying to support backwards and forwards
compatibility along the dimension of extra configuration data vs
ostree commit). And the tool used to apply the configuration data
(i.e. Ansible) will be shipped in the ostree.

If the extra configuration data is not included in the ostree in the
Endless case, then there's a question of how is it shipped. I'm
planning to ship it in a separate ostree ref; using ostree to
transport it seems appropriate given that this data is closely tied to
the main ostree, and then we'll get plenty of nice features for
managing such data: P2P updates, ref redirection, ...

The OS upgrade process would then look something like:
 1. Pull latest ostree ref
 2. Pull corresponding configuration data ref
 3. Make a new ostree commit that combines the configuration data and
ostree into a single tree
 4. Deploy that new tree
 5. chroot into the new deployment and run the configuration manager
(i.e. Ansible) to apply all the extra configuration
 6. Mark the new deployment as active for next boot
 7. Reboot

Compared to the current "pull, deploy, reboot" path, the added
complexity here is a little worrying, but thinking more I presume it's
actually rather similar to the case of Silverblue updates where the
user has layered RPMs on top, which is presumably something like:
 1. Pull latest ostree ref
 2. Make a new ostree commit that combines the user's layered RPMs and
the base ostree into a single tree
 3. Deploy that new tree
 4. chroot into the new deployment and run the rpm post install scripts
 5. Mark the new deployment as active for next boot
 6. Reboot

In terms of static deltas I'm hoping that the process described above
for Endless would work OK. There would be a local ref available
corresponding to the upstream pristine ostree, and even though it
isn't actually deployed in that exact form, I'm assuming ostree would
consider this ref as a base for obtaining and applying any applicable
static deltas that take us to the latest ostree commit, before we do
the layering.

As you go deeper, many more challenges emerge, such as how to assemble
and test the extra configuration, how to differentiate between changes
that you want to only apply for new installs vs changes you want to
affect existing users too, and so on.

But the above covers the core idea from the ostree standpoint.
Comments would be very welcome.

Daniel