Re: Updating the OS data that ostree doesn't manage

["Colin Walters" <walters verbum org>]

On Tue, Feb 18, 2020, at 12:25 PM, Will Manley wrote:
> Thanks for sharing.  It sounds like a sensible design given the 
> constraints you've described.  I've made a few comments below about how 
> we handle our deployment process.  Our use-case is a lot simpler than 
> yours as we have a lot of control over the devices, and the devices 
> contain little state, but in the spirit of sharing I thought I'd write 
> it up:

Thanks for this too!

> Regarding managing /var: This mostly consists of updating permissions, 
> creating directories, etc.    We rely on systemd-tmpfiles for this 
> which runs at boot time to perform these updates.

Right - rpm-ostree today for example synthesizes tmpfiles.d entries from RPM directories.

> Regarding managing /sysroot/ostree: We have a systemd unit called 
> post-upgrade-cleanup.service which is run at boot after a deploy.  This 
> allows us to perform housekeeping including running `ostree admin 
> cleanup`.

Yep, this is a sensible approach.

That all said at some point we may need to add a generic "before reboot" hook in ostree for
https://github.com/ostreedev/ostree/issues/1026

> Regarding managing other filesystems/partitioning: We use loopback 
> mounts for user-data.  This means that it's easier to make changes like 
> adjusting the relative sizes of these filesystems without having to 
> perform online adjustments to /.

Ah.  Yeah, systemd is creating homed which does this too but I have to say at least the Red Hat storage team 
said to never use loopback mounts for anything production.  (I know storage developers find it useful for 
testing though)

We are also still debating the size for / for FCOS at the moment
https://github.com/coreos/coreos-assembler/pull/924#issuecomment-589268886
as it relates to users creating custom partitions.

> We've not yet had to make any changes to our rootfs that require 
> repartitioning or reformatting.  I can't see that this could be done in 
> a safe atomic way without Android style A/B filesystems.

Yeah, can't be done atomically, but I know some people have written code to boot into the initramfs and do 
destructive operations from there like rewriting/upgrading the root filesystem type or block device setup.  
It's still a "don't turn off your computer now"  type situation but if you're only doing it rarely, at least 
that risk is mitigated.

> I think it's worth splitting changes to /var and to /etc conceptually 
> rather than considering them together. It seems to me that they are 
> quite different in the way they're handled by ostree.  In particular 
> there is a separate /etc per deploy, which makes changes atomic there, 
> while /var is shared, meaning that you have to take a lot more care 
> making modifications there.

Agreed!