Re: Deployment when having separate development streams

[Colin Walters <walters verbum org>]

On Thu, Oct 19, 2017, at 09:52 AM, Tobias Frost via ostree-list wrote:
> The embedded device is some kind of company-standardized piece of
> hardware, so it will be used in very different contexts, sharing mayor
> portions of the embedded software (e.g kernel, basic tools, libs, ...)
> that will be combined with other (specialized) parts, developed by the
> groups that will then use the embedded hardware into their project. IOW,
> the deployed software will be the base system + specialized part. That
> can be merged into one part (e.g can be then on one ostree branch) if
> this helps. Does it or would it be better to keep them separated?

Hmm...is this separation like "base OS" and "userspace apps"?  Or are the "parts"
more like drivers?
 
> As said, the hardware will be used company wide. That means when it will
> be produced only the base system can be pre-installed in the factory.
> The second part will then be deployed later on, when the hardware is put
> into the final device. We'd like to use ostree for that too, if
> possible.

Ah, right.  So this is certainly an interesting topic.  flatpak today uses
ostree for desktop apps.  A lot of the container world is standardizing on OCI,
(see Kubernetes etc.).  But OTOH, OCI doesn't have any deltas format.  There
are ways to ameliorate that cost in the primary target audience there of
datacenters, but it's obviously more painful outside of that context, particularly
in embedded systems deployed in remote areas etc.

There are a lot of choices here.   I could certainly imagine something a bit more
`docker` like that pulled from ostree.   Supporting things like that is
why the "get files from A to B with deltas" is a separate layer in libostree from
"manage bootloader and OS" - flatpak only uses the former.

I may at some point in the future be looking at a system to add ostree-style deltas to OCI, but
that's not near term.

But none of the above may apply if I'm not quite understanding your case,
see below:

> The challenge now is that the base system advances independently of the
> second part.  While we can control the relation between the base system
> and the 2nd part in terms of minimum version used as base, we will face
> situations where we get different base systems pre-installed and then
> need to generate the deltas up to the latest version.
>
> (Bx denoting base system version, Sx the combined work with the
> specialized part, the version to be deployed; as said it could also be
> only the specialized part if it helps)

Hum.  I am a bit confused - are you planning for a Sx instance "merged" into the same
filesystem tree or not?  

Let's look at a real world example here; for rpm-ostree with package layering,
we generate a new commit client side - I used e.g. `rpm-ostree install powerline`
since I think the shell prompts are cool.  It's part of my host.  `rpm-ostree upgrade`
fetches from *both* sources.

In this case, the update streams are distinct (this has its own issues,
see e.g. https://github.com/projectatomic/rpm-ostree/issues/415 )

So when you're saying "combined work", is it operating like this, where
there is a single filesystem tree that's a union? 

This contrasts completely with e.g. flatpak/docker/etc. as there each
container has its own "base image", updated asynchronously from the host.

Now there's an entirely different model possible - one where one *does*
use "containers" in the sense that the apps have their own filesystem tree,
but they're updated with the host.   I'm not aware of anyone trying this
today, but I bet it would work pretty well to have e.g.:

/usr/share/containers/$appname/{usr,etc}

which is baked into the single ostree branch, and point a systemd unit file at it
with just:

RootDirectory=/usr/share/containers/$appname
(And a number of the other systemd unit file directives are useful here)

> We do only support "known" version mixes, so S2 would always include B2
> and never B3 nor B1.

This sounds like merged trees on the server - or are you thinking a client
side tool for this?

> While we for sure can always release a new combined software and
> maintain it on an os-tree branch, it can happen that we will have
> different versions of the base system where we'd need base our deltas
> on. Would that be possible with ostree?

Hm, since you're concerned about deltas, you must be thinking these
trees are merged on the server side.   In that case, if I'm understanding
everything correctly - yes, you'd currently need distinct deltas for each base.
An ostree delta goes from an *exact* commit A to an *exact* commit B,
and in a "merged" scenario you'd need to support going from/to
multiple of those.

The delta design though did consider "recursive" deltas, which
I haven't gotten back to but would probably help a lot here.  It's
a way to trade back server CPU/storage for some additional
per-client overhead.

Now hm...we could also consider here teaching ostree about
the fact that conceptually there are multiple filesystem trees involved
here, and rather than making a delta from a merged tree (Mx, a
combination of Bx and Sx) to another "My" merge, we retain
the commits for Bx and Sx in the commit metadata, and teach
the pull code to fetch Bx and Sx separately and do the union client
side.

This would be very much like if for rpm-ostree rather than fetching
RPMs it fetched ostree branches.  (That's...a whole big topic)

Does any of this help?