Re: Adapting an existing distribution to OSTree

[Colin Walters <walters verbum org>]

Hi,

On Wed, Jun 15, 2016, at 05:46 AM, Leon Anavi wrote:
> Hi,
>
> I am a working on the integration of OSTree with GENIVI SOTA project in 
> Automotive Grade Linux (AGL).

Great!  I'm really interested in helping this type of thing succeed.

> The client downloads the 
> binaries from the server, and then hands them over to the system for 
> installation. 

In OSTree these phases are intentionally mixed - the client downloads
updates incrementally *unpacked* into the repo, without affecting the current system,
and "deployment" is a second phase which also doesn't affect the running
system other than to make the hardlink farm and a new bootloader entry.

I see this as an advantage over e.g. yum/apt-get and other package
managers which stage the compressed packages and then unpack them
only after all are downloaded, leading to a transient space usage spike.

It's also an advantage over dual partition schemes because it avoids
doubling the OS footprint.

> After reviewing several open source solutions for 
> installation strategies during the AGL face to face meeting in May the 
> interested parties decided to try OSTree. More technical details about 
> the discussions are available in AGL wiki and mailing list:
> https://wiki.automotivelinux.org/agl-distro/may2016-f2f
> https://lists.linuxfoundation.org/pipermail/automotive-discussions/2016-May/002077.html

I'm not finding a lot of details on the comparison there but I would
point out:
https://ostree.readthedocs.io/en/latest/manual/related-projects/

I'd be happy to accept links to other projects - I was unaware of mender.io
for example.

> AGL is Yocto/OE based distribution. I have already integrated RVI SOTA 
> client which is written in the Rust programming language to the upstream 
> of AGL.

Neat.  I would love to be able to use more Rust but I think for various
conservative reasons the ostree stack is going to remain C for quite a
while.  First I'd like to move more in the direction of using Linux-specific features/syscalls,
which is certainly possible with rust but then I'm fighting `libstd` in the
same way we're avoiding `GFile` mostly now,
and second hard requiring Rust would close out some more conservative targets
right now.

That said I'd be fine having whatever we need to enable Rust bindings to
libostree in the core.  I think a lot of that hopefully comes for free from
https://github.com/gi-rust ?

> Now I am trying to adapt AGL as an OSTree-enabled system and to generate 
> OSTree commits from the Yocto builds. I have already read chapter 
> "Adapting existing mainstream distributions" from the official 
> documentation, I tried out GNOME continuous in a virtual machine and I 
> looked for hints from the bash scripts in directory tests. I still need 
> more information.

GContinuous is more of a demo.  My employer uses OSTree indirectly
via rpm-ostree which in turn derives content from production
distributions like CentOS and Fedora.  See http://www.projectatomic.io/download/
for links to VMs and ISO installers.

But that said the distinction between ostree and higher level tools
like rpm-ostree is precisely to enable sharing code with other
distributions.

> Could you please provide me more practical guidelines and examples for 
> converting an image built with bitbake to OSTree based image?

The quick answer is I can't justify a lot of time on Yocto right now, but I
can help with questions.  I already have to spend a lot of time integrating
with the Fedora/CentOS ecosystems.

If you want to try building custom content with a CentOS base using
rpm-ostree, see:
https://github.com/projectatomic/rpm-ostree/blob/master/docs/manual/compose-server.md

More details on Yocto:

What GContinuous does is take the target root dir from bitbake,
tars it up, then imports it again into ostree.  See:
https://git.gnome.org/browse/gnome-continuous-yocto/tree/meta-gnomeos/classes/gnomeos-contents.bbclass?h=gnomeostree-3.14-dizzy
and then commits it into ostree:
https://git.gnome.org/browse/gnome-continuous/tree/src/js/tasks/task-build.js#n1173

It's quite slow, but the goal of that system is to use Yocto only for a slow-changing
base, then have fast-moving content on top using a highly custom build
system that uses git+ostree.

This could certainly be improved and ideally upstreamed into Yocto core.
To really make OSTree fast, one needs to do things like GContinuous
is doing where each package/component is stored as a separate branch,
and then unioned to compute the final tree.  We're increasingly doing
more things like this in rpm-ostree.   It basically needs a lot
of integration into the buildsystem.