Re: Adapting an existing distribution to OSTree

[Leon Anavi <leon anavi konsulko com>]

Hi Colin,

On 15.06.2016 16:18, Colin Walters wrote:
> Hi,
>
> On Wed, Jun 15, 2016, at 05:46 AM, Leon Anavi wrote:
> > Hi,
> >
> > I am a working on the integration of OSTree with GENIVI SOTA project in
> > Automotive Grade Linux (AGL).
> Great!  I'm really interested in helping this type of thing succeed.

Thank you for your prompt reply and the provided additional information. 
I hope that we will be able to move this project forward.

> > The client downloads the
> > binaries from the server, and then hands them over to the system for
> > installation.
> In OSTree these phases are intentionally mixed - the client downloads
> updates incrementally *unpacked* into the repo, without affecting the current system,
> and "deployment" is a second phase which also doesn't affect the running
> system other than to make the hardlink farm and a new bootloader entry.
>
> I see this as an advantage over e.g. yum/apt-get and other package
> managers which stage the compressed packages and then unpack them
> only after all are downloaded, leading to a transient space usage spike.
>
> It's also an advantage over dual partition schemes because it avoids
> doubling the OS footprint.
>
> > After reviewing several open source solutions for
> > installation strategies during the AGL face to face meeting in May the
> > interested parties decided to try OSTree. More technical details about
> > the discussions are available in AGL wiki and mailing list:
> > https://wiki.automotivelinux.org/agl-distro/may2016-f2f
> > https://lists.linuxfoundation.org/pipermail/automotive-discussions/2016-May/002077.html
> I'm not finding a lot of details on the comparison there but I would
> point out:
> https://ostree.readthedocs.io/en/latest/manual/related-projects/
>
> I'd be happy to accept links to other projects - I was unaware of mender.io
> for example.
>
> > AGL is Yocto/OE based distribution. I have already integrated RVI SOTA
> > client which is written in the Rust programming language to the upstream
> > of AGL.
> Neat.  I would love to be able to use more Rust but I think for various
> conservative reasons the ostree stack is going to remain C for quite a
> while.  First I'd like to move more in the direction of using Linux-specific features/syscalls,
> which is certainly possible with rust but then I'm fighting `libstd` in the
> same way we're avoiding `GFile` mostly now,
> and second hard requiring Rust would close out some more conservative targets
> right now.
>
> That said I'd be fine having whatever we need to enable Rust bindings to
> libostree in the core.  I think a lot of that hopefully comes for free from
> https://github.com/gi-rust ?

I have already added Yocto/OE recipe for building the client in AGL 
using layer meta-rust. I believe that if in long term Rust is 
contentious for the SOTA client, ATS Advanced Telematic Systems GmbH, 
the company working on GENIVI SOTA project, may be interested in 
rewriting it in C.

I am trying to make progress step by step. Therefore my first step was 
to create the Yocto/OE recipes for building up to date version of OSTree 
in AGL. The second step was to get familiar with OSTRee by trying it out 
with AGL running on Intel NUC. I successfully initialized OSTree 
repository for a directory with text files, added a few commits, 
synchronized it with another repository on the same device with ostree 
pull-local, synchronized with a repository on another machine using 
ostree pull. I have also generated static-delta from one repository and 
applied them offline to another. My next step is to convert AGL to 
OSTree based system.

> > Now I am trying to adapt AGL as an OSTree-enabled system and to generate
> > OSTree commits from the Yocto builds. I have already read chapter
> > "Adapting existing mainstream distributions" from the official
> > documentation, I tried out GNOME continuous in a virtual machine and I
> > looked for hints from the bash scripts in directory tests. I still need
> > more information.
> GContinuous is more of a demo.  My employer uses OSTree indirectly
> via rpm-ostree which in turn derives content from production
> distributions like CentOS and Fedora.  See http://www.projectatomic.io/download/
> for links to VMs and ISO installers.
>
> But that said the distinction between ostree and higher level tools
> like rpm-ostree is precisely to enable sharing code with other
> distributions.
>
> > Could you please provide me more practical guidelines and examples for
> > converting an image built with bitbake to OSTree based image?
> The quick answer is I can't justify a lot of time on Yocto right now, but I
> can help with questions.  I already have to spend a lot of time integrating
> with the Fedora/CentOS ecosystems.
>
> If you want to try building custom content with a CentOS base using
> rpm-ostree, see:
> https://github.com/projectatomic/rpm-ostree/blob/master/docs/manual/compose-server.md
>
> More details on Yocto:
>
> What GContinuous does is take the target root dir from bitbake,
> tars it up, then imports it again into ostree.  See:
> https://git.gnome.org/browse/gnome-continuous-yocto/tree/meta-gnomeos/classes/gnomeos-contents.bbclass?h=gnomeostree-3.14-dizzy
> and then commits it into ostree:
> https://git.gnome.org/browse/gnome-continuous/tree/src/js/tasks/task-build.js#n1173

I am looking at the provided links. Right now I want to manually adapt 
AGL to OSTree-based system. Following the recommendations from the 
documentation I have moved the directories and I added rules for 
systemd-tmpfiles.

What are the exact steps to manually initialize OSTree repository for 
the whole system, add a commit and create a new deployment?

I have tried by executing the following commands directly on the 
embedded device:
ostree admin os-init testos
ostree commit -b testos/foo/bar --tree=ref=testos/foo/bar 
--subject="Initial commit" --body="commit 1"
ostree admin deploy testos/foo/bar –os=testos

But I got the following error:
error: openat: No such file or directory

Best regards, Leon

> It's quite slow, but the goal of that system is to use Yocto only for a slow-changing
> base, then have fast-moving content on top using a highly custom build
> system that uses git+ostree.
>
> This could certainly be improved and ideally upstreamed into Yocto core.
> To really make OSTree fast, one needs to do things like GContinuous
> is doing where each package/component is stored as a separate branch,
> and then unioned to compute the final tree.  We're increasingly doing
> more things like this in rpm-ostree.   It basically needs a lot
> of integration into the buildsystem.
>
>   
>
> _______________________________________________
> ostree-list mailing list
> ostree-list gnome org
> https://mail.gnome.org/mailman/listinfo/ostree-list

--
Leon Anavi
Software Engineer
konsulko.com