App image experiments

[Alexander Larsson <alexl redhat com>]

Today I've been looking at the app image part of the application stuff
we were talking about on the hackfest.

Basically, given a "base profile" as a directory with the basic
dependencies and one or more app images, merge these two into a single
tree inside a separate namespace and run the app from there.

Given that we don't currently have any union fs upstream the only way
we can do this currently is to use bind mounts. So, I've written a
small app to test this. It takes a directory with the base os and
a set of squashfs images, then it clones to get a new namespace
where it mounts a tmpfs. Inside it we mount loopback mount the images
and create a directory "root" where we build up the merged structure.

Any dir/file just in one of the sources gets bind-mounted in, but
if any directory is in multiple sources we create it on the tmpfs and
merge in the the children recursively.

It also mounts /proc, and bind-mounts $home and /dev. We also disable
all kind of SUID and new caps stuff.

Code at: https://github.com/alexlarsson/run-merged

In order to set up some kind of test base environment I used this:

 mkdir /opt/base_os
 yum install nautilus --installroot=/opt/base_os/F18 --releasever=18
 yum remove nautilus --installroot=/opt/base_os/F18 --releasever=18

It gives me a base os that has everything nautilus requires at runtime.
Obviously a real base os needs to be much more carefully constructed,
but
this is a simple test.

Then i can create a test app with:

 yumdownloader nautilus --releasever=18
 yumdownloader nautilus-extensions --releasever=18
 yumdownloader gtk3-devel --releasever=18

 mkdir -p /tmp/app_image
 rpm2cpio nautilus-3.6.3-4.fc18.x86_64.rpm | (cd /tmp/app_image && cpio
-id )
 rpm2cpio nautilus-extensions-3.6.3-4.fc18.x86_64.rpm |
(cd /tmp/app_image && cpio -id )
 rpm2cpio gtk3-devel-3.6.4-1.fc18.x86_64.rpm | (cd /tmp/app_image &&
cpio -id )
 mksquashfs /tmp/app_image nautilus.squashfs

Then i start run-merged:

 ./run-merged /opt/base_os/F18 ./nautilus.squashfs
 sh-4.2$ mount | wc -l
 5307

Yowzers! 5000 bind mounts! But i can run gtk3-demo and it works.

Of course, now comes the hard parts:

 sh-4.2$ nautilus
 (nautilus:15428): GLib-WARNING **: getpwuid_r(): failed due to unknown
user id (1000)
 (nautilus:15428): GLib-GIO-ERROR **: Settings schema
'org.gnome.nautilus.preferences' is not installed

The first one is because of no /etc/passwd i guess.
The second one is harder. It fails because glib-compile-schemas has not
run in the merged /usr/share/glib-2.0/schemas where the nautilus rpm put
it. We need to run the schema compilation and things like that during
image construction, and probably put it in a different prefix.

Basically, this is where we need to solve the "search path problem".

The code inherits some code from colins linux-user-chroot app. Does it
seem sane? Can anyone thing of a better approach to do the merging?

Also, I worry a bit about the loopback mounted files. There is nothing
prohibiting the user to modify the fs images while the app is running,
is there? Does the kernel really keep working if the directory metadata
in the filesystem changes under its feet? Is there some way to get some
kind of COW snapshot of the image to mount?

/ Alex