Re: roadmap/todo for sdk and tooling

[Alexander Larsson <alexl redhat com>]

On tis, 2015-01-13 at 23:19 -0500, Matthias Clasen wrote:
> Hey Alex,
>
> I think it would be great if we could put together a small
> roadmap/todo list for xdg-app and surroundings. If only because I'm
> itching to help out, and a roadmap would help a lot with that.

I don't really have a nice roadmap, but I have a lot of TODO things
scattered around. I realize this is not great, especially if more
people get involved. So, yeah, making some kind of roadmap seems like
a good plan. I'm gonna start by typing out a huge mail with stuff that
needs doing/researching.

We need to figure out the story on how to create apps with our
runtime. I mean, in theory anyone can compose a correctly structured
app dir manually, and we should of course let people do that. But we
also want a story that makes it simple to reuse/share bundling
work. Ideally it will also lead to people using identical binaries for
bundled stuff, thus sharing space via ostree dedup. My plans for this
atm are very vague, but in general we would create some kind of
collection of pre-made rpms that are built do bundle in /self and that
rely on the org.gnome.Platform dependencies.

We should create a bunch of app bundles for gnome apps. Both to make
it easy to run them, and to verify that our chosen machinery works
well.

At some point we want the gnome infrastructure to build the official
builds of the sdk, etc. Although we should probably wait until things
are a bit more settled before doing this.

Initial download of an application via ostree is currently way to
slow. I'm working on adding support for from-empty delta snapshots to
speed this up.

xdg-app needs a bunch of more commands. For instance, we need to be
able to uninstall things, and maybe roll back or check out particular
app versions. Also listing what is installed, and getting details about
them would be nice.

xdg-app build also needs more tooling. Right now it doesn't do the
final step that takes the built directory and pushes it into a
repository. We need to figure out how this should look and implement
that.

Right now, any exported files are symlinked into the exports directory
during install/upgrade. This is part of the solution for desktop
integration (desktop files, dbus services, etc), but not enough. This
needs to be extended with support for running triggers when needed, as
well as specific triggers for things like update-desktop-database,
gtk-update-icon-cache and update-mime-database. We also need to ensure
that XDG_DATA_DIRS is set to point into the exports so that the shell
and others will see the installed apps.

We need to look at the layer above xdg-app. We want some kind of
periodic update system, and it probably needs an ui. We also need to
extend the pure ostree repositories with "user" level metadata,
probably based on appdata, so that things like gnome-software can give
a nice "app store" front for the lower level stuff.

Dbus right now always uses an abstract unix socket for the session
bus. This doesn't work with apps that disable the network, so we need
to figure out some way to make gnome or the distros start dbus with a
non-abstract unix socket in /run instead. See my recent mails to the
dbus list about this.

The sdk right now has a build of mesa, but i had to disable the radeon
driver for some build reason. This has to be fixed. Also, we need to
ensure that the built libEGL works, as I have not tried this. Maybe we
also need to forward the drirc config to the sandbox.

The mesa in the sdk is build in a special location /usr/lib/GL. This
is because it can then easily be replaced, which is important as it
needs to match the host OS driver version. This way it is easily
replacable with non-mesa drivers. The plan I have here is to allow
some kind of sub-runtime that gets mounted over /usr/lib/GL in the
sandbox. So, you would download/install a new GL driver built for a
particular runtime+host, then the user would register that with
xdg-app, pairing it with a particular runtime. This would allow
runtime-switching of the gl driver without having to modify the base
runtime. This needs more thinking and research.

xdg-app-helper, which is the thing that sets up the app runtime
environment doesn't set up access to the Wayland socket (it only does
X11, pulseaudio and dbus atm). This needs to be extended to the
wayland socket, and we need to verify that wayland-from-sandbox works.

DNS resolve doesn't currently work inside the sandbox, because there
is no resolve.conf. Long term i think Lennarts work on
systemd-resolved is the right solution here. But for now we should
just try to at least have /etc/resolv.conf from the host injected into
the sandbox. I've not done this yet because it is a bit tricky. We
can't just bind-mount /etc/resolv.conf into the namespace because that
would not propagate updates from this file into the sandbox. Such
updates are typically done via rename-replace, which means the bind
mount keeps seeing the old file. I've had this idea of a workaround
that keeps a copy of the resolv.conf in /run somewhere and updates it
in-place when the real one updates. Then a bind mount could work. We
probably want a session dbus service singleton you can use to request
this.

dconf "seems" to work currently, but only if you give the app access
to $HOME, and even then its not quite right because it can't read the
system dconf databases or the special files in
"/run/user/1000/dconf". Its even unclear if it is the right thing to
do in general to write app-specific config into a global
database. Maybe it should instead write into /var, which is the
per-app writable space (that persists over updates). This needs
research.

Gvfs needs some work to make the volume monitor work in a sandboxed
world. Right now the client loads files from
/usr/share/gvfs/remote-volume-monitors directly, but these are host os
defined. We need to change this and make the client read these via
dbus APIs instead.

The gvfs fuse mount is in /run/user/1000/gvfs. We need to optionally
(if configured) make the app see this.

Accessibility uses custom dbus busses (/run/user/1000/at-spi2*), and
we need to propagate the right ones into the apps. Also, we need to
look at exactly what goes over these, and if we need to change it in
the future when apps are more heavliy sandboxed. i.e. is this a
security risk?

The sdk includes libproxy, but its built in a pretty "dumb" mode. We
should figure out the best way to instead allow it to talk to some
kind of proxy daemon running in the session. This way we have
centralized control and don't have to duplicate and update the proxy
conf in each app.

Right now there is no cups lib in the image and no printing works
(disabled in gtk3). We need to figure out in general how printing
should work for sandboxed apps. Maybe we can somehow make this work
over dbus, or maybe run some kind of proxy cupsd in the session. I
don't think we want to have the printer configuration in each app.

I'm not sure what the current state is, but we want propagate timezone
settings, and possibly even timezone files (they update regularly)
from the host os to the app. Needs research.

Some apps use syslog or the journal APIs to log stuff. We need to set
things up so that this works in a nice way in the sandbox. Needs
research.

I've ignored colord for now, but maybe this needs some kind of
integration with apps.

One nice affect of the glick2 one-loopback-mounted-file-per-app design
was that you could remove or update an app without problems while it
was being run (the file is gone/replaced but the mout is still
there). Ostree allows efficient concurrent deployment of two versions
of an app, which would allow you to keep an old running app installed
while still updating the on-disk one to the latest. However, it is
hard to know when you can safely remove the old version. I had an idea
how this could be done, but unfortunately some kernel semantics that
it relied on changed, so it broke. I want to do some more research
into this to see if there is some way we can know for sure that no app
is running with a particular subdirectory mounted. This way we could
delay remove on update if the app is running.

Further in the future we need to start looking into harder
sandboxing. This means at least adding support for:
* creating a cgroup for the app (i.e. a user systemd unit)
* switch to using kdbus with a limited dbus access
* Use selinux to further isolate the app
* Make pulseaudio better at isolation
* Drop X11 and use wayland only
* Start creating dbus "portal" APIs to integrate with the desktop
  in a sandboxed way.

There is also a bunch of work needed on the sdk/platform images.
For example:

* research libsecret
  I just added this as a simple dependency for webkit, but we need to
  figure out how to make it work right/securely in a sandboxed world.

* package libcanberra
  gtk and others use this, so we should probably package it and make
  sure it works with our pulse integration

* package gstreamer
  Also, when done update webkit to enable the video/audio stuff

* package enchant, dependencies and spellchecker dicts
  Lots of things use this, and it would be nice to share the
dictionaries.
  Also, when done update webkit to enable spell-checking

Also in general we need to carefully consider what we ship in the
platform, because we have to support it a long time, while still
making it useful to a maximum number of people. For instance, right
now it has no scripting language. Should we perhaps add python? If so,
what version, which modules, and how do we handle apps that need a
different version.

We also might want to think about how we handle locale data, because
it is very large, and each user only uses one locale. Maybe we can
split it out somehow.



-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                            Red Hat, Inc 
       alexl redhat com            alexander larsson gmail com 
He's a sword-wielding ninja matador for the 21st century. She's a 
time-travelling cigar-chomping mermaid with the power to bend men's 
minds. They fight crime!