Re: Can't launch apps, fork() returns ENOMEM

[Alan Coopersmith <alan coopersmith oracle com>]

On 04/ 2/18 12:15 PM, Daniel Drake wrote:
> Discussions here have lead to glibc's posix_spawn() being recently
> reimplemented to use clone(CLONE_VM) while solving practically all the
> danger there.

We've done something similar in the Solaris libc recently, so I'd
support using posix_spawn if possible.

> Then I considered making glib just call posix_spawn() directly when
> appropriate. There are a number of things in that API which let
> relevant actions be done in the child process before it does the exec
> - managing file descriptors, signal handlers, etc. The big issue there
> is that glib lets you pass an arbitrary GSpawnChildSetupFunc to be run
> in the child before the exec(), this is used by gnome-shell, and there
> is no equivalent hook in posix_spawn().
>
> In gnome-shell app_child_setup() this function is used to send stdout
> and stderr to journald. Also in glib gdesktopappinfo.c child_setup()
> the same mechanism is used to set GIO_LAUNCHED_DESKTOP_FILE_PID in the
> child process to it's own process PID (can't find the background
> here).
>
> I think there are 2 viable paths forward:
>
>  1. Eliminate the child_setup calls from these codepaths to allow
> posix_spawn() to be used. The gspawn code already allows for file
> descriptor redirection but this would have to be exposed via
> additional parameters to g_desktop_app_info_launch_uris_as_manager() -
> an API change. If GIO_LAUNCHED_DESKTOP_FILE_PID can't be reimplemented
> another way then we could first exec a binary wrapper that sets this
> env variable before execing the app itself.

This also seems like the better path to me.  It does seem strange that
you need to set an environment variable in the child process to tell
it what it's PID is - perhaps it's for children of that child to find
the spawn point of their family?

>  2. Propose to http://www.opengroup.org/austin/ that posix_spawn()
> grows the capability to call a user-specified child setup func. There
> is some unused padding in posix_spawnattr_t which could be used to
> store the function pointer and data pointer. Then implement this in
> glibc. I guess this is a lengthy process and it's not a great idea
> from the glibc standpoint where such code would run in dangerous
> context.

The Austin Group does not like specifying API that no one has yet
implemented and proven to be working and useful - they'd rather you
get one of the Linux, BSD, or Unix libc's to first implement and show
the design works, then standardize it.

-- 
        -Alan Coopersmith-               alan coopersmith oracle com
         Oracle Solaris Engineering - https://blogs.oracle.com/alanc