Re: g_thread_exit()

[Chris Vine <vine35792468 gmail com>]

On Wed, 27 Jun 2018 08:29:00 -0500
richard boaz via gtk-list <gtk-list gnome org> wrote:
> hello,
>
> (not sure if this is a glib question, or a linux kernel question, any
> helpful hints appreciated...)
>
> i have a server-side program that is responsible for managing hundreds
> (increasing daily) of IoT connections, where this program listens for a
> connection request and then spawns a separate thread to manage the incoming
> real-time data feed using g_thread_try_new().
>
> i have placed a 30 second timeout on the socket and when this is breached,
> the socket is closed and the thread is exited using g_thread_exit().
>
> however, with so many connections coming and going over all time,
> eventually, spawning a new thread results in g_thread_try_new() failing
> with the error: "Error creating thread: Resource temporarily unavailable".
> i have upped the max threads per process to 95K, but this is just a
> stop-gap measure that merely delays the onset of this problem.
>
> my question: is g_thread_exit() guaranteed to reduce the kernel resource
> pool counter by 1?  my experience seems to suggest that it does not, so my
> next question is: how to avoid the problem (i.e., not receive this false
> error) when the total number of threads existing at any given moment does
> not exceed max threads per process?
>
> or is it the (unfortunate) reality that the max threads per program
> represents the total number of threads over all time, exited or not?
>
> thanks in advance for any pointers to a solution...

Are you calling g_thread_join() on the extinct thread to reclaim
pthread and glib resources?  If that is problematic you might consider
using pthreads directly and detaching threads at the outset.

On a different issue, one thread per connection is not scalable on most
hardware.  If your program is glib based, using GIO's asynchronous
operations with a glib main loop is likely to give you much better
results.  GIO uses a dedicated thread pool where it needs to hand work
off.

This may require you to rewrite your application, and it may involve
you dealing with "callback hell" (there is no easy way of implementing
await semantics with coroutines if you are using glib), but it will
probably prove worth it if you are otherwise trying to run nearly
100,000 threads.  Kudos for even getting near that number of native
threads on reasonable hardware.

Chris