GNOME.org
 

Re: regarding scaling libsoup server

And please don't get me wrong. libsoup is an awesome library, I use it a lot in my projects, I can tell the same about GLib, but they are just not suitable for everything. They have not been designed with scaling & networking as a top priority so they don't scale. If you have to watch 10 things in the GTK UI or embed a small webserver in the app or periodically access some HTTP API, they are perfect. But if you need to have guarantee that your mainloop will never lock for a while due to underlying call that was not supposed to be locking (seen that in the wild with DNS resolver) or will efficently handle tens of thousands of connections, there are just solutions that were designed with that purpose as the primary goal.

m.

2017-05-02 21:37 GMT+02:00 marcin saepia net <marcin saepia net>:
And here is sample of dicussion you will end up if you try to rely on glib for project that needs to seriously scale:

https://lists.freedesktop.org/archives/spice-devel/2016-January/026068.html

I don't have much more time for proving things that are quite obvious for anyone who wrote such stuff, that has be enough for now.

m.

2017-05-02 21:28 GMT+02:00 Jens Georg <mail jensge org>:
On Tue, 2017-05-02 at 18:05 +0200, marcin saepia net wrote:
> poll does not scale.
>
> epoll does

It's always nice to have some references for random claims.

>
> m.
>
> 2017-05-02 17:12 GMT+02:00 Sergio Villar Senin <svillar igalia com>:
> > O Sáb, 29-04-2017 ás 12:57 +0000, marcin saepia net escribiu:
> > > Hello Allen,
> > > Generally speaking, forget about doing serious scaling using any
> > > library or tool that uses GLib underneath. Regardless of many
> > other
> > > limitations, it does not use epoll for checking descriptors'
> > state
> > > and you are likely to run into c10k problem even if you use
> > multiple
> > > cores.
> > > I suggest taking a look at Elixir/Phoenix, Java/Netty or Go (in
> > that
> > > order).
> > > Marcin
> >
> > I don't have a strong knowledge about epoll but glib main loop uses
> > poll whenever available.
> >
> > BR
> >
> > >
> > > sob., 29.04.2017, 13:36 użytkownik Allen Rintoul <arkjcc gmail co
> > m>
> > > napisał:
> > > > Hello, 
> > > >
> > > > I am building a websocket application using libsoup. During my
> > > > testing I observed the cpu which runs the main loop getting
> > pegged
> > > > at around 300 websocket connections and beyond ~1000
> > connections I
> > > > am observing client connection failures.
> > > >
> > > > Is there a way to spread the connections across all the cpus?
> > or is
> > > > it by design all libsoup functions needs to be executed in the
> > same
> > > > thread?
> > > >
> > > > I also see from perf top that more than 30 percent of the time
> > is
> > > > spend on g_list_sort_with_data(). I have an i5-4440S CPU @
> > 2.80GHz
> > > > (4 threads) and 16G RAM. While the cpu running main loop is
> > getting
> > > > pegged the other 3 cores are more or less idling.
> > > >
> > > > Any thoughts on scaling libsoup to make use of all the cores?
> > > >
> > > > Thank you
> > > > Allen
> > > >
> > > >
> > > >
> > > > _______________________________________________
> > > > libsoup-list mailing list
> > > > libsoup-list gnome org
> > > > https://mail.gnome.org/mailman/listinfo/libsoup-list
> > > >
> > >
> > > _______________________________________________
> > > libsoup-list mailing list
> > > libsoup-list gnome org
> > > https://mail.gnome.org/mailman/listinfo/libsoup-list
> > _______________________________________________
> > libsoup-list mailing list
> > libsoup-list gnome org
> > https://mail.gnome.org/mailman/listinfo/libsoup-list
> >
>
> _______________________________________________
> libsoup-list mailing list
> libsoup-list gnome org
> https://mail.gnome.org/mailman/listinfo/libsoup-list
_______________________________________________
libsoup-list mailing list
libsoup-list gnome org
https://mail.gnome.org/mailman/listinfo/libsoup-list


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]