Re: regarding scaling libsoup server

[Jens Georg <mail jensge org>]

Hi,

thanks, I just realized that my request sounded a bit rude. Apologies.
I was just looking for pointers.

> I can also recommend this:
> http://chimera.labs.oreilly.com/books/1230000000545/index.html
>
> m.
>
> 2017-05-02 21:47 GMT+02:00 marcin saepia net <marcin saepia net>:
>
> > And please don't get me wrong. libsoup is an awesome library, I use
> > it a lot in my projects, I can tell the same about GLib, but they
> > are just not suitable for everything. They have not been designed
> > with scaling & networking as a top priority so they don't scale. If
> > you have to watch 10 things in the GTK UI or embed a small webserver
> > in the app or periodically access some HTTP API, they are perfect.
> > But if you need to have guarantee that your mainloop will never lock
> > for a while due to underlying call that was not supposed to be
> > locking (seen that in the wild with DNS resolver) or will efficently
> > handle tens of thousands of connections, there are just solutions
> > that were designed with that purpose as the primary goal.
> >
> > m.
> >
> > 2017-05-02 21:37 GMT+02:00 marcin saepia net <marcin saepia net>:
> >
> > Here it is:
> http://www.webcitation.org/6ICibHuyd?url=http://www.kegel.com/c10k.html
> > [1]
> >
> > And here is sample of dicussion you will end up if you try to rely
> > on glib for project that needs to seriously scale:
> https://lists.freedesktop.org/archives/spice-devel/2016-January/026068.html
> > [2]
> >
> > I don't have much more time for proving things that are quite
> > obvious for anyone who wrote such stuff, that has be enough for now.
> >
> > m.
> >
> > 2017-05-02 21:28 GMT+02:00 Jens Georg <mail jensge org>:
> > On Tue, 2017-05-02 at 18:05 +0200, marcin saepia net wrote:
> > > poll does not scale.
> > >
> > > epoll does
> >
> > It's always nice to have some references for random claims.
> >
> > > m.
> > >
> > > 2017-05-02 17:12 GMT+02:00 Sergio Villar Senin
> > <svillar igalia com>:
> > > > O Sáb, 29-04-2017 ás 12:57 +0000, marcin saepia net escribiu:
> > > > > Hello Allen,
> > > > > Generally speaking, forget about doing serious scaling using
> > any
> > > > > library or tool that uses GLib underneath. Regardless of many
> > > > other
> > > > > limitations, it does not use epoll for checking descriptors'
> > > > state
> > > > > and you are likely to run into c10k problem even if you use
> > > > multiple
> > > > > cores.
> > > > > I suggest taking a look at Elixir/Phoenix, Java/Netty or Go
> > (in
> > > > that
> > > > > order).
> > > > > Marcin
> > > >
> > > > I don't have a strong knowledge about epoll but glib main loop
> > uses
> > > > poll whenever available.
> > > >
> > > > BR
> > > >
> > > > >
> > > > > sob., 29.04.2017, 13:36 użytkownik Allen Rintoul
> > <arkjcc gmail co
> > > > m>
> > > > > napisał:
> > > > > > Hello,
> > > > > >
> > > > > > I am building a websocket application using libsoup. During
> > my
> > > > > > testing I observed the cpu which runs the main loop getting
> > > > pegged
> > > > > > at around 300 websocket connections and beyond ~1000
> > > > connections I
> > > > > > am observing client connection failures.
> > > > > >
> > > > > > Is there a way to spread the connections across all the
> > cpus?
> > > > or is
> > > > > > it by design all libsoup functions needs to be executed in
> > the
> > > > same
> > > > > > thread?
> > > > > >
> > > > > > I also see from perf top that more than 30 percent of the
> > time
> > > > is
> > > > > > spend on g_list_sort_with_data(). I have an i5-4440S CPU @
> > > > 2.80GHz
> > > > > > (4 threads) and 16G RAM. While the cpu running main loop is
> > > > getting
> > > > > > pegged the other 3 cores are more or less idling.
> > > > > >
> > > > > > Any thoughts on scaling libsoup to make use of all the
> > cores?
> > > > > >
> > > > > > Thank you
> > > > > > Allen
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > libsoup-list mailing list
> > > > > > libsoup-list gnome org
> > > > > > https://mail.gnome.org/mailman/listinfo/libsoup-list [3]
> > > > > >
> > > > >
> > > > > _______________________________________________
> > > > > libsoup-list mailing list
> > > > > libsoup-list gnome org
> > > > > https://mail.gnome.org/mailman/listinfo/libsoup-list [3]
> > > > _______________________________________________
> > > > libsoup-list mailing list
> > > > libsoup-list gnome org
> > > > https://mail.gnome.org/mailman/listinfo/libsoup-list [3]
> > >
> > > _______________________________________________
> > > libsoup-list mailing list
> > > libsoup-list gnome org
> > > https://mail.gnome.org/mailman/listinfo/libsoup-list [3]
> > _______________________________________________
> > libsoup-list mailing list
> > libsoup-list gnome org
> > https://mail.gnome.org/mailman/listinfo/libsoup-list [3]
>
>
>
> Links:
> ------
> [1] 
> http://www.webcitation.org/6ICibHuyd?url=http://www.kegel.com/c10k.html
> [2] 
> https://lists.freedesktop.org/archives/spice-devel/2016-January/026068.html
> [3] https://mail.gnome.org/mailman/listinfo/libsoup-list