Re: [gnet-dev] zero-copy interface?

[Jeff Garzik <jgarzik pobox com>]

Tim Müller wrote:
> On Mon, 2005-11-07 at 21:21 -0500, Jeff Garzik wrote:
>
> Hi Jeff,
>
>
> > However, a showstopper problem for me is the amount of data copying that 
> > occurs in GNet, which really impacts CPU usage in my application.
> >
> > I'm interested in modifying GNet as follows, in [my] priority order:
> >
> > * figure out how to send integrate sendfile(2) support.  I see its in 
> > the TODO.
>
>
> I'm not convinced that's easily doable or actually useful for all but a
> very few applications. As far as I can see, the main problem is the
> GIOChannels layer in GLib, and that doesn't really provide for nifty
> things like that, does it? You could get the socket fd from GTcpSocket
> and handle writes / sendfile() yourself though.

Yeah, I should probably just modify GLib to add sendfile(2) support. 
Apache provides a sample Win32 implementation and a portable Unix one, 
which is helpful.  I need to modify GLib to support epoll(4) anyway, if 
someone hasn't already done that.

Unfortunately today's sendfile(2) in many OS's is largely for the single 
case of fs-based fd -> socket-based fd.  One must either provide the 
obvious software fallback, or return "not implemented" if one chooses 
the walk the less-travelled paths.


> > * figure out some way for app to provide buffer for async write, rather 
> > than having GNet memdup it
>
>
> That looks like it should be fairly easy to do, and it's also a good
> thing to have. We could add new API for that, maybe something like:
>
> void
> gnet_conn_write_direct (GConn* conn, gchar* buffer, gint length,
>                         GDestroyNotify buffer_destroy_func);
>
> That way you could for example pass an mmap()ed buffer and unmap that in
> your buffer destroy function when it's been written. Not sure how that
> compares to sendfile() performance-wise, but it should make quite a bit
> of a difference in any case.

The write-completion callback works just fine as a sequence point, 
indicating GNet is done with the buffer.

Just need to tell GNet not to memdup the write buffer, and to not free 
it when the write completes.


> > * figure out some way for app to provide buffer for async read, rather 
> > than having GNet allocate and manage it
>
>
> What's the use case you have in mind for that?  GConn reads into a fixed
> per-instance buffer, the only expensive thing involved might be a
> g_memmove() when bytes have been processed, but that you can avoid if
> you read bytes in any 'denomination' rather than a fixed amount or until
> the end of the line.

Several points here:
* g_memmove() in process_read_buffer() isn't a big deal, since its 
working on small chunks that are probably in L1/L2 cache.

* The bigger issue is that the app is forced to copy all bytes it cannot 
act upon immediately, due to GNet's fixed per-instance buffer.

* Also, g_realloc() copies.  Large copies, though certainly it reaches a 
stable size.

* In a smart app that does its own "page management", a memory range 
successfully filled by read(2) could live on for quite a while in an 
app-managed cache or memory pool.


Overall, the reason I chose GNet rather than write my own async network 
I/O lib is that it maps quite naturally to how things work inside the 
kernel:  everything is asynchonous, with I/O completion callbacks. 
GNet's gnet_conn_read[line]() and gnet_conn_write() are very nice 
interfaces.

I'm interested in using GNet in production TCP servers.  All the 
standard open source servers that handle large loads -- http, ftp, pop, 
imap, smtp, nntp -- use a few common techniques (kill copies, use mmap, 
use sendfile, use IP TOS, use MSG_MORE) to keep CPU usage at a minimum.

The GNet design is pretty optimal.  Just need a few small tweaks.  Need 
to let the app manage read/write buffers, figure out a good GLib-or-GNet 
solution for sendfile, and if I'm really lucky, figure out how to do 
MSG_MORE.

Regards,

	Jeff