Re: Streams API [Was: Re: Plans for gnome-vfs replacement]

[Alexander Larsson <alexl redhat com>]

On Thu, 2006-09-21 at 02:25 -0400, Hans Petter Jansson wrote:
> On Wed, 2006-09-20 at 10:07 +0200, Alexander Larsson wrote:
> > On Tue, 2006-09-19 at 16:24 -0400, Hans Petter Jansson wrote:
> 
> > > Of course, not all applications need such functionality, but that's
> > > just a matter of wrapping this in a higher-level, more rigid API.
> 
> > In my reading of code using gnome-vfs i haven't seen a single use of
> > anything like this. Could you perhaps describe some real-life examples
> > of how/where this would be used?
> 
> Well, a peer-to-peer filesharing application is the most extreme example
> I can think of. Such programs typically need encryption, rate control,
> ad-hoc protocol implementations, lots of TCP and UDP sockets and in
> decentralized networks, routing facilities. Now, we don't need to
> provide all of this, especially not in glib, but having the basic
> building blocks there would be nice.
> 
> Another example is instant messaging applications, which require
> encryption (typically an SSL layer), rate control (so pasted text can be
> sent slowly, lest you trigger the server's anti-flood filter) and ad-hoc
> protocols. In fact, I think Gaim is a good example of how the platform's
> lack of a good, low-level async I/O API leads to broken applications:
> 
> Gaim (at least version 1.5, which I inspected) has UI that blocks on
> network I/O, poor error handling and convoluted protocol code. It also
> features an interesting approach to "async" DNS lookups: Fork a child
> process and do lookups there, using pipes to communicate with it.
> Unfortunately, pipe communication was done using blocking posix calls,
> so the end result was a main process that could still block on DNS
> lookups.

It still strikes me as very specialized code that is not really useful
for anywhere near the majority of applications.

> It's possible that we're talking past each other here, by the way. I was
> thinking of the streams API as lower-level than VFS, so not something
> you'd see a need for perusing gnome-vfs consumers.
> 
> My mental image is:
> 
> [posix] <- [Async I/O] <- [Streams] <- [VFS]

This is not quite how I see it. The VFS is a module that implements and
hands out streams, so if anything streams are above the vfs. Normal
local file streams will use blocking posix calls for blocking i/o, and a
simple threadpool to implement async calls (or posix aio if supported).
For a non-local file the stream would be a socket connection to the vfs
daemon for that particular mountpoint. Such a stream will use a simple
protocol (dbus, or a custom one) that just sends packages and waits for
the reply by blocking or by using the mainloop. Nowhere is there any
need for a special async i/o system.

You might be thinking of the implementation of the backends though. Here
things are a bit more complicated as we must implement the actual
protocols. Each backend will be different, because they'll be using
various existing libraries having special requirements. Perhaps some
could use some general framework. Although it seems to work fine in
gnome-vfs without it, as the stuff being implemented is really pretty
simple.

> > > Also, what's your thinking on supporting asynchronous DNS lookups,
> > > representing IP addresses, TCP/UDP sockets, etc? We currently don't
> > > have any clear platform solution for that.
> 
> > These are interesting questions, and some applications would surely
> > benefit from such code (although others will consider it bloat). I don't
> > want to intermix these questions with the vfs layer though. They are not
> > really strongly related, only TCP sockets are slightly related to
> > streams, and not really in a very interesting way (its just another
> > possible stream implementation).
> 
> If you're going to keep implementations of network protocols in glib
> (which I assume you'd have to for some VFS modules), you'll have to
> implement this anyway. Why not make it available to applications?
>
> I'm making the assumption that you'll keep VFS modules in glib because
> you mentioned it would have "odd dependencies" earlier.

I specifically talked about "odd dependencies" as a reason for not
having it in glib. ATM I think its better to have the actual vfs
implementation outside of glib, and just the vfs daemon protocol
implementation in glib. However, nothing is set in stone yet.

Its true that some of the backend implementations might have uses for
some of these things. And, if we manage to create high quality APIs and
implementations of these they might fit in glib. That is a separate
discussion for each such implementation IMHO, and up to the glib
maintainers. I'm somewhat scared by this from exposing to much of the
stuff gnome-vfs needed internally as public APIs, and later having
problems maintaining them due to stability guarantees and just bad
initial decisions. Anyway, I see no particular need for such things to
be in glib to allow the implementation of the vfs, but if they are there
they could certainly be useful. In fact, they might be useful for the
vfs implementation even if they were in a separate library like
"libgnetwork" or something.

Also, we will be using various third party libraries like libsmb and
neon to implement the backends. These libraries already do some of the
things you talk about internally, and it would be hard to make them use
our own implementation of e.g. a socket abstraction.

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                            Red Hat, Inc 
                   alexl redhat com    alla lysator liu se 
He's a lonely pirate barbarian who knows the secret of the alien invasion. 
She's a radical antique-collecting safe cracker in the wrong place at the 
wrong time. They fight crime!