Re: Asyncronous life helper (Was: EggDBus)

[David Zeuthen <david fubar dk>]

On Sat, 2008-12-27 at 20:35 +0100, Alexander Larsson wrote:
> On Sat, 2008-12-27 at 04:21 -0500, Freddie Unpenstein wrote:
> > This is a side-topic, raised by developments in handling DBus, but
> > something I feel is worth asking... Is there any mechanism for making
> > working with asynchronous stuff easier?
> > 
> > Quite often I've had a situation where I've needed to collect several
> > pieces of information from various asynchronous sources, and then act
> > on them as a group only once some portion of that information is
> > available. This is becoming increasingly important as we start working
> > with more and more remote DBus services, web services,
> > Corba/Orb/whatever-its-presently-called services, and the list goes
> > on. Data life cycle and flow management keeps being a problem.
> > 
> > You need to:
> > - use a signal to catch data when it becomes available
> > - store that data somewhere
> > - distribute that data to other places when it becomes available
> > - clean up the data when it's no longer needed
> > - safely handle data sources failing or disappearing
> > - manage re-try timeouts, age invalidation timeouts, etc.
> > 
> > I sort of feel that what's needed, is a GWish which is a higher-level
> > GBoxed embodying the whole data life cycle management, and I'm
> > wondering whether there are any plans or similar ideas in the works,
> > or even just whether anyone else has a working example of tackling a
> > similar situation.
> > 
> > My starter concept is this; GWish will
> > - hold a piece of data, when it's available
> > - can be asked to obtain its data if it doesn't have it
> > - can be dependant on other GWish's (dependant data)
> > - ask any depedant GWishes for theirs when it is asked
> > - invoke a callback when ALL dependant data is available
> > - can be given the data to hold at any time (usually by the callback)
> > - signal when its data is ready (ie. when its data is set)
> > - only ask dependant GWish's when it itself is asked
> > - provide a helper function to perform the ask, and then wait
> > - allow the data to be invalidated without destroying the GWish
> > - be used internally (or at least faked externally) by async
> > properties
> > - propagate data invalidation (unless blocked by a flag or ignored)
> > - allow some dependant data to be flagged as optional
> > - allow the data set callback to "fail" after requesting more data
> > - optionally count a dependency as a reference on that GWish
> > - use weak references to clean up non-referencing dependencies
> > - support relevant timeouts with automatic timeout cleanup
> > - optionally destroy itself if a dependancy gets destroyed
> > 
> > An application can then tie together various pieces of required data
> > from any source (either GWish-compatible or wrapped in an external
> > GWish), even user interaction, by having the final GWish dependant on
> > other GWishes, some representing other intermediate shared data, and
> > others representing source data, or simply stages of processing to
> > ensure certain events occur in order regardless of what order their
> > data becomes available.
> 
> This sounds quite nice actually. There are many cases where something
> like this would have helped me.
> 
> Another possible name for this is GFuture, as such objects are about
> information or other things that'll be availible in the future.

This sounds very similar to how it's handled in ACE

http://www.dre.vanderbilt.edu/Doxygen/Stable/ace/classACE__Future.html

One thing I did in EggDBus as compared to GIO, was to make the async
functions return a pending call id, e.g.

 guint some_async_function (const gchar         *stuff,
                            GCancellable        *cancellable,
                            GAsyncReadyCallback  callback,
                            gpointer             user_data);

so you can write

 foo->pending = some_async_function("stuff", NULL, async_ready, NULL);

and later

 egg_dbus_connection_pending_call_cancel (con, foo->pending);

to cancel. Or you can do

 egg_dbus_connection_pending_call_block (con, foo->pending);

to block. It's not farfetched, I think, to have a GFuture base class so
you can do

 GFuture *future;
 future = g_future_get (foo->pending_call);
 g_object_set_data (future, "other-stuff", other_stuff, g_object_unref);

and the GFuture object would be unreffed when the pending call is
finished.

In libgio, all async functions currently don't return anything. I think
we can make them return a pending call id without breaking ABI.

> One major problem about doing things like this in C is that you
> constantly have to define and allocate small structs to keep the data
> you need for the completo operation. If we could somehow make this
> easier that would be a large win. I don't know a great approach to do
> that offhand though. Some form of user data is one possibility, but I'd
> really like it to betype safe and without using strings as API.

If you want type safety you could subclass GFuture and do this

 subclassed_future = my_subclassed_future_new (...);
 g_future_set (foo->pending_call, G_FUTURE (subclassed_future));

but for simple stuff we can have convenience stuff on GFuture just like
we do for GSimpleAsyncResult.

> I guess a GFuture/GWish needs to be some form of container that you can
> grow in size by adding a struct of your own data to it. Maybe we can do
> it via a macro g_future_push_struct (future, struct LocalOpData,
> free_op_data) which grows the size of the object and puts the struct at
> the end. You'll pop it at end to free it.
> 
> Of course, such serial/stack-like data management might not be general
> enought to handle things like parallel async operations. Maybe just a
> way to add the struct (undefined where) to the GFuture and then you'll
> get a pointer to the struct inside the GFuture in the completion
> callback...

I think a lot of this would be simpler with fibers....

      David