Re: Asyncronous life helper (Was: EggDBus)

[Alexander Larsson <alexl redhat com>]

On Sat, 2008-12-27 at 04:21 -0500, Freddie Unpenstein wrote:
> This is a side-topic, raised by developments in handling DBus, but
> something I feel is worth asking... Is there any mechanism for making
> working with asynchronous stuff easier?
>
> Quite often I've had a situation where I've needed to collect several
> pieces of information from various asynchronous sources, and then act
> on them as a group only once some portion of that information is
> available. This is becoming increasingly important as we start working
> with more and more remote DBus services, web services,
> Corba/Orb/whatever-its-presently-called services, and the list goes
> on. Data life cycle and flow management keeps being a problem.
>
> You need to:
> - use a signal to catch data when it becomes available
> - store that data somewhere
> - distribute that data to other places when it becomes available
> - clean up the data when it's no longer needed
> - safely handle data sources failing or disappearing
> - manage re-try timeouts, age invalidation timeouts, etc.
>
> I sort of feel that what's needed, is a GWish which is a higher-level
> GBoxed embodying the whole data life cycle management, and I'm
> wondering whether there are any plans or similar ideas in the works,
> or even just whether anyone else has a working example of tackling a
> similar situation.
>
> My starter concept is this; GWish will
> - hold a piece of data, when it's available
> - can be asked to obtain its data if it doesn't have it
> - can be dependant on other GWish's (dependant data)
> - ask any depedant GWishes for theirs when it is asked
> - invoke a callback when ALL dependant data is available
> - can be given the data to hold at any time (usually by the callback)
> - signal when its data is ready (ie. when its data is set)
> - only ask dependant GWish's when it itself is asked
> - provide a helper function to perform the ask, and then wait
> - allow the data to be invalidated without destroying the GWish
> - be used internally (or at least faked externally) by async
> properties
> - propagate data invalidation (unless blocked by a flag or ignored)
> - allow some dependant data to be flagged as optional
> - allow the data set callback to "fail" after requesting more data
> - optionally count a dependency as a reference on that GWish
> - use weak references to clean up non-referencing dependencies
> - support relevant timeouts with automatic timeout cleanup
> - optionally destroy itself if a dependancy gets destroyed
>
> An application can then tie together various pieces of required data
> from any source (either GWish-compatible or wrapped in an external
> GWish), even user interaction, by having the final GWish dependant on
> other GWishes, some representing other intermediate shared data, and
> others representing source data, or simply stages of processing to
> ensure certain events occur in order regardless of what order their
> data becomes available.

This sounds quite nice actually. There are many cases where something
like this would have helped me.

Another possible name for this is GFuture, as such objects are about
information or other things that'll be availible in the future.

One major problem about doing things like this in C is that you
constantly have to define and allocate small structs to keep the data
you need for the completo operation. If we could somehow make this
easier that would be a large win. I don't know a great approach to do
that offhand though. Some form of user data is one possibility, but I'd
really like it to betype safe and without using strings as API.

I guess a GFuture/GWish needs to be some form of container that you can
grow in size by adding a struct of your own data to it. Maybe we can do
it via a macro g_future_push_struct (future, struct LocalOpData,
free_op_data) which grows the size of the object and puts the struct at
the end. You'll pop it at end to free it.

Of course, such serial/stack-like data management might not be general
enought to handle things like parallel async operations. Maybe just a
way to add the struct (undefined where) to the GFuture and then you'll
get a pointer to the struct inside the GFuture in the completion
callback...