Re: [Straw] Fixing async IO reliability

[Tuukka Hastrup <Tuukka Hastrup iki fi>]

On Fri, 24 Nov 2006, Tuukka Hastrup wrote:
> I'm a big fan of asynchronous IO in favor of threaded concurrency, but in 
> Straw it seems to be the cause of complexity and lack of reliability in 
> the networking as seen in the code and bug reports. Perhaps the async code 
> can be fixed for good, perhaps the feeds should be updated in separate 
> worker threads. I have described a couple of issues below, all of which 
> would disappear if threads were used instead. Consider this an offer to 
> work on these issues, whether threading is to be used or not.

I've updated the Straw wiki page with information about a prototype that 
uses threaded networking IO. 

http://live.gnome.org/Straw


These two issues are patched in the old IO code:

> 1. Loading feeds takes a very long time or times out

> 2. Limited download speed


Simply enabling threaded address lookups fixed the following except for 
IPv6:

> 3. Feed URIs with IP addresses don't work, IPv6 and /etc/hosts don't work


The threaded networking code supports https and file URIs but is currently 
missing support for proxies, passwords and a limit on the number of 
connections it makes simultaneously.


> 4. Feed parsing hangs everything

> I suppose changing from asyncore and ADNS to threading would require small 
> changes all over the code. However, the functionality is mainly in 
> LookupManager.py, URLFetch.py, and SummaryParser.py. I'd hope the 
> threading model wouldn't get too complex if a worker thread was spawned 
> for each feed update. The thread could independently perform hostname 
> lookup, content downloading and feed parsing before in the main 
> thread inserting the results into the user interface.

Of these, I only changed URLFetch.py so feed parsing remains as it is for 
now. MainloopManager got a function "schedule" to call functions in the 
main thread. This is for the worker threads to communicate back.

-- 
-- Trying to catch me? Just follow up my Electric Fingerprints
-- To help you: Tuukka Hastrup iki fi
                http://www.iki.fi/Tuukka.Hastrup/