Re: Updating GUI during long operation

[Kip Warner <kip thevertigo com>]

On Thu, 2013-04-25 at 22:18 -0700, Simon Feltman wrote:
> A brief look at the source and it seems there is nothing calling
> Gdk.threads_init which I think is needed. 

Hey Simon. I'm actually calling GObject.threads_init() over in Main.py.
I've been told by others that this is enough, whereas others have said
your call is more appropriate. I think what I'll do is just play it safe
and use both as the faq suggests in the order given:

  <https://live.gnome.org/PyGObject/IntrospectionPorting#Threads>

> Similarly, use
> Gdk.threads_add_idle instead of GObject.idle_add for scheduling GUI updates
> from worker threads. The necessity of these is still somewhat unclear to me
> at the moment though.

I'll give that a try. I'm not sure what the difference is either, but
the link above seems to recommend it too so it can't hurt to try.

> Enough questions have come up recently with Python and GTK+ threading that
> I've added a wiki page here:
> https://live.gnome.org/PyGObject/Threading
>
> This is very much a work in progress and I appreciate updates or validation
> if you find any of this useful.

Thanks a lot and I'll definitely give it a look.

> The threading model in Python is pretty bad and you might consider a
> different technique (please read up on the Python GIL if you don't already
> know how threading in Python works). If the operation is IO bound, look
> into using async IO with Gio. If it is CPU bound, take a look at the Python
> "processing" module or even numpy for certain things. These techniques will
> make the app much more responsive because other threads won't be blocked by
> the GIL.

In my case, my worker thread is actually both CPU and I/O intensive as
it generates md5 checksums of large files on disk. I could try with a
non-blocking approach, but it might mean re-factoring a fair amount of
code.

> A surprising example is as follows:
>
> > > > sum(range(1000000))
>
> This seems to completely block any other Python threads for the entirety of
> the operation. I think the reason is because the GIL is held the whole time
> and Python is not giving time to other threads for the duration of the call
> (this includes GTK+ update callbacks written in Python).

I think so too.

> If you split the operation up, you can gain interactivity with other
> threads because Python will interleave statement execution with giving time
> to other threads (at the cost of performance).
>
> > > > total = 0
> > > > for i in range(0, 1000000, 100):
> ...     total += sum(range(i, i + 100))

Yes. That's a good observation. One should note that it's not guaranteed
because the implementation could always change one day, but I understand
what you are saying.

> In this sense Python is really always single threaded in regards to running
> pure Python code in which none of the statements ever release the GIL (IO
> related functions should be fine though). And if there are not any calls
> which release the GIL, Python threads might just be adding overhead and
> confusion.

Yeah, I know what you mean. I once wrote a VM and I realized that
probably most of them for most languages are ultimately single threaded.

> Finally, the same example using numpy will allow the same operation to run
> completely unblocked by the GIL in its own thread, so you keep full
> interactivity with actual concurrent processing of the threads (and the
> operation is about 10 times faster than the first example).
>
> > > > numpy.sum(numpy.arange(1000000))

I'll definitely earmark numpy. It could be very useful in the future.
Thanks Simon.

-- 
Kip Warner -- Software Engineer
OpenPGP encrypted/signed mail preferred
http://www.thevertigo.com