Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.


What you mentioned is good information to start hunting. Was the CPU time related to IO wait at all? Always get accurate numbers before performance tuning. "Measure, measure, measure" or so the mantra goes.

Unfortunately, the symptom you see regarding IO will very likely change under a different processing model. If the problem is truly CPU bound then you will only be starting IO requests after you were done processing. This means valuable time is wasted while waiting for the pages to be loaded into the buffers. The code will just be blocking while this is going on.

What could be done easily is every time an item starts processing it could asynchronously begin loading the next image using gio. This means the kernel can start paging that file into the vfs cache while you are processing the image. This of course would still mean you are limited to a single processor doing the scaling. But if the problem is in fact cpu bound, that next image will almost always be loaded by time you finish the scale meaning you've maximized the processing potential per core.

To support multi-core, like it sounds like you want, a queue could be used to store the upcoming work items. A worker per core, for example, can get their next file from that queue. FWIW, I wrote a library, iris[1], built specifically for doing work like this while efficiently using threads with minimum lock-contention. It would allow for scaling up threads to the number of cores and back down when they are no longer needed.



-- Christian

Mark wrote:
On Fri, Aug 28, 2009 at 10:45 PM, Christian Hergert<chris dronelabs com>  wrote:
Have you profiled to see if the bottleneck is CPU bound?  If its IO bound,
you will only cause more contention by adding threading.

At minimum, using a thread (or async gio) to load files and another thread
that just thumbnails might be a good idea.


-- Christian

Mark wrote:
  is there anyone here that has experienced with this? is there
anyone i an work with on this? actually anyone that could help me
through this? i'm brand new in glib and new to programming in general.
I would like to give this a try but i'm afraid i can't do this alone.
So any information about this, help and thoughts would be nice.


I haven't done io profiling but i did calculate the disc usage for
those 1927 files. and every benchmark was WAY below what my hdd could
handle (Spinpoint F1 1TB hdd and it can handle roughly 100MB/sec).

As for the CPU. i did "profiling" there. i just opened up gnome's
system monitor tool and just watch it during the benchmark. That gave
the result of 1 core running at 100% and 3 other cores idle thus there
is a cpu bottleneck with thumbnail generating.

About your suggestion for loading in a thread and thumbnailing in
another thread.. don't you get in big problems if the pc is
thumbnailing faster then it's loading? Or is that a impossible


[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]