Re: Speeding up thumbnail generation (like multi threaded). Thoughts please.

[Mark <markg85 gmail com>]

On Tue, Sep 29, 2009 at 9:46 PM, Dr. Michael J. Chudobiak
<mjc avtechpulse com> wrote:
> On 09/29/2009 12:00 PM, Mark wrote:
>>
>> Well, the benchmarks ran above are resizing 1927 wallpaper sized
>> images to a max width or height of 200 and that function clearly
>> loses.
>> The solution to this is extremely simple. Grab this file:
>>
>> http://git.gnome.org/cgit/gnome-desktop/tree/libgnome-desktop/gnome-desktop-thumbnail.c
>> and replace the line:
>>       scaled = gnome_desktop_thumbnail_scale_down_pixbuf (pixbuf,
>>                                                  floor (width * scale +
>> 0.5),
>>                                                  floor (height * scale +
>> 0.5));
>> with a function that is (a lot) faster.
>
>
> Mark,
>
> Have you tried benchmarking changes to Nautilus, rather than your own
> benchmark program? (That was the original point of this thread, right? I
> forget now...)
>
> My understanding of gnome-desktop-thumbnail.c is that the "problem function"
> gnome_desktop_thumbnail_scale_down_pixbuf is only called if an external
> script returns an over-sized thumbnail (e.g., like a video file returning
> the first frame). In most cases (such as jpegs and pngs)
> _gdk_pixbuf_new_from_uri_at_scale will be used to return a correctly-sized
> thumbnail instead, so the performance of
> gnome_desktop_thumbnail_scale_down_pixbuf is a non-issue.
>
> _gdk_pixbuf_new_from_uri_at_scale should use the jpeg scaling trick (in the
> jpeg loader), followed by the gdk scaling routines (in the generic pixbuf io
> file).
>
> Also, if you're playing with benchmarks, you might test to see if
> gnome_desktop_thumbnail_scale_down_pixbuf is superior when downscaling by
> huge factors (say, from 16000x16000 to 16x16). The code comments certainly
> imply that. The gdk down-scaling routines can blow up and freeze the system
> under these conditions.
>
> Anyway, great to see someone looking at the issues... these are just my
> thoughts.
>
>
> - Mike
>

hehe that was the idea indeed ^_^ and i will continue with that.
I will test the large factors tomorrow.

For now i'm happy with 100% cpu usage on all my cores (4).
with the code posted in my previous message i only had 70% cpu usage
so there was a bottleneck and it wasn't the HDD nor the CPU.
Now that's fixed with giving each thread more then one (5 actually)
images of it's own before locking and refilling the queue of 5 so now
there is 100% cpu usage in the multi threaded benchmark.

http://codepad.org/PKnp69qW

be sure to adjust the same things as in my last post and also look at:
numOfThreads variable which is set at 4 right now for my case but
anyone that's going to test this might need more or less depending on
your cpu.

Guess how long 1927 images took to get thumbnailed now ^_^

22 seconds! and since the SPU is the bottleneck now this could
probably be faster. To the one earlier in this thread that said disc
io might prevent much faster thumbnail generation.. i hereby present
you with the proof that it can. 22 seconds.