benchmarking cairo/gtk themes

[Benjamin Otte <in7y118 public uni-hamburg de>]

Hi,

as some of you may be aware, I've recently had fun poking at the Gtk theme
torturer. This last weekend it became a bit more obsessive.
My heavily hacked repository is now available at
git://people.freedesktop.org/~company/git/gtk-theme-torturer

Examples of what you can do with it are here:
http://www.freedesktop.org/~company/stuff/cairo-benchmark-themes.png
and unless Benjamin deleted it already, here:
http://benjamin.sipsolutions.net/theme-perf/

The code requires libgda 1.2, since it stuffs all the results into an sql
database for easier processing.

What do you need to know, if you wanna play with it?
1) There's a script named test_themes, that tests all the installed
themes, collects the data and puts it into the default database.
2) The default database is an SQLite database named results.sql. To start
over, simply delete that file.
3) You can look at that file with your favourite database tool or ...
4) You can use the create-image tool to create those funky screenshots
seen above.

create-image needs some introduction. You call it like this:
./create-image -p $PRIMARY [-s $SECONDARY] [LIMITING-OPTIONS] out.png
It will then proceed to create the file out.png printing on the left side
the column you set as $PRIMARY and using as the header the column you set
as $SECONDARY. If you have set any LIMITING-OPTIONS, the generated image
will be limited to those results.

Still confused? Some examples:
http://www.freedesktop.org/~company/stuff/cairo-benchmark-themes.png
  was created like this:
  create-image -p theme -s test
http://benjamin.sipsolutions.net/theme-perf/Redmond.png
  create-image -p engines-version -s test --theme=Redmond
http://benjamin.sipsolutions.net/theme-perf/tests/vscrollbar-slider-big.png
  create-image -p engines-version -s theme --test=vscrollbar-slider-big

If there's any questions about this, don't hesitate to ask :)


So what did we learn from running all these weird tests?

First of all, I am pretty sure that the current way of profilingprovides
reliable results. The values keep pretty much the same every time someone
runs the test and the tests pretty much match the expectations.

Second, we've found lots of nasty things in the theme engines. ;)

And last but not least, cairo really is slow for non-trivial drawing
operations. They seem to be - as a rough first guess - 5 times slower than
they were before. See
http://benjamin.sipsolutions.net/theme-perf/tests/vscrollbar-slider-big.png
and compare the "none" values (that is gtk-engines 2.6 where no engine
used cairo) to the "2.7.7" values (which is current CVS, where most
engines use cairo).
On the good side, cairo is able to draw a single-color grey rectangle
pretty much as fast as raw X. :)


Benjamin