Re: Better coverage from cairo performance suite (and some results)

[Federico Mena Quintero <federico ximian com>]

On Thu, 2006-10-05 at 11:32 -0700, Carl Worth wrote:
> [I hope the performance-list readership doesn't mind me cross-posting
> performance-related cairo threads like this. Let me know.]

Not at all!  This is really interesting stuff.  It's great to see that
Cairo is getting love with a plan.

That should be our new motto.  Love with a plan.

> It would definitely help to have some visualization tools be built up
> around this, (there are a myriad of different useful plots that could
> be formed by varying a single variable---backed, operation, source
> type, or operator). Vladimir even made a start at an SVG-producing
> script earlier that could form the basis here.

Are those SVGs available anywhere?

I would make a simple plot like this:

- One plot per backend; one point cloud in each plot.  You can probably
put related backends in the same plot, such as image-rgba and image-rgb.

- The horizontal axis is the number of your performance run (1 with the
original code, 2 after the first change, 3 after the second change,
etc.).

- The vertical axis is time.

- Points are colored based on the source and operation:  solid-rgb-over
is black, solid-rgba-over is green, etc.

I guess that's a good way to condense a lot of information in a small
space, while still giving you a good overview of the big picture.  You
probably don't need individual plots for each run unless you are digging
into a particular optimization.

You can then make your brain do "black and green points should be close
together, since they are essentially a fancy memcpy()", etc.

[  0]    image-rgba               paint_solid_rgb_over-64      0.008  0.03%   100
[  1]    image-rgba             paint_solid_rgb_source-64      0.008  0.02%   100
[  2]    image-rgba              paint_solid_rgba_over-64      0.040  0.59%   100
[  3]    image-rgba            paint_solid_rgba_source-64      0.008  0.04%   100
[  4]    image-rgba               paint_image_rgb_over-64      0.028  0.06%   100
[  5]    image-rgba             paint_image_rgb_source-64      0.027  0.11%   100

Is [0] really 100 iterations of 0.008 milliseconds each?

One *huge* problem I had in the initial Pango benchmarks is that they
had way too much noise from the rest of the system.  I was getting
wildly different numbers each time.  The problem is that the test was
running only for a few seconds; simply increasing the number of
iterations so that the whole test suite runs for a few minutes (instead
of less than 10 seconds) gave me very stable numbers.

How long does the test suite take to run on your machine?  If it takes
seconds rather than minutes, I'd be somewhat suspicious of the numbers
it gives you.  If so, just increase the number of iterations.  You may
not get accurate timings if num_iters*time_per_test is very close to the
kernel's HZ.

This is *excellent* stuff, Carl.  Keep us posted :)

  Federico