Re: Make'em suffer!

[Eero Tamminen <ext-eero tamminen nokia com>]

Hi,

ext Manu Cornet wrote:
> And what do you think about displaying the average time values instead
> of the total time values, so that one could directly compare results
> obtained with different numbers of iterations?

IMHO average values are not so useful, a single bad value can
change it a lot.  Therefore I would prefer seeing:
- median: the time the test-case "normally" takes
- minimum: what is fastest possible time
- maximum: with above, the range between which values fluctuate
  (potential effect of rest of the system like context switching,
  X server internal scheduling etc)
- deviation: how reliable is the median (or average) value and the test
  producing them (tests should be run on idle machine, test code
  should not have random elements)

If deviation is too large (because there were not enough iterations
or they fluctuated too much), the results are worthless and no time
should be spent trying to analyze them.

To get these, it would be better if the tester and post-processing parts
were separated.  I mean the tester should produce raw values and 
post-processor then decides what it calculates from them and how
it presents them.


	- Eero