Re: Gtk+ unit tests (brainstorming)

On Wed, 25 Oct 2006 12:40:27 -0500, Federico Mena Quintero wrote:
> Cairo has a very nice testing framework.  We can surely steal ideas from
> it.

I don't know that cairo's "make check" stuff has much worth
stealing directly. But I will share some experiences at least.

There are some things I really don't like in cairo's "make check"
suite right now:

1. We are exporting a (very few, very tiny) test-specific functions in
   the library just so the test suite can get at it. These are
   explicitly not "part of" the API, but they do exist. They're mostly
   on the order of flipping an internal bit to change the behavior to
   exercise some specific piece of the library. But this approach
   really doesn't scale. It would probably be better to do this kind
   of thing in a "test build" as Havoc recently mentioned, (cairo
   currently doesn't have such a build).

2. The tests take forever to link. Each test right now is a separate
   program. I chose this originally so that I could easily execute
   individual tests, (something I still do regularly and still
   require). The problem is that with any change to the library, "make
   check" goes through the horrifically slow process of using libtool
   to re-link the hundred or so programs. One idea that's been floated
   to fix this is something like a single test program that links with
   the library, and then dlopens each test module (or something like
   this). Nothing like that has been implemented yet.

We do have log files and HTML output, both of which are very useful,
but the format of each is very ad-hoc and nothing that I would
recommend anybody copying. That stuff is also very strongly oriented
toward tests which can be verified by "visual diff" of two
images---this works well for cairo, of course, but wouldn't generalize

We also did some work on hooking gcov up to our makefiles and getting
lovely HTML reports of how much coverage their is in the test
suite. As Havoc points out, this is really important, (but sadly, we
haven't yet made a good effort at using this data and writing directed
tests to improve the coverage). I'm not even 100% sure the gcov stuff
in the Makefiles works right now, but it might still be useful for
someone looking to how to start that kind of thing, (though the gcov
documentation might be even better).

> > - for all widget types, create and destroy them in a loop to:
> >    a) measure basic object setup performance
> >    b) catch obvious leaks
> >    (these would be slowcheck/perf tests)
> Yeah.  GtkWidgetProfiler (in gtk+/perf) will help with this.  Manu
> Cornet has improved it a lot for his theme engine torturer, so we should
> use *that* version :)

Something that is worth stealing is some of the work I've been doing
in "make perf" in cairo. I've been putting a lot of effort into
getting the most reliable numbers out of running performance tests,
(and doing it is quickly as possible yet). I started with stuff that
was in Manu's torturer with revamping from Benjamin Otte and I've
further improved it from there.

Some of the useful stuff is things such as using CPU performance
counters for measuring "time", (which of course I didn't write, but
just got from liboil---thanks David!), and then some basic statistical
analysis---such as reporting the average and standard deviation over
many short runs timed individually, rather than just timing many runs
as a whole, (which gives the same information as the average, but
without any indication of how stable the results are from one to the

The statistical stuff could still be improved, (as I described in a
recent post to performance-list), but I think it is a reasonable
starting point.

Oh, and my code also takes care to do things like ensuring that the X
server has actually finished drawing what you asked it to, (I think
GtkWidgetProfiler does that as well---but perhaps with a different
approach). My stuff uses a single-pixel XGetImage just before starting
or stopping the timer.

> > as always, feedback is appreciated, especially objections/concerns
> > regarding the ideas outlined ;)

Excellent stuff. Testing is really important and often
neglected. Never forget the truth that Keith Packard likes to share

	Untested code == Broken code


Attachment: pgpafVpiAJYRj.pgp
Description: PGP signature

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]