Re: Gtk+ unit tests (brainstorming)

[Federico Mena Quintero <federico ximian com>]

On Wed, 2006-10-25 at 17:52 +0200, Tim Janik wrote:

> while analysing the need for a testing framework and whether it makes sense
> for GLib and Gtk+ to depend on yet another package for the sole purpose of
> testing, i made/had the following observations/thoughts:

Wooo!  Thanks for working on this, Tim; it would be *fantastic* to unify
all the disparate tests that we have right now, as well as write new
ones.

> - for reasons also mentioned in the afformentioned blog entry it might
>    be a good idea for Gtk+ as well to split up tests into things that
>    can quickly be checked, thoroughly be checked but take long, and into
>    performance/benchmark tests.
>    these can be executed by make targets check, slowcheck and perf
>    respectively.

This is a good idea.  For testing the file chooser, it helped a lot to
write some "black box tests" which are essentially smoke tests.  They
just test the very basic functionality of the chooser, without going
into the whole test suite.  The black box tests helped me fix a bug with
many ramifications quickly, since at all times I could ensure that none
of the basic functionality was broken.

Performance tests should definitely be separate.

> - for time bound tasks it can also make sense to fork a test and after
>    a certain timeout, abort and fail the test.

Yes.  Hans Petter uses this approach for hiw Flow testing, and it's very
cool.

> - homogeneous or consistent test output might be desirable in some contexts.
>    so far, i've made the experience that for simple make check runs, the most
>    important things are that it's fast enough for people to run frequently
>    and that it succeeds.

One thing about this... it would be good to keep megawidgets in separate
tests, but then to have "make check" run all the tests for all
megawidgets.  That is, I'd like a single
"test-all-the-stuff-in-the-treeview" and a single
"test-all-the-stuff-in-the-file-chooser":  when coding in one, I
normally don't care about the other one.

So if all the test-* programs spit the same output, we can collect it
easily from "make check".

Cairo has a very nice testing framework.  We can surely steal ideas from
it.

> - GLib based test programs should never produce a "CRITICAL **:" or
>    "WARNING **:" message and succeed.

Definitely.  I have some simple code in autotestfilechooser.c to catch
this and fail the tests if we get warnings/criticals.

> 5- simple helper macros to indicate test start/progress/assertions/end.
>     (we've at least found these useful to have in Beast.)

For the file chooser tests, I just return a boolean from the test
functions, which obviously indicates if the test passed or not.  What
sort of macros are you envisioning?

> - for a specific widget type, test input/output conditions of all API
>    functions (only for valid use cases though)
> - similarly, test all input/output conditions of the Gdk API
> - try setting & getting all widget properties on all widgets over the full
>    value ranges (sparsely covered by means of random numbers for instance)
> - try setting & getting all container child properties analogously

I believe the OSDL tests do this already.  We should steal that code;
there's a *lot* of properties to be tested!

> - check layout algorithms by layouting a child widget that does nothing but
>    checking the coordinates it's layed out at.

Oooooh, nice idea.  This would be *really* nice even for non-widget
stuff, such as all the tricky layout going on in GtkTreeView and its
cell renderers.

> - generically query all key bindings of stock Gtk+ widgets, and activate them,
>    checking that no warnings/criticals are generated.

Nice.

> - for all widget types, create and destroy them in a loop to:
>    a) measure basic object setup performance
>    b) catch obvious leaks
>    (these would be slowcheck/perf tests)

Yeah.  GtkWidgetProfiler (in gtk+/perf) will help with this.  Manu
Cornet has improved it a lot for his theme engine torturer, so we should
use *that* version :)

> as always, feedback is appreciated, especially objections/concerns
> regarding the ideas outlined ;)

Again, thanks for working on this.  It will be reasurring to have a good
way to plug in new tests as we add new code to gtk+.  After the
infrastructure in autotestfilechooser was done, it was quite fun to add
new tests; before that, it was extremely cumbersome to start with a
blank sheet of paper every time I wanted to test something.

  Federico