RFC: GLib testing framework
- From: Tim Janik <timj imendio com>
- To: Gtk+ Developers <gtk-devel-list gnome org>
- Subject: RFC: GLib testing framework
- Date: Thu, 1 Nov 2007 13:25:29 +0100 (CET)
hi All.
with lots of help from Sven Herzberg and others, i've designed a unit test
framework for GLib, which Sven and i plan to implement in the following weeks.
i'll post about testing framework/utilities for Gtk+ at a later point.
feedback is greatly appreciated.
QUICK READERS: two short test code examples are at the end of this mail;
the proposed test framework API is attached as testapi.h;
more elaborate example code is attached as testapi.c.
to present the main design requirements up front:
- the best place for a testing framework is libglib.so. that way, the framework
can reuse lots of logic already supplied by glib, and also all of glib can
benefit from it. i.e. tests under glib/glib/TESTDIR can't use libgobject.so
for example. if people want some GObject-ified API at some point (which
i'm not currently planning to implement), this can be put into libgobject.so
since it depends on libglib.so.
- tests should run on all systems (to ease portability testing and debugging),
not just selected developer machines. that way they can aid in bug reporting.
it's already a nuisance that many developers need an extra tree for docu
building, so a testing framework should not come with additional or optional
dependencies for basic functionality.
(this basically rules out optional use of external testing frameworks.)
also, external test frameworks easily become unmaintained over long periods
(sourceforge and google are full of examples) or at some point introduce
arbitrary dependencies. GLib/Gtk+ cannot afford that, putting the framework
into glib itself also ensures that it's properly maintained.
- the testing framework must be very simple and understandable to allow every
developer easy access to debugging of other peoples code.
E.g. having magic prefix and trailing macros around test functions, makes
reading, understanding and debugging of test code harder and should be
avoided. (E.g. Check has this, and we consider it a showstopper)
- by default, tests shouldn't be forked-off by for performance reasons and
because forks are hard to debug. (nevertheless, we'll introduce API to
allow explicit forking where test require this functionality.)
- the test framework must make it easy to run single tests isolated and shell
into gdb (this is particularly tricky when many tests are compiled into the
same binary to reduce link time.)
- unit tests need to be quick to be useful, otherwise people don't use them,
the testing framework needs to allow for that:
http://blogs.gnome.org/timj/2006/10/23/23102006-beast-and-unit-testing/trackback/
- linking tests or test suites should be fast. other projects (cairo, beast)
have made the experience that linking test functions into many isolated
programs or shared libraries wastes too much build time as the suites grow.
- there is also a place for performance tests and slow but thorough tests.
however to not interfere with frequent quick tests, those probably should
run as part of different Makefile rules or part of a different test mode.
- many more detailed requirements are listed in the summary of last years
testing framework discussion:
http://mail.gnome.org/archives/gtk-devel-list/2006-November/msg00039.html
one thing that became clear in that discussion is that many people out
there want full testing reports that list the number of failed tests
instead of abort-on-first-failure behavior, so the framework should allow
for this.
we've identified these main testing scenarios that should be well covered:
1) it should be easy and very quick to run a subset of the test suite, so
developers can run relevant tests before every commit. e.g. allow:
make -C gtk+/gtk/testtreeview test
2) something unknown broke or seems fishy. run the whole test suite as
fast as possible, to roughly spot the area of concern:
make -C gtk+/ test-report
3) we can't stand having a spare machine idling around.
run the whole test suite thoroughly over night (includeing lengthy brute
force tests, performance tests, everything):
make -C gtk+/ full-report
4) we're interested in performance changes, because a critical core component
changed, or we want to compare performance between two releases:
make -C gtk+/ perf-report
5) run the test suite as part of make distcheck. this is probably best
achieved by hooking up "make test" to "make check" and let automake
take care of the rest.
as said, these are *main* scenarios, sub variants like:
make -C gtk+/gtk/testtreeview perf-report
and easily running test cases in gdb should of course also be possible.
the API is for the most part designed according to established concepts found
in the xUnit testing framework family (JUnit, NUnit, RUnit) see [1],
which in turn is based on smalltalk unit testing concepts [2].
that is:
- tests (a test method) are grouped together with their fixture into
test case objects: GTestCase
- a test fixture consists of fixture data and setup and teardown methods to
establish the environment for the test functions.
we use fresh fixtures, i.e. fixtures are newly set up and torn down around
each test invokation to avoid dependencies between tests.
- test cases can be grouped into test suites (GTestSuite), to allow subsets
of the available tests to be run.
suites can be grouped into other suites as well.
- we provide an extended set of assertions for strings, ints and floats
that allow printing of assertion arguments upon failures to reduce
the need for debugging:
g_assert_cmpfloat (arg1, cmpop, arg2);
g_assert_cmpint (arg1, cmpop, arg2);
g_assert_cmpstr (arg1, cmpop, arg2);
used like:
g_assert_cmpstr ("foo", !=, "faa");
g_assert_cmpfloat (3.3, <, epsilon);
g_assert() is still available of course, but using the above variants,
assertion messages can be more elaborate, e.g.:
** testing.c:test_assertions(): assertion failed '(3.3 < epsilon)': (3.3 < 0.5)
- we'll provide a test binary wrapper that will take care of creating
machine readable test reports.
- we intend to implement test report post processing tools, e.g. to
generate html charts of performance test results.
to make up for the lack of introspection support in C (like Java/JUnit have it)
we make use of preprocessor macros and a hierarchical naming scheme:
- test cases and test suites have to be given names, so individual tests
can be adressed on the command line (e.g. to debug a specific test within
a huge test binary). this way, test cases or suites within a testing
binary can be referred to via pathnames:
/gtksuite/treeviewsuite/columnsuite/test text cell renderer
/gtksuite/windowsuite/test window title
etc.
- test suites can be created implicitely by registering a new test case
with it's full pathname. this avoids fiddling with lots of case/suite
object references just to establish hierarchical grouping, e.g.:
g_test_add_func ("/misc/assertions", test_assertions);
registers the test function test_assertions as test case "assertions"
and implicitely creates the test suite "misc".
- assertions make use of __FILE__, __LINE__, __PRETTY_FUNCTION__ to provide
elaborate error messages.
- in the spirit of g_new() and g_slice_new(), g_test_add() accepts a fixture
type argument, to provide type safety for test functions that take a fixture
data argument.
- planned test runner interface:
> gtester --help
Usage: gtester [OPTIONS] testbinary [testbinaries...]
Run all test binaries and produce a report file gtester.log.
Options:
-l List all available testpaths of a testbinary.
-p <testpath> Run test suites and cases below <testpath>.
-m <mode> Run tests in mode <mode>. Possible modes:
perf - run performance measurements,
slow - run tests that need lots of time,
quick - run tests quickly (default).
-o <logfile> Save testing log as <logfile>.
-k, --keep-going Do not abort upon first test failure.
-q, --quiet Quiet, suppres output from test binaries.
--seed <rand> Random seed, specify this to force repeatable
test results using g_test_rand_range().
- note that similar to gtester, testbinaries will themselves also support
-l, -m, -p and --seed, so individual tests can be debugged in gdb without
starting gtester.
- the testbinary and gtester
since this email is quite long already, for the rest, i'll just
introduce the shortest possible test program and an extended example:
==============shortest-test-program==============
static void
test_number_assertion (void)
{
g_assert_cmpint (4, ==, 2 + 2);
}
int
main (int argc,
char *argv[])
{
g_test_init (&argc, &argv, NULL);
g_test_add_func ("/misc/number assertion", test_number_assertion);
return g_test_run();
}
==============
the following is a test program that maintains a test specific fixture
around running a test case:
==============fixture-test-program==============
typedef struct {
gchar *string;
} Stringtest;
static void
stringtest_setup (Stringtest *fix)
{
fix->string = g_strdup ("foo");
}
static void
stringtest_test (Stringtest *fix)
{
g_assert_cmpstr (fix->string, ==, "foo");
}
static void
stringtest_teardown (Stringtest *fix)
{
g_free (fix->string);
}
int
main (int argc,
char *argv[])
{
g_test_init (&argc, &argv, NULL);
g_test_add ("/misc/stringtest", Stringtest, // <- fixture type
stringtest_setup, stringtest_test, stringtest_teardown);
return g_test_run();
}
==============
conceptually, the Stringtest structure and the stringtest_setup, stringtest_test
and stringtest_teardown functions (plus assertion macros if you may), equate the
TestCase object introduced in JUnit [3] and many other frameworks. the proposed
type safe version of g_test_add() is the shortest and most convenient notation
i could come up with to implement the concept in plain C.
apart from library dependency issues, this _could_ be implemented as GObject or
a GInterface, however without providing significant benefits (a GType ID
isn't needed here, and we'd not want to do cross library boundary inheritance
of test case objects).
i've attached two files:
testapi.h intended public API additions to implement in libglib.so
(except for g_test_queue_unref).
testapi.c example code, showing off how the API from testapi.h is
intended to be used. note that main2() reimplements the
functionality from main(), albeit avoiding the testpath
convenience API. this demonstrates the maintenance savings
by using the testpath API.
given the above scope description and terminology introductions, the API in
testapi.h should be self revealing. where it isn't, please come back to me
with your questions.
[1] "xUnit Test Patterns: Refactoring Test Code" by Gerard Meszaros.
book: http://www.amazon.com/dp/0131495054
wbesite: http://xunitpatterns.com/
(the website is really good as it described many patterns
introduced in the book)
[2] "Simple Smalltalk Testing: With Patterns" by Kent Beck.
link: http://www.xprogramming.com/testfram.htm
[3] "JUnit Test Infected: Programmers Love Writing Tests" by Eric Gamma and
Kent Beck
link: http://junit.sourceforge.net/doc/testinfected/testing.htm
(Excellent introduction to automated unit testing)
---
ciaoTJ#include <glib.h>
typedef struct GTestCase GTestCase;
typedef struct GTestSuite GTestSuite;
/* assertion API */
#define g_assert_cmpstr(s1, cmp, s2) do { if (g_strcmp0 (s1, s2) cmp 0) ; else \
g_assertion_message_cmpstr (G_LOG_DOMAIN, __FILE__, __LINE__, G_STRFUNC, \
#s1 " " #cmp " " #s2, s1, #cmp, s2); } while (0)
#define g_assert_cmpint(n1, cmp, n2) do { if (n1 cmp n2) ; else \
g_assertion_message_cmpnum (G_LOG_DOMAIN, __FILE__, __LINE__, G_STRFUNC, \
#n1 " " #cmp " " #n2, n1, #cmp, n2, 'i'); } while (0)
#define g_assert_cmpfloat(n1,cmp,n2) do { if (n1 cmp n2) ; else \
g_assertion_message_cmpnum (G_LOG_DOMAIN, __FILE__, __LINE__, G_STRFUNC, \
#n1 " " #cmp " " #n2, n1, #cmp, n2, 'f'); } while (0)
int g_strcmp0 (const char *str1,
const char *str2);
// g_assert(condition) /*...*/
// g_assert_not_reached() /*...*/
/* report performance results */
void g_test_minimized_result (double minimized_quantity,
const char *format,
...) G_GNUC_PRINTF (2, 3);
void g_test_maximized_result (double maximized_quantity,
const char *format,
...) G_GNUC_PRINTF (2, 3);
/* guards used around forked tests */
guint g_test_trap_fork ();
void g_test_trap_assert_passed (void);
void g_test_trap_assert_failed (void);
void g_test_trap_assert_stdout (const char *stdout_pattern);
void g_test_trap_assert_stderr (const char *stderr_pattern);
/* initialize testing framework */
void g_test_init (int *argc,
char ***argv,
...);
/* run all tests under toplevel suite (path: /) */
int g_test_run (void);
/* hook up a simple test function under test path */
void g_test_add_func (const char *testpath,
void (*test_func) (void));
/* hook up a test with fixture under test path */
#define g_test_add(testpath, Fixture, fsetup, ftest, fteardown) \
((void (*) (const char*, \
gsize, \
void (*) (Fixture*), \
void (*) (Fixture*), \
void (*) (Fixture*))) \
(void*) g_test_add_vtable) \
(testpath, sizeof (Fixture), fsetup, ftest, fteardown)
/* measure test timings */
void g_test_timer_start (void);
double g_test_timer_elapsed (void); // elapsed seconds
double g_test_timer_last (void); // repeat last elapsed() result
/* automatically g_free or g_object_unref upon teardown */
void g_test_queue_free (gpointer gfree_pointer);
void g_test_queue_unref (gpointer gobjectunref_pointer);
/* provide seed-able random numbers for tests */
long double g_test_rand_range (long double range_start,
long double range_end);
/* semi-internal API */
GTestCase* g_test_create_case (const char *test_name,
gsize data_size,
void (*data_setup) (void),
void (*data_test) (void),
void (*data_teardown) (void));
GTestSuite* g_test_create_suite (const char *suite_name);
GTestSuite* g_test_get_root (void);
void g_test_suite_add (GTestSuite *suite,
GTestCase *test_case);
void g_test_suite_add_suite (GTestSuite *suite,
GTestSuite *nestedsuite);
int g_test_run_suite (GTestSuite *suite);
/* internal ABI */
void g_assertion_message (const char *domain,
const char *file,
int line,
const char *func,
const char *message);
void g_assertion_message_expr (const char *domain,
const char *file,
int line,
const char *func,
const char *expr);
void g_assertion_message_cmpstr (const char *domain,
const char *file,
int line,
const char *func,
const char *expr,
const char *arg1,
const char *cmp,
const char *arg2);
void g_assertion_message_cmpnum (const char *domain,
const char *file,
int line,
const char *func,
const char *expr,
long double arg1,
const char *cmp,
long double arg2,
char numtype);
void g_test_add_vtable (const char *testpath,
gsize data_size,
void (*data_setup) (void),
void (*data_test) (void),
void (*data_teardown) (void));
/* GLib testing framework examples
* Copyright (C) 2007 Tim Janik
*
* This work is provided "as is"; redistribution and modification
* in whole or in part, in any medium, physical or electronic is
* permitted without restriction.
*
* This work is distributed in the hope that it will be useful,
* but WITHOUT ANY WARRANTY; without even the implied warranty of
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
*
* In no event shall the authors or contributors be liable for any
* direct, indirect, incidental, special, exemplary, or consequential
* damages (including, but not limited to, procurement of substitute
* goods or services; loss of use, data, or profits; or business
* interruption) however caused and on any theory of liability, whether
* in contract, strict liability, or tort (including negligence or
* otherwise) arising in any way out of the use of this software, even
* if advised of the possibility of such damage.
*/
#include "testapi.h"
/* test assertion variants */
static void
test_assertions (void)
{
g_assert_cmpint (1, >, 0);
g_assert_cmpint (2, ==, 2);
g_assert_cmpfloat (3.3, !=, 3.4);
g_assert_cmpfloat (7, <=, 3 + 4);
g_assert (TRUE);
g_assert_cmpstr ("foo", ==, "foo");
g_assert_cmpstr ("foo", !=, "faa");
gchar *fuu = g_strdup_printf ("f%s", "uu");
g_test_queue_free (fuu);
g_assert_cmpstr ("foo", !=, fuu);
g_assert_cmpstr ("fuu", ==, fuu);
}
/* quickly test a wide range with randomization */
static void
test_randomized (void)
{
guint i;
/* test a few randomized integers instead of 2^64 */
for (i = 0; i < 100; i++)
{
guint64 num = (guint64) g_test_rand_range (0, 18446744073709551615.0);
char *numstr = g_strdup_printf ("%llu", num); // forward conversion
guint64 snum = g_ascii_strtoull (numstr, NULL, 0); // backward conversion
g_assert_cmpint (num, ==, snum);
g_free (numstr);
}
}
/* do measurements and report performance */
static void
test_performance (void)
{
guint n_calls;
/* seconds should be minimized */
g_test_timer_start();
for (n_calls = 0; n_calls < 11; n_calls++)
g_usleep (500);
double secs = g_test_timer_elapsed();
g_test_minimized_result (secs, "bogus loop consumed %f seconds", secs);
/* calls should be maximized */
g_test_timer_start();
for (n_calls = 0; g_test_timer_elapsed() < 10; n_calls++)
g_free (g_strdup ("foo"));
double calls_per_second = n_calls / g_test_timer_last();
g_test_maximized_result (calls_per_second, "string copies per second: %f", calls_per_second);
}
/* fork out for a failing test */
static void
test_fork_fail (void)
{
if (g_test_trap_fork())
{
g_assert_not_reached();
}
g_test_trap_assert_failed();
}
/* fork out to assert stdout and stderr patterns */
static void
test_fork_patterns (void)
{
if (g_test_trap_fork())
{
g_print ("some stdout text: somagic17\n");
g_printerr ("some stderr text: semagic43\n");
}
g_test_trap_assert_passed();
g_test_trap_assert_stdout ("*somagic17*");
g_test_trap_assert_stderr ("*semagic43*");
}
/* run a test with fixture setup and teardown */
typedef struct {
guint seed;
guint prime;
gchar *msg;
} Fixturetest;
static void
fixturetest_setup (Fixturetest *fix)
{
fix->seed = 18;
fix->prime = 19;
fix->msg = g_strdup_printf ("%d", fix->prime);
}
static void
fixturetest_test (Fixturetest *fix)
{
guint prime = g_spaced_primes_closest (fix->seed);
g_assert_cmpint (prime, ==, fix->prime);
prime = g_ascii_strtoull (fix->msg, NULL, 0);
g_assert_cmpint (prime, ==, fix->prime);
}
static void
fixturetest_teardown (Fixturetest *fix)
{
g_free (fix->msg);
}
/* register and run all tests */
int
main (int argc,
char *argv[])
{
g_test_init (&argc, &argv, NULL);
g_test_add_func ("/misc/assertions", test_assertions);
g_test_add_func ("/misc/randomized", test_randomized);
g_test_add_func ("/misc/performance", test_performance);
g_test_add_func ("/fork/failing test", test_fork_fail);
g_test_add_func ("/fork/output patterns", test_fork_patterns);
g_test_add ("/misc/primetoul", Fixturetest, fixturetest_setup, fixturetest_test, fixturetest_teardown);
return g_test_run();
}
/* main() rewritten with same tests using manual registration
* instead of the testpath based conveninence API
*/
int
main2 (int argc,
char *argv[])
{
GTestCase *tc;
g_test_init (&argc, &argv, NULL);
GTestSuite *rootsuite = g_test_create_suite ("root");
GTestSuite *miscsuite = g_test_create_suite ("misc");
g_test_suite_add_suite (rootsuite, miscsuite);
GTestSuite *forksuite = g_test_create_suite ("fork");
g_test_suite_add_suite (rootsuite, forksuite);
tc = g_test_create_case ("assertions", 0, NULL, test_assertions, NULL);
g_test_suite_add (miscsuite, tc);
tc = g_test_create_case ("randomized", 0, NULL, test_randomized, NULL);
g_test_suite_add (miscsuite, tc);
tc = g_test_create_case ("performance", 0, NULL, test_performance, NULL);
g_test_suite_add (miscsuite, tc);
tc = g_test_create_case ("failing test", 0, NULL, test_fork_fail, NULL);
g_test_suite_add (forksuite, tc);
tc = g_test_create_case ("output patterns", 0, NULL, test_fork_patterns, NULL);
g_test_suite_add (forksuite, tc);
tc = g_test_create_case ("primetoul", sizeof (Fixturetest),
(void(*)(void)) fixturetest_setup,
(void(*)(void)) fixturetest_test,
(void(*)(void)) fixturetest_teardown);
g_test_suite_add (miscsuite, tc);
return g_test_run_suite (rootsuite);
}
static void
test_number_assertion (void)
{
g_assert_cmpint (4, ==, 2 + 2);
}
int
main3 (int argc,
char *argv[])
{
g_test_init (&argc, &argv, NULL);
g_test_add_func ("/misc/number assertion", test_number_assertion);
return g_test_run();
}
typedef struct {
gchar *string;
} Stringtest;
static void
stringtest_setup (Stringtest *fix)
{
fix->string = g_strdup ("foo");
}
static void
stringtest_test (Stringtest *fix)
{
g_assert_cmpstr (fix->string, ==, "foo");
}
static void
stringtest_teardown (Stringtest *fix)
{
g_free (fix->string);
}
int
main4 (int argc,
char *argv[])
{
g_test_init (&argc, &argv, NULL);
g_test_add ("/misc/stringtest", Stringtest, // <- fixture type
stringtest_setup, stringtest_test, stringtest_teardown);
return g_test_run();
}
// gcc -O2 -Wall `pkg-config --cflags glib-2.0` -c testapi.c
[
Date Prev][
Date Next] [
Thread Prev][
Thread Next]
[
Thread Index]
[
Date Index]
[
Author Index]
