Comments on glib testing framework

[Tommi Komulainen <tommi komulainen nokia com>]

Hi,

Me and Xan Lopez just tried out using the glib testing framework in a
separate module (in our case, hildon.) Following are notes from our
experiment.

Overall it's a fairly decent testing framework for API level testing.
However writing test cases for user interactions, especially on
composite black box widgets can be a bit cumbersome.


Documentation is missing (highlevel and API reference.) There was no
guide for how to set up your own project to use the glib testing
framework (I'll send a quick guide separately) or higher level overview
of setup/teardown, available asserts, gtk helpers, running, parameters,
etc. Given the testing framework is still young this is understandable.

For custom objects/widgets it would be nice to be able to have some
generic tests for things such as:
      * default object properties; create an object, verify all
        properties have default values
      * interfaces; for all interfaces provided by an object, if
        possible verify the API contract holds
              * gtk_editable_set_editable(FALSE) + cut() shouldn't
                delete text
              * gtk_container_add(parent, child) => g_assert
                (child->parent == parent)
              * ...

make -k test probably shouldn't abort gtester on first failing assertion

glib/gtestutils.h references g_object_unref() which requires gobject.

It would be nice to have g_assert_cmpenum and g_assert_cmpflags to
pretty print the name of enum/flags rather than plain numerical value
(See example code for cmpenum attached, needs to be in gobject.) For
example:

        assertion failed (mode == GTK_SIZE_GROUP_HORIZONTAL): (3 (GTK_SIZE_GROUP_BOTH) == 1 (GTK_SIZE_GROUP_HORIZONTAL))


While we are on this topic, does it make sense to add something like
g_enum_get_name (GType, gint)? The g_enum_* functions can be used to do
this, but are a bit cumbersome to use.

gtk_test_create_widget is nice, but unfortunately won't be usable in the
cases where a widget does more than a g_object_new call in its _new
function (GtkSpinButton, text variants of comboboxes, GtkAspectFrame,
any other third party widget, ...). Maybe not too important, but worth
noting.

gtk_test_teardown. See bug #507256

There are two technologies that could help in the creation of testcases:
      * A way to record a series of events and replay them at will.
        (Already exists?)
      * A way to record either the static aspect (screenshot) or an
        extended period of time (video, or series of images matched with
        a checkpoint) and compare it against a pre-defined image/video.

Combining both features some pretty complex tests could be defined very
easily; for example: Build a menu, populate it, replay a series of
events and check that the menu aspect matches the data previously
defined. A simple UI could be developed to create this kind of graphical
unit testing.

The downside of this approach is that it would be theme/font size/etc
specific.


-- 
Tommi Komulainen                            <tommi komulainen nokia com>

#include <gtk/gtk.h>

#define g_assert_cmpenum(n1,cmp,n2,t)   do { gint64 __n1 = (n1), __n2 = (n2); \
                                             if (__n1 cmp __n2) ; else \
                                               g_assertion_message_cmpenum (G_LOG_DOMAIN, __FILE__, __LINE__, G_STRFUNC, \
                                               #n1 " " #cmp " " #n2, (t), __n1, #cmp, __n2); } while (0)

void
g_assertion_message_cmpenum (const char     *domain,
                             const char     *file,
                             int             line,
                             const char     *func,
                             const char     *expr,
                             GType           enum_type,
                             gint            arg1,
                             const char     *cmp,
                             gint            arg2)
{
  GEnumClass *enum_class;
  GEnumValue *val1, *val2;
  char *a1, *a2, *s;

  enum_class = g_type_class_ref (enum_type);
  g_assert (G_IS_ENUM_CLASS (enum_class));

  val1 = g_enum_get_value (enum_class, arg1);
  val2 = g_enum_get_value (enum_class, arg2);

  a1 = val1 ? g_strdup_printf ("%d (%s)", arg1, val1->value_name) : g_strdup_printf ("%d", arg1);
  a2 = val2 ? g_strdup_printf ("%d (%s)", arg2, val2->value_name) : g_strdup_printf ("%d", arg2);

  s = g_strdup_printf ("assertion failed (%s): (%s %s %s)", expr, a1, cmp, a2);
  g_free (a1);
  g_free (a2);

  g_assertion_message (domain, file, line, func, s);
  g_free (s);

  g_type_class_unref (enum_class);
}

int
main (int argc, char **argv)
{
  gint mode;

  g_type_init ();

  mode = GTK_SIZE_GROUP_HORIZONTAL;

  g_assert_cmpenum (mode, ==, GTK_SIZE_GROUP_BOTH, GTK_TYPE_SIZE_GROUP_MODE);

  return 0;
}