Re: Test Framework Mini Tutorial

[Kouhei Sutou <kou cozmixng org>]

Hi,

I'm developing Cutter(*) that is an unit testing framework
for C to write tests more easily and simply.
(*): http://cutter.sf.net/

In <Pine LNX 4 62 0712131809200 16734 master birnet private>
  "Test Framework Mini Tutorial" on Thu, 13 Dec 2007 18:14:57 +0100 (CET),
  Tim Janik <timj imendio com> wrote:

> The main goals in writing tests are:
> - In example (test) driven development (EDD or TDD), an example or test
>    program is written first, and then the newly used API gets implemented.
>    This ensures early testability and gives programmers early feedback on
>    their implementation.
> - For untested legacy code, new tests are written to get about to be changed
>    code portions under automated tests to catch behavior changes as code is
>    changed.
> - Most tests are written to ensure basic functionality checks of the code
>    area in change. Tests usually cannot perform comprehensive checks, but
>    can check for specific known to be tricky corner cases or often test a
>    representative subset of an API.
> 
> In general, working on code that is sufficiently under automated tests
> makes programmers feel much more confident about their changes and helps
> to spot errors shortly after introduction. So well tested code bases
> tend to increase productivity and fun in working with the code.

I agree with you.

And I think that there is an another important
thing. Writing tests shouldn't bother us. It's better that
we can write tests with fun. If writing tests bothers us, we
will not write tests. :<


> 3) Implement the actual test function, possibly taking a fixture argument.
>     Tests should try to avoid duplicating logic or tests and often consist
>     of a series of calls and checks to use a component and verify its
>     behavior, e.g.:
>       string = g_string_new ("first");
>       g_assert_cmpstr (string->str, ==, "first");
>       g_string_append (string, "last");
>       g_assert_cmpstr (string->str, ==, "firstlast");

If this assertion fails, we will get output like the
following:

  /teststring: **
  ** ERROR:(teststring.c:123):test_string: assertion failed (string->str, ==, "firstlast"): ("first last" == "firstlast")
  zsh: abort      ./teststring


I think the output '("first last" == "firstlast")' is too
difficult to find difference between an actual result
("first last") and an expected result ("firstlast") by our
eye diff.

In Cutter, the test will be written like the following(*):

  string = g_string_new ("first");
  cut_assert_equal_string ("first", string->str);
  g_string_append (string, "last");
  cut_assert_equal_string ("firstlast", string->str);

(*) An expected result is written first because it's
    important rather than actual test target code
    (string->str). In reading tests, we will be interested
    in an expected result. e.g. an expected result is
    "first" in the first assertion. An expected result is
    "firstlast" after "last" is appended. But an actual test
    target code (string->str) isn't changed.

And outputs like the following:

  ...
  1) Failure: test_string
  <"firstlast" == string->str>
  expected: <firstlast>
   but was: <first last>
  ./test-string.c:123: test_string()
  ...

Cutter formats an expected result and an actual result to
find difference between them. This will help us to debug the
problem. And the problem line is formated with Emacs
friendly format. i.e. FILE_NAME:LINE: OTHER INFO.

> 6) Hook up the test in the test program. The simplest test program is:
>       int
>       main (int   argc,
>             char *argv[])
>       {
>         gtk_test_init (&argc, &argv); // initialize test program
>         g_test_add_func ("/TestProgramName/Test Case Name", test_case_test_func);
>         return g_test_run();
>       }
>     The g_test_add() function can be used to hook up tests with Fixtures:
>       g_test_add ("/scanner/symbols",        // test case name
>                   ScannerFixture,            // fixture structure type
>                   NULL,                      // unused data argument
>                   scanner_fixture_setup,     // fixture setup
>                   test_scanner_symbols,      // test function
>                   scanner_fixture_teardown); // fixture teardown

I think self registering tests mechanism will bother us. We
may forget to register new test.

Cutter collects tests automatically. And Cutter can find
fixture setup/teardown function too. Cutter has an example
test for a stack implementation. The example test is used in
Cutter tutorial(*). Here is an complete test:
(*) http://cutter.sourceforge.net/reference/tutorial.html

http://cutter.svn.sourceforge.net/viewvc/cutter/trunk/sample/stack/test/test-stack.c?view=markup

  #include <cutter.h>
  #include <stack.h>

  void test_new_stack (void);
  void test_push (void);
  void test_pop (void);

  static Stack *stack;

  void
  setup (void)
  {
      stack = NULL;
  }

  void
  teardown (void)
  {
      if (stack)
          stack_free(stack);
  }

  void
  test_new_stack (void)
  {
      stack = stack_new();
      cut_assert(stack_is_empty(stack));
  }

  void
  test_push (void)
  {
      stack = stack_new();
      cut_assert_equal_int(0, stack_get_size(stack));
      stack_push(stack, 100);
      cut_assert_equal_int(1, stack_get_size(stack));
      cut_assert(!stack_is_empty(stack));
  }

  void
  test_pop (void)
  {
      stack = stack_new();

      stack_push(stack, 10);
      stack_push(stack, 20);
      stack_push(stack, 30);

      cut_assert_equal_int(3, stack_get_size(stack));
      cut_assert_equal_int(30, stack_pop(stack));
      cut_assert_equal_int(2, stack_get_size(stack));
      cut_assert_equal_int(20, stack_pop(stack));
      cut_assert_equal_int(1, stack_get_size(stack));

      stack_push(stack, 40);
      cut_assert_equal_int(2, stack_get_size(stack));
      cut_assert_equal_int(40, stack_pop(stack));
      cut_assert_equal_int(1, stack_get_size(stack));
      cut_assert_equal_int(10, stack_pop(stack));
      cut_assert_equal_int(0, stack_get_size(stack));
      cut_assert(stack_is_empty(stack));
  }

We just define setup, teardown, test functions. We only want
to write test. We don't want to other things like
registration. Cutter will do other things instead of us.

> 8) Execute the tests.
>     Currently, GLib and Gtk+ support four Makefile targets related
>     to tests, one of which is hooked up to automake's check rule:

>       make test-report	# run all tests recursively,
>                          # ignore errors, generate test-report.xml

'make test-report' outputs like the following lines:

  /testglib/Types Sizes:                                               OK
  /testglib/Various Strings:                                           FAIL

Is it too verbose? If we have many tests, the first test
result will be out of screen. I think that we just want to
know about failed tests. We will not consider about
succeeded tests. They means that no more works.

Cutter just outputs "." for a succeeded test, "F" for a
failed test. (There are other marks.) And Cutter outputs
summary for tests. The following output is GLib tests
written with Cutter(*):

  % ./run-test.sh
  ..................................................................N.........................................

  1) Notification: test_arg_double_de_DE_locale
  Cannot set locale to de_DE, skipping
  ./option.c:396: test_arg_double_de_DE_locale()

  Finished in 17.272884 seconds

  107 test(s), 1239192 assertion(s), 0 failure(s), 0 error(s), 0 pending(s), 1 notification(s)

(*) http://cutter.svn.sourceforge.net/viewvc/cutter/trunk/sample/glib/


There is a GRegex test in the GLib tests:
  http://cutter.svn.sourceforge.net/viewvc/cutter/trunk/sample/glib/regex.c?view=markup

This test is rewritten the original GRegex test(*) with Cutter.
(*) http://svn.gnome.org/svn/glib/trunk/tests/regex-test.c

IMO, I could write a GRegex test that is more clearer and
easy to read rather than the original GRegex test.


Thanks,
--
kou