[Vala] [RFC] Testing framework improvements

[Jan Hudec <bulb ucw cz>]

Hello Folks,

The current binding for the testing framework has two disadvantages compared
to similar frameworks in other languages (C#, Java, Python, etc.):
 - One has to manually add each test case.
 - It's hard to actually use fixtures, because there is no GLib.Test.add or
   GLib.Test.add_vtable. (it's possible via new TestCase, but you loose the
   convenience of using the path)

I have three (alternative) ideas how to improve it:

Option 1) The test code would be compiled to a library and a special runner
program would use the .gir file to find test methods by their name and create
and run test cases from them using the gobject-introspection library.

  Pros:
    + No changes to vala
    + Somewhat useful ouside of vala
    + Runner is generic, so advanced alternate runner interfaces are possible
  Cons:
    - The invocation process is somewhat complicated
    - Enforces naming (but so does JUnit, python unitest and perl Test::Unit)

Option 2) The test code would be compiled to a library and a tool would
generate a runner program from the .gir file.

  Pros:
   + No changes to vala
   + Somewhat useful outside of vala
  Cons:
   - Complicates building (two stages needed)
   - Enforces naming

Option 3) The test methods would be decorated with [TestMethod] attribute and
their containing class with [TestClass(path=...)] attribute. Valac would
generate test registration code from that.

  Pros:
   + Matches C# (NUnit)
   + No extra tools involved
   + No naming restrictions
  Cons:
   - Needs to be implemented in valac (or is there a plugin system to add
     attributes already?)
   - Probably more complicated.
To show what I think the code would be like, the first sample from
http://live.gnome.org/Vala/TestSample would convert to:

Under option 1 and 2:

  class Tests : GLib.TestFixture {
    public void test_string_addition() {
      assert ("foo" + "bar" == "foobar");
    }
  }

Under option 3:

  [TestClass]
  class Tests {
    [TestMethod]
    public void string_addition() {
      assert ("foo" + "bar" == "foobar");
    }
  }

Obviously the main point is, that the function is not (does not have to be --
it still could) static and that new object is constructed before each
invocation and destroyed again after the test completes.

So, what do you people think?

Regards,
Jan