Gtkvts conclusions

[Iago Toral Quiroga <itoral igalia com>]

Hi,

I've been playing a bit with gtkvts and I would like to share my
conclusions about it:

Gtkvts is two things at the same time:

a) A set of tests for GTK+-2.0 developed using TET3 testing framework
API.
b) A set of basic command line tools for dealing with these tests.

A. The tests
---------------------------------

1. Provides a _huge_ amount of tests. 

This might seem pretty good, but I am not sure about it. Too many tests
can be worse than too few because it leads (or may lead) to:
    - Too much execution time.
    - Too much code to maintain and document.
    - Redundant tests.
    - And as a consequence, it may become hardly usable, and finally
lead people to not use them at all, which is worse than having no tests.

I see tests as a tool for maintainers. Imagine you are maintaining a
package and you want to execute your tests before doing a commit or a
new release to ensure everything is ok and you find your tests take
hours to execute or sometimes hang and you have to manually Crtl-C them,
etc... I would bet such maintainer will not use nor maintain these tests
in the future. IMHO, this is something we should avoid.

2. These tests have been developed using the TET API, but are mostly
framework independent because they use templates for all the TET related
stuff. The only thing tied to TET inside the tests that I have found is
the TET_ROOT environment variable usage, which is avoidable.

3. Some tests could be redundant or unnecessary. I think we (or GTK+
guys, or together with them) should discard tests that are not really
useful for the reasons I wrote in 1).

4. Some tests have a functional approach (they generate events and send
them to widgets, etc). Maybe these tests should better be done using a
tool like Dogtail. These are not unit tests but functional tests.

B. The tools
-----------------------

- A tool for checking bugzilla bugs. Some tests have a bug id 
assigned, so you can use this tool to check the results of 
the tests regarding these bugs. I miss an option to search 
for specific bugs. Very simple.

- A tool for tests coverage. Outputs the percentage of classes 
functions and assertions tested/documented. It does not point 
out which code is really tested or not, nor which functions 
or classes have tests or how complete these tests are (or
I did not find info about this). To me, seems too simple to be 
useful. Tools like gcov produce much more interesting information.
The good thing is it takes care about documentation too.

- A tool to generate the real tests (.c) from test definitions (.inp). 
Pros: 
   - You can extend the approach to get rid of the TET framework
     and create framework independent test cases. 
   - You can extend the approach to create several backends for
     several testing frameworks.
   - All tests for GTK+ already done by Sun would become available
     for usage with other testing frameworks.
   - The tool adds support to avoid repetition of some testing
     code when doing the same test with different parameters.
Cons:
   - Tests execution takes more time. You have to first create 
     the real tests (.c) from test definition files (.inp) 
     applying appropriate templates for your framework (TET, Check,
     etc.)
   - It needs some development to get it working like this. Nowadays
     this generator is tied to the TET framework. This also has to do
     with Mafefiles generated for tests, which are hardcoded 
     within the tool code (no template for this) and are tied to
     TET too. I'll post a more specific "todo" later.
   - Managing, executing and configuring the tests would be a bit 
     more messy because it involves more steps and code generation.

- It also provides some simple libraries to help with test development.
One of these libraries (librobot) is oriented to event generation, 
which is something more related to "funcional" tests rather than unit 
tests. Maybe this kind of stuff and tests should better be achieved 
using a tool specially designed for funcional interaction like Dogtail.

Hope you find this information useful, and feel free to make any
comments or correct anything that you think is wrong or not complete.

Iago.