Re: Gtkvts conclusions

[Iago Toral Quiroga <itoral igalia com>]

El jue, 03-08-2006 a las 11:51 +0200, Iago Toral Quiroga escribió:
> I've been playing a bit with gtkvts and I would like to share my
> conclusions about it:
> 
> Gtkvts is two things at the same time:
> 
> a) A set of tests for GTK+-2.0 developed using TET3 testing framework
> API.
> b) A set of basic command line tools for dealing with these tests.
> 

Hi all,

I've spent some more time working with gtkvts test suite, but I have
found some things that I didn't like too much:

- Some tests don't even compile (for example, missing ';')
- When added -Wall almost every test dumps a huge ammount of warnings.
- Some tests are bad implemented (for example, I found a loop with its
arguments swapped: for (i=0; i++; i<N), error messages that do not
correspond with the test that's being done, etc)
- Some tests fail but the programmers are not sure if they should (they
wrote comments in the source files about things like this)
- The tests don't follow any code conventions.
- Seems as if some tests are redundant.
- Although Fernando Herrera was told that LSB people were maintaining
this test suite, I haven't see any commits since August, even though
some of the tests don't even compile.
- The tests mix unit testing and functional testing. The latter are done
'by hand' instead of using specific tools like Dogtail.
- The tests are generated from a framework-independent language (.inp).
This provides some other issues:
   - Increases compilation time (about 3 min in my box), due to .inp
-> .c conversion.
   - Test cases are not .c files but .inp files, which makes edition
with highlight mode harder.
   - Hard integration with autotools, because the Makefiles for the
tests are generated at "make" time. Indeed, they currently are
generating Makefiles directly in the .inp - .c conversion process.
   - The parser for the .inp -> .c conversion is an had-hoc 44K parser
written in C and it would not be easy to mantain.
   - One change to a test implies you need to do the .inp -> check
conversion for _all_ the tests again -> very discouraging for tests
programmers/maintainers.
   - Of course, when you get compilation errors, the compiler shows the
line number of the .c file, but you need the line number in the .inp
file. This makes it harder to fix compilation issues.

In a few words, I think it is better to start the Gtk+ testing suite
from scratch instead of re-using gtkvts. Gtkvts is still a huge
collection of tests, but I'm not confident in them because of all the
things I have already said before. I think we can do something better if
we start from scratch.

That said, I will retake my patch to add "Check" support to Gtk+ and
start working on some unit tests.

Iago.