Re: QA Guidelines (was: Looking for QA Coordinator and PR point person for GNOME 1.4)

[Maciej Stachowiak <mjs eazel com>]

Derek Simkowiak <dereks realloc net> writes:

> -> The GNOME Foundation Board, the GNOME 1.4 Release Team, and other
> -> people have discussed ways to raise the bar on quality for GNOME
> -> 1.4.
> 
>      Maciej,
> 	I'm not subscribed to the gnome-1.4-list so I'm sending this to
> you personally.  Feel free to forward it where appropriate.

Actually, this discussion belongs on gnome-hackers; unit tests are a
subject much broader than 1.4.

> 	The single most productive thing (in my experience) for software
> quality is Unit Tests.  Every object should have a test module that tests
> the maxima, minima, and error handling of its methods.  Every program
> should have a "test" Makefile target that tests all of the methods used in
> the program.

I agree these help a lot. The main thing I am looking for at this
point is for someone to coordinate integration tests, since that is
all we really have time to do for 1.4. But more unit tests are a great
goal for the future.

> 	Perhaps a formalized standard for putting unit tests in Gnome
> software would be a good first step for improved quality.  Futhermore,
> perhaps the Gnome Foundation should adapt the standard, "If your program
> does not have complete unit tests, we won't ship it as part of the
> standard Gnome distribution."

That would be pretty harsh given the current state of unit tests in
GNOME.

> 	Beyond that, we could implement pre-checkin tests in the CVS
> repository.  CVS can run arbitrary actions when doing a checkin--things
> like doing a test compile, running unit tests, etc.  For instance, the
> Mozilla project shames people who checkin code that breaks the build.  We
> could have the CVS repository run unit tests, or confirm that the
> appropriate documentation is in place (fishing for ideas here..?).

At Eazel we run tinderbox which gives you a good but not certain idea
of who broke the build, or `make distcheck' or `make check' (these run
tests and test that installation works right) or even the rpm
build. It's been effective as a tool for shaming people. That's
probably less load than running tests on every checkin.

> 	Finally, the only way to test something is if you know what the
> expected result is.  I think the Gnome Foundation should be a little
> tougher on documentation standards (with easy tools to produce that
> documenation; gtk-doc is fairly difficult to use).

Now this is pretty relevant to 1.4 - if you have specific suggestions,
send to gnome-1.4-list. Federico suggested in a recent foundation
board meeting that the release coordinators should reject packages
which do not come with complete docs.

 - Maciej