Thoughts on testing and outreach

[Eitan Isaacson <eitan ascender com>]

Hi,

I think a few stuff were left in the air about the huge testing
challenge we face, and how to put the outreach program to play in that
direction. I would like to organize a few of my own thoughts, and get
other people's opinions on this. I'm not appending this to the existing
thread in desktop-devel, because I think this might be less specific
than that conversation. If you are in a rush, jump to the conclusion at
the bottom.

In my mind there are a few things we mean when we say "testing", here
they are:

UNIT TESTS

These are small non-runtime tests that are usually invoked by "make
check". They are ideal for shared libraries. There are plenty of GNOME
modules out there that take advantage of autotools integration of these
kinds of tests, and the GNOME build brigade provides test results during
it's (nightly?) automated builds. In my mind we have, or potentially
have, good coverage of these kinds of tests.
It's very much like tinderbox, you could see it in action here:
http://build.gnome.org

AUTOMATED GUI TESTS

These are runtime tests that automate user interaction, and have
different methods of asserting the test results. This is what the talk
about is in desktop-devel[1]. I think everyone there is in agreement
that - the status quo sucks, we need an open and collaborative space for
automated testing, maybe something like build.gnome.org. And a blessed
single testing toolkit.
Coincidentally, automated GUI testing usually relies heavily on our
accessibility framework. But let's not get confused, the main purpose of
most of these tests is not accessibility, a developer will put these
tests in place to squash conventional bugs. Will has a hidden agenda
when he pushes for these kinds of tests in other GNOME modules, we get
to exercise an application's accessibility "for free".

ACCESSIBILITY TESTS

Unlike the top two categories, accessibility tests don't have a specific
method, it's just any kind of test that tests for accessibility, they
could be unit tests, or GUI tests. They could also be a third kind of
test, and that is external poking, like at-poke and accerciser do. They
could be automated, and they could be manual. I think this is a very raw
but fertile ground. A good example of a start is Peter Parente's
validation plugin[2] for Accerciser, it will give a lint-style
accessibility test.

CONCLUSION

In my opinion, our outreach program should focus on the third meaning of
"testing" - accessibility testing. Besides the obvious test schemas and
accerciser extensions, I think this also entails work inside other
modules, and adding to their testing story a sleuth of tests that test
nothing other than accessibility. It could be as simple as writing a
test that searches for unrelated labels in a glade file, this script
could be duplicated across every module that uses glade, and run during
"make check". Or a dogtail sequence that simply listens for the right
events as buttons get clicked.

Think about it - I haven't. [3]

Eitan.

1.
http://mail.gnome.org/archives/desktop-devel-list/2008-February/msg00103.html
2. http://pithy.tumblr.com/post/23941873
3. Stephen Colbert