Re: Thoughts on testing and outreach

["Steve Lee" <steve fullmeasure co uk>]

I'd add 2 forms of user testing:

*  'Ad hoc' or 'guerilla' testing often exposes problems not always
caught by methodical test strategies. AFAIUI this is users playing
fast and loose with the UI, trying to catch it out.

* real world usage as users may exercise stuff it in unexpected ways
(I think Orca has benefited greatly from this).

I agree that we should concentrate on accessibility tests. While it
seems to make sense to leverage the effort and cover the the forms of
testing I think that will dilute what we need to achieve as the a11y
group and the message of the outreach program. Even when we are
producing tests that have obvious general test application I think we
should remain foucussed for now (other groups can always use and
contribute). Once we have good mileage there we can embrace the other
testing and share work.

That said, the AT's and infrastructure should have good unit/automated
test suits for regression etc.

Steve Lee

On 23/02/2008, Eitan Isaacson <eitan ascender com> wrote:
> Hi,
>
>  I think a few stuff were left in the air about the huge testing
>  challenge we face, and how to put the outreach program to play in that
>  direction. I would like to organize a few of my own thoughts, and get
>  other people's opinions on this. I'm not appending this to the existing
>  thread in desktop-devel, because I think this might be less specific
>  than that conversation. If you are in a rush, jump to the conclusion at
>  the bottom.
>
>  In my mind there are a few things we mean when we say "testing", here
>  they are:
>
>  UNIT TESTS
>
>  These are small non-runtime tests that are usually invoked by "make
>  check". They are ideal for shared libraries. There are plenty of GNOME
>  modules out there that take advantage of autotools integration of these
>  kinds of tests, and the GNOME build brigade provides test results during
>  it's (nightly?) automated builds. In my mind we have, or potentially
>  have, good coverage of these kinds of tests.
>  It's very much like tinderbox, you could see it in action here:
>  http://build.gnome.org
>
>  AUTOMATED GUI TESTS
>
>  These are runtime tests that automate user interaction, and have
>  different methods of asserting the test results. This is what the talk
>  about is in desktop-devel[1]. I think everyone there is in agreement
>  that - the status quo sucks, we need an open and collaborative space for
>  automated testing, maybe something like build.gnome.org. And a blessed
>  single testing toolkit.
>  Coincidentally, automated GUI testing usually relies heavily on our
>  accessibility framework. But let's not get confused, the main purpose of
>  most of these tests is not accessibility, a developer will put these
>  tests in place to squash conventional bugs. Will has a hidden agenda
>  when he pushes for these kinds of tests in other GNOME modules, we get
>  to exercise an application's accessibility "for free".
>
>  ACCESSIBILITY TESTS
>
>  Unlike the top two categories, accessibility tests don't have a specific
>  method, it's just any kind of test that tests for accessibility, they
>  could be unit tests, or GUI tests. They could also be a third kind of
>  test, and that is external poking, like at-poke and accerciser do. They
>  could be automated, and they could be manual. I think this is a very raw
>  but fertile ground. A good example of a start is Peter Parente's
>  validation plugin[2] for Accerciser, it will give a lint-style
>  accessibility test.
>
>  CONCLUSION
>
>  In my opinion, our outreach program should focus on the third meaning of
>  "testing" - accessibility testing. Besides the obvious test schemas and
>  accerciser extensions, I think this also entails work inside other
>  modules, and adding to their testing story a sleuth of tests that test
>  nothing other than accessibility. It could be as simple as writing a
>  test that searches for unrelated labels in a glade file, this script
>  could be duplicated across every module that uses glade, and run during
>  "make check". Or a dogtail sequence that simply listens for the right
>  events as buttons get clicked.
>
>  Think about it - I haven't. [3]
>
>  Eitan.
>
>  1.
>  http://mail.gnome.org/archives/desktop-devel-list/2008-February/msg00103.html
>  2. http://pithy.tumblr.com/post/23941873
>  3. Stephen Colbert
>
>  _______________________________________________
>  gnome-accessibility-list mailing list
>  gnome-accessibility-list gnome org
>  http://mail.gnome.org/mailman/listinfo/gnome-accessibility-list
>


-- 
Steve Lee
--
Open Source Assistive Technology
www.fullmeasure.co.uk