Proposal: splitting out the app wrappers into a separate tarball/repo, and how to organize dogtail-tests

[David Malcolm <dmalcolm redhat com>]

[cross-posting to build-brigade-list as I'd like feedback on how best to
structure this with jhbuild etc: see near the end of this post]

As Dogtail development moves on, I think we're going to want to
stabilize the core APIs, and eventually touch these as little as
possible.  

However, one problem with this is the dogtail/apps/wrappers
subdirectory.  These are full of little workarounds for dealing with
specific apps (capturing these workarounds and distro-specific
differences in one place, rather than forcing them to be cut and pasted
throughout the tests) and I suspect we're going to want to continually
make changes to this part of the code, even when the rest of the
codebase is done.

So I think we should move this out of the main dogtail module and into
somewhere else.  Doing so will break much of the existing example code,
unfortunately.

What do we have here?  (lengthy, so skip this list to get to the
proposal)
- dogtail/apps/categories.py:  my feeling is that this was an
experiment, and can be moved elsewhere or got rid of
- dogtail/apps/wrappers/epiphany.py: short and somewhat hackish
- dogtail/apps/wrappers/evolution.py: quite a lot here, useful
- dogtail/apps/wrappers/gcalctool.py: not particularly useful
- dogtail/apps/wrappers/gnomepanel.py: appears to contain some
Fedora-specific code
- dogtail/apps/wrappers/kicker.py: just a stub
- dogtail/apps/wrappers/konqueror.py: just a stub
- dogtail/apps/wrappers/mozilla.py: just a stub
- dogtail/apps/wrappers/nautilus.py: short and somewhat hackish
- dogtail/apps/wrappers/yelp.py: short and somewhat hackish
- examples/abiword-test.py: never finished
- examples/crack.py: never finished
- examples/evolution-test-composing-html.py: never finished, but would
make a good test of the evolution wrapper.
- examples/evolution-test-configuring-exchange.py: works, but assumes
evolution is already configured and running.  Uses evolution wrapper.
- examples/evolution-test-configuring-imap-smtp.py: works, but assumes
evolution is already configured and running.  Uses evolution wrapper.
- examples/evolution-test-first-time-wizard.py: works IIRC; assumes
evolution is running, and may blow away your configuration.  Uses
evolution wrapper.
- examples/evolution-test-sending-email.py: works.  Uses evolution
wrapper.
- examples/evolution-test-survives-email-CAN-2005-0806.py: works IIRC.
Uses evolution wrapper.
- examples/evolution-test-switching-components.py: works, but doesn't
test anything (infinite loop).  Uses evolution wrapper.
- examples/filechooser-stress-test.py: works; another infinite loop test
that should be turned into a resource usage test.  Doesn't use wrappers.
- examples/firefox-test-browsing-local-html-file.py: not finished; would
use mozilla wrapper.
- examples/gcalctool-test-fibonacci.py: silly demo, using gcalctool
wrapper
- examples/gedit-test-utf8-procedural-api.py: doesn't use wrappers.  
- examples/gedit-test-utf8-tree-api.py: doesn't use wrappers
- examples/gnome-panel-test-starting-every-app.py: silly non-test; uses
gnomepanel wrapper
- examples/google-search.py: silly non-test; uses epiphany wrapper
- examples/i18n-test.py: actual example of using the i18n API, rather
than a test case; doesn't use app wrappers.  Should stay in examples
subdirectory
- examples/nautilus-test-icon-view-collage.py: silly non-test; uses
nautilus wrapper
- examples/no-help-at-all.py: silly non-test that shows off the tree
API; doesn't use app wrappers.  Should stay in examples subdirectory
- examples/rhythmbox-test.py: unfinished stub
- examples/test-events.py: simple example of using atspi event API, for
debugging purposes.  Should stay in examples subdirectory


Where should it go?  I think it would be useful to have a common
location where Dogtail tests can live (although e.g. Frysk is storing
its tests within its main source tree).  There's already a dogtail-tests
module in GNOME CVS.  So I propose moving the bulk of the examples above
to that module, with some kind of subdirectory structure.  Note that the
current tests in dogtail-tests all use the wrappers, and so these may
need fixing up too (there aren't many so this should be easy).

With that in mind, I think it makes sense to add the app wrappers
somewhere inside dogtail-tests, possibly with some Python packaging. I'm
not yet sure how to organize this.  We need something that allows for:
- jhbuild to be able to grab the wrappers and a collection of tests, and
run them
- ability to grab the wrappers and make them available to out-of-tree
tests
- easy use of the wrappers when developing tests.

Possible approaches:  (i) have a per-application directory of tests (as
we do now) and store the wrapper for each app within each directory.
(ii) gather all the wrappers into some Python package that can be
installed as a package by a distribution, or rolled up as a
nicely-versioned tarball, be a single module within jhbuild etc

We probably should store desktop-wide integration tests in another
subdirectory of dogtail-tests too.

Thoughts?  How does this play with the various automatic jhbuild/garnome
tinderbox approaches?  How should this look in the jhbuild modulesets?
How could it interact with GARNOME?

Hope this helps
Dave