Re: dogtail-devel distribution of dogtail tests

[David Malcolm <dmalcolm redhat com>]

On Sat, 2006-07-29 at 19:05 +0200, Frederic Peters wrote:
> Hello,
> 
> As some of you may already know I am working on integrating dogtail
> tests into jhbuild, and more importantly, jhbuild.bxlug.be so test
> results are available for everyone to see.
Thanks!
http://jhbuild.bxlug.be/  is looking very nice.

Based on your blog (which is at
http://www.0d.be/2006/07/30/and-now-for-dogtail-support/ for those who
haven't seen it), I went looking for the dogtail stuff, here's the one
log I could find:
http://jhautobuild.0d.be/builds/2006-07-30-0000/longview  [1]

Am I right in thinking that's a 1-module stack on top of the system
stack?

Have you seen this bug:
http://bugzilla.gnome.org/show_bug.cgi?id=318531
? ("Support dogtail scripts in GNOME tinderbox").  

This was dormant for a while: we were waiting on a refactoring of
jhbuild to separate out how software is downloaded/checked out from how
it is built (looks like this landed after the recent big explosion of
source code control systems, git, mercurial etc).

There's a proposed moduleset attached to that bug which layers on top of
a gnome moduleset: I just attached a new one, updating it to the new
jhbuild DTD, this time for gnome-2.16  
What do you think of the proposed moduleset?  It seems to me to be very
compatible with your proposals below.  Should we integrate the tests
into the main moduleset, or have a separate one?

> I am now wondering about the way dogtail tests could be distributed to
> make this efficient.  Ideally jhbuild would checkout 'gedit tests',
> run them and upload results, they would then be linked from gedit
> module page.
> 
> Unfortunetaly for the moment there are gedit tests in dogtail/examples/
> and in dogtail-tests/gedit/; perhaps even in 108, I didn't check.
> 
> I have a proposal, but note I started using dogtail yesterday, I
> definitely lack experience.  Here is the modest proposal :
> 
> Dogtail tests for a given module should be commited to dogtail-tests/
> module-name/.
Sounds good: similar to what's there already.

Whether tests should live with the software they test, or in a separate
module is a much-debated area.  I think we should allow modules to have
tests live inside them, if possible: an example of a module with inline
dogtail tests is Frysk ( http://sourceware.org/frysk/ ) where the tests
for the java-gnome GUI are deeply embedded within the rest of the test
suite:
http://sources.redhat.com/cgi-bin/cvsweb.cgi/frysk-gui/frysk/gui/test/dogtail_scripts/?cvsroot=frysk

The caveat is that such a module can't be within the dependency tree for
dogtail, or you'll run into circular dependency problems (I think).

With my "distribution" hat on I have a preference for out-of-module test
suites since it makes packaging easier:  I can run the tests on an
installed package on a live distribution, without having to deal with a
build system.  Though obviously this is heading off-topic for a
tinderbox-style setup.


> 
> Tests spanning more than one module should go into dogtail-tests/
> jhbuild-meta-module-name/, if it makes sense, for example tests
> touching most desktop modules would go in meta-gnome-desktop/.
Again, sounds good.

> 
> For other tests, common sense and a bit of imagination is enough,
> things like tomboy-evolution-integration for tomboy tests touching
> evolution integration, what a surprise.

+1

> 
> Once scripts are sorted, the second phase would be a common way to run
> them.  Each module directory could have a Makefile with a rule check,
> dogtail could have a helper script to merge all test suites from a
> directory and run them, other ideas are possible.

This could work.  I'm not a great fan of Makefiles, I'm wondering if
there's a more pythonic way of doing it, perhaps using pyunit?

> Third phase is for those tests to provide results in a consistent way,
> if the whole test run could output to a single XML file, it would be
> nice for my integration work.

BTW, have you see the dogtail-run-headless script?  In particular, we're
hoping to have test runs automatically generate screencasts so that you
can investigate what when wrong on a failure.  See:
http://bugzilla.gnome.org/show_bug.cgi?id=346220


In your blog you said "Dogtail still lacks a structured repository of
tests and that part will be mandatory if we want them to be automated.".
What else is needed to get this up-to-speed?

> 

> 
> End of proposal.  See how it lacks polish ? :)  Comments welcome.

Your site is already looking awesome!  Keep up the good work!

Hope this helps
Dave

[1] all those "failed to send buffer" messages look disturbing; need to
investigate that; we greatly tightened up error checking in CVS HEAD
recently, would be nice to trap everything