Re: dogtail-devel distribution of dogtail tests

[Frederic Peters <fpeters 0d be>]

David Malcolm wrote:

> http://jhautobuild.0d.be/builds/2006-07-30-0000/longview  [1]

http://jhautobuild.0d.be/builds/2006-07-30-0000/logs/dogtail-gedit-tests/
looks nicer since it has formatting for PyUnit XML output.


> Am I right in thinking that's a 1-module stack on top of the system
> stack?

Yes, this is a really simple module I made, a tarball with
gedit-saving-tests.py as only content.


> There's a proposed moduleset attached to that bug which layers on top of
> a gnome moduleset: I just attached a new one, updating it to the new
> jhbuild DTD, this time for gnome-2.16  
> What do you think of the proposed moduleset?  It seems to me to be very
> compatible with your proposals below.  Should we integrate the tests
> into the main moduleset, or have a separate one?

Totally compatible with my idea, yes.  For the record LDTP tests are
doing it like this:

  <testmodule id="ldtp-gedit-test" type="ldtp">
    <branch repo="ldtp.freedesktop.org" module="pyautosuite/gedit"/>
    <dependencies>
      <dep package="ldtp"/>
      <dep package="gedit"/>
    </dependencies>
    <testedmodules>
      <tested package="gedit"/>
    </testedmodules>
  </testmodule>

The differences are
 1) <testmodule> instead of <autotools>, I don't remember the exact
    code but it does something like ldtp-run tests.xml
 2) <testedmodules> so it is possible to associate tests to modules.
    (out-of-module scenario)

It should not be difficult for LDTP and Dogtail to standardize on a
Makefile target here.


> Whether tests should live with the software they test, or in a separate
> module is a much-debated area.  I think we should allow modules to have
> tests live inside them, if possible: an example of a module with inline
> dogtail tests is Frysk ( http://sourceware.org/frysk/ ) where the tests
> for the java-gnome GUI are deeply embedded within the rest of the test
> suite:
> http://sources.redhat.com/cgi-bin/cvsweb.cgi/frysk-gui/frysk/gui/test/dogtail_scripts/?cvsroot=frysk
> 
> The caveat is that such a module can't be within the dependency tree for
> dogtail, or you'll run into circular dependency problems (I think).

Or adding more logic so there is a second run for the tests; but this
is much more intrusive to jhbuild.


> With my "distribution" hat on I have a preference for out-of-module test
> suites since it makes packaging easier:  I can run the tests on an
> installed package on a live distribution, without having to deal with a
> build system.  Though obviously this is heading off-topic for a
> tinderbox-style setup.

I also prefer out-of-module suites; easier to handle in a tinderbox
setup.


> > Once scripts are sorted, the second phase would be a common way to run
> > them.  Each module directory could have a Makefile with a rule check,
> > dogtail could have a helper script to merge all test suites from a
> > directory and run them, other ideas are possible.
> 
> This could work.  I'm not a great fan of Makefiles, I'm wondering if
> there's a more pythonic way of doing it, perhaps using pyunit?

I don't know anything from pyunit that would work out of the box for
everything but gedit-saving-tests.py is a good start; it is then quite
easy for a script to import every python file and look for TestCase
classes.


> BTW, have you see the dogtail-run-headless script?  In particular, we're
> hoping to have test runs automatically generate screencasts so that you
> can investigate what when wrong on a failure.  See:
> http://bugzilla.gnome.org/show_bug.cgi?id=346220

I didn't see it but the idea is so good I also thought about it :)
Istanbul bugs 349177 (start recording from command line) and 349561
(use correct display/screen) are somehow related; I provided a patch
for the first one and it has just been applied.

My own scripts is a hacked version of Debian (I think) xvfb-run,
modified so it basically does:

 istanbul --record-file xxx.ogg &
 ISTPID=$!
 run-tests
 RETVAl=$?
 kill -15 $ISTPID
 wait $ISTPID
 kill $XVFBPID


> In your blog you said "Dogtail still lacks a structured repository of
> tests and that part will be mandatory if we want them to be automated.".
> What else is needed to get this up-to-speed?

Moving tests to dogtail-tests/ and arranging Makefile (or anything
else, LDTP tests are running their own command, it is easy to add a
new one for dogtail).



        Frederic