Re: notes from QA BOF today

[Dave Malcolm <dmalcolm redhat com>]

On Mon, 2005-10-10 at 13:10 -0400, Luis Villa wrote:
> So, we had a QA BOF at the summit. Because the dogtail guys were
> there, and they are very interested in getting dogtail into a jhbuild
> tinderbox, that became most of the thrust of the BOF. A dump of my
> notes:
> 
> things we're blocking on to run dogtail tests in tinderbox:
> * makefile-without-configure module for pyspi/dogtail in jhbuild or
> track down an autotools guru who can be volunteered (or 'touch
> autogen.sh; cvs commit autogen.sh' ;)

I've created bugs 318534 and 318535 to cover the unjhbuildability  (if
that's a word) of pyspi and dogtail respectively.

I've created bug 318531 as a tracker bug for getting tinderbox building
working.  This has the above two bugs as blockers (I like tracker bugs).
I've attached a patch to the gnome-2.14.modules file to 318531 which
adds pyspi and dogtail (is this committable, or will this break the
build for everyone?).  

> * create dogtail-tests/ to contain module-specific tests and
> meta-tests. should tag this with same patterns as gnome for gnome
> releases, so you can check out the 2.14 tests forever

I've imported this into GNOME CVS, with an initial testcase (which has
gcalctool doing multiplication tables).  Doesn't yet work, due to the
nonjhbuildability of dogtail.

I also attached a proposed "gnome-2.14-testsuite.modules" file to 318531
which contains this testcase as a module.  I've deliberately kept the
name agnostic regarding which test automation system is in use - it can
contain both LDTP and Dogtail scripts, depending on what people feel
comfortable working with.

> * create a test module type in jhbuild (or just do same crappy
> autogen/configure hack)

jhbuild seems to happily deal with the lack of any autotool files; my
testcase module has a Makefile with a simple:

all:
 	name-of-test.py

layout, which succesfully invokes the test in jhbuild when I run
jhbuild buildone dogtail-tests/gcalctool

It invokes the test script, but fails due to it not finding dogtail
installed in the correct python location.

> * running headless: figure out xvfb v. xfake, maybe just use whichever
> is installed; need to get a full session going- what exactly are the
> needs for the session? could jhbuild run gnome-session (inside session
> run test scripts) or could maybe do the nat-jeff session:
> http://nat.org/2005/october/#Keep-It-Simple-Stupid instead of a real
> session
> 

I filed this as bug 318537 and added it to the tracker bug.

> things we'd really like:
>  * recording tools that record scripts for maintainers to play back
> (for reproduction purposes and regression testing) and potentially
> also video. Could do a sabayon-style editor to trim extraneous things.
> Probably need to require playback/verification/duplication before
> upload.
I filed this as bug 318538; it's not on the tinderbox tracker bug.

> * 'real' jhbuild integration- do it per module with separate column
> instead of fake modules; this might require quite a bit of jhbuild
> rearch, depending on exact goals.

> performance/memory/etc. stuff:
> * get the infrastructure running need to talk to the performance
> people; tell them 'we're going to be running all the apps in an
> automated fashion, what do you want to know?' examples include:
> bootchart.org, valgrind

I didn't any bugs for these at this time - the idea of this list is to
try to reduce the total number of open bugs, right :-P

Dave