Re: GNOME's testing strategy for GUIs

[Qi-Bo Paul Mei <Paul Mei Sun COM>]

Hi Brad,

I got this mail from Willima Walker, maybe it is late reply. I am in Sun 
as the QE/RE manager for desktop team. My team has done lots of testing 
for gnome applications, actually we are finding ways how to do the 
automation testing.

The principle for automation tools we are using is that it  is easy and 
can be helpful for our project testing quickly. For firefox Tinderbox 
testing, we are using LDTP for the nightly build check, becasue we can 
leverage LDTP community work on it. Recently mozilla community is using 
the Eggplant for their automation testing, we also evaluated and tried 
it. For the A11Y automation testing, we are using Orca for the 
automation work. For flash automation testing, we also try the ULLU tool 
developed by flash team.

For gnome automation testing, we have done one evalutaion comparison 
about dogtail vs LDTP, please see attachment file.

We don't include the strongwind in here, because we aren't familiar with 
it. :-)  Please let us know your comments.

Because dogtail is integrated into Solaris and is easy for our 
automation development and maintaining, we preferred to use dogtail for 
gnome automation now. We have finished some automation works for gnome 
application sanity testing, it is running for our nightly build testing 
now. We are tracking the testing, maintaining these automation testing 
codes, and analyse our result and data.

Actually it is still in the evaluation and trial phase, we are 
collecting data about ROI for our automation work and decide how to do 
it. We hope to get more resource to do the automation development work, 
and especially hope to leverage resource from gnome community. I think 
this is very good start to discuss about this.

Thanks,
Paul.
------------------------------------------------------------------------

Subject:
Re: GNOME's testing strategy for GUIs
From:
Brad Taylor <brad getcoded net>
Date:
Thu, 14 Feb 2008 11:09:15 -0800
To:
Willie Walker <William Walker Sun COM>

To:
Willie Walker <William Walker Sun COM>
CC:
"desktop-devel-list gnome org" <desktop-devel-list gnome org>


Hi Willie,

First off, in the spirit of full disclosure, I should mention that I
work for Medsphere, the company who backs the development of the GPL
testing tool, Strongwind[1].



> I might be opening a big can of worms with this question, and I 
> apologize if someone is already working in this space and I just don't 
> know it.
>   

Ken VanDine and I met at SCALE this last week and discussed some plans
regarding automated testing in GNOME.  We identified a few rough spots:


* Many tools are available for automated testing, and GNOME needs to
annoint one with holy gnome pee so that it will become an acceptable
dependency for development.  Whether this is LDTP, Dogtail, Strongwind,
or others, a clear decision needs to be made so that people can go
forth with creating tests.

Having a mix of testing tools in use will only make it more difficult
for folks to contribute to, and maintain a large body of tests.
      * Someone needs to step up and write good, thorough documentation 
for
developing tests.  To my knowledge, no current testing tool has this.

* A culture of automated testing needs to be developed and fostered
inside of GNOME so that maintainers feel real benefit in maintaining
a high-quality test library.  Maintenance is not a trivial thing.

* Distributions who already have automated tests for GNOME applications
need to have a place to contribute and maintain these tests
_publicly_, and have them shared by all.  This was made clear in
discussions with folks at this year's GNOME Summit.


Obviously, there is a lot of work that needs to be done, but these
challenges are by no means insurmountable.

The first thing that needs to come into the picture are good docs.  I've
commited myself to contribute a Gnome Journal article on writing
Strongwind tests as a first start into this field.

Cheers,

-Brad

--
[1] http://medsphere.org/projects/strongwind/


_______________________________________________
desktop-devel-list mailing list
desktop-devel-list gnome org
http://mail.gnome.org/mailman/listinfo/desktop-devel-list

Title: QE/RE/dogtail ldtp review - JDS-BJ

QE/RE/dogtail ldtp review

From JDS-BJ

LDTP - Dogtail - Comparison Notes

--Patrick Gu updated 2007-05-10

Results: LDTP is good but to users on lastest Snv, strongly recommand Dogtail.

LDTP - Dogtail

Items	Description	LDTP - 0.7.0	Dogtail - 0.6.0	Note
Package and Installation on Snv	How easy install and use on solaris	Need take 0.5 day to build new workable build when ldtp version update. Need take 0.5 day release package by Dave. Latest night build(broken now): /net/allstar.prc.sun.com/export/storage/download/opensolaris/ldtp	Already intregated in opensolaris. SUNWgnome-gui-test packages which delivers Dogtail. * dogtail/distro.py: Add patch from Brian Cameron to detect Solaris platform. So most scripts work from community on solaris after comments Version or package check line(packageDb.getVersion). We don't need take time to build and maintain package.	Dogtail win
Widget Access	Both of them are dynamic access. The ability of accessing wadget.	It's no way to access 'noname' widget but there are a lot of 'noname' widget in applications.	Access 'noname' widget by relations as workaround. For example，	Dogtail win
Stability	Chance of breaking scripts between builds	Ldtp version update will break testing everytime.	Dogtail work fine during vermillion 51 to 55. See if those sricpts still work after vermillion update to gnome 2.17	Dogtail win
Designing test scripts	Availability of sufficient functions for writing test scripts			Tie
Report	Availability of sufficient functions to produce reports/logs in desired format	change to python logging module	Python unittest frame or testoob(need extra install)	Ldtp win
Record	Ability to record user action and produce a executable python test script			Tie
Replay	Ability to replay the recorded / written script			Tie
Documentation				Tie
S10Ux(Gnome2.6) support				Ldtp win
Code complexity and Maintainability				Dogtail win
Community Support				Ldtp win
Learning Curve				Tie