Hi Brad,
I got this mail from Willima Walker, maybe it is late reply. I am in Sun
as the QE/RE manager for desktop team. My team has done lots of testing
for gnome applications, actually we are finding ways how to do the
automation testing.
The principle for automation tools we are using is that it is easy and
can be helpful for our project testing quickly. For firefox Tinderbox
testing, we are using LDTP for the nightly build check, becasue we can
leverage LDTP community work on it. Recently mozilla community is using
the Eggplant for their automation testing, we also evaluated and tried
it. For the A11Y automation testing, we are using Orca for the
automation work. For flash automation testing, we also try the ULLU tool
developed by flash team.
For gnome automation testing, we have done one evalutaion comparison
about dogtail vs LDTP, please see attachment file.
We don't include the strongwind in here, because we aren't familiar with
it. :-) Please let us know your comments.
Because dogtail is integrated into Solaris and is easy for our
automation development and maintaining, we preferred to use dogtail for
gnome automation now. We have finished some automation works for gnome
application sanity testing, it is running for our nightly build testing
now. We are tracking the testing, maintaining these automation testing
codes, and analyse our result and data.
Actually it is still in the evaluation and trial phase, we are
collecting data about ROI for our automation work and decide how to do
it. We hope to get more resource to do the automation development work,
and especially hope to leverage resource from gnome community. I think
this is very good start to discuss about this.
Thanks,
Paul.
------------------------------------------------------------------------
Subject:
Re: GNOME's testing strategy for GUIs
From:
Brad Taylor <brad getcoded net>
Date:
Thu, 14 Feb 2008 11:09:15 -0800
To:
Willie Walker <William Walker Sun COM>
To:
Willie Walker <William Walker Sun COM>
CC:
"desktop-devel-list gnome org" <desktop-devel-list gnome org>
Hi Willie,
First off, in the spirit of full disclosure, I should mention that I
work for Medsphere, the company who backs the development of the GPL
testing tool, Strongwind[1].
> I might be opening a big can of worms with this question, and I
> apologize if someone is already working in this space and I just don't
> know it.
>
Ken VanDine and I met at SCALE this last week and discussed some plans
regarding automated testing in GNOME. We identified a few rough spots:
* Many tools are available for automated testing, and GNOME needs to
annoint one with holy gnome pee so that it will become an acceptable
dependency for development. Whether this is LDTP, Dogtail, Strongwind,
or others, a clear decision needs to be made so that people can go
forth with creating tests.
Having a mix of testing tools in use will only make it more difficult
for folks to contribute to, and maintain a large body of tests.
* Someone needs to step up and write good, thorough documentation
for
developing tests. To my knowledge, no current testing tool has this.
* A culture of automated testing needs to be developed and fostered
inside of GNOME so that maintainers feel real benefit in maintaining
a high-quality test library. Maintenance is not a trivial thing.
* Distributions who already have automated tests for GNOME applications
need to have a place to contribute and maintain these tests
_publicly_, and have them shared by all. This was made clear in
discussions with folks at this year's GNOME Summit.
Obviously, there is a lot of work that needs to be done, but these
challenges are by no means insurmountable.
The first thing that needs to come into the picture are good docs. I've
commited myself to contribute a Gnome Journal article on writing
Strongwind tests as a first start into this field.
Cheers,
-Brad
--
[1] http://medsphere.org/projects/strongwind/
_______________________________________________
desktop-devel-list mailing list
desktop-devel-list gnome org
http://mail.gnome.org/mailman/listinfo/desktop-devel-list
QE/RE/dogtail ldtp review
From JDS-BJ
LDTP - Dogtail - Comparison Notes
--Patrick Gu updated 2007-05-10
Results: LDTP is good but to users on lastest Snv, strongly recommand Dogtail.
LDTP - Dogtail
Items
Description
LDTP - 0.7.0
Dogtail - 0.6.0
Note
Package and Installation on Snv
How easy install and use on solaris
Need take 0.5 day to build new workable build when ldtp version update. Need take 0.5 day release package by Dave. Latest night build(broken now): /net/allstar.prc.sun.com/export/storage/download/opensolaris/ldtp
Already intregated in opensolaris. SUNWgnome-gui-test packages which delivers Dogtail. * dogtail/distro.py: Add patch from Brian Cameron to detect Solaris platform. So most scripts work from community on solaris after comments Version or package check line(packageDb.getVersion). We don't need take time to build and maintain package.
Dogtail win
Widget Access
Both of them are dynamic access.
The ability of accessing wadget.
It's no way to access 'noname' widget but there are a lot of 'noname' widget in applications.
Access 'noname' widget by relations as workaround. For example，
Dogtail win
Stability
Chance of breaking scripts between builds
Ldtp version update will break testing everytime.
Dogtail work fine during vermillion 51 to 55. See if those sricpts still work after vermillion update to gnome 2.17
Dogtail win
Designing test scripts
Availability of sufficient functions for writing test scripts
Tie
Report
Availability of sufficient functions to produce reports/logs in desired format
change to python logging module
Python unittest frame or testoob(need extra install)
Ldtp win
Record
Ability to record user action and produce a executable python test script
Tie
Replay
Ability to replay the recorded / written script
Tie
Documentation
Tie
S10Ux(Gnome2.6) support
Ldtp win
Code complexity and Maintainability
Dogtail win
Community Support
Ldtp win
Learning Curve
Tie
_______________________________________________
desktop-devel-list mailing list
desktop-devel-list gnome org
http://mail.gnome.org/mailman/listinfo/desktop-devel-list