Some notes on LDTP and Dogtail

[David Malcolm <dmalcolm redhat com>]

I just noticed this criticism of me on Nagappan's blog [1]; copied
inline here:

> DogTail ideas copied from LDTP !!!  
> Do you believe ? One of the main Dogtail Author Dave Malcolm was
> interacting with me in person in GUADEC 2005. He collected all the 
> details from me !!! and in Boston submit, I came to know that Dogtail
> has been released. When I checked the list of authors Dave Malcolm was
> one among them and most of the CVS commits in dogtail project was done
> by him. In GUADEC 2005, I requested Dave to get some RedHat QA
>engineers to involve in LDTP project, he assured me that he will try
> his level best. So, I assumed that he will try to get somebody for
>:LDTP, Actually what happened is totally reverse. Dave tried copying
> the ideas from LDTP and gave it as DogTail ;) 

I'm sorry to have to post this to desktop-devel-list but I don't blog
and I think it might be worth a detailed public explanation of the
LDTP/Dogtail situation as I see it.


We've been looking at automated testing for the whole time I've been at
Red Hat (since well before LDTP's original announcement).  The framework
developed by LDTP isn't the first Free/Open Source testing framework,
and it isn't the first one to leverage the excellent a11y layer
available in GNOME [2].  Thanks to everyone who's worked on the a11y
layer, especially everyone at Sun.  


When LDTP first was announced, we were excited, and looked at it.
Unfortunately, we quickly concluded that the framework they had
implemented was unsalvageable.  It was 
(i) a large chunk of C code (why C??!!! - Python, C#, Java - anything
but C, please, I have too many large chunks of C code in my life
already) that 
(ii) implemented application maps, as the sole way of communicating with
applications:  application maps are a legacy of the approach used by
automation tools in the Microsoft Windows environment, where you don't
have the rich metadata about the GUI available.  It's a big complex
unnecessary thing that the scriptwriter has to keep up-to-date.  It's
never up-to-date, and your scripts break regularly for various reasons.
(iii) It used a totally custom system for driving tests - why not use a
real scripting language?

As far as we could see, once you've eliminated the extraneous elements
like the application maps, there wasn't anything of use in the project,
so we continued to work on our own internal system.

At this point, we could have approached the LDTP project and said that
we admire the spirit of their project but want to totally reimplement it
from scratch.  In retrospect, maybe we should have done.  I wonder what
their response would have been; from my experience in the open source
software world I wouldn't have expected them to go for it.

The LDTP framework went through a number of releases.  We continued to
build our own system, and observe theirs.

Nagappan and the rest of the LDTP team have done a good job of improving
their framework: for example, creating a Python wrapper that removes
objection (iii) above.

In the process the LDTP team has uncovered a number of a11y bugs.  This
is valuable work, and kudos to the LDTP team for doing this: this has
been of use to the Evolution project, for example.

However I think that in spite of their good work in this area the
framework has enough deep flaws (mostly (ii) above, but partly (i) as
well), sufficiently so that, given a blank slate today I'd be doing a
rewrite from scratch, rather than trying to fix the framework.

I attended GUADEC 2005, and was in a difficult situation: our framework
was at the time an internal project by Red Hat's Quality Assurance team,
and I can't talk about internal projects.  I did my best in my questions
and interaction with Nagappan to raise my concerns with LDTP's technical
approach: the usage of application maps, and the usage of C.  I went
back to the QA team within Red Hat and tried to come up with a way
forward.  The approach we settled for was to open-source Dogtail, and
make it available to the various open source communities: GNOME,
mozilla, OpenOffice.org etc - and to our competitors.

Right now I'm half-wishing we hadn't open-sourced it.  Looking at it
from a cynical business standpoint, I think Dogtail gives Red Hat a
significant competitive advantage over companies who are trying to use
LDTP to automate their regression testing.  But ultimately I want GNOME,
the Mozilla Project, OpenOffice.org and friends to have some tools to
help get good releases out of the door, and I'm glad we open-sourced our
framework.

Some possible ways forward: I have the beginnings of an emulation layer
written that fakes the LDTP framework, taking the interface exposed by
the Python bindings and reimplementing it on top of Dogtail (which
leverages the various quality-of-life-for-scriptwriter things that
Dogtail adds to the mix: automatic highly-readable logging, the
robustness of the search algorithm, the avoidance of hard-coded sleeps,
etc).  I've been concentrating on improving Dogtail lately, but I could
finish that up and try to port LDTP's scripts using it.  It would be a
few hundred lines of Python when finished.
Or we could accept that we now have multiple Free/Open Source tools
available, and script-writers and package maintainers can use whatever
tool they feel most comfortable with to automate their apps.


Dave Malcolm



[1]:
http://nagappanal.blogspot.com/2005/10/dogtail-ideas-copied-from-ldtp.html

(dated 24th October)

[2]: For example, the Culchie app that Matthew Garrett wrote in
September of 2004:
http://www.advogato.org/person/mjg59/diary.html?start=55

[3]: oh, and CVS sucks.  This was so much easier when we doing it on our
internal Subversion server :-(