Macro recorder using AT-SPI

[Tessa Lau <tlau-gnome tlau org>]

Hi all,

I'm trying to write a macro recorder using the accessibility interface,
AT-SPI.  A macro recorder requires two things from the underlying
platform:

(1) tracking what the user has done: which buttons were clicked, what
text was typed, etc.

(2) programmatically invoking user actions, such as clicking on buttons
and typing in text

I've run the demo programs that come with at-spi in the test/ directory,
and used the at-poke program to introspect into running programs such as
the gnome-panel and galeon.  I've also read up on the AT-SPI API to see
what can be done with the current interface.

Based on what I've read, #2 is easy to do; at-poke for example is able
to push buttons and invoke menu items programmatically.  I'll be able to
read the at-poke source and figure out how it's done.

However, I'm having trouble figuring out how to do #1.  I've written
test programs that register global listeners on the a11y registry,
listening for various events.  I've tried listeners on:

A) "mouse:button" -- but the event source is always the desktop, even
when I click in an application, so I can't tell what button was invoked

B) "Gtk:GtkWidget:button-press-event" --- here I get about 25 event
notifications each time I push a button (e.g., the foot menu in
gnome-panel).  I only pushed the button once; why do I get so many event
notifications?  Anyway this only seems to work for certain buttons.  If
I bring up the "Search for Files" tool, none of the buttons in that
dialog box generate event notifications with this listener.

C) "Gtk:GtkButton:clicked" --- I tried setting a listener specifically
for GtkButtons.  Having to set a listener for every type of
Gtk widget would be suboptimal, but I'll do it if that's what it takes.
Even before clicking on anything, I get hundreds of event notifications
of the "clicked" event on what appear to be the buttons in my WindowList
applet.  I didn't click them, but this event is firing anyway.  That
won't work for a macro recorder.

Ideally I'd like to set a listener for whenever the user invokes a
widget, whether it be clicking on a checkbox, pressing a button, or
selecting a menu item.

Can someone tell me how to do this with AT-SPI?  (I'm using the Python
ORBit bindings to speak to the Accessibility interface via CORBA, but
examples using the AT-SPI C API would be fine, as it's straightforward
to translate between the two.)

Thanks,
--Tessa