Querying top panel widget positions?

[Chris Spencer <chris k spencer protonmail com>]

Hi,

Is there any way to programmatically lookup the position of widgets displayed in the Gnome top panel, as well as UI elements within those widgets?

I'm trying to build a UI test automation framework for some Gnome Shell panel widgets. I'd like to write a script that can click on a panel widget, confirm a popup appears, verify certain text exists in the popup, etc.

I maintain a Gnome Shell extension, and it's become immensely expensive, not to develop, but to test and maintain. There are so many different distros with varying incompatible Gnome versions, and testing them all by hand is virtually impossible. Fixing a bug for one Gnome version introduces a bug in another version. It's gotten to the point where I almost have to abandon the project because it's too difficult to do basic integration and regression testing on Gnome, but I really want to find a solution.

Gnome is one of the most widely used desktop environments in the world, so I find it hard to believe no one else has encountered this problem befpre.

I've found some tools like xdotool and pyautogui for programmatically clicking elements. However, without knowing exactly where widgets are rendered, the script would be clicking blind. And even if I could reliably click a location, those tools don't let me read DOM values for the UI elements at that location.

I've also found some tools like xwininfo that can query some window positions and do a little introspection, but this seems specific to X and treats the entire top panel as one huge window, with no ability to introspect any of the widgets inside of it.

I had hoped the Dogtail project might be helpful in this regard, but it seems to be dead  with virtually no documentation. I was able to build it, but I couldn't find any ability within it to introspect the top panel.

Are there any other tools or libraries I could potentially use? Does anyone have any alternate suggestions for how I might automate Gnome UI testing?

Regards,

Chris