Re: [orca-list] [VINUX] Call to Arms

[Trevor Saunders <trev saunders gmail com>]

Hi,

On Fri, Jul 23, 2010 at 12:27:35PM +0200, Pi?eiro wrote:
> From: Trevor Saunders <trev saunders gmail com>
>
> > Here's an idea for regression testing.
> > Can we setup a server somewhere to automatically run the test sweet
> > every so often, and post the results somewhere or mail them to people
> > who will look at the results and take the needed action?  This
> > shouldn't be too hard to setup if we can get a machine to use, I'm
> > willing to help with this.  The benefits as I see it are that a person
> > doesn't have to spend the time the tests take waiting, and people who
> > aren't comfortable setting up the test sweet can help with reading
> > over the results.
>
> Instead of setup a new server, one option would first check if we
> could execute this regression tests on the current gnome integration
> system, the GNOME buildot [1].

ok, wasn't aware of this stuff.

> After all, it is already compiling orca every ofter [2], so we could
> check if we can add this regression tests in the check step, and show
> a summary, as gtk+ [3] or cairo [4] does.

this would be convenient, but orca's test sweet seems to touch a lot
of other things which will probably make this complicated.

> The main problem could be that this servers doesn't run a X server. In
> order to run some of the tests in some modules (cairo and gtk) they
> use xvfb (Virtual Framebuffer 'fake' X server). Not sure if at-spi
> would work fine with it. It would be also required to check if we
> could run at-spi there.

not sure about the frame buffer issue, it seems like it should work,
but has anyone tested this?  I think this does point out an issue
though, orca's test sweet clearly depends on at-spi, but the tests
also test gcalctool, firefox open office, at first gues I doubt build
servers have OO or firefox.

> But this is a possibility that probably we should investigate.

yes, long term it seems good,  not sure how easy to setup it will be.

> Opinions? Ideas?

ok, I took a quick look at the test sweet here is what I'm thinking.
1. we can setup something like I suggested quickly days / week or two,
and use it in the short term, since its pretty easy to setup and
manage.
2. long term it sems like a lot of what orca is testing could be
tested in a better way.  What I mean is this we have test scripts that
test firefox by starting firefox and tabbing n times and asserting
orca says a predefined string.  That's a simplification, but roughly
accurate I believe.  The big problem with this is that if the ui for
that app changes, but is still fine, say you have to tab an extra time
or one less.  orca's test sweet will break, and since this depends on
software we have to deal multiple versions of we will have include all
sorts of ugly hacks in our test sweet.
SO my question is this can we push a lot of the testing of other apps
out of orca's test sweet and into at-spi, or better yet have the apps
themselves test it?  I'm mean we get gcalctool to run there regression
tests with at-spi running and gail and atk-bridge loaded, then they
have tests based on there spec for the ui that they run asserting that
the ui provides the correct information to at-spi.  SInce there
dealing with the change to the ui already, this shouldn't e a problem
for them, and possibly the tests can even be autogenerated from the
spec. If this were to hapen then at-spi can just test that it passes
signals correctly, and orca can test that when scripts for app x are
running and event foo is recieved the right thing happens.  Of course
this will be hard to get to, is pretty dreamy perfect world stuff.
Which leaves us running tests on apps that don't test themselves.  So
the question is how can we test them best, I think our current
framwork is decent in this case but ideas are welcome.  While it would
be nice to test this on the gnome build boxen I'm not sure they're
setup for this kind of testing.

Trev


> [1] http://build.gnome.org/
> [2] http://build.gnome.org/orca
> [3] http://build.gnome.org/builders/gtk%2B-RHEL5/builds/410/steps/gtk%2B%20check/logs/summary
> [4] http://build.gnome.org/builders/cairo-RHEL5/builds/413/steps/cairo%20check/logs/summary
>
> BR
>
> ===
> API (apinheiro igalia com)