Re: [gnome-desktop-testing] A thought about integrating Mago to desktop project source

[Willie Walker <William Walker Sun COM>]

Hi Tim:

Getting testing more tightly integrated would be awesome.  There are at 
least a couple different ways to look at it, and it would be nice to try 
to have a test solution that helps with both:

1) I'm a developer and I want to test my changes before committing them.

2) I'm an integrator building from tarballs and would like to test how 
well everything plays together.

Other people also run manual tests (i.e., a human sits behind the 
machine and runs through a set of testing steps, recording the results).

Since you're already proposing a separate repository from GNOME git for 
tests, one thing that is also of interest to some people is Testopia. 
Testopia seems pretty nice for maintaining/organizing tests as well as 
recording results and provide result summaries.  I believe it also has 
some way to integrate with Bugzilla.  I'm curious if there might be a 
way to integrate with Testopia into the environment.  For example:

1) The tests would be stored in Testopia.

2) The testing framework would pull the tests from Testopia, run them,
and optionally push test results back to Testopia.

3) The GNOME 'make test' targets could be set up to refer to the 
Testopia repository.  Imagine also that more diligent projects might use 
Bugzilla to track patches and would require a pointer to a successful 
test run before the patch is committed.

4) Something else would provide the integrators with a way to get to the 
same tests (or perhaps various subsets of them - a smoke test, an 
exhaustive test, etc.).

Or, maybe just keep the tests in the bsr/git repository as they are now, 
but somehow try to find a way to integrate the test result reporting in 
Testopia.

Will

Tim Miao wrote:
> Hi there,
>
> The GNOME desktop testing project/Mago is stepping forward, more and
> more people are involving and contributing to this great project. Ara
> did make a great presentation on this year GUADEC, and presentation
> attracted much people there, people show their interests on this
> automating testing framework.
>
> This reminds me, could we integrate this automation testing framework to
> the desktop project source code? To add a testing line in makefile, so
> that each time when developer changes the code, builds the application,
> the automation testing will be executed to make sure there will be no
> serious regression introduced.
>
> Mago/GNOME desktop testing has set up a bzr/git repository to store the
> automation test cases for some application, and test cases are still in
> developing. To introduce the automation testing to project source will
> help both project owner to find out bugs and Mago/GNOME desktop testing
> team to implement more automated test cases. Everyone will benefit from
> this, I think.
>
> I discussed this idea with Ara in irc channel, she also agreed with this
> proposal, so I'd like to send this idea out for getting more feedbacks.
> Any thoughts, suggestions and comments are always welcome. If GNOME
> community could accept this, I think that would be a great thing for
> both developers and QAs.
>
> Thank you.
> Regards.
> -Tim Miao
> Sun Desktop QA Team