QA Team

[Sriram Ramkrishna <sri ramkrishna me>]

Hello Release Team!

We are planning on spinning up a QA Team and we're looking for some
feedback on how we can integrate with the release team.  QA will
greatly help us improve the quality of our releases, find out
regressions before the public discovers them, and provide feedback to
maintainers and designers.

For instance, we would like to define the components that you consider
should be QA'd as part of the release.  Since you control what
components go into GNOME it stands to reason that you would be
interested in knowing the quality of each of the packages that is part
of that release.

Our plan right now is to split between automated tests and manual
tests.  We have not really defined what that is, but it would be
helpful to start with the list of components that you would be
interested it.  It could just be a jhbuild list gnome-suites-core-3.12
or something.  But it would be good to get the direction.

The idea is to test everything wtih as much automation as possible
while finding interaction bugs in the manual test.  Now that we have a
continuously integration thanks to Colin, I think we have the
infrastructure to make this a reality.

Phase 1:
* define the list of components that will be QA'd
* define how QA interacts with the release team - what tests would
release team want to decide the quality of a release
* define interaction between release and engagement team (who are
doing the release notes) in order to define the features, regressions,
and hatever else we might be interested in.

Phase 2:
* define the tests for each of the components
  * what will we test, and what won't we test
  * possible call for volunteers to define manual tests
* UI automation tests will be done with dogtail
* UI interaction tests will be done with a team of contributors
* Unit tests will conceived and implemented by module maintainers for libraries

Phase 3:
* Document how to write UI automation tests using dogtail
* Document how to download an image, and run manual UI tests
* Define manual automated tests for each of the components

Phase 4:
* Call for volunteers to manual interaction tests on specific images
* Call for volunteers to write automated UI tests
* Write automated tests and integrate them into the continuous build server

Phase 5:
Profit!

This is sort of a rough idea of what I'm planning.

Vincent - I would love to get feedback from the opensuse QA person.