application testing via InstalledTests

[Colin Walters <walters verbum org>]

Hi,

I'd like to talk about applying
https://wiki.gnome.org/Initiatives/GnomeGoals/InstalledTests
to application testing.

Now, the executive summary on InstalledTests is "improved make check
framework".  Its focus is on rigorous automation.

At the moment in GNOME, many shared libraries like glib and libsoup have
test suites via "make check" historically, and a growing
number are additionally implementing InstalledTests.

But few applications have test suites.  What application testing
exists tends to live in external downstream repositories.

Having implemented one of the fastest, most efficient and reliable
continuous deployment systems in the world[1] for GNOME, I'd really like
it if more applications had tests to take advantage of this =)

https://bugzilla.gnome.org/show_bug.cgi?id=721349

is a bug for Epiphany, which in turn links to some tests Matthias wrote
for gnome-software.

These tests happen to be implemented using a mixture of Dogtail and
DBus; they work by launching the application, talking to it via DBus,
and checking the results.

Your tests don't even necessarily need to spawn your app - you can
also install test binaries for internal components.  For example,
Epiphany could install a small test binary that checked the
behavior of the HTTP cache directly.  

What gnome-continuous will take care of is automating execution
of the tests.  But of course you can also use
"gnome-desktop-testing-runner" inside jhbuild, as intended.

The reason I want to see more application tests is because applications
are what people actually use.  A good example is that while GLib has
some basic tests for GSocket, if we run through the Epiphany tests,
we're *also* testing GSocket, as well as libsoup, and gtk+, etc.

Application testing thus creates a powerful virtuous loop, testing
everything at once.

So please take a look at the tests Matthias wrote for gnome-software,
and consider writing something similar yourself to test your
applications!

If you have questions about implementation, you can always catch me and
several others on IRC in #gnome-hackers (EST time), or just reply to
this mail.

Thanks!

Footnotes:

[1] If you know of any other project that comes close, please send me a
    link