Re: New GnomeGoal proposal: InstalledTests

[Colin Walters <walters verbum org>]

So, two weeks later.  A quick status update on prototyping this:

First, I converted gjs to have an "installed-tests" subdirectory:
https://git.gnome.org/browse/gjs/commit/?id=54538f3da91b25ed20c9deaeda00d57fb252227c

That worked, however I got some pushback:
https://bugzilla.gnome.org/show_bug.cgi?id=698935

Meanwhile, I tried doing this for both glib:
https://bugzilla.gnome.org/show_bug.cgi?id=699079
and GTK+:
https://bugzilla.gnome.org/show_bug.cgi?id=699601

Of those, the glib patches are unreviewed as of yet (but work in local
testing).  Much more exciting is that I now have the GTK+ reftests under
automation; they run after every build, and I'm taking screenshots along
the way:
http://build.gnome.org/ostree/buildmaster/tasks/integrationtest/saved-20130510.1/work-gnome-ostree-gnome-ostree_buildmaster_x86_64-runtime-disk/screenshot-7.png

(I manually saved that particular task run, otherwise it'd eventually be
rotated out by later integration test runs)

I also converted the ostree tests to be installed:
https://git.gnome.org/browse/ostree/commit/?id=e9a181c67d7d9bccc4aafd172b14996ae4dc5d43

So, is this useful and worth the additional maintenance as a different
option from "make check"?  I strongly believe so.  For the core modules
like glib/gtk+, it definitely felt conclusive from this thread that we
should keep "make check" working.  Sometimes that was quite un-obvious
and tricky, and so for "my" module ostree I decided to just burn the
bridges and have the tests only work installed.

Next steps:

* Add --enable-installed-tests to jhbuild
* Teach Debian's Autopkgtest how to run these
* Improve gnome-desktop-testing's runner binary "gnome-desktop-testing-runner"
* More tests.  More!

The specification is still open to revision of course too.