Re: New GnomeGoal proposal: InstalledTests

[Dan Winship <danw gnome org>]

On 04/25/2013 10:21 AM, Colin Walters wrote:
> See:
>
> https://live.gnome.org/GnomeGoals/InstalledTests

> Installed tests go in /usr/bin/testname-installed-test.

That seems unnecessary, since they're not run as part of ordinary system
operation, and it will pollute autocompletion. I think it would be
better to have some other standard location for them.

Also, they need to include the package name (either as a path component,
or as part of the test name), to avoid conflicts. (eg, glib and libsoup
both have a "proxy-test".)

We could install both the test binaries and a Makefile into, eg,
/usr/libexec/$pkgname/tests/, and then you can just run "make -C
/usr/libexec/$pkgname/tests check" to run all of them. (And maybe have a
wrapper for that so you just type "run-installed-tests glib".)

> They should exit with code 0 on success, and any other code on
> failure.

I recently learned that automake's test harness supports two special
exit statuses: 77 for "skipped" (it's not currently possible/meaningful
to run the test, eg, because of a missing run-time dependency, or
because this system doesn't support the feature being tested), and 99
for "hard failure" (something went wrong and the test can't continue,
but the failure is not related the thing that's being tested). "Skipped"
is definitely useful. "Hard failure" is useful in theory (eg, "disk
full"), but less useful in practice since it would be a pain to
distinguish those sorts of failures in most places.

-- Dan