Re: New GnomeGoal proposal: InstalledTests

[Simon McVittie <simon mcvittie collabora co uk>]

On 25/04/13 15:45, Colin Walters wrote:
> Though you've got me thinking: maybe instead of
> the /usr/bin/foo-installed-test naming scheme, we should
> have /usr/share/installed-tests/foo.desktop.  Desktop files by their
> nature already:
>
> 1) Are what we use to launch binaries
> 2) Have metadata

I like this idea, and I imagine it wouldn't be hard to hook this into
other test harnesses (e.g. describe each test in that directory as a
Debian autopkgtest, or get Jenkins CI to run them, or whatever).

Having these test binaries on $PATH seems bad, particularly if they're
allowed to damage $HOME when run without precautions, so having them
canonically go in ${pkglibexecdir} (with the exact location being an
implementation detail because of the use of .desktop files) would
probably be better.

> * Run as non root
> * Assume the presence of a logged in session (or sometimes autocreate
>   one with Xvfb/dbus-launch)
> * Are nondestructive

I think the default should be that tests are non-destructive, run as
non-root, do not require a session, do not require networking and so on;
if any of these are not true, they should be declared via something
analogous to autopkgtest's Restrictions.

The currently-defined Restrictions in autopkgtest are rw-build-tree,
breaks-testbed, needs-root and build-needed. The two build ones are not
relevant for installed tests. GNOME could use needs-root, breaks-testbed
(if it breaks the system) and breaks-home (if it trashes $HOME), perhaps?

    S