Re: New GnomeGoal proposal: InstalledTests

[Travis Reitter <travis reitter collabora co uk>]

On Thu, 2013-04-25 at 16:41 +0100, Simon McVittie wrote:

This general plan looks really good to me. I think it'd help us really
tighten up quality at the boundaries of our projects.

> On 25/04/13 15:45, Colin Walters wrote:
> > Though you've got me thinking: maybe instead of
> > the /usr/bin/foo-installed-test naming scheme, we should
> > have /usr/share/installed-tests/foo.desktop.  Desktop files by their
> > nature already:
> >
> > 1) Are what we use to launch binaries
> > 2) Have metadata
>
> I like this idea, and I imagine it wouldn't be hard to hook this into
> other test harnesses (e.g. describe each test in that directory as a
> Debian autopkgtest, or get Jenkins CI to run them, or whatever).
>
> Having these test binaries on $PATH seems bad, particularly if they're
> allowed to damage $HOME when run without precautions, so having them
> canonically go in ${pkglibexecdir} (with the exact location being an
> implementation detail because of the use of .desktop files) would
> probably be better.

Additionally, I wouldn't want hundreds of installed tests block shell
tab-completion from being useful.

> > * Run as non root
> > * Assume the presence of a logged in session (or sometimes autocreate
> >   one with Xvfb/dbus-launch)
> > * Are nondestructive
>
> I think the default should be that tests are non-destructive, run as
> non-root, do not require a session, do not require networking and so on;
> if any of these are not true, they should be declared via something
> analogous to autopkgtest's Restrictions.
>
> The currently-defined Restrictions in autopkgtest are rw-build-tree,
> breaks-testbed, needs-root and build-needed. The two build ones are not
> relevant for installed tests. GNOME could use needs-root, breaks-testbed
> (if it breaks the system) and breaks-home (if it trashes $HOME), perhaps?

We wouldn't need to support these types of tests immediately, but I do
think it'd be useful to plan for them. There are classes of tests which
I wouldn't want to run on my live system but which I would want run
automatically on a test system. Eg, a test that includes switching to
airplane mode and back to check that the system dims out network-related
functionality and re-enables it once the network is back up.

-Travis