Re: New GnomeGoal proposal: InstalledTests

[Colin Walters <walters verbum org>]

On Thu, 2013-04-25 at 16:30 +0200, Bastien Nocera wrote:

> What are the requirements in terms of what tests are allowed to do? I'm
> guessing they shouldn't touch the running environment, 

That's a good question.  The gnome-ostree testing infrastructure uses
qemu with a copy-on-write qcow2 disk snapshot for all tests; they're run
as an autologged in user with an active desktop session.

It'd absolutely be possible to have tests which mutate the environment,
or even run as root and possibly trash the system.

But the primary thing I'm focusing on now is transitioning most of
GNOME's set of tests which tend to have these characteristics:

* Run as non root
* Assume the presence of a logged in session (or sometimes autocreate
  one with Xvfb/dbus-launch)
* Are nondestructive

Almost all of the GLib, gjs, and pygobject test cases are of that form;
by having them be installed, builders and integrators can fully automate
running them however often they choose.

Though you've got me thinking: maybe instead of
the /usr/bin/foo-installed-test naming scheme, we should
have /usr/share/installed-tests/foo.desktop.  Desktop files by their
nature already:

1) Are what we use to launch binaries
2) Have metadata

If other people like that idea, I'll look at reworking the testing code
to use .desktop files.

> create files in
> the user's home directory, etc. Are we allowed to access the network for
> example?

I don't think we should write network-dependent tests at this time;
they'd be a huge source of unreliability.  

Now it'd obviously be very cool to have e.g. a static copy of a
wikipedia page hosted in GNOME infrastructure, and have a test which
automates running Epiphany against that page.

But let's learn to walk before we run =)

> Do we have (makefile) infrastructure to allow GTests to run both
> uninstalled (during make *check) and installed? 

Not at this time; that'd be nice, but I think the jhbuild model
mostly obviates the need for uninstalled tests, because jhbuild by
nature is a sandboxed environment independent from the underlying
system.

cd ~/src/glib
jhbuild make   # note: runs make install
~/build/bin/glib-proxy-installed-test

So if you broke glib, you still have the underlying system to fall back
on.  

Now for things like NetworkManager that jhbuild is incapable of
handling, I can certainly imagine how uninstalled tests would be very
convenient for developers.  And it would likely make sense for some of
those tests to support running them both installed and uninstalled.

But the point of the gnome-ostree model is precisely to test those types
of things that have historically had very bad testing coverage because
jhbuild is incapable of handling them, and "sudo make install" and
hacking up distribution packages also sucks.