Re: New GnomeGoal proposal: InstalledTests

[Martin Pitt <martin pitt ubuntu com>]

Simon McVittie [2013-04-25 16:41 +0100]:
> I think the default should be that tests are non-destructive, run as
> non-root, do not require a session, do not require networking and so on;
> if any of these are not true, they should be declared via something
> analogous to autopkgtest's Restrictions.

Right, and each of these limitations makes it harder (and thus less
likely) for developers to actually run them regularly. Running tests
as root is not much of a drag, but jumping from "sudo make check" to
"have to build and boot a VM and run tests there" is a major
degradation of turnaround time.

Fortunately there are nice and proven ways to avoid touching the
production system's bits, like

 - GSETTINGS_BACKEND=memory

 - Set $HOME, $XDG_CONFIG_HOME, $XDG_DATA_HOME, $XDG_RUNTIME_DIR to
   temporary locations (especially $HOME now works with glib 2.36)
   (example: gvfs)

 - Run stuff on a temporary private session and/or system D-BUS
   (example: upower, gvfs)

 - If root tests need to change some files in /etc, /var/lib/foo/
   etc., run them through 'unshare -m' and put tmpfs mounts over these
   paths (example: gvfs)

 - If the program depends on a particular state/behaviour of things
   like logind or polkit, use python-dbusmock to mock them, instead of
   actually creating active/inactive sessions or actually suspending
   the machine (example: gnome-settings-daemon, udisks)

 - If you need to talk to a network server, start one locally with a
   local apache/ssh/ftp/etc. server (example: gvfs)

 - If you need to format/create/remove/hotplug block devices, use
   scsi_debug instead of trashing the actual hard disk (example:
   udisks)

 - If you need to test network device handling, use
   mac80211_hwsim/hostapd/dnsmasq instead of trashing the actual
   system's network (example: upcoming NetworkManager tests; they are
   not upstream yet, you can look at them at [1] for now.

I found that in most cases you can get away with not having to run
tests in a VM, and still retain the possibility of doing a lot of
those "intrusive" kinds of test bed setup.

Martin

[1] http://bazaar.launchpad.net/~network-manager/network-manager/ubuntu/files/head:/debian/tests/

-- 
Martin Pitt                        | http://www.piware.de
Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)