Re: New GnomeGoal proposal: InstalledTests

[Colin Walters <walters verbum org>]

On Sat, 2013-04-27 at 15:40 +0200, Martin Pitt wrote:

>  (II) installed tests on installed system: regular verification of
>  daily image builds, self-check of installed systems and ostree
>  snapshots

But note that if we're talking about black box testing which is what
most of GNOME's tests are, the "II" case is effectively a superset of
all of the other cases; it's the most flexible and powerful.  It also is
the one that is easiest to match closely with what normal users actually
get.

You're talking about this "II" case as something "daily".  But I've
optimized the gnome-ostree build system now to the point where the build
server consistently gets to running installed tests in qemu 3-4 minutes
(sometimes less) after a git commit.   This is running in and testing
the real tree that users will download; it's not a weird partially
assembled pseudo-environment like jhbuild or a mock/pbuilder root.

This isn't hypothetical - while we were at the GTK+ hackfest this week
the gnome-ostree system detected that
https://git.gnome.org/browse/glib/commit/?id=ddb0ce14215cd62c7a2497d6cf9f2ea63c40ebb5
broke gnome-shell in minutes.

And this number is only going to go down; with various optimizations,
I'm pretty sure I could get a case like a git commit to gnome-shell that
only modifies a .js file booting within 30 seconds of the push
notification of the commit.  For example, by reusing a cached copy of
the previous build, so it's just a matter of "make && make install".

At that speed, a lot of the other test forms become superfluous.  They
were defensive measures necessary because package-based systems suck at
continuous integration.

> So it would be good if we can have all four and provide standard ways
> of running them.

At the moment in GNOME we have a lot of BB, and I'm pushing hard for II.
Per smcv's comment, I believe projects like Debian could effectively
reuse II tests in autopkgtest by simply ignoring the source tree aspect
and running the installed binaries.

IB can be subsumed by II if the build system allows just installing the
libraries, or just installing the tests.

Something like:
$ cd ~/src/glib
$ jhbuild make --enable-installed-tests  # builds and installs glib *and* tests
$ <hack hack hack>
$ jhbuild make --disable-installed-tests

Now you have a newer libglib.so in your build root, on which you can run the
previous binary tests.

> I fully agree, and I don't think there is much reason to drop
> uninstalled tests in favor of installed ones; In most cases it should
> be easy to support both. That's the model how development (should)
> work, after all: You write a test for a new feature, then implement
> it, and run "make check" (or "sudo make check") until it works; it
> would be a step backwards to require installing the tests (and
> clobbering your system) first.

But what if you had a build system that made it really easy and reasonably
fast to go from "list of git repositories" to "bootable VM"?  I do, and
it's changed my whole perspective on development and testing =)

Anyways, as far as an executive summary here:  Module maintainers
can/should keep supporting BB at their discretion, but are you OK with
the proposed II model?