Re: New GnomeGoal proposal: InstalledTests

[Colin Walters <walters verbum org>]

On Fri, 2013-04-26 at 22:56 +0200, Florian Müllner wrote:
> On Fri, Apr 26, 2013 at 9:44 PM, Colin Walters <walters verbum org> wrote:
> > Anyways...dropping tarballs is not going to happen soon, but I'm happy
> > to consider what we need to do to lay the foundations now.
>
> More short-term, would it make sense to leverage the build servers to
> run make distcheck when a new tag is pushed and take care of uploading
> the generated tarballs?

Maybe.  The "make dist" part would probably be relatively doable to
automate for most modules that don't have many "sysdep" type
dependencies.

But the "check" part is harder because we'd need a well defined,
reproducible environment for tests which goes back to the installed
tests design.  Currently it's completely up to the individual module
maintainers what's running on *their* laptop when they type "make
distcheck".  There's some modules out there that I'm sure have only ever
been distchecked on Ubuntu or OpenSUSE for example.

> (Though there's some temptation for maintainers to be lazy and create
> a tag without making sure that distcheck actually passes ...)

Yeah, that's an issue.  But there's another related and also important
one:  Remember, tests don't pass or not in an abstract sense; the only
pass or not *in a specific environment*. 

It's entirely possible that distcheck passes on *your* laptop but not on
the build servers.  For example, you have a 32 bit installation, the
build server is 64 bit.  This is the kind of issue that can be a
tremendous pain to debug because then you want access to the environment
so you can run gdb, etc.

Years of thought on this made me realize that it'd actually be far
better if the build server just spit out binaries as fast as it could,
and the tests ran automatically, continuously after that, tagging builds
which passed tests.  If you need to debug something, you *can* download
the binaries straight from the build server ("buildmaster"), and debug
those locally.  Which is how the gnome-ostree build system works.  But
that only came after several failed build systems attempting to do (part
of) something like what you're describing.

Going back to the top though, "make dist" seems somewhat plausible for
automation, the biggest question is what the environment is (jhbuild on
top of Ubuntu/Fedora?).