Re: jhbuild continuous integration testing: status and plans

[Simon McVittie <simon mcvittie collabora co uk>]

On 08/03/13 18:44, Colin Walters wrote:
> On Fri, 2013-03-08 at 18:02 +0000, Simon McVittie wrote:
> > On 08/03/13 14:45, Colin Walters wrote:
> > > This gets back to something I want to figure out how to do, which is
> > > standardize metadata for tests (e.g. "this test requires a public
> > > Internet connection", "this test requires a logged in desktop session",
> > > "this test requires root", etc.).
> >
> > It might be worth discussing this with the authors of
> > <http://dep.debian.net/deps/dep8/>
>
> Yeah, I saw it; the core conflict between what I want and that is that
> the requirement for being run from an unpacked source tree.

DEP8/autopkgtest assumes that tests being run from inside a source tree
will be the common case, because, well, right now it is :-) (make
installcheck, etc.)

> Basically, if one is making an operating system, binary compatibility is
> of foremost importance.  Thus, for example, I want to run *old* already
> built glib test binaries against *new* libglib.

DEP8 tests can work like this with sufficient support from the package's
upstream developer (telepathy-gabble-tests 0.17.x from experimental
works like this). I'd like to make dbus do this too, but it looks like a
comprehensive version will take quite a while due to bugs like
<https://bugs.freedesktop.org/show_bug.cgi?id=37849>.

Maemo's continuous integration system (CITA) enforced the approach you
advocate, and my experience from that system is that it requires
relatively intrusive changes to most packages to turn tests designed for
"make check" into something you can run like that. That's fine if it's
something the package's upstream maintainer wants, but if they don't,
maintaining it as a distribution patch is pretty awkward.

I might still have Maemo's libsoup patchset somewhere if you want to
make its tests work like that...

    S