Re: jhbuild continuous integration testing: status and plans

[Tristan Van Berkom <tvb gnome org>]

On Fri, Mar 8, 2013 at 12:48 AM, Martin Pitt <martin pitt ubuntu com> wrote:
> Hello fellow GNOMErs,
>
> after the first round of discussions a month ago[1] about the jhbuild
> CI building/testing [2] I'd like to give some status update.

Along the same topic, I wanted to bring to light something that
was recently raised in EDS bugzilla [0].

Vadim filed this bug, and apparently copied the makefile
from another module... I was wondering if standardizing
this would be interesting for the Continuous Integration
servers... the html output it creates is certainly interesting
to have, an example of the output can be found here [1].

My thoughts were mostly that in that proposed patch, the
included Makefile is a bit crack and hacked out, I would
be interested in standardizing that a bit.

My thoughts were:

  a.) Create an m4 macro implementing the gtester rules
       which could be included in gnome-common and easy
       to integrate into various GNOME modules

  b.) Perhaps extend the .doap format (who defines the
       .doap format anyway ?).. so that the build server
       (or jhbuild directly perhaps) could introspect whether
       a module supports the 'make gtester-report' target.

       In that way, the CI server could automatically collect
       these reports only for modules which support them.

Is it interesting for more modules to implement the more fine grained report ?

Did I miss some important information (i.e. is this already done somewhere
at least partially and I'm not aware of it) ?

Any additional thoughts on this ?

Cheers,
        -Tristan

[0]: https://bugzilla.gnome.org/show_bug.cgi?id=695688
[1]: http://dl.dropbox.com/u/8686253/test-report.html

> Since then, Jean-Baptiste has worked quite a bit on the scripts:
> Updating git checkouts as well as building modules are now
> parallelized (building still respects the dependency tree), so that we
> can now rebuild all the 160 modules in something like 15 minutes now.
> This brings us much closer to useful commit-level testing. It now also
> uses "jhbuild sysdeps" and a much smaller hardcoded list of extra
> build dependencies (which are not exposed by the module lists).
> If a build or test fails it now uses jhbuild's -C option to restart
> with a clean checkout, to avoid tripping over build system cruft from
> autotools file changes.  I spent some time chasing down long-standing
> failures in some modules, as well as filing bugs for newly identified
> regressions.
>
> Since the announcement, the system has stabilized a lot, and the set
> of test failures is now quite stable.
>
> The thing that hurts most currently is that the machine is behind a
> proxy.  This causes quite a lot of test failures (like libgdata's
> youtube test), as well as spurious build failures like [3]. We do plan
> to move this machine into the DMZ soon, so that it has unrestricted
> network access. I'd like to postpone sending out notifications until
> that happens, as otherwise we'll spam people too much about irrelevant
> issues.
>
> Once that happens, we'll set up email notifications for state changes
> (e. g. "pass → fails to build", or "test fail → pass") and send them
> to Jean-Baptiste and myself first, to give this some more real-world
> testing. If that has a low enough noise ratio, we were planning to
> mail the maintainers of the affected modules (if the module has a
> .doap file), with some hint to notify us if they want to opt out of
> notifications.
>
> Then we can investigate for some time how this works, and debate if
> filing bugs would be better or not.
>
> Does that sound like an acceptable plan?
>
> Thanks,
>
> Martin
>
> [1] https://mail.gnome.org/archives/desktop-devel-list/2013-February/msg00025.html
> [2] https://jenkins.qa.ubuntu.com/view/Raring/view/JHBuild%20Gnome/
> [3] 
> https://jenkins.qa.ubuntu.com/view/Raring/view/JHBuild%20Gnome/job/jhbuild-amd64-libgee/92/artifact/libgee.log
> --
> Martin Pitt                        | http://www.piware.de
> Ubuntu Developer (www.ubuntu.com)  | Debian Developer  (www.debian.org)
>
> _______________________________________________
> desktop-devel-list mailing list
> desktop-devel-list gnome org
> https://mail.gnome.org/mailman/listinfo/desktop-devel-list