Re: Announcement/RFC: jhbuild continuous integration testing

[Tristan Van Berkom <tvb gnome org>]

On Thu, Feb 14, 2013 at 3:12 PM, Martin Pitt <martin pitt ubuntu com> wrote:
> Hello Tristan,
>
> Tristan Van Berkom [2013-02-14  6:42 +0900]:
> > Upon reading this particular part (and I noticed before you are
> > using mostly jhbuild mechanics), it leads me to wonder, how
> > granular exactly are these rebuilds ?

Hi !

Thanks for answering in detail (and Colin and Emmanuele too, very
interesting stuff).

> Right now, every 15 minutes. Sometimes longer, when the previous run
> is still running.
>
> > I think ideally it would be great if builds could be triggered by
> > commit. In other words, commits are serialized chronologically and
> > each and every commit should trigger an entire rebuild, each rebuild
> > should build everything in the moduleset "up to the latest commit"...
> > separately, one after the other.
>
> That is indeed the long-term plan, but there's still some work to be
> done before we can do that. The machine we are running this on has 64
> 2.7 GHz cores and 64 GB of RAM, that really isn't a bottleneck right
> now. The main two problems right now are that the "jhbuild update"
> stage takes some 5 minutes to update all the ~ 160 git trees, and
> that jhbuild build doesn't parallelize at all, i. e. build modules
> which don't depend on each other could build in parallel.
>
> Once we solve both, and we dramatically reduce the time of one run
> from "several hours" (which is currently needed if e. g. a glib change
> happens, which rebuilds pretty much everything) to < 15 minutes.
>
> > The way I imagine this works now (and this is a big assumption,
> > correct me if I'm wrong), is that a commit in a given module triggers
> > a jhbuild build, which would mean that:
> >
> >    a.) Several commits could have been made in a given module
> >         by the time jhbuild actually runs... meaning we dont know
> >         which of the given commits in that lapse of time actually
> >         caused the fault.
>
> That's right. It's massively better to know that one commit in these
> 15 minutes triggered it than "something in the past week", but still
> not perfect as you say.
>
> >    b.) Module "foo" triggers a rebuild... and while jhbuild builds,
> >         it also pulls in new changes from module "bar", in this
> >         case it's possible that a recent commit in module "bar"
> >         caused another module "baz" to be effected,  but in the
> >         end it's module "foo" who is blamed (since module "foo"
> >         essentially /triggered a rebuild/)
>
> That's a trickier thing. For most commits, one should actually be able
> to build them independently, but sometimes those "in between" breaks
> are inevitable. Say, you make an API change in a library and then
> update your application to the new API, then in between you will get a
> build failure. The next iteration should fix it again.
> We have that problem independently of the frequency we build stuff of
> course, as we can always hit a "bad time".

As someone mentioned/proposed earlier in this thread, this kind of temporary
error could probably be ruled out with a timeout (perhaps not a real timeout,
but a measurement in elapsed time between commits).

In other words, no need to alert people if a breakage was almost immediately
addressed and fixed.

I'm sure with some more time and development we'll find the right approach
and refine things further (may include some kind of graph theory, trying some
builds with different modules' changes applied in different orders and
eliminating
false breakages this way).

Anyway, very exciting work, thank you for doing this :)

Cheers,
    -Tristan


PS: One of the fun things this will allow is... to hand out build breaker
awards (something we used to do in a company I worked at, was to
hand out an "award" to the committer which introduced the most
build breaks this month, mostly just for giggles).