Re: Announcement/RFC: jhbuild continuous integration testing

On Thu, Feb 14, 2013 at 3:12 PM, Martin Pitt <martin pitt ubuntu com> wrote:
Hello Tristan,

Tristan Van Berkom [2013-02-14  6:42 +0900]:
Upon reading this particular part (and I noticed before you are
using mostly jhbuild mechanics), it leads me to wonder, how
granular exactly are these rebuilds ?

Hi !

Thanks for answering in detail (and Colin and Emmanuele too, very
interesting stuff).

Right now, every 15 minutes. Sometimes longer, when the previous run
is still running.

I think ideally it would be great if builds could be triggered by
commit. In other words, commits are serialized chronologically and
each and every commit should trigger an entire rebuild, each rebuild
should build everything in the moduleset "up to the latest commit"...
separately, one after the other.

That is indeed the long-term plan, but there's still some work to be
done before we can do that. The machine we are running this on has 64
2.7 GHz cores and 64 GB of RAM, that really isn't a bottleneck right
now. The main two problems right now are that the "jhbuild update"
stage takes some 5 minutes to update all the ~ 160 git trees, and
that jhbuild build doesn't parallelize at all, i. e. build modules
which don't depend on each other could build in parallel.

Once we solve both, and we dramatically reduce the time of one run
from "several hours" (which is currently needed if e. g. a glib change
happens, which rebuilds pretty much everything) to < 15 minutes.

The way I imagine this works now (and this is a big assumption,
correct me if I'm wrong), is that a commit in a given module triggers
a jhbuild build, which would mean that:

   a.) Several commits could have been made in a given module
        by the time jhbuild actually runs... meaning we dont know
        which of the given commits in that lapse of time actually
        caused the fault.

That's right. It's massively better to know that one commit in these
15 minutes triggered it than "something in the past week", but still
not perfect as you say.

   b.) Module "foo" triggers a rebuild... and while jhbuild builds,
        it also pulls in new changes from module "bar", in this
        case it's possible that a recent commit in module "bar"
        caused another module "baz" to be effected,  but in the
        end it's module "foo" who is blamed (since module "foo"
        essentially /triggered a rebuild/)

That's a trickier thing. For most commits, one should actually be able
to build them independently, but sometimes those "in between" breaks
are inevitable. Say, you make an API change in a library and then
update your application to the new API, then in between you will get a
build failure. The next iteration should fix it again.
We have that problem independently of the frequency we build stuff of
course, as we can always hit a "bad time".

As someone mentioned/proposed earlier in this thread, this kind of temporary
error could probably be ruled out with a timeout (perhaps not a real timeout,
but a measurement in elapsed time between commits).

In other words, no need to alert people if a breakage was almost immediately
addressed and fixed.

I'm sure with some more time and development we'll find the right approach
and refine things further (may include some kind of graph theory, trying some
builds with different modules' changes applied in different orders and
false breakages this way).

Anyway, very exciting work, thank you for doing this :)


PS: One of the fun things this will allow is... to hand out build breaker
awards (something we used to do in a company I worked at, was to
hand out an "award" to the committer which introduced the most
build breaks this month, mostly just for giggles).

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]