Re: Announcement/RFC: jhbuild continuous integration testing

[Tristan Van Berkom <tvb gnome org>]

On Fri, Feb 15, 2013 at 7:57 AM, Olav Vitters <olav vitters nl> wrote:
> On Thu, Feb 14, 2013 at 07:12:10AM +0100, Martin Pitt wrote:
> > That is indeed the long-term plan, but there's still some work to be
> > done before we can do that. The machine we are running this on has 64
> > 2.7 GHz cores and 64 GB of RAM, that really isn't a bottleneck right
> > now. The main two problems right now are that the "jhbuild update"
> > stage takes some 5 minutes to update all the ~ 160 git trees, and
> > that jhbuild build doesn't parallelize at all, i. e. build modules
> > which don't depend on each other could build in parallel.
>
> Could you perhaps do a test that does 10 git checkouts at once (real
> ones, while things are updated and so on)? I think you might eventually
> run into issues with the bandwidth between Red Hat NOC and Canonical
> NOC.
>
> If you see a bottleneck problem, please say so so that it can be worked
> on at the same time as parallelizing jhbuild. E.g. there was a plan to
> mirror git.gnome.org, but there are also a few GNOME servers @
> Canonical that could be reused.

Fixing that bottleneck also sounds like one of the first steps towards
setting up a patch queue approach, i.e. set up a local mirror of all
remote git repos which is periodically updated... and fixup jhbuild
to optionally update from those local mirror instead of accessing
remote ones all the time.

Of course, making jhbuild parallelize the actual module builds is
a bit more complex (probably just a bit of python scripting ? not sure).

Another thought, if this gets integrated in a 'mostly a feature of jhbuild'
fashion, this can have some really interesting additional benefits.

Many projects that use the gnome platform libs already do create
their own modulesets, and can then easily set this up on their own
build server for their own project (improving the gnome developer
story in a big/real way), first thing that comes to mind is the osx
builds of Clutter and GTK+ (which are mostly just customized jhbuild
modulesets), not sure about win32... do we use jhbuild inside MSYS
I can't remember ?

Cheers,
      -Tristan