Re: Build sheriffs for GNOME

[Tristan Van Berkom <tristan upstairslabs com>]

On Sun, 2016-01-24 at 05:18 +0100, Vamp898 wrote:
> Hi, 

Hi Shirakawa,

I see you replied off list, I am returning this to the list because it
is a good question to answer and it's rather wasteful to compose a full
reply to this and not have the benefit of it being recorded in the
archives for posterity. Also if we shake the tree on the list, maybe
something interesting will fall out :)

> I hope this mail doesn't sound too stupid (I'm not that into the
> GNOME build system). 
>
> Jhbuild checks out the git repository and builds them. 
>
> I personally would say that even master should be buildable,
> otherwise it's not of much use, or is it? 

No it does not sound stupid at all, without a deeper understanding of
how the projects inside GNOME function when they are under active
development combined with some understanding of the history of how
things were done before we had jhbuild, it is perfectly understandable
that you might think that is what jhbuild is for.

In fact, it is even possible that some people within GNOME have
presented jhbuild to you as if it were a tool you could use to always
build and "obtain" the latest master of everything, and that it should
always work, that if it did not build it was in fact a problem that
some module could be blamed for. Things are a bit more complex than
just this, though.

I would in fact argue that jhbuild is exactly the opposite of this, one
of the great benefits of jhbuild is that we find incompatibilities and
breakages early on in the cycle rather than in the last few weeks
before release time.

> I rather have an old eds than none (and if master is broken due to
> changes in eds, it's like having nothing). 

There is one thing to take in context here: GNOME has never had a
policy which mandates ABI or even API stability for ALL projects which
participate in GNOME. API stability is only a hard requirement for core
platform modules who have opted in and advertised this, understandably
because these platform components are relevant also to the real world
outside of GNOME. Historically we have a competitive platform, and lets
hope that with work and discipline our platform will again be a viable
and top choice for mobile, IVI and other embedded sectors (but I am
digressing...).

The point is, for any project in GNOME which does not advertise API
stability, these projects are entirely free to change the API contract
during the course of a release cycle. Of course when this happens,
there are periods where other projects in GNOME depending on these will
be broken, and this is not incorrect or undesired. This does not happen
that frequently, but it's reasonable to expect that some applications
wont build with "everything master" for periods of weeks at a time.

We could mandate that every project which provides an API be always API
stable and extend this rule beyond core platform components, but this
would generate a lot more work for some not funded volunteer driven
projects, and it's an entirely different discussion.

> I'd say touching git (in terms of reverting commits) is the worst
> idea. 
>
> Jhbuild should take care of that. Jhbuild should find ( with the help
> of tags in git?) what is the latest thing that still builds in
> combination and check out that. 

It's not possible for jhbuild or continuous to distinguish at this
time:
  a.) When an application is broken
  b.) When a library broke that application
  c.) When a library changed their API contract and the application
      is rightfully broken in a transitional period

Further, it would not be desirable for jhbuild to automatically do what
you suggest, because then we would not see the breakage, exposing that
breakage as early as possible is one of the primary values jhbuild has
given us.

In order to shed some light, I want to share a little bit of history of
how things were done before we had jhbuild, because I will argue that
jhbuild is not a delivery system of latest GNOME, but that does not
decrease the value of jhbuild, it has really, really been a great step
forward in our development cycles.

So, before jhbuild, most developers of GNOME projects had their own way
of building their project against bleeding edge versions of other
components. Everyone had a different OS with different third party
dependency versions installed, and everyone did everything by hand.

What that means, is usually I would have a few lines of environment
script I would use to build:

  PREFIX=/opt/devel
  PATH=${PREFIX}/bin:$PATH
  LD_LIBRARY_PATH=${PREFIX}/lib:${LD_LIBRARY_PATH}
  PKG_CONFIG_PATH=${PREFIX}/lib/pkgconfig:${PKG_CONFIG_PATH}
  ACLOCAL_FLAGS="-I ${PREFIX}/share/aclocal"

  export PATH LD_LIBRARY_PATH PKG_CONFIG_PATH ACLOCAL_FLAGS

Personally I would mostly be building Glade, so to do that, usually I
would have to get the latest version of GTK+, so usually I would have
to update atk, cairo, pango, gdk-pixbuf and gtk+, and I would have to
do so in that order specifically.

I would source my environment script, and then for each checkout in
order I would have to:

  cvs update
  ./autogen.sh --prefix=/opt/devel
  make && make install

Sometimes, or usually even, it was enough to take the latest bleeding
edge tarball for each module, building from cvs is always a bit more
tricky and the tarball usually installs well (especially considering we
had some gtk-doc churn and updating things to build from cvs and have
all that gtk-doc stuff in place was just a pain).

Usually, after having setup my relocated build sandbox once, I would
build the least components possible, I would first try to build the
latest GTK+, and then get annoyed that it required a new pango, cairo
or atk.

Sometimes, GTK+ would not build even after passing configure, so I
would have to figure out what happened, usually it meant that GTK+
forgot to update configure.in with it's latest dependencies, so I would
hunt down the dependency and maybe end up rebuilding the whole stack up
to GTK+ ensuring I had the latest of everything before complaining
about any breakage to the GTK+ maintainers.

Pretty much nobody except the release team was building "all of GNOME",
and that was happening near the end of the release cycle.

Then shit started to hit the fan. Then, in our hypothetical EDS analogy
of choice, we finally found out, 2 or 3 weeks before release day, that
libfolks would not build. The gnome-contacts guys depending on libfolks
would be happily publishing release tarballs with their shiny new UI
changes claiming that everything works fine. It turns out that when
Milan (hypothetical situation) clearly announced on d-d-l that there
was going to be a change in the API contract, nobody maintaining
libfolks and gnome-contacts actually checked out the latest version of
EDS and rebuilt their code against the new version !

So, such were our problems, and jhbuild fixed a lot of this stuff:

  o First, jhbuild introduced the 'bootstrap' command, this ensured
    that we all had the same versions of autotools and external
    dependencies and such so you didnt have (many) situations anymore
    where the build would work on one persons system and not on another
    - everyone had exactly the same set of third party dependencies in
    their sandbox.

  o Suddenly it became fairly easy to build a given component in GNOME,
    jhbuild tracked the inter module dependencies and built them, with
    the correct environment and in the correct order all by itself.

    Many false bugs are eliminated in this step, did you remember to
    set the environment properly when building GTK+ after building
    pango ? did you remember to build GTK+ after pulling the latest
    gdk-pixbuf ?

    All of this was finally automated.

  o Cross module API churn related breakage was suddenly much easier
    to track, developers of high level components like gnome-contacts
    would suddenly notice that the API has changed in the components
    they depend on, and right away ! months before the release team
    was left with the mess of deciding "Are we going to ship the last
    cycles version of EDS or are we going to just forgo shipping
    gnome-contacts at all this cycle" ?

    While this question is still technically left in the hands of the
    release team, it happens much, much less now that we are capable
    of identifying churn early on and addressing it before it falls
    on the release team.

In closing, jhbuild automation has indeed solved a lot of problems
related to coordinating releases of these inter-related projects, all
of which have separate goals and separate priorities but have in common
that they can be used in the context of an integrated GNOME Desktop
Environment.

What jhbuild is not, is a delivery system to users who want to try the
latest bleeding edge version of every project in GNOME together. Even
though it is possible that 80% of the time or even more, every
project's master branch happens to build together harmoniously, it's
simply a fallacy to presume that it can always be so.

Thank you Shirakawa for raising the question, please do not feel shy to
post directly to the list in the future as it benefits everyone to have
it out in the open :)

Best Regards,
    -Tristan