Re: GNOME Continuous and build breakage

[Matthias Clasen <matthias clasen gmail com>]

On Thu, Dec 10, 2015 at 1:46 PM, Emmanuele Bassi <ebassi gmail com> wrote:
> Hi all;
>
> I've been meaning to discuss this with the release team for a while,
> and I probably already annoyed a bunch of people on IRC, so here goes.
>
> I'd like the r-t to give its blessing to volunteers that decide to act
> as "build sheriffs" on Continuous builds. If we exclude the issues
> with the build machine itself throwing a fit — something that usually
> gets fixed by Colin kicking it — the vast majority of build breakages
> come from GNOME projects issues.
>
> What usually happens when a build goes into perma-red (i.e. it keeps
> failing over the same component) is that somebody on the #testable IRC
> channel (usually me or Colin Walters) tags the module inside the
> Continous manifest, opens a bug, and hopes that a fix get applied and
> communicated on the channel so that the tag gets reverted.
>
> This is not enough, and it does not raise the bar in keeping
> Continuous (and thus GNOME) building. It actually lowers it a fair
> bit, to the effective point that *nobody* cares about Continuous
> builds.
>
> I want this to change. I want to be able to revert failing commits on
> the offending modules, if they are hosted on GNOME infrastructure, if
> they fail for more than N hours, and *then* open a bug about it.
> Ideally, I want to tag only modules that are *not* hosted on GNOME
> infrastructure, as they are beyond our control and commit
> capabilities. In short, I want to ensure that GNOME maintainers become
> a bit more proactive in giving a crap about their modules breaking on
> something that is not their own computers.
>
> This obviously will need to be discussed on d-d-l, but I'd like to get
> some feedback from a limited audience, and hopefully have the release
> team backing this initiative — especially in the hope that we can have
> more than one build sheriff, to cover more time zones, and avoid
> perma-red build failures going on for more than two or three hours,
> instead of half a day.

This sounds ok to me - I think a policy of pinging the relevant
maintainer on irc first before reverting is a good idea (I know I
break things occasionally, and would appreciate a ping if I don't see
the breakage myself). I'd be happy to help out with this as well