[ Revised Proposal ] Continuous Builds in GNOME

[Tristan Van Berkom <tristan upstairslabs com>]

On Tue, 2016-06-07 at 22:24 +0200, Sébastien Wilmet wrote:
[...]
> > This of course, requires real thought and engineering, but I wanted
> > to
> > at least offer some technical input, some starting point that could
> > be
> > explored - this wont be solved without real engineering, trial and
> > error.
> It's similar to the "master/master-stable-next/master-stable"
> branches
> concept that I thought about in this previous sub-thread. See those
> two
> mails:
> https://mail.gnome.org/archives/desktop-devel-list/2016-June/msg00002
> .html
> https://mail.gnome.org/archives/desktop-devel-list/2016-June/msg00008
> .html
>
> It doesn't look like rocket science to me, and it would get us closer
> to
> continuous delivery.

Honestly I hadn't put a huge amount of thought to this, but I did
figure in that once there is API churn, there will be multiple
variations/combinations that are desirable to build - one might want to
build a certain app which has not yet adapted to an API change, in
which case they need an amalgamation of branches which gets them their
app building without the API change in a lower level library/service.
On the other hand someone who wants bleeding edge of the given
library/service will want it with the API change included... but I
admit these considerations are possibly not worthwhile for our basic
goals: improve CI for development cycles and help have a more friendly
starting point for onboarding newcomers.

I also woke up this morning to an interesting conversation on #gnome-
hackers, which I missed due to timezone skew, but made me think of
something cheaper also more inline with what you suggest with master-
stable, master-stable-next etc.

So here it goes, after reading the interesting conversation I think
that we are close to something that is a step forward which will not
cost very much at all, I think that the main point we disagree on is
reverts in *master*, and the idea that *master* is something that can
be forced to be always stable.

For some of us, Milan and I at least, I think we disagree most
particularly on dirtying master, this creates work especially if we
have to pick up the pieces after some API change or such, and it
dirties the mainline development history which is a bad thing in
general, it makes bisections dirty and history tracking dirty, we've
come to have some discipline over the years to even avoid any
whitespace in commits because we value the history that much, we want
to know that every commit in mainline is relevant.

Simply put, if master is broken, it needs to be fixed the correct way,
and a maintainer has never advertised master as being stable until we
tag it as stable, we want to keep this.

So what I thought up after reading that IRC conversation, is that there
is no reason at all why all of this has to be done in master, and no
justification to want this behavior on master either - we just want to
be able to provide a closest-to-master as possible build that always
builds, and we want that to be the "easy way" that we advertise to
newcomers - more mature developers when they grow up might want to
build master anyway and see the actual breakages in real time.

Sorry I dont have time to spend hours reducing this huge text and being
all concise, but I want to contribute to this, so here goes:

Cheap proposal for a build-bot/release-team/build-sherif owned
integration branch
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
So what I would propose here is a setup with some automation, similar
to the proposed master-stable but I would call it 'integration' and
with a few changes.

Starting with our current workflow, all our projects/modules use the
master branch as mainline development, feature branches are merged into
master and eventually come release time, master is tagged and branched
as stable and we release tarballs for each release tag.

 [ feature ]   |
             \ |
              \
          [ master ]
               |
               |
       [ release tag ]
                \
               |  \
 [ feature ]   |    \
           \   |      \
            \  |        \
             \ |      [ release branch ]
              \|               |
               |               |
           [ bugfix ] -------> |
               |               |
               |      [ bugfix release ]

I dont think the above should change at all; patch flow for production
continues to flow in this traditional way that we have come to work
with very well. Master remains mainline.

What I would suggest, in addition to our setup, is that there is
a new 'integration' branch for every module in the moduleset, this
integration branch is automatically managed.

Workflow for contributors can look more like this:
        [ integration ]   [ master ] [ bugzilla patches ]
               |               |        /
               |               |      /  patches possibly
               |               |    /    made against
               |               |  /      integration branch
               |               |/
               |               |
  recommended  |        [ release tag ]
  build branch |               |\
      for      | <-----------  |  \
  newcommers   |    Auto CI    |    \
               |               |      \
  No commits   |               |        \
     here      |               |      [ release branch ]
               |               |               |
               |               |               |


Some assumptions I would point out about the above:

  o The integration branch has a strict commit policy, which
    is to say only the continuous automated merge process
    is every allowed to commit here - I would even go as far
    as to lock out maintainers from messing with the integration
    branch - This is really for an automated service, and build
    sheriffs can control the integration branch.

    In this case I really want a hard block, if I ever accidentally
    try to push something to 'integration' directly, I want an error,
    'integration' is not where patches and work enters the work flow.

  o Commits are never *reverted* from the integration branch,
    instead of this, the history of the integration branch is
    automatically *rebuilt* from master, and done so on a
    continuous basis.

    This can mean that the build bots or build sheriffs can
    launch a script which rebuilds the history of 'integration'
    based on master, but excluding the offending commit, IMO
    it's interesting to see 'integration' as the closest possible
    mirror to master which 'builds at all times'

Similar to Sébastien's suggestions about 'master-stable', the default
for running jhbuild would be to use 'integration' to build the bleeding
edge, with the added side effect of blocking all commits from any
committer.

A contributor can already choose pretty easily with jhbuild to build 3
or 4 modules directly from master if that's what bakes their cake, but
it's not the default behavior.

I think in the big picture, with this proposal we maintain the prestine
meaning of what master/mainline should be, and we get all the benefits
of a constantly building bleeding edge - in any case 'integration'
would be exactly the same thing that people in these threads are
proposing for 'master', it satisfies their requirement without
interfering with mainline.

It adds a little bit more work in setting this up:

   o Need a script to automatically/continuously rebuild
     'integration' from master
   o CI of the integration branch should be able to blacklist
     broken commits and "omit" them from the history
   o Some complex hooks need to be implemented gnome-wide
     in the git repositories, ensuring that any direct commits
     to the integration branch are blocked, with the exception
     of build sheriffs and the CI service.

There are also added benefits, making things easier for people
babysitting the build:

Build sheriffs never need to chase down maintainers explicitly before
blacklisting a broken commit, the blacklist (formerly 'reverts in
master') would simply effect the integration branch; maintainers should
be notified via bug reports that certain commits have been 'rejected
from integration' - this of course raises a red flag to the maintainers
because, any patches still rejected by integration cannot logically be
merged come release time.

I think the above outlined overhead may cost us... say 2 or 3 weeks of
scripting and setup from a single dedicated full time engineer, modulo
a week or so depending on whether we can find someone who is already
familiar with the gnome-continuous current setup and who can easily
push this through in our infrastructure.

Any thoughts ?

Cheers,
    -Tristan