How to make releases with BuildStream: Feedback Wanted

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

TL;DR:

  In general, we build and test against master and specific branches in
  git, and then last minute we convert to tarballs and "hope for the
  best", this is fragile and makes it difficult to ever have any hope
  of running CI against what we're actually going to release in advance
  of release day.

  Can we rid ourselves of any part of this process and tie things
  closer together ? How ?


Hi all,

    So I started a writeup of a proposal to kill off tarball
distchecking from the release process, but then realized that even if
we did, it would not solve the fundamental problem I hoped it might
solve (although it might make maintainer life a bit easier and allow us
to push out releases more easily, I think it's besides the point).

Basically, that fundamental problem I'm alluding to is that we are
essentially maintaining two builds at all times; and this gives us some
significant overhead and friction.

Currently the situation is that developers build and test their
software against whatever is the chosen git branch in JHBuild or now
BuildStream, but this is largely disconnected with what gets into the
release, which usually remains untested until it blows up in our face
at the last minute - this situation would not change by choosing git
tags instead of tarballs (although it might grease the pipeline a bit
by making module releases easier to produce).

So, I'd like to explore here what our options are, sorry if this is
going to be another long email.


Maintaining both in the same repository
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is not my favorite approach to the problem, however in the absence
of discussion and feedback it might be the default approach which is
least disruptive, this; or maintaining the status quo and
reimplementing the jhbuild conversion script to work with BuildStream.

I dont like the idea of producing a separate static release set
generated with some scripts, I find this makes the release process
clunky and disparate from the real build metadata we maintain in git,
also it means that we almost *never* test the release until the last
minute.

The downside to maintaining both in the same branch/repo is a bit more
verbosity in the BuildStream project, i.e. instead of listing only one
source for a given element, we could introduce a `release` conditional
statement, allowing one to build in "release mode":

    (?):
    - (release == True):
        sources:
        - kind: tar
          url: downloads.gnome.org:gtk+/3.22/gtk+-3.22.26.tar.xz
          ref: 61eef0d320e541976e2dfe445729f12b5ade53050ee9de6184235cb60cd4b967
    - (release == False):
        sources:
        - kind: git
          url: git.gnome.org:gtk+
          track: gtk-3-22
          ref: 4de0e4b1edbeada94589af61791315c213cdc050

Instead of the more brief and nice:

    sources:
    - kind: git
      url: git.gnome.org:gtk+
      track: gtk-3-22
      ref: 4de0e4b1edbeada94589af61791315c213cdc050

I dont like the idea of maintaining things this way; but it does seem
better than maintaining two vectors of software versions but hiding
that fact behind some release team conversion scripts, at least we
could potentially run CI on what we're actually going to release next
as well as on the master branches.

Also, it should be noted that automatic tracking of new tarball
releases available on a tarball server is not implemented in
BuildStream yet, this is largely because there exist various methods
for publishing tarballs and we probably need to provide separate
tarball plugins for interfacing with those (e.g. sourceforge cannot be
crawled ftp style but offers it's own web API for discovering available
versions and download urls).


Drop the tarballs from the released build metadata
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This would simplify things a lot, but would also severely change the
nature of our releases - I wish this were an option but I am doubtful
that it is.

One of the more interesting things BuildStream gives us is the `bst
track` ability (needs implementing for tarballs as described above)
which lets us automatically update the project with the latest
references to source tracking branches (for git: latest git commit sha
on a given branch or symbolic tag).

This means that in theory; we can easily:

  o Continuously build things with the latest of tracking branches
  o Update the tracking branches when maintainers request that
    (i.e. GTK+ team can say "GTK+3 should now be built from the
     gtk-3-22)
  o Make a snapshot of the overall project by running `bst track`
  o Branch our project ("modulesets") when it's time to make a major
    point release
  o Commit and tag snapshots to the release branches of our project

In this way, a release becomes a tag in the upstream "gnome-modulesets" 
repository, which is just a tag of a snapshotted release, one which
guarantees that if anyone builds GNOME at that specific snapshot tag,
they will build _exactly the same thing_

Even better, is that with CI we can eventually get automated builders
to run CI and "propose release candidate snapshots" automatically.

The main problems with this approach however, is that we completely
disconnect what GNOME release team builds and releases, from what
module maintainers have declared as a "release" (if someone were to
collect and build whatever tarballs maintainers have published in the
real module releases, they would not get the same thing that we have
published).


Maintain _only_ the release modulesets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This is a bit of a straw man for the sake of completeness.

This is very much like saying "If you did not roll a tarball for your
release, your software developement just didnt happen", from a
downstream perspective, I think this is totally just and correct.

This would mean we would only ever maintain tarballs in the the
upstream gnome modulesets project; and developers would use workspaces
to wire up to their git repositories; and build and test only against
modules who *have* released a tarball.

This is mostly a non-starter in my opinion because it completely
defeats the purpose of CI, it delays integration for developers and
causes problems to be discovered too late in the cycle.


Middle Ground ?
~~~~~~~~~~~~~~~
I wonder if it's important for the release team to be making builds of
"exactly the tarballs or release tags which module maintainers
release".

Who are the consumers of the static release build metadata anyway ?

As far as I can see:

  o Downstream consumers consume tarballs or release tags directly
    from the upstream modules who produce them
  o It is still interesting to publish which new tarballs of GNOME
    modules have been released at a given time (i.e. release time)
  o It is still interesting to publish the collective NEWS entries

But, is it important that the static build metadata we include in a
release is based on the explicit versions which maintainers have
"pinned" as release versions ?

At face value it seems dilligent to do, but it looks like:

  o the cost is fairly high, what we release is disparate from what we
    test

  o nobody as far as I can tell *cares* about these static release
    modulesets anyway, nobody is building from the exact tarball
    build metadata that we've produced to match an exact release
    version (or, is this true ?)

So, maybe it's possible to find some middle ground where we continue to
publish the collective NEWS and new release versions published by
module maintainers, but only publish build metadata for a tested
"snapshot" of the latest git commit shas at the moment of a release ?


Any other thoughts on how we could potentially simplify this process ?

Cheers,
    -Tristan