GNOME Build situation and BuildStream

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

TL;DR: We are working to improve build tooling for GNOME software development, and
       are very interested in GNOME developer community feedback on the BuildStream
       approach we are taking, and how to achieve a successful transition.


Hi all,

    By now many participants of this list are already aware of our efforts on
the BuildStream tooling from reading my blog posts ([0] and [1]), which we aspire
to use in GNOME to improve GNOME's developer experience as well as improving the
process around maintaining GNOME releases.

At this time I would like to start a more open conversation about our plans,
ensure that we are all aligned on what the best approach should be for building
GNOME and also look at how we can implement a better build story for GNOME,
hopefully over the following months.

There is a lot to say here as it's a big topic, so to structure this a bit,
I'll talk about:

  o What are the existing use cases of building GNOME ?
    o What are the pain points related to these use cases ?
  o How do we plan to address these pain points using BuildStream ?
  o How would we implement BuildStream in GNOME with a smooth transition ?


What are the existing use cases of building GNOME ?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  o A developer uses JHBuild to build and hack on parts of
    GNOME on their local laptop or desktop machine, which
    can be any linux distro.

    Pain Points:

    * Non deterministic set of dependencies cause builds to fail in
      random unpredictable ways, which can be off putting for newcomers
      and also an annoyance to seasoned developers who want to fix some
      specific bug, but get caught up fixing other bugs instead.

    * It's difficult to debug core components such as the interactions
      of gnome-keyring, Linux-PAM, GDM, GNOME Session, GNOME Shell.

      Manual tinkering is needed, you either need a separate machine or
      a VM you've setup manually to recreate login scenarios and
      gnome-initial-setup scenarios, ensuring a seamless handoff to the
      initial user with their online accounts setup and all of this.

  o The release team needs to build and publish GNOME releases

    Pain Points:

    * Non deterministic dependencies again make things unclear as to
      what exactly we are releasing.

      E.g., we might know that this vector of GNOME module versions
      work well on some specific distro it has been tried with, but
      we can't know for sure that the result of a JHBuild of GNOME
      will behave the same on any person's distro.

      By the same logic, it becomes difficult as time passes to
      build older releases of GNOME in the future on more modern
      dependency sets.

    * Current tooling does not allow for any distinction from
      a specific version of something (be it a commit sha in a git
      repository or a specific tarball), and a symbolic branch name.

      With JHBuild (or flatpak-builder for that matter), you must
      either specify a symbolic branch, or a specific commit.

      To advertise a specific release involves a lot of tedious
      editing of build metadata to communicate new versions of a stable
      or development release set manually.

  o The Flatpak maintainers currently maintain their own set
    of build metadata to build the GNOME SDK and runtime.

    Pain Points:

    * Arguably the flatpak maintainership should not be accountable
      for maintaining the GNOME Runtime and SDK builds. I think we
      mostly agree by now that it would be great to have the GNOME
      release team have control of their own flatpak builds.

      There is however a large overlap of libraries and tooling
      which must appear in the GNOME Runtimes/SDKs and must also
      appear on an operating system running GNOME (libraries and
      services to support the display manager, the shell,
      the session, control center, etc.).

      Maintaining these sets of build metadata in separate formats
      and separate repositories is both burdensome and error prone:
      releases are not communicated atomically and nothing guarantees
      that GNOME 3.24 moduleset versions and GNOME 3.24 flatpak runtime
      versions coincide at all times.

  o Application module maintainers need to either build Flatpaks
    of their module or at least provide some flatpak-builder json
    for a third party to build it on your behalf.

    Pain Points:

    * Here there is not much in terms of pain points for the author
      of a Flatpak application. As long as the only bundled format
      your application wants to support is Flatpak, you'll only
      need to maintain one set of build metadata.

  o The CI story of building GNOME on automated build machines.

    Pain Points:

    * Here again the problem of maintaining multiple sets of
      build metadata in various formats comes up for the release team.

      We should ideally be using the same build metadata for building
      GNOME releases as we do for CI.


How do we plan to address these pain points using BuildStream ?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


  For Developers
  ~~~~~~~~~~~~~~
  First off, BuildStream never allows the build to be contaminated
  by host dependencies or tooling in any way. This means that for a given
  GNOME release moduleset, every developer will be using exactly the same
  set of dependencies and the same tools to build with; any and all failures
  perceived by one developer will be reproducible by any other developer
  regardless of what is installed on their host.

  Secondly, in addition to being able to quickly setup a shell environment
  and test application code inside a sandbox (similar to how people generally
  use jhbuild), we will also by default provide some build metadata to allow
  users to easily spin up a full VM of GNOME. This means developers can more
  easily test their components against a fully assembled GNOME environment
  with the latest master or latest stable of everything in GNOME, also it will
  become a lot easier to hack on core components which deal with the login
  scenario (i.e. GNOME Session, GDM, GNOME Keyring etc).

  We care a lot about the developer experience, so our aim is to achieve
  a comfortable developer experience for dealing with build failures and
  to accelerate your edit/compile/test cycles. We hope that the user will
  be able to modify a line of code in glib, and have a new booting VM of
  a fully functional GNOME system to test within minutes.


  For the Release Team
  ~~~~~~~~~~~~~~~~~~~~


    Deterministic builds
    ~~~~~~~~~~~~~~~~~~~~
    There are a few ways that the GNOME Release story can also be improved
    by using a virtually entirely deterministic build system.

    o We always know that what is released by GNOME is entirely reproducible
      by anyone who wants to try a release of GNOME.

    o We have much stronger provenance information about not only what
      dependencies were chosen to build GNOME on top of, but also how
      they were built, when compared to the minimal bound dependency model
      we have where JHBuild interrogates the host for a 'sysdep'

      We expect that this will be a great help to any distributions who
      wish to integrate GNOME. By examining our own complete reference builds
      it becomes easier to identify integration bugs both on GNOME's side and
      on the given supporting distro's side.

    o All GNOME releases or "reference builds", become entirely repeatable
      in the future. This should mean that using build metadata to build
      the entire GNOME 3.24.X and create a bootable image should produce
      exactly the same thing in 2017 as it will in 2027


    Tracking latest branch
    ~~~~~~~~~~~~~~~~~~~~~~
    Beyond deterministic builds and repeatability, I should also point out
    the `bst track` feature which automates the update of a specific revision
    from a symbolic tracking name (usually a branch).

    So in BuildStream, Source plugins[2] are what import external data in the
    pipeline, usually for the purpose of building, and usually Sources represent
    VCSes or some other means for obtaining external data. Every different kind
    of Source may be attributed both "ref" and "track" attributes:

      o ref: A reference to an exact version, every time the same "ref" is used,
             the imported data should be bit for bit identical as the last time.

      o track: A symbolic name which the plugin can use to update "ref", usually
               this is a branch name for a VCS.

    For the purpose of running `bst build`, all involved sources must have a "ref"
    already resolved (using git for instance, this means ensuring there are commit
    shas specified for every module you will build). However, as long as modules
    also specify a "track" attribute, their refs can be updated by invoking `bst track`.

    We expect this can save time for a release team member who might want to
    use the following kind of workflow:

      o Branch the GNOME moduleset repository for a given stable release, or
        use master branch for current dev.

      o Manually specify only the appropriate branch names for each module
        in the major point release (e.g.: all GNOME 3.24 branches).

      o Periodically make a release:

        - Checkout the appropriate modulesets branch locally
        - run `bst track` on the whole pipeline, automatically updating the commit shas
        - run `bst build` and make sure it at least builds
        - git commit the modulesets with the `bst track` results, probably with
          a commit message mentioning the release of GNOME "3.24.4" for example

    Coincidentally, a lot of the above is also interesting for a CI setup where
    a build machine would:

      o Checkout the appropriate modulesets branch locally
      o run `bst track` on the whole pipeline, automatically updating the commit shas
      o run `bst build` and any additional tests on the result
      o report any failures
      o if the build succeeds, branch, commit and push a release candidate branch
        for human review


    Build Variants
    ~~~~~~~~~~~~~~
    Ideally soon, will come a time where the GNOME release team will be responsible
    for rolling out the GNOME Flatpak SDKs in addition to the regular release
    metadata.

    This basically means for GNOME as a whole, we will start to have numerous
    "build plans"

       Plan A
       ~~~~~~
       - Stage a deterministic and bootable base system
         (ideally produced by a third party distro)
       - Build GNOME platform libraries on top of that
       - Build GNOME core modules and services like GDM
       - Deploy a bootable disk image

       Plan B
       ~~~~~~
       - Stage a deterministic flatpak SDK: freedesktop-sdk-images
       - Build some dependencies not provided by freedesktop-sdk-images
       - Build GNOME platform libraries on top of that
       - Filter and separate parts of the output (the Debug, SDK and Locale)
       - Deploy these separated results to an ostree repo, i.e. a Flatpak Runtime & SDK

    Our view is that the above two "build plans" can be described in the same
    build metadata (or "BuildStream project"), and that modifying the upstream
    GNOME build metadata in one place should immediately be reflected in any
    builds of either Flatpak runtimes or GNOME bootable images.

    Our vehicle for achieving this is called "variants", the semantics of
    "variants" are described in more detail in the documentation[3]. There are various
    ways to achieve configurability in build systems, however we have found that:

      o In build systems which treat every configuration as a "variable"
        and then allow build scripts to make conditional statements on
        any variable, we arrive at a combinatorial explosion of complexity
        where the project as a whole is virtually impossible to validate
        since it purports to support a staggering number of possible
        configurations (think of the buildroot configuration system
        or gentoo's "use flags").

      o In build systems which avoid this configurability, we end up duplicating
        the same metadata in cases where one module can be built in more than
        one way, and so we have a lot more metadata to maintain.

    Our concept of "variants" is the compromise, every element may itself
    advertise a fixed number of named configurations which it supports and
    consequently reduces the number of supported configurations. Conversely,
    one may depend on an element in such a way that is ambivalent of the
    variant, which avoids the need for duplication of any build metadata.

    Long story short, we have a very nice plan to allow GNOME to maintain
    a tree of build metadata which produces both Flatpak runtimes and bootable
    GNOME systems from the same metadata and software, demonstrating this
    is next on our milestones.


  For CI
  ~~~~~~
  Here I am unsure what the best way forward is for CI in GNOME. Clearly
  we want to be running installed tests and whatever other tests we can
  run on a booted GNOME disk image, and clearly we dont want to maintain
  separate sets of build metadata to do any of this.

  We may want to repurpose the existing GNOME continuous framework to
  just replace the build procedure with a BuildStream build of the GNOME
  release metadata. Alternatively, I think that running a gitlab instance
  for the CI cases may get us something superior as we can leverage
  a lot of gitlab features and maybe even offer an opt-in pre-merge CI
  experience for participating GNOME projects.

  Further than this, I should mention there is some movement to implement
  Source plugins to represent distro packaging sources, e.g. the dpkg
  source plugin[4]. With this plugin in place, I suspect it will be
  very easy to run tests of building and running GNOME against various
  third party distributions and versions of those. Imagine we could have
  a board somewhere which displays which distributions GNOME builds on
  without failing tests (we could always know when GNOME master is failing
  against debian sid, or latest fedora, etc).



How would we implement BuildStream in GNOME with a smooth transition ?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Ideally we would like to demonstrate all of this working in a side branch
of the GNOME release modulesets, without being disruptive or calling for
a flag day where everyone needs to switch over atomically.

While we were able to achieve something close to that for the Baserock
project which now has some automation to build converted BuildStream
projects beside the YBD tooling, the situation for GNOME is a bit more
complex since we already have two important sources to convert from (the
gnome-sdk-images repository and the release team's JHBuild modulesets).

So, I'm testing the water in advance to see how the community feels
about this plan in general, and also to gather opinions on what exactly
we should be doing next.

This is not the whole roadmap[5], but what I have on the menu specifically
for the cases of building GNOME and satisfying GNOME related requirements,
which is my top priority for the next months and listed in the order I
would currently have them done:

What we have on the menu right now, in the order I would have them done:

  o Complete the Source implementations needed for Flatpak JSON (almost done,
    bzr is supported by flatpak-builder and we need only add some more tests
    to merge bzr support in BuildStream)

  o Complete the Source implementations needed for converting JHBuild
    modulesets (I think we will only need svn to complete this list, possibly
    but hopefully not cvs or hg).

  o Experiment with Flatpak deployments

    Most of what BuildStream will have to do to generate Flatpak output for
    runtimes and applications is straight forward, just building into some custom
    prefixes and encoding some metadata files for Flatpak to read.

    Looking through flatpak-builder code however there are some customizations
    like post-install manipulation of build output specifically for the locale
    data to end up where Flatpak wants it to be.

  o Create conversion script for JHBuild modulesets.

    Here I would by default use a debian base system to build on top of if
    the development of the dpkg plugin[4] works soon. Then I would create
    a script which parses the JHBuild XML and outputs a BuildStream project.

    This is going to need some additional static supporting metadata:

    - We will need to maintain a list of modules (or 'elements') which should
      be built for the GNOME Flatpak SDK/Runtime but not for the GNOME Releases,

      E.g. non-GNOME dependencies which need to be added to the runtime but are
      not necessarily provided by the freedesktop runtime.

    - Conversely, we need the list of things to be included in GNOME Releases
      but are not built in the GNOME Flatpak runtime/SDK.

      E.g. core GNOME components such as gdm and gnome-session

    With this script in place, we should be able to continuously produce GNOME
    releases which boot in a VM, and also produce the GNOME SDK/runtimes for Flatpak
    without requiring a "flag day" to switch over.

  o Developer Experience

    The developer experience needs work before I can confidently say that it
    can replace JHBuild for the regular day to day requirements of developers.

    What we absolutely need for this is:

      o Developer Workspaces[6]

        These will allow the developer to override a Source plugin in their
        local checkout of the GNOME release metadata, so that for example,
        you build everything directly from gits or tarballs, but you build
        gedit from a local directory of your choosing.

      o Dual Cache Key Modes[7]

        The shape of this might change a bit, but the gist of it is that
        in a release setting, one wants to be sure that everything was rebuilt
        in the case that any of its dependencies have changed (this is currently
        the default for calculating whether something needs to be rebuilt).

        In a developer setting however, you want to go all the way down to
        glibc (or lets say glib) and add one printf() statement, or change a line
        of code, and then you want to rebuild only glib and start some application
        or run a fresh new VM with that modified glib, without rebuilding the
        whole world to get there.

        These two scenarios are obviously at odds with eachother so we will want
        some user configuration to make the decision of what to build and when.

    In addition to the above, it will be nice to also tie up:

      o Artifact Cache Sharing[8]

        Authenticated users or CI machines should be able to upload the
        results of individual builds. In this way when a developer wants
        to hack on a given module, they really only need to build the
        module(s) they are concerned with but can reuse (download) the build
        results of any other builds, provided those were built once and
        uploaded to an artifact share.

  o CI Investigation and implementation

    I prefer to keep this as the last piece of the puzzle, as stated above
    it's quite possible that we discover more powerful ways to leverage CI
    using gitlab, or it may be that refactoring current GNOME Continuous to
    build with the same build metadata maintained by the release team is
    the way to go.

    In either case I think the requirements here are:

      o Use the same metadata as release team, not a separate format/repo

      o Should be easily repeatable. As in anyone should be able to setup
        a new instance to run CI on GNOME in their home by following some
        steps in a README somewhere, ideally this should not take all day
        and it should be reliably repeatable.

      o Should not have any regressions over the current solution. Which
        means installed-tests should run, some screenshots proving a booted
        system and some apps, fast build times preferred over rebuilding
        everything on every dependency change.

    Asides from the above, it will be nice to:

      o Have also full builds run (i.e. rebuilds of anything with a
        changed dependency)

      o Have CI builds contribute to a shared artifact cache which users
        can tap into in order to save cycles on their own machines.


In Closing
~~~~~~~~~~
I am of course very excited about what we can accomplish and I very much
hope that you will share in my enthusiasm.

Does this look like a lot of work to do ? Yes it does, however there is
already a great deal of work done and momentum is on our side. Thanks to
Codethink sponsoring this project, we also fully expect to have budget to
work on this full time for some months to come, so I'm confident that we
can deliver on our promises.

Feedback and involvement in any form are greatly appreciated, are
there parts of the picture you think we've missed ? Please reply and
tell us about them :)

Any questions about what we have created so far and how it works ? Please
reply and ask about these !

Best Regards,
    -Tristan


[0]: https://blogs.gnome.org/tvb/2017/02/06/introducing-buildstream-2/
[1]: https://blogs.gnome.org/tvb/2017/04/10/buildstream-progress-and-booting-images/
[2]: https://buildstream.gitlab.io/buildstream/#sources
[3]: https://buildstream.gitlab.io/buildstream/format.html#variant-conditionals
[4]: https://gitlab.com/BuildStream/buildstream/issues/10
[5]: https://wiki.gnome.org/Projects/BuildStream/Roadmap
[6]: https://wiki.gnome.org/Projects/BuildStream/Roadmap/Workspaces
[7]: https://wiki.gnome.org/Projects/BuildStream/Roadmap/CacheKeyModes
[8]: https://wiki.gnome.org/Projects/BuildStream/Roadmap/ArtifactSharing