GNOME Modulesets in BuildStream - Baby Steps

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi !

It's been a couple months since meeting at GUADEC, and forgive me if
I'm still not up to speed with things, but looking at this wiki page:

  https://wiki.gnome.org/Schedule

...suggests that the dust is settling for the current 3.26 release and
we have not really started with 3.28 - so I hope this is a good time to
start talking about a timeline and structure for migrating our
modulesets to BuildStream format.

Thinking of the bigger picture, there are a lot of moving parts (I wont
enumerate it all in exhaustive detail here) and trying to land
everything at once could be a potential disaster (not to mention, put
immense stress on myself).

I think if we make sure things are getting better with each step, and
that we dont introduce more than one drastic UX change for GNOME
developers, then we're doing it right.

So what I would really like to do, is to start with migrating only the
JHBuild modulesets as is; in advance of making the new GNOME
BuildStream project support generating the GNOME Flatpak SDK, and
making bootable GNOME VM images from the same project data.

Also, this will give GNOME developers a chance to try building what
they normally build with JHBuild using BuildStream instead, well in
advance of the 3.28 release, which is also important.

So currently what I would envision for this first step would be to
continue using the ostree repos we generate from debian packages
(available for builds on i386, x86_64, armhf and Aarch64), which very
easily provides everything we currently describe in our sysdeps JHBuild
moduleset (sysdeps just become a list of packages used to create a
runtime to build on).

These are the immediate improvements I can see:

  o Exactly the same base system for everyone - no inconsistency
    and no host dependencies.

  o The artifact sharing features; downloadable individual builds
    in the case that they have been built by an automated server
    (this will be easy enough to setup almost right away - not
    real CI, but at least availability of shared builds)

  o Powerful parallelization of builds, if your laptop or
    build machine can handle that kind of thing (otherwise
    control the number of maximum parallel builds).

  o Default behavior of strict build order, I noticed Matthias
    mention in #release-team the other day that some time was
    lost due to JHBuild reusing cached builds (I guess this
    just means forgetting to rm -rf the install prefix with
    JHBuild, though).

  o Ability to branch / tag an exact GNOME release from git, and
    use `bst track` to automatically try release candidates
    using the latest git commit sha for a given module branch.

These are where I think we achieve feature parity with JHBuild:

  o Ability to launch and debug graphical (or other) applications
    you build, via `bst shell`

  o Ability to build from your own git clone, requiring users
    to call `bst workspace open` to create workspaces for
    the modules they want to work on

  o Optionally allow lax, non-strict build order; so developers
    can still do their `jhbuild buildone` kind of workflow without
    having to rebuild everything.

And depending on how you look at it, this is a regression:

  o Having a constant, relocated install prefix on the host, since host
    tools and libraries just never enter the picture with BuildStream,
    this part just doesnt fit.

    Of course `bst shell` will still generate one on the fly, and
    one can be created with `bst checkout`... However; since it was
    never built against dependencies on your host system, it doesnt
    really bear any resemblance to the JHBuild prefix.

What would be lost I think, is if some people currently use the jhbuild
prefix and do really invasive hacks on their host, to say; run gnome-
session or GDM built by JHBuild - I think this UX is lost until we come
up with bootable VM images - the redeeming part of course being when we
*do* get to generating VM images - developers will be testing on
something more relevant than an invasively hacked host system.

Am I missing something that maybe people are doing with JHBuild that I
might not have accounted for ?

Does this sound like a reasonable way forward to others ?

I should be able to setup something (or ensure the existing conversions
are still working nicely) in less than a week for us to look at
internally and consider, as I would expect other release team members
will want to play with this well in advance of considering rolling it
out to a wider audience.

Looking forward to any feedback !

Cheers,
    -Tristan