Re: Release team now using gnome-build-meta repository, not JHBuild

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi all,

This will be a long email, so TL;DR:

   o At this time you can keep using JHBuild for these few cases where
     testing is very difficult, you can even do so indefinitely.

   o For these few modules who dont benefit directly at development
     time from the release team's official build metadata, the effort
     should be very low.

   o We will have a way for you to test core components soon, and it
     will be superior in many ways, but will inevitably lose *some*
     of the convenience which JHBuild offered.


On Mon, 2018-01-22 at 14:31 -0600, mcatanzaro gnome org wrote:
> > On Mon, Jan 22, 2018 at 12:56 PM, Carlos Soriano <csoriano gnome org> 
> wrote:
> > What's the workflow for those before a proper solution is done? Or 
> > are the developers of those modules expected to maintain JHbuild on 
> > the meantime?
>
> Thanks Carlos; this is an important question. Let me provide a bit more 
> context for our decision to stop maintaining JHBuild now, even though 
> BuildStream is not yet be usable for running some modules.
[...]

   I'd like to thank Michael for chipping in here while I was enduring
a travel itenerary which takes over 20 hours all told, and recovering f
rom it's side effects.

There have been some discussion about the pain points for testing the
few, albeit very important and central components of the GNOME UX on
your own developer laptop.

First of all, to avoid blowing things out of proportion, we are talking
about the maintenance of a few configure arguments for a handful
modules here. These modules are not easily verifiable and testable on
your laptop using BuildStream, *yet*.

Yes the release team demands that you update the gnome-build-meta
repository for these for our purposes of delivering integrated builds
and releases of GNOME as a whole, and these few core modules might not
yet be benefiting from those builds as developers - but these configure
arguments don't change immensely often, and whatever work the release
team did before in terms of maintaining them (mostly this is manually
checking the status of tarball converted builds at release time), is
not benefiting your experience as a developer on these modules
anyway.

A better way to test an integrated GNOME is coming, but for the time
being:

  Please update the gnome-build-meta repository to help us ensure that
  it's building. And feel free to continue to use JHBuild modulesets
  without those modulesets being considered in the integrated releases.


With that out of the way, in the long term; yes it's also fine if you
continue to use JHBuild for your own development workflow preferences,
but I am very driven to convince you that ultimately we will have a
better way of doing things both for the release team and for all GNOME
developers, and these visions should be aligned. So, I'll try to
alleviate some of your concerns regarding the long term picture of
development.


To start, here is a general outline of how the tooling differs in their
core values, and how these tools are effectively crossed in purposes:


   On JHBuild
   ~~~~~~~~~~
   JHBuild is a tool which prioritizes developer convenience,
   attempting to build the "latest of GNOME" on top of a loosely
   defined abstract system.

   Various tricks are in place in the GNOME modulesets, for instance
   modules which we *can* build, but avoid trying if we have the chance
   to determine that a local pkg-config file is found for it.

   The result is that you can mostly test your software on your
   machine, and for the majority of high level apps, this is good
   enough for the developer to test and cover the majority of use cases
   in manual testing.


   On BuildStream
   ~~~~~~~~~~~~~~
   BuildStream is a tool which puts determinism of builds first and
   allows absolutely no room for unknown factors contaminating your
   integrated build result.

   Various safe guards are put in place to ensure that you are never
   building or testing a result that cannot be reproduced exactly on
   another laptop.

   The result is that you are never uncertain about which module
   changed from one build to the next when a bug was introduced,
   and you are never writing software against the old version of
   a changed component which happens to be lying around on your
   system, causing last minute pain at release time.

   Whether GNOME is broken or is working well, you always know that
   the integrated result of a GNOME build is exactly what you expect
   it to be.


I should also point out that from a release team perspective, our
priority has to be delivering an integrated GNOME where it's components
always work well with eachother at release time, not a hand full of
components which happened to work well in their respective maintainers
hacked distro setups.


Developers in FOSS have historically filed the integration work under
the "not my problem" category, and the result of that is a lot of back
and forth with the downstream distros which ensues until *they* finally
get a well functioning integrated result, which is probably at least a
year if not two years behind what we're working on now. The software
that reaches distro standards of quality is often end of life by the
time the user ever sees it.

We recognize that this is really hard work, though; and as such
we're trying to push the envelope in the tooling to bridge the gap
between the two camps: development and integration.

We have to educate developers that integration matters and that running
a whole GNOME session against the undefined distro packages which
happen to be on their laptop is not sufficient, but more than this we
have to make it actually easy to accomplish.

So to the (hopefully temporary) dissapointment of some:

  No - BuildStream will never enable you to run the software you built
  against your host runtime in the ad-hoc way that JHBuild has enabled
  this, but this is a very good thing.

We will however be providing VM images (which can even be flashed onto
hardware and used without the VM) which a developer can easily use to
test their module against an actually relevant set of dependencies,
i.e. the ones which we expect GNOME to be released against in the next
cycle.

Also to clarify, the work on VM images is not theoretical or some kind
of work in progress. This was tested and working over 6 months ago but
cannot just easily be repeated while switching to the new structure of
build metadata in GNOME.

The following example has probably regressed due to API changes over
the last 7 months but the original (outdated) element.bst file which
creates a bootable GNOME VM image is here:

    https://gitlab.com/BuildStream/buildstream-tests/blob/build-gnome/elements/gnome/gnome-system-image.bst

The x86image element type used in the above is defined in the 
bst-external repository and is not outdated, but needs to be
reintegrated into a better defined build of real upstream gnome, its
(not very great) documentation, including the shell commands which are
run in the sandbox, can be found here:

    http://buildstream.gitlab.io/bst-external/elements/x86image.html#module-elements.x86image

It is loosely based on an audit of the `wic` code from the yocto
project and by using some modern fsck features in combination with the
syslinux bootloader, works entirely without the need of root
privileges.

The work which needs to be done here is a matter of:

  o Providing a fully bootable base system in the freedesktop-sdk
    project which also produces the new base runtimes for flatpak

  o Integrating the GNOME build metadata on top of that bootable
    minimal system (this means stuff like defining our PAM setup
    and such).

  o Just using the existin x86image element to generate the images

I dont think this is very far away now...

So yes we lose some convenience, but we gain other convenience and on
top of it, we actually test a relevant integrated whole which we get to
define, instead of an ad-hoc test which doesn't really prove much
(except that it "works for you, now, on your unsupported
frankendebian").

Consider the case of the newcommer who wants to scratch their itch,
there is a problem with unlocking GOA at login time:

  o The newcomer just has to sprinkle some debugging statements
    across gnome-keyring, GDM and gnome-session code to debug
    the "PAM conversation", build and boot the VM and figure out
    what is breaking.

    In this case the newcomer need not be a beginner, lowering
    the barrier of entry here makes it that much easier for a
    seasoned GNOME developer who usually doesnt hack on gnome-shell
    to write a patch and fix things.

  o There is no question about what the PAM configuration is
    on the host; we have decided it in our integrated system.

    This also gives us some opportunity to demostrate a template
    or model PAM configuration - we can make a statement that:

       "GNOME works when the system is configured like this"

    And even if distros dont configure the same way, this is
    immensely beneficial for them as a template, and for us internally
    to stay on the same page during development cycles.


Kind Regards,
    -Tristan