Re: GNOME Modulesets in BuildStream - Baby Steps

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Wed, 2017-10-11 at 22:30 +0900, Tristan Van Berkom wrote:
> Hi !

Here is a follow up and status report.

Was hoping to get this out a whole week ago, but with significant
BuildStream maintenance overhead, and dealing with librsvg now using
rust, this was a bit delayed.

Summary first: things are working very well and at least to my
satisfaction (more details in post scriptum, trying to not be overly
verbose here).

Regarding the overall transition, there will have to be a cut off point
at which we stop maintaining the jhbuild modulesets and doing the
conversions, and we start maintaining the new BuildStream project
directly instead, and this has to happen before we start supporting
building the Flatpak GNOME SDK from our release modulesets.

I think the best would be to:

  a.) Start releasing using BuildStream immediately
  b.) Make an announcement of this on d-d-l and any relevant places
      also immediately
  c.) Keep the conversions going, and maintain the project in
      JHBuild format for a couple of months, so as to not be too
      disruptive to GNOME developers
  d.) Recommend that developers start to use BuildStream instead
      of JHBuild to build GNOME, so they have some time to adjust
      before the cut off.
  d.) Decide on a cut off date for JHBuild, perhaps end of 2017 is
      a good target - well in advance of the 3.28 stable release,
      and also unblocks my work on delivering a Flatpak SDK at the
      same time.

I have not yet rolled out a GNOME release, but I would be happy to be
the first one to make a GNOME release using BuildStream to validate
what builds and ensure that I have a better and clear picture of the
entire problem space, maybe Javier can help me the first time around.

After that I would be happy to help another release team member make
the next development release using BuildStream and be there "on the
line" in case there is any frustration with the tooling.

Does this sound like a sensible plan ?


Cheers,
    -Tristan


P.S.:

Some additional status of BuildStream builds of GNOME follows here.


Getting Started Wiki
~~~~~~~~~~~~~~~~~~~~
Javier has helped a lot with the refresh for:

  https://wiki.gnome.org/Newcomers/BuildSystemComponent

And I've completed that work, the draft is available at:

  https://wiki.gnome.org/jjardon/BuildSystemComponentBuildstream

There are some things which may change in that wiki over time,
depending on the kind of workflow we want to recommend for developers,
and as some bugs get fixed in BuildStream.

I intentionally omitted the `bst build --track` option, which allows
one to track the latest sources and build the latest in one go, because
semantics will change to make this UX more comfortable. These related
issues are presently being addressed:

  https://gitlab.com/BuildStream/buildstream/issues/113
  https://gitlab.com/BuildStream/buildstream/issues/117
  https://gitlab.com/BuildStream/buildstream/issues/131
  https://gitlab.com/BuildStream/buildstream/issues/129


Status of builds
~~~~~~~~~~~~~~~~
I believe we've reached build parity with the 3.27.x releases, and
better.

For instance we were able to catch the issue of gnome-recipes not
building with the newer meson version we have in the modulesets, thanks
to BuildStream guaranteeing that we only ever build things against the
correct version of their dependencies. 

Thanks Matthias for fixing this !

I will run another build from scratch tonight and report the full build
logs again, which I expect will have the same number of failing builds
or less.

Interestingly I think there is no need for any manually tagging of
modules to 'skip' with BuildStream, we simply build with e.g.:

  bst --on-error continue build core/meta-gnome-core.bst

And the build log should report nicely what it was unable to build, and
what we could not try to build due to failed dependencies.


Status of conversions
~~~~~~~~~~~~~~~~~~~~~
The conversion script seems to be running seamlessly, automatically
updating the read-only repository at:

   https://gnome7.codethink.co.uk/gnome-modulesets.git

However, since I recently changed some things in BuildStream (a
significant change in the format which needed to be done pre 1.0), I
did not completely fix it for patches.

I fell short of this because:
  o Current 3.28 modulesets dont have those patches against mozjs
    anymore.
  o I hope that we wont have to do the conversions for very much
    longer.

However, I can fix this so that if we add patches to jhbuild, they
are automatically converted to BuildStream format nicely.

Nice example:

    Just today, I tried to build gnome-recipes with Matthias's fix, 
    and found that it fails to depend on gnome-online-accounts.

    I committed a fix to jhbuild 3.28 modulesets to make recipes
    properly depend on GOA.

    5 minutes later, I ran `git pull` in my gnome-modulesets.git
    repository, and that issue is fixed.

I'm quite satisfied with this :)