Re: Improving the way we build nightly apps

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Christian,

    I'm sorry to hear this from you especially, because I am very
interested in the things we can do for projects like Builder and it's
users, I remain very hopeful that you are going to be one of the people
most excited about what we can accomplish with BuildStream.

On Tue, 2017-02-28 at 15:50 -0800, Christian Hergert wrote:
> On 02/28/2017 03:18 PM, Michael Catanzaro wrote:
[...]
> You make it sound like there is already unanimous support for this 
> (considering I've seen no discussion on the topic, I find that hard
> to 
> take at face value)?

You are right,

I should mention that before making our final commitments to developing
BuildStream back in November, we did have some lengthly discussion with
Alex and GNOME release team members. Michael was a part of that
discussion and that would be why it seems this way.

It's a bit of a chicken-and-egg situation, discussing this with the
wider GNOME community without already having something that works is
something I wanted to avoid especially while the project was in it's
infancy, but we've been working on this and have something that is
working very well so far, features are still lacking and will be
presented in public as things develop.

> Having configurations that are generated from meta-configurations
> makes 
> IDE tooling a giant PITA. The .json files are our primary
> configuration 
> format for Builder going forward.

Sorry I dont follow what you mean by configurations generated from
meta-configurations.

I'm not sure if this is where you draw this conclusion from but: We do
plan to provide an automated conversion primarily as a reliable
migration path, I would not recommend constant migrations back and
forth as a matter of process.

> The further we abstract from that, the more difficult we make our 
> newcomer story. If they aren't in the projects git repository, this
> ends 
> up being as bad as jhbuild today where we have to synchronize 
> configuration changes between separate modules.
>
> I really don't want to see a development workflow where after
> checking 
> out a GNOME module we have to look up in an external system how to
> build 
> the thing, just to have to push changes back to said system when the 
> developer changes a project setting.
>
> If we do end up with something (other than the flatpak json files)
> that 
> end up in each projects module, then it is paramount that we have
> enough 
> information to tie together how to configure the project, how to
> execute 
> flatpak-builder, what build system to use, what runtime and sdk to
> wire 
> up, and of course, without having a pre-processed .in file.
>
> I don't see any reason the aforementioned project can't do that, but
> I 
> expect some bike-shedding about where configurations live. I hope
> this 
> serves as a major counter-point to the ultimately centralized choice.


Ok.
    I think Michael (and perhaps myself ?) framed this incorrectly with
centralized vs decentralized.

In fact in last year's Flatpak/GNOME Software hackfest in London, I was
pushing for exactly this scenario where Flatpak app json be hosted in
GNOME app modules.

Since we started with the BuildStream project this has not really
changed, I do completely endorse that for modules in GNOME which
produce "apps" (many, many modules do not fall into that category), the
maintainers of these modules should be in control of their build
related metadata.

Regarding centralization: I think that we can all agree that it's
suboptimal to have three separate sets of build metadata for building
what is essentially the "GNOME SDK/Runtime" and that whether this
runtime is to be installed directly on a base operating system image or
deployed as a Flatpak SDK/Runtime, it should still be the same thing
that we are maintaining.

So I think we need to think more about the bigger picture than just the
apps, and what process surrounds that:

 o How does the release team track what specific version of a GNOME
   "app" that is endorsed in an official GNOME release ?

   This becomes a little more tricky when the release team cannot
   control what commit sha to choose in one moduleset, currently
   we are using tarballs and sha256 sums in the release modulesets.

   One thing which is central to the BuildStream plan is something I'm
   calling "recursive pipelines" for now; which is basically a way to
   cascade projects and say that "a given element that I depend on, is
   actually another BuildStream project".

   With this approach, the release modulesets (which will be able to
   produce a VM on demand) should be able to depend on GNOME modules as
   BuildStream projects, where each 'app producing' GNOME module
   contains it's own build metadata. Referring to a GNOME module with
   specific commit sha and/or release tag ensures deterministic
   revisions of the build metadata therein.

   It's also plausible that we want to throw the whole aspect of
   release team blessing "apps" out the window with Flatpak and
   decouple Flatpak apps from our GNOME release story entirely
   (although I doubt this is desirable).

 o Can we wrap up a deterministic and repeatable bootable operating
   system image, with all of the core GNOME Apps installed in the
   Flatpak which is installed in that image ?

   Here I'm thinking, release team has blessed specific release tags
   of everything, and they can announce: just run this command and you
   will get an image for your VM of GNOME version x.y.z. We know that
   you will see _exactly_ the same thing in your VM as we saw when we
   blessed the official release.

   The problem of deploying a full OS with Flatpak SDKs and Apps
   installed in that bootable image is certainly a tricky one, but it
   is merely a technical one which I'm sure we can overcome elegantly.


To get more into the topic of this discussion, I'm a bit sad about the
timing of this proposal, as I had to spend 2 or 3 weeks putting out
fires for another project and otherwise would have probably by now had
a Flatpak BuildStream plugin which can produce the same deployments
that flatpak-builder does by embedding the desired metadata in a build
result and deploying to ostree (and all of that at a much lower
maintenance cost compared with flatpak-builder, Flatpak is too
important of a project to get bogged down in the details of building).

I would hope to relinquish maintenance of the flatpak deployment plugin
to the flatpak developer team so that Alex can be in control of that.

That said, as I am not there "right now" I cannot very well try to
block this change (which I originally endorsed anyway), however I will
have to come back soon with yet another proposal to migrate (with easy
automation) these very same json files in GNOME modules to BuildStream
projects.

Best Regards,
    -Tristan

PS:

Christian, you seem to have drawn some strange conclusions (generated
configurations and .in files ?) and as such I really hope you will join
us in #buildstream on IRC so that we might have a more natural
conversation and I can help to alleviate your concerns.

Also it would be great to have more input about what things will make
life better for Builder and it's users, and potential users.