TL;DR: We are working to improve build tooling for GNOME software development, and
are very interested in GNOME developer community feedback on the BuildStream
approach we are taking, and how to achieve a successful transition.
Hi all,
By now many participants of this list are already aware of our efforts on
the BuildStream tooling from reading my blog posts ([0] and [1]), which we aspire
to use in GNOME to improve GNOME's developer experience as well as improving the
process around maintaining GNOME releases.
At this time I would like to start a more open conversation about our plans,
ensure that we are all aligned on what the best approach should be for building
GNOME and also look at how we can implement a better build story for GNOME,
hopefully over the following months.
There is a lot to say here as it's a big topic, so to structure this a bit,
I'll talk about:
o What are the existing use cases of building GNOME ?
o What are the pain points related to these use cases ?
o How do we plan to address these pain points using BuildStream ?
o How would we implement BuildStream in GNOME with a smooth transition ?
What are the existing use cases of building GNOME ?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
o A developer uses JHBuild to build and hack on parts of
GNOME on their local laptop or desktop machine, which
can be any linux distro.
Pain Points:
* Non deterministic set of dependencies cause builds to fail in
random unpredictable ways, which can be off putting for newcomers
and also an annoyance to seasoned developers who want to fix some
specific bug, but get caught up fixing other bugs instead.
* It's difficult to debug core components such as the interactions
of gnome-keyring, Linux-PAM, GDM, GNOME Session, GNOME Shell.
Manual tinkering is needed, you either need a separate machine or
a VM you've setup manually to recreate login scenarios and
gnome-initial-setup scenarios, ensuring a seamless handoff to the
initial user with their online accounts setup and all of this.
o The release team needs to build and publish GNOME releases
Pain Points:
* Non deterministic dependencies again make things unclear as to
what exactly we are releasing.
E.g., we might know that this vector of GNOME module versions
work well on some specific distro it has been tried with, but
we can't know for sure that the result of a JHBuild of GNOME
will behave the same on any person's distro.
By the same logic, it becomes difficult as time passes to
build older releases of GNOME in the future on more modern
dependency sets.
* Current tooling does not allow for any distinction from
a specific version of something (be it a commit sha in a git
repository or a specific tarball), and a symbolic branch name.
With JHBuild (or flatpak-builder for that matter), you must
either specify a symbolic branch, or a specific commit.
To advertise a specific release involves a lot of tedious
editing of build metadata to communicate new versions of a stable
or development release set manually.
o The Flatpak maintainers currently maintain their own set
of build metadata to build the GNOME SDK and runtime.
Pain Points:
* Arguably the flatpak maintainership should not be accountable
for maintaining the GNOME Runtime and SDK builds. I think we
mostly agree by now that it would be great to have the GNOME
release team have control of their own flatpak builds.
There is however a large overlap of libraries and tooling
which must appear in the GNOME Runtimes/SDKs and must also
appear on an operating system running GNOME (libraries and
services to support the display manager, the shell,
the session, control center, etc.).
Maintaining these sets of build metadata in separate formats
and separate repositories is both burdensome and error prone:
releases are not communicated atomically and nothing guarantees
that GNOME 3.24 moduleset versions and GNOME 3.24 flatpak runtime
versions coincide at all times.
o Application module maintainers need to either build Flatpaks
of their module or at least provide some flatpak-builder json
for a third party to build it on your behalf.
Pain Points:
* Here there is not much in terms of pain points for the author
of a Flatpak application. As long as the only bundled format
your application wants to support is Flatpak, you'll only
need to maintain one set of build metadata.
o The CI story of building GNOME on automated build machines.
Pain Points:
* Here again the problem of maintaining multiple sets of
build metadata in various formats comes up for the release team.
We should ideally be using the same build metadata for building
GNOME releases as we do for CI.
How do we plan to address these pain points using BuildStream ?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~
For Developers
~~~~~~~~~~~~~~
First off, BuildStream never allows the build to be contaminated
by host dependencies or tooling in any way. This means that for a given
GNOME release moduleset, every developer will be using exactly the same
set of dependencies and the same tools to build with; any and all failures
perceived by one developer will be reproducible by any other developer
regardless of what is installed on their host.
Secondly, in addition to being able to quickly setup a shell environment
and test application code inside a sandbox (similar to how people generally
use jhbuild), we will also by default provide some build metadata to allow
users to easily spin up a full VM of GNOME. This means developers can more
easily test their components against a fully assembled GNOME environment
with the latest master or latest stable of everything in GNOME, also it will
become a lot easier to hack on core components which deal with the login
scenario (i.e. GNOME Session, GDM, GNOME Keyring etc).
We care a lot about the developer experience, so our aim is to achieve
a comfortable developer experience for dealing with build failures and
to accelerate your edit/compile/test cycles. We hope that the user will
be able to modify a line of code in glib, and have a new booting VM of
a fully functional GNOME system to test within minutes.
For the Release Team
~~~~~~~~~~~~~~~~~~~~
Deterministic builds
~~~~~~~~~~~~~~~~~~~~
There are a few ways that the GNOME Release story can also be improved
by using a virtually entirely deterministic build system.
o We always know that what is released by GNOME is entirely reproducible
by anyone who wants to try a release of GNOME.
o We have much stronger provenance information about not only what
dependencies were chosen to build GNOME on top of, but also how
they were built, when compared to the minimal bound dependency model
we have where JHBuild interrogates the host for a 'sysdep'
We expect that this will be a great help to any distributions who
wish to integrate GNOME. By examining our own complete reference builds
it becomes easier to identify integration bugs both on GNOME's side and
on the given supporting distro's side.
o All GNOME releases or "reference builds", become entirely repeatable
in the future. This should mean that using build metadata to build
the entire GNOME 3.24.X and create a bootable image should produce
exactly the same thing in 2017 as it will in 2027
Tracking latest branch
~~~~~~~~~~~~~~~~~~~~~~
Beyond deterministic builds and repeatability, I should also point out
the `bst track` feature which automates the update of a specific revision
from a symbolic tracking name (usually a branch).
So in BuildStream, Source plugins[2] are what import external data in the
pipeline, usually for the purpose of building, and usually Sources represent
VCSes or some other means for obtaining external data. Every different kind
of Source may be attributed both "ref" and "track" attributes:
o ref: A reference to an exact version, every time the same "ref" is used,
the imported data should be bit for bit identical as the last time.
o track: A symbolic name which the plugin can use to update "ref", usually
this is a branch name for a VCS.
For the purpose of running `bst build`, all involved sources must have a "ref"
already resolved (using git for instance, this means ensuring there are commit
shas specified for every module you will build). However, as long as modules
also specify a "track" attribute, their refs can be updated by invoking `bst track`.
We expect this can save time for a release team member who might want to
use the following kind of workflow:
o Branch the GNOME moduleset repository for a given stable release, or
use master branch for current dev.
o Manually specify only the appropriate branch names for each module
in the major point release (e.g.: all GNOME 3.24 branches).
o Periodically make a release:
- Checkout the appropriate modulesets branch locally
- run `bst track` on the whole pipeline, automatically updating the commit shas
- run `bst build` and make sure it at least builds
- git commit the modulesets with the `bst track` results, probably with
a commit message mentioning the release of GNOME "3.24.4" for example
Coincidentally, a lot of the above is also interesting for a CI setup where
a build machine would:
o Checkout the appropriate modulesets branch locally
o run `bst track` on the whole pipeline, automatically updating the commit shas
o run `bst build` and any additional tests on the result
o report any failures
o if the build succeeds, branch, commit and push a release candidate branch
for human review
Build Variants
~~~~~~~~~~~~~~
Ideally soon, will come a time where the GNOME release team will be responsible
for rolling out the GNOME Flatpak SDKs in addition to the regular release
metadata.
This basically means for GNOME as a whole, we will start to have numerous
"build plans"
Plan A
~~~~~~
- Stage a deterministic and bootable base system
(ideally produced by a third party distro)
- Build GNOME platform libraries on top of that
- Build GNOME core modules and services like GDM
- Deploy a bootable disk image
Plan B
~~~~~~
- Stage a deterministic flatpak SDK: freedesktop-sdk-images
- Build some dependencies not provided by freedesktop-sdk-images
- Build GNOME platform libraries on top of that
- Filter and separate parts of the output (the Debug, SDK and Locale)
- Deploy these separated results to an ostree repo, i.e. a Flatpak Runtime & SDK
Our view is that the above two "build plans" can be described in the same
build metadata (or "BuildStream project"), and that modifying the upstream
GNOME build metadata in one place should immediately be reflected in any
builds of either Flatpak runtimes or GNOME bootable images.
Our vehicle for achieving this is called "variants", the semantics of
"variants" are described in more detail in the documentation[3]. There are various
ways to achieve configurability in build systems, however we have found that:
o In build systems which treat every configuration as a "variable"
and then allow build scripts to make conditional statements on
any variable, we arrive at a combinatorial explosion of complexity
where the project as a whole is virtually impossible to validate
since it purports to support a staggering number of possible
configurations (think of the buildroot configuration system
or gentoo's "use flags").
o In build systems which avoid this configurability, we end up duplicating
the same metadata in cases where one module can be built in more than
one way, and so we have a lot more metadata to maintain.
Our concept of "variants" is the compromise, every element may itself
advertise a fixed number of named configurations which it supports and
consequently reduces the number of supported configurations. Conversely,
one may depend on an element in such a way that is ambivalent of the
variant, which avoids the need for duplication of any build metadata.
Long story short, we have a very nice plan to allow GNOME to maintain
a tree of build metadata which produces both Flatpak runtimes and bootable
GNOME systems from the same metadata and software, demonstrating this
is next on our milestones.
For CI
~~~~~~
Here I am unsure what the best way forward is for CI in GNOME. Clearly
we want to be running installed tests and whatever other tests we can
run on a booted GNOME disk image, and clearly we dont want to maintain
separate sets of build metadata to do any of this.
We may want to repurpose the existing GNOME continuous framework to
just replace the build procedure with a BuildStream build of the GNOME
release metadata. Alternatively, I think that running a gitlab instance
for the CI cases may get us something superior as we can leverage
a lot of gitlab features and maybe even offer an opt-in pre-merge CI
experience for participating GNOME projects.
Further than this, I should mention there is some movement to implement
Source plugins to represent distro packaging sources, e.g. the dpkg
source plugin[4]. With this plugin in place, I suspect it will be
very easy to run tests of building and running GNOME against various
third party distributions and versions of those. Imagine we could have
a board somewhere which displays which distributions GNOME builds on
without failing tests (we could always know when GNOME master is failing
against debian sid, or latest fedora, etc).
How would we implement BuildStream in GNOME with a smooth transition ?
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~
Ideally we would like to demonstrate all of this working in a side branch
of the GNOME release modulesets, without being disruptive or calling for
a flag day where everyone needs to switch over atomically.
While we were able to achieve something close to that for the Baserock
project which now has some automation to build converted BuildStream
projects beside the YBD tooling, the situation for GNOME is a bit more
complex since we already have two important sources to convert from (the
gnome-sdk-images repository and the release team's JHBuild modulesets).
So, I'm testing the water in advance to see how the community feels
about this plan in general, and also to gather opinions on what exactly
we should be doing next.
This is not the whole roadmap[5], but what I have on the menu specifically
for the cases of building GNOME and satisfying GNOME related requirements,
which is my top priority for the next months and listed in the order I
would currently have them done:
What we have on the menu right now, in the order I would have them done:
o Complete the Source implementations needed for Flatpak JSON (almost done,
bzr is supported by flatpak-builder and we need only add some more tests
to merge bzr support in BuildStream)
o Complete the Source implementations needed for converting JHBuild
modulesets (I think we will only need svn to complete this list, possibly
but hopefully not cvs or hg).
o Experiment with Flatpak deployments
Most of what BuildStream will have to do to generate Flatpak output for
runtimes and applications is straight forward, just building into some custom
prefixes and encoding some metadata files for Flatpak to read.
Looking through flatpak-builder code however there are some customizations
like post-install manipulation of build output specifically for the locale
data to end up where Flatpak wants it to be.
o Create conversion script for JHBuild modulesets.
Here I would by default use a debian base system to build on top of if
the development of the dpkg plugin[4] works soon. Then I would create
a script which parses the JHBuild XML and outputs a BuildStream project.
This is going to need some additional static supporting metadata:
- We will need to maintain a list of modules (or 'elements') which should
be built for the GNOME Flatpak SDK/Runtime but not for the GNOME Releases,
E.g. non-GNOME dependencies which need to be added to the runtime but are
not necessarily provided by the freedesktop runtime.
- Conversely, we need the list of things to be included in GNOME Releases
but are not built in the GNOME Flatpak runtime/SDK.
E.g. core GNOME components such as gdm and gnome-session
With this script in place, we should be able to continuously produce GNOME
releases which boot in a VM, and also produce the GNOME SDK/runtimes for Flatpak
without requiring a "flag day" to switch over.
o Developer Experience
The developer experience needs work before I can confidently say that it
can replace JHBuild for the regular day to day requirements of developers.
What we absolutely need for this is:
o Developer Workspaces[6]
These will allow the developer to override a Source plugin in their
local checkout of the GNOME release metadata, so that for example,
you build everything directly from gits or tarballs, but you build
gedit from a local directory of your choosing.
o Dual Cache Key Modes[7]
The shape of this might change a bit, but the gist of it is that
in a release setting, one wants to be sure that everything was rebuilt
in the case that any of its dependencies have changed (this is currently
the default for calculating whether something needs to be rebuilt).
In a developer setting however, you want to go all the way down to
glibc (or lets say glib) and add one printf() statement, or change a line
of code, and then you want to rebuild only glib and start some application
or run a fresh new VM with that modified glib, without rebuilding the
whole world to get there.
These two scenarios are obviously at odds with eachother so we will want
some user configuration to make the decision of what to build and when.
In addition to the above, it will be nice to also tie up:
o Artifact Cache Sharing[8]
Authenticated users or CI machines should be able to upload the
results of individual builds. In this way when a developer wants
to hack on a given module, they really only need to build the
module(s) they are concerned with but can reuse (download) the build
results of any other builds, provided those were built once and
uploaded to an artifact share.
o CI Investigation and implementation
I prefer to keep this as the last piece of the puzzle, as stated above
it's quite possible that we discover more powerful ways to leverage CI
using gitlab, or it may be that refactoring current GNOME Continuous to
build with the same build metadata maintained by the release team is
the way to go.
In either case I think the requirements here are:
o Use the same metadata as release team, not a separate format/repo
o Should be easily repeatable. As in anyone should be able to setup
a new instance to run CI on GNOME in their home by following some
steps in a README somewhere, ideally this should not take all day
and it should be reliably repeatable.
o Should not have any regressions over the current solution. Which
means installed-tests should run, some screenshots proving a booted
system and some apps, fast build times preferred over rebuilding
everything on every dependency change.
Asides from the above, it will be nice to:
o Have also full builds run (i.e. rebuilds of anything with a
changed dependency)
o Have CI builds contribute to a shared artifact cache which users
can tap into in order to save cycles on their own machines.
In Closing
~~~~~~~~~~
I am of course very excited about what we can accomplish and I very much
hope that you will share in my enthusiasm.
Does this look like a lot of work to do ? Yes it does, however there is
already a great deal of work done and momentum is on our side. Thanks to
Codethink sponsoring this project, we also fully expect to have budget to
work on this full time for some months to come, so I'm confident that we
can deliver on our promises.
Feedback and involvement in any form are greatly appreciated, are
there parts of the picture you think we've missed ? Please reply and
tell us about them :)
Any questions about what we have created so far and how it works ? Please
reply and ask about these !
Best Regards,
-Tristan
[0]: https://blogs.gnome.org/tvb/2017/02/06/introducing- buildstream-2/
[1]: https://blogs.gnome.org/tvb/2017/04/10/buildstream- progress-and-booting-images/
[2]: https://buildstream.gitlab.io/buildstream/#sources
[3]: https://buildstream.gitlab.io/buildstream/format.html# variant-conditionals
[4]: https://gitlab.com/BuildStream/buildstream/ issues/10
[5]: https://wiki.gnome.org/Projects/BuildStream/Roadmap
[6]: https://wiki.gnome.org/Projects/BuildStream/Roadmap/ Workspaces
[7]: https://wiki.gnome.org/Projects/BuildStream/Roadmap/ CacheKeyModes
[8]: https://wiki.gnome.org/Projects/BuildStream/Roadmap/ ArtifactSharing
_______________________________________________
desktop-devel-list mailing list
desktop-devel-list gnome org
https://mail.gnome.org/mailman/listinfo/desktop- devel-list