Re: GNOME Continous and xdg-app

[Alexander Larsson <alexl redhat com>]

Also CCing gnome-os-list, as this seems related there

On mån, 2015-10-26 at 21:56 -0700, Giovanni Campagna wrote:
> Hello all,
>
> At the GNOME Summit this year I had time to make a quick and dirty
> (but 
> working) patch to evolve gnome-continous to produce xdg-apps. The
> code 
> is at https://bugzilla.gnome.org/show_bug.cgi?id=756443, and Colin 
> asked me to present it on the ML, so here it is.

That is pretty awesome! 

I've been thinking about how to solve the bundling issue for xdg-app
for a while now. xdg-app by itself is independent of how you produce
your bundles. As long as you can somehow compose a directory tree with
your files in it we can distribute and run it. I have experimented with
various ways to build things, such as yocto, rpms, copr, makefiles,
simple build scripts, etc.

In an xdg-app world it will become much more important for upstreams to
build their own dependencies, so we *must* have a simple story on how
to do this. They should not have to figure out all this manually like
We've been doing, nor have to rely on some distro like thing.

There are some important properties we want to have:

 * Super-easy to use. 
   This implies shared repo of bundling metadata that you can
   cherry-pick from. And the metadata has to be minimal and easy to 
   understand.
 * Push bundling work upstream.
   Avoid complex out of band metadata like spec-files, instead make 
   upstream maintain this.
 * Repeatable builds.
   Cross-app ostree deduplication is very nice, but only works if 
   binaries are identical. We need to work on reproducible builds, or
   possibly sharing pre-built binaries. This is especially important
   in incremental builds of the same app, as it makes updates smaller.
 * Support partial module dependencies.
   Many dependencies are partial, like you need the library, but not
   the binaries and the docs from the dependency. We need to be able
   to pick just the parts we want in the final bundle.

My current idea on this is actually very similar to your continuous
work. First of all I want to base this on Colins build API:

 https://github.com/cgwalters/build-api/blob/master/build-api.md

So, lets assume that all dependencies are conforming to this (or have a
shim that does so). We can try to push any work needed for this
upstream so that the upstream maintainers can maintain it. For non-
cooperative modules we can share a repo of shims.

We then have a tool that takes a manifest listing modules, build order
and configure switches. These are built in order with "xdg-app build",
and then at the end we clean up non-wanted files. We would also
checksum all the ingoing data in each step so that we can cache steps
to make repeated builds faster (similar to Dockerfiles caches).

The options that need to be specified should be very minimal, but we
need at least:
  tarball/vcs uri
  commit-id/tag/branch
  tarball checksums for verification
  configure options
  optional patches
  list of extra files to clean up at end 

I want to make this available as a very generic xdg-app build tool,
which is shipped with (or next to) xdg-app itself. 

Obviously this is very similar to gnome-continuous, but I'd like to
make some changes to how it works. 

First of all, I'd like to avoid having the tool depend on gjs or any
other technology that is not on the freedesktop runtime/sdk as I want
non-gnome apps to be able to use this tool too.

Secondly I would like to rely on the make/make install cycle to do
everything that a module needs, rather than having to rely on
externally maintained post-install scripts (and triggers). This is nice
because upstream already maintains and tests this and we don't have to 
duplicate it. This means we can't use DESTDIR, and the per-module
split-and-recombine-at-the-end. Instead it'll do a series of regular
make&&make install and then a clean-up at the end.

Third, as the goal is not continuous integration I want to rebuild the
entire app from the first changed dependency each time, rather than
building just the changed module and recombining. This is more
reproducible, and less likely to run into issues if dependencies change
ABI/API. Avoiding rebuilding unchanged dependencies will help us with
reproducible builds.

Sadly, it seems to me that this will pretty much amount to re-
implementing a subset of gnome-continuous. I don't think it will be a
huge piece of work though.

[1] https://github.com/alexlarsson/gnome-sdk-bundles
[2] https://copr.fedoraproject.org/coprs/alexl/fedora-runtime/
[3] https://copr.fedoraproject.org/coprs/alexl/fedora-apps/
[4] https://github.com/alexlarsson/nightly-build-apps/.sh

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                            Red Hat, Inc 
       alexl redhat com            alexander larsson gmail com 
He's a witless crooked inventor on the wrong side of the law. She's a 
provocative mutant museum curator with the power to bend men's minds. 
They fight crime!