Re: Fwd: wip/new-model review

["Jasper St. Pierre" <jstpierre mecheye net>]

On Tue, Jul 16, 2013 at 1:01 PM, Colin Walters <walters verbum org> wrote:

On Thu, 2013-07-11 at 14:51 -0400, Jasper St. Pierre wrote:

>
> I've pushed a reworked set of patches for this fixing most of the
> other issues to wip/new-model.

So I cherry-picked more easy commits from the branch, and those are on
master now.

The first one I ran into trouble with was this:

https://git.gnome.org/browse/gnome-ostree/commit/?h=wip/new-model&id=7a7ce68c830a4b8839ea84e2b4f119dc5f5ac4a6

The problem is that I very frequently do these two operations:

$ cd ~/src/checkouts/glib
$ ln -s $(pwd) /src/build/ostbuild/work/overrides
$ cd ~/build
$ ostbuild make -x builddisks build

This is my edit-compile-debug workflow with gnome-ostree locally.  All
we do is "build", i.e. new binaries in the repo.  Then I switch to a
running VM which I've configured to point at a webserver on the host,
and do "ostree admin upgrade -r" there.

Here I don't want to update the cached disk images and run
smoketest/integrationtest; I'll just use a VM interactively.

$ ostbuild make -x zdisks resolve fetchAll=true

I run this command periodically on my laptop.  I fetch all the git
repos, do a build, update my local disk images and run
smoketest/integrationtest on them, just to be sure that all works
locally too.

However I exclude zdisks because it's really really slow, and not useful
for local builds.

I think this sort of flexibility makes sense for developers, and I'm not
sure how to keep it without retaining some sort of skip operation.

OK. I think this is going to be important about deciding how we want to proceed. Ideally, the autobuilder is very very simple:

Once some git repos are updated, kick off a bunch of tasks. Build every component, and then resolve it into a tree, do some testing, and generate a zdisk.

Unfortunately, this is slow (rebuilding the kernel on a gnome-control-center change isn't fun), so we pull an optimization where we reuse the last build if it hasn't changed, assuming that it means nothing. Sometimes, this is a leaky optimization -- if we have an soname bump, we should rebuild everything, but we don't, so we manually force some components to rebuild in that case.

There's also some tasks, like "bdiff", which explicitly want to say "compare this and the last build we did".

The big question is how we share this information. Up until new-model, it's been shared across the entire system in this thing called "jsondb", which we both want to get rid of. The way we do this in new-model is symlinks: the autobuilder fully controls a directory structure, and when a build starts, it symlinks in this thing called "last-build", which points to the last build Tasks that used to go out-of-band to the jsondb now just look for this symlink instead.

The make model of "run some of these tasks" can naively be implemented by you specifying the last build and other such parameters on every command line, but that feels weak.

I'd instead like to replace "make", the high-level primitive, with a bunch of different utilities based around core workflows.

We might have a command for "take this build, and rebuild it, but with this set of overrides", and it might look like:

  $ ostbuild build --base builds/20130717.05/ --override glib-testing.json -o builds/glib-testing-build/

which takes the base build and its snapshot, applies the overrides provided, and builds everything that changed into a new build. This could easily now be used for trybuilds or similar.

We also might have a way for us to run a few tasks in a builddir, and just assume that everything is set up until that point. If, for instance, a smoketest/integrationtest fails and we want to inspect locally, there might be:

  $ ostbuild test builds/glib-testing-build/

to run all tests, and:

  $ ostbuild integrationtest builds/glib-testing-build/

to run a specific kind of one. Both integrationtest and smoketest don't really need a previous build; in the new model, everything those tasks need is in the builddir itself.

We, of course, probably wouldn't shut off access to things like "run-task" meaning you could build custom things if you wanted to, but overall I feel better focusing on making the high-level workflows better, rather than knowing that you have to skip the zdisks pass for speed.

--
  Jasper