Re: Fwd: wip/new-model review

["Jasper St. Pierre" <jstpierre mecheye net>]

On Thu, Jul 18, 2013 at 10:58 AM, Colin Walters <walters verbum org> wrote:

On Wed, 2013-07-17 at 20:16 -0400, Jasper St. Pierre wrote:

>
> The big question is how we share this information. Up until new-model,
> it's been shared across the entire system in this thing called
> "jsondb",

Yes...and repo/ and cache/.  Plus src/.

>   $ ostbuild test builds/glib-testing-build/

Would this command log to stdout?  Or would it log to
builds/glib-testing-build/integrationtest/ ?  If the latter would it
delete any previous logs there?

I don't particularly care enough. I haven't started to build these workflows, but I'm amenable to doing whatever you want.

 

> We, of course, probably wouldn't shut off access to things like
> "run-task" meaning you could build custom things if you wanted to,

Right.  I think we're going to of necessity have two layers:

1) A layer that knows how to run "tasks" in a subprocess (recursively),
   and handles logging to file(s).  Ideally the latter is something
   systemd journal like.
2) A layer that knows about builds.

I think this is wrong. Every single task, including from resolve to zdiff, knows about a general concept of a "build". Ideally, it's the only input/output of a task, and ideally it would be a well-defined directory structure, with some well-defined files (the snapshot, the current status, the last build for optimizations, etc.). Obviously, since I'm sort of wreaking havoc, throwing stuff around for new-model, it's not very well-defined, but I'm hoping to clean that up.

The thing that I want to do is remove this central lookaside of "builds" that everything pulls from, and require the developer commands to specify which build they're working on.

So, I have some build somewhere, and I can smoketest/integrationtest it.

Or, if I have two builds, I can bdiff them (the bdiff implementation is a bit ugly, as it relies on the "last-build" implementation detail, so you can't diff two arbitrary builds with the current system, but that should change)

I want to basically take "build management" out of the tasks and into the invoker -- in this case, the autobuilder has its own idea of how to manage these builds (with the builds/, tasks/, results/ structure we talked about in-person, which I should probably write up somewhere)

But putting that in the autobuilder means that either I fix make to do the same, or I just punt on it and dummy up the implementation. For time reasons, I just chose to dummy up make.

>  but overall I feel better focusing on making the high-level workflows
> better, rather than knowing that you have to skip the zdisks pass for
> speed.

I agree with that goal.  So leaving the trybuilding stuff aside, your
suggestion is basically we have a new toplevel command that does
a specific thing?

$ ostbuild make       # resolve, build
$ ostbuild smoketest  # resolve, build, builddisks, smoketest
$ ostbuild export     # all of it except bdiff

"resolve" is the special one, and I'm not really comfortable with how I chose to handle it. Perhaps you might be able to chime in here.

The issue with "resolve" is that it's not really part of a build. The only essential part of a build is the snapshot, and in the new model, it's the "id" of a build. If you have the snapshot, you have a complete description of a build, and can remake the rest of everything at any time. (ignoring things like network status, repositories moved, build dates in binaries, etc.)

Right now, the autobuilder creates a skeleton dir, runs resolve, and if resolve says nothing changed, we remove the directory and don't bump the build number.

Additionally, for disk space / network optimization reasons, we keep the git repos outside of a build, and symlink them into the build when the autobuilder creates it, along with the other build skeleton files (last build, ostree repo).

The autobuilder, of course, has its own directory structure, and knows it very well. The issue is that the developer tools don't really have that, and don't really know about "src", "repo" and "last-build", etc.

I thought passing "last-build" in was fine, but for the other two, we should probably just load everything from a config file, similar to jhbuildrc (~/.ostbuildrc?)

So these developer-focused commands would exclude tasks mostly only
useful for the web UI, like bdiff.  Except I could imagine it being kind
of useful to do that locally...like pull new stuff (e.g. systemd and
glib), look at the changes.  Eh, we can skip that for now.

Yeah, as I said above, bdiff is also a special case since it has two builds as inputs, instead of one. Right now it serves the autobuilder case correctly by relying on the "last-build" model. We should probably make the task take two builds, and then in the autobuilder always run it so that it bdiffs between last-build and the current build.

Hopefully this cleared things up about the challenges I'm facing and what I want the new model to be.

--
  Jasper