Re: Dual cache key calculation and build planning modes

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Wed, 2017-07-05 at 15:10 +0200, Jürg Billeter wrote:
> Hi Tristan,
>
> thanks for the write-up. It looks like a reasonable plan for handling
> dual cache keys / build planning modes. That said, I'm not very fond of
> having such a non-predictable build planning mode at all. If there is
> no other way to fulfill the requirements, so be it, but maybe the
> following idea would be an acceptable alternative.

I'm not sure I'm fond of the alternative proposal here, but I'm not
sure I understand exactly how it works or how it solves the problem in
a cleaner, more comprehensive way.

I should stress that keeping the user facing interfaces as simple as
possible is a big priority.

That said, if it's only the predictability of builds passing or failing
with such an approach; maybe something else can be cooked up to address
that concern; for instance, we might be able to pin certain groups of
elements to not behave this way, when plotting the build planning mode.

This way one could select only a specific group of elements suitable
for weak planning mode ?

> The underlying reason why weak build planning modes will (mostly) work
> is that shared library interfaces are reasonably stable, at least in
> the GNOME ecosystem. Instead of using weak cache keys, hoping for the
> best, and manually intervening when something breaks, we could supply a
> bit more information to BuildStream such that it can avoid rebuilding
> reverse dependencies for minor library changes without losing the
> predictability we have right now. It could even become the default
> mode.
>
> My proposal is to add a secondary ref to sources of library elements.
> That secondary ref points to the oldest revision that has the same API
> and ABI as the primary ref (and doesn't contain any major bugs that
> would break build tools). Whenever such a library is staged as a
> dependency of a build process (directly or indirectly), the artifact
> built from the secondary ref is used. 

Currently the solution handles stable vs. experimental, and which
versions of things to build/combine together by:

  A.) Revisioning the buildstream project itself with something
      like git, so we have stable vs dev branches of a buildstream
      project.

  B.) Using stable branch names as tracking refs for stable branches
      of the buildstream project itself.

      So when you are building stable branches of every element in
      the project, these interfaces are inherently never expected
      to break (however mileage of course will vary, pinning to exact
      release tags or tarballs is also a possible countermeasure).

Would this also mean we would have to have 2 tracking references ?

I'm open to trying to find another less complex solution, but this is starting to look like a more complex 
project definition than a simpler one.

> The artifact built from the
> primary ref is used only when composing a system.

I'm not sure what you are proposing with this exactly, are you saying
that we have some kind of semantic for assembling artifacts together,
which have never previously been built against eachother ?

Also, I dont know what a "system" is in this context, we dont have such
a concept really, maybe you are alluding to what happens with a
'compose' element.

> This will require building all libraries with differing refs twice, but
> only when building from scratch. Artifact sharing should be very
> effective with this approach.
>
> I haven't worked out all the details yet but I would appreciate input
> on whether it makes sense to continue in this direction or if this is
> not suitable to fulfill the requirements. One question is how to handle
> bumps of secondary refs for fully automated build systems. Is a manual
> action on interface breaks and additions acceptable?

I'm not completely averse to manual intervention on interface breaks
and have considered this for different reasons in the past.

I'm not sure how this would work in practice, especially in conjunction
with `bst track` (how do we determine that a tracked source has passed
the manually specified version boundary, in a VCS agnostic way ?).

Cheers,
    -Tristan