Re: [BuildStream] Proposal: Decouple source tracking and building

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Jürg,

> On Oct 21, 2019, at 5:01 PM, Jürg Billeter <j bitron ch> wrote:
[...]
> > I suspect that the main motivator to remove this is because it makes
> > state handling complex. State handling has been a bit of a mess for a
> > long time and update_state() is maddening for most readers.
> >
> > I think we’ve learned that in the current architecture; when we teach
> > BuildStream to do really cool things (like the feature under
> > discussion), we risk making update_state() worse.
> >
> > I would much prefer that we have the expectation that complexity is
> > only going to increase and that we will always make BuildStream do
> > really cool things, and we should refactor with that expectation so
> > that complexity doesn’t bottleneck in functions like update_state(),
> > but instead concerns are neatly separated. Maybe this is unrealistic
> > ?
>
> While it should certainly not be a surprise that the complexity of a
> software project typically increases over time, we shouldn't dismiss
> proposals for simplifications just because the complexity increase is
> normal. If we've added a feature in the past that turns out to be less
> useful than expected (and/or conflicts with future features), I think
> it's very healthy to reevaluate whether the feature is worth it (we
> also always have to consider stability promises, of course).
>
> I don't know whether `bst build --track` is or isn't worth it. However,
> I think it's a valid discussion.
>
> And yes, we're working towards the removal of `_update_state()`. This
> should help reduce the complexity involved in the combined track+build
> pipeline. However, the track+build pipeline support will not suddenly
> become completely trivial.

So yes, of course trade offs have to be made sometimes, and also refractors should happen continuously.

I’ll try to be clearer (have been replying from smart phone over the day):

* I do think that in the current confusing form of state handling, removing this feature is desirable, even 
at the cost of losing a luxury feature (it’s not a base feature, but I think BuildStream is a luxury car, not 
just any vehicle or build script).

* I do not believe that the complexity of how state handling is expressed in the codebase is justified. Maybe 
I’m wrong about this (hence why I ask in my last reply if my thinking is unreasonable), but I have a very 
hard time believing that we cannot come up with better separation of concerns in scheduling and state 
handling.

I would much rather be having a conversation about how we can make the codepaths manageable, if they were 
more manageable already, I doubt we’d be having this conversation at all (we’d be happily improving 
performance and adding more luxurious features).

Cheers,
    -Tristan



> Cheers,
> Jürg