Re: [BuildStream] Proposal: Decouple source tracking and building

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Jürg,

> On Oct 21, 2019, at 3:21 PM, Jürg Billeter <j bitron ch> wrote:
>
> Hi Tristan,
>
> > On Mon, 2019-10-21 at 11:37 +0900, Tristan Van Berkom via buildstream-
> > list wrote:
> > [...]
> >
> > So, the main reason I added this long ago was indeed because I
> > planned to use it in a very specific CI situation (besides the fact
> > that I think it’s just a very cool feature, which is indeed
> > subjective).
> >
> > Consider a CI scenario where you have many people committing to the
> > master branch of many git repositories simultaneously and you just
> > don’t have the horse power to run builds of all reverse dependencies
> > for each commit (a scenario I happen to be facing right now
> > actually), and you want to report build failures as fast as possible.
>
> Would it be possible to use git post-receive hooks or triggers to
> update the 'refs' in the BuildStream project instead of periodically
> tracking all git repos? That would likely be faster overall.
>
> This wouldn't work for external git repositories, of course. However,
> in that case you would probably anyway want git mirroring to local
> servers and that could again trigger the 'ref' updates.

You could write some scripts which update the YAML yes, you might also do that for tarballs, and for other 
source types as needed, replication much of the value that BuildStream already supplies with its tracking 
feature. But do we *want* that to be the user story for the build solution we’re creating and want everyone 
to use ?

The tracking feature is very convenient and valuable already and we should leverage it.

Honestly, I do think this is a valuable feature enough to justify the complexity it adds.

I suspect that the main motivator to remove this is because it makes state handling complex. State handling 
has been a bit of a mess for a long time and update_state() is maddening for most readers.

I think we’ve learned that in the current architecture; when we teach BuildStream to do really cool things 
(like the feature under discussion), we risk making update_state() worse.

I would much prefer that we have the expectation that complexity is only going to increase and that we will 
always make BuildStream do really cool things, and we should refactor with that expectation so that 
complexity doesn’t bottleneck in functions like update_state(), but instead concerns are neatly separated. 
Maybe this is unrealistic ?

Cheers,
   -Tristan

> Cheers,
> Jürg