Re: [BuildStream] Proposal: Decouple source tracking and building

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

I replied a bit soon as I see now others have already raised the issue of lost parallelism.

I still do think it is an important feature, and feel that it might be a bit embarrassing to not have this 
feature, considering everything else can be well parallelized, so why not tracking also.

Cheers,
    -Tristan

> On Oct 19, 2019, at 11:56 AM, Tristan Van Berkom <tristan vanberkom codethink co uk> wrote:
>
> Hi,
>
> > > On Oct 18, 2019, at 10:56 PM, Chandan Singh <chandan chandansingh net> wrote:
> > Hi all,
> > tl;dr: Remove `--track-*` options from `bst build` and simplify codebase.
> > ---
> > I wanted to propose a clearer separation between tracking sources and building
> > them in BuildStream. Currently `bst build` offers a few `--track-*` options,
> > all related to whether or not BuildStream should attempt to track sources
> > before starting the build.
> > In my opinion, these are two different kinds of operations, and don't really
> > need to be part of the same pipeline.
>
> I disagree with this.
>
> The reason why we can track and build in the same session is an optimization, we should not have to wait 
> until all the elements finish tracking before being able to build the first elements which reach a 
> consistent state. I understand the code would be easier to maintain if we didn’t have this feature but I 
> think it really provides good value.
>
> I do however agree that the UX for this is undesirable and could be improved, one suggestion somebody made 
> on IRC some months ago was to make things a bit scriptable for the more complex invocations, and for 
> running multiple commands in a single session (so we could also have build + checkout for instance, which 
> has been requested as a convenience by some users).
>
> This might mean removing some of the more complex CLI options (or not) and adding a new option to read 
> commands from a file (or a quoted string on the command line, or stdin or such).
>
> Not sure how popular this idea might be, just throwing it out there.
>
> Cheers,
>   -Tristan
>
>
> > My concerns about this can be broken down
> > into two broad areas:
> > 1. Simplifying scheduling/update_state() logic
> > 2. UI/UX
> > First, let's talk about scheduler. At present, when we start a build, we expect
> > the elements being tracked to be resolved only after they've been tracked. This
> > means that we can't be sure that all elements in the pipeline are fully resolved
> > and ready to be built.
> > Decoupling tracking and build operations should simplify the `update_state()`
> > logic as it would eliminate one of the needs for doing so.
> > Moving on to UI/UX issues, `bst build --track-*` essentially duplicates
> > functionality from `bst source track`. Hence we end up with two ways of doing
> > the same thing and have to support both.
> > Since tracking sources and building are rather different kinds of operations,
> > I am proposing to have them clearly separated. More concretely, I am proposing
> > to remove the following options from `bst build`:
> > * `--track`
> > * `--track-all`
> > * `--track-except`
> > * `--track-cross-junctions`
> > * There's also `--track-save` which is deprecated and ignored
> > `bst source track` already supports all functionality offered by the above
> > options, so that won't require any additional features.
> > I can understand the desire to update all source references in a CI build, or
> > for testing purposes. But, I think the same can be achieved by running
> > `bst source track` followed by `bst build`. Can people think of any other use
> > cases that would be negatively affected by this change, other than having to
> > type two commands instead of one?
> > Let me know what you think.
> > Thanks!
> > Chandan
> > _______________________________________________
> > buildstream-list mailing list
> > buildstream-list gnome org
> > https://mail.gnome.org/mailman/listinfo/buildstream-list