Re: [BuildStream] Proposal: Decouple source tracking and building

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi,

> On Oct 18, 2019, at 10:56 PM, Chandan Singh <chandan chandansingh net> wrote:
>
> Hi all,
>
> tl;dr: Remove `--track-*` options from `bst build` and simplify codebase.
>
> ---
>
> I wanted to propose a clearer separation between tracking sources and building
> them in BuildStream. Currently `bst build` offers a few `--track-*` options,
> all related to whether or not BuildStream should attempt to track sources
> before starting the build.
>
> In my opinion, these are two different kinds of operations, and don't really
> need to be part of the same pipeline.

I disagree with this.

The reason why we can track and build in the same session is an optimization, we should not have to wait 
until all the elements finish tracking before being able to build the first elements which reach a consistent 
state. I understand the code would be easier to maintain if we didn’t have this feature but I think it really 
provides good value.

I do however agree that the UX for this is undesirable and could be improved, one suggestion somebody made on 
IRC some months ago was to make things a bit scriptable for the more complex invocations, and for running 
multiple commands in a single session (so we could also have build + checkout for instance, which has been 
requested as a convenience by some users).

This might mean removing some of the more complex CLI options (or not) and adding a new option to read 
commands from a file (or a quoted string on the command line, or stdin or such).

Not sure how popular this idea might be, just throwing it out there.

Cheers,
    -Tristan


> My concerns about this can be broken down
> into two broad areas:
>
> 1. Simplifying scheduling/update_state() logic
> 2. UI/UX
>
> First, let's talk about scheduler. At present, when we start a build, we expect
> the elements being tracked to be resolved only after they've been tracked. This
> means that we can't be sure that all elements in the pipeline are fully resolved
> and ready to be built.
>
> Decoupling tracking and build operations should simplify the `update_state()`
> logic as it would eliminate one of the needs for doing so.
>
> Moving on to UI/UX issues, `bst build --track-*` essentially duplicates
> functionality from `bst source track`. Hence we end up with two ways of doing
> the same thing and have to support both.
>
> Since tracking sources and building are rather different kinds of operations,
> I am proposing to have them clearly separated. More concretely, I am proposing
> to remove the following options from `bst build`:
>
>  * `--track`
>  * `--track-all`
>  * `--track-except`
>  * `--track-cross-junctions`
>
>  * There's also `--track-save` which is deprecated and ignored
>
> `bst source track` already supports all functionality offered by the above
> options, so that won't require any additional features.
>
> I can understand the desire to update all source references in a CI build, or
> for testing purposes. But, I think the same can be achieved by running
> `bst source track` followed by `bst build`. Can people think of any other use
> cases that would be negatively affected by this change, other than having to
> type two commands instead of one?
>
> Let me know what you think.
>
> Thanks!
> Chandan
> _______________________________________________
> buildstream-list mailing list
> buildstream-list gnome org
> https://mail.gnome.org/mailman/listinfo/buildstream-list