Re: [BuildStream] Responsive, but not overly verbose UX - not-in-scheduler

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Daniel,

On Thu, 2019-04-18 at 08:46 +0100, Daniel Silverstone via buildstream-list wrote:
> tl;dr
> -----
>
> We need to be better at feedback to the user when not in the scheduler.  I
> propose a combination of a context manager, a lightweight thread, a little
> configuration, and some rules around UX messaging, to ameliorate things.
>
> For context, a user of a fast system with a small project (7000ish elements,
> via one junction, already cached in `.bst`) might see (much like today) a
> sequence such as:
>
>     [--:--:--] START   Loading elements
>     [00:00:02] SUCCESS Loading elements
>     [--:--:--] START   Resolving elements
>     [00:00:04] SUCCESS Resolving elements
>     [--:--:--] START   Resolving cached state
>     [00:00:02] SUCCESS Resolving cached state
>
> Where the `Resolving elements` phase provided a progress indicator during
> the operation.
>
> A user of a slower system might, for the same project see something more like:
>
>     [--:--:--] START   Loading elements
>     [00:00:02] START   > Checking out junction: junctions/debian-junction.bst
>     [00:01:20] SUCCESS > Checking out junction

The fact that we don't have any feedback about the fetching of
junctions, and also that the user settings (e.g. fetchers) has no
impact on the junction fetching is rather buggy indeed (I use a system
monitor, and I feel rather offended that BuildStream is causing network
activity to occur without telling me about it or allowing me to police
it in a way that is consistent with all other fetch operations).

This is what I had in mind in order to fix these two problems:

  * Pop up the status bar right away instead of deferring it

  * Have the Loader trigger a callback mentioning that it requires a
    fetch operation (or multiple fetch operations) to be performed.

  * Stream run the scheduler on it's behalf to fetch the junction

It should be feasible to fetch as many junctions which were encountered
in the first pass load of one project at once in parallel, rather than
serializing potential fetches at once (all while respecting the
configuration of how many fetches the user has allowed to occur in
parallel).

This way, if we enhance progress indicators in some way in the future,
it can be applied equally and consistently to all operations which
support progress reporting (Source plugins could optionally support
such a reporting API one day, the loader might provide a counter of how
many elements were loaded so far, etc).

What do you think, is there a reason we need to add an additional
logging mechanism for early logging that I overlooked, or a reason that
it would be more desirable ?

Cheers,
    -Tristan