GNOME.org
 

Re: [BuildStream] Partial local CAS

Hi Sander,

On Tue, 2018-11-20 at 16:24 +0000, Sander Striker wrote:

Hi,

On Tue, Nov 20, 2018 at 10:50 AM Tristan Van Berkom <tristan vanberkom codethink co uk> wrote:

Hi Sander, Jim...

On Fri, 2018-11-16 at 17:28 +0000, Sander Striker wrote:

Hi Tristan,
 
I am not clear if we're violently agreeing or very much disagreeing :).  More inline.


I think that we agree on most points here.


Good to hear.
 

I feel very strongly that the default imperative to have the artifacts
in the local cache as a result of running `bst build` should not be
changed just because remote execution is active.
 
We may only disagree on this point.


That is likely the case.  It's a fairly fundamental disagreement though :).


I'm not sure I understand why you feel so strongly as to want to change
this fundamental behavior, especially given that at this point, I think
we are only arguing over a default behavior.

I'll go on to elaborate more why I really think it would be ridiculous
to change this, but feel that I might be insensitive and missing
something from your perspective which might explain why you feel
strongly about this.


As Jim points out:

On Mon, 2018-11-19 at 13:38 +0000, Jim MacArthur via BuildStream-list wrote:

On 16/11/2018 11:42, Tristan Van Berkom via BuildStream-list wrote:

Most importantly I just think it is important to bare in mind that
remote execution is just an optimization, and the way people use remote
execution will not always match our expectations.
 

 
We have at least once discussed using remote execution for cases when 
software cannot be built on the local machine; C code which cannot be 
cross-compiled, for example. In these cases, not only would remote 
execution be required for a build, but it's likely the resulting 
artifacts would be of no use on the local system either.


It is true that remote execution is required for building things which
require an execution environment unsupported by the host, but I also
don't think that we should break the expectation to "have something you
just built" just because that thing which you built is not runnable on
your host.


I would argue that it is a false expectation to begin with, which we
should reset as soon as possible.  That all artifacts that result
from a bst build of a pipeline, including any intermediate artifacts,
are implicitly available on local disk seems overly broad.


Subtle difference here I should point out, which might be a
misunderstanding so far in this thread: The only artifacts which are
really expected to exist locally after a build, are the target(s) and
their runtime dependencies; i.e. the explicit product of the build.

Intermediate objects (i.e. build-only dependencies) are always built on
demand, only if the artifacts which build-only depend on them cannot be
pulled.


Currently (before remote execution), whether an artifact is built by a
machine in CI and downloaded from an artifact server, or built locally
as a result of `bst build`, the expectation is that `bst build` gets
you the artifact; remote execution is not really much different than
the artifact having already been built by a third party and then
shared.


I would argue that the expectation should be that the artifact is
built after the invocation of `bst build`.  And that the artifact is
_guaranteed_ to be available locally after a `bst pull`.
We could (and probably should) introduce the convenience of having
`bst shell` or `bst [artifact] checkout` do an implicit pull.


This is a misunderstanding of the special nature of `bst build`
compared to all of the other, more granular commands.

The `bst build` command has always been all inclusive of everything
that might have to be done for a given target, i.e. `bst build` is
basically "show time" for BuildStream.

The fundamental underlying reason(s) for this is:

  * Primarily an optimization

    It makes no sense to run bst build and bst pull separately when we
    can potentially save time by parallelizing build and pull
    operations.

    This is precisely why tracking operations can be enabled during
    a build session, and likewise fetching, so that we do not have to
    block on completion of all track or fetch operations until we start
    building something.

    Likewise, it makes no sense to block on building of everything if
    that happens remotely, before we start pulling the results that we
    intend to use locally.

  * Less than the above, but essentially command line user convenience.

    As a user, I don't want to remember to run multiple commands to
    do the "get it done" thing which is `bst build`, I just want to run
    `bst build` and know that my target, including runtime
    dependencies, is ready for a `bst shell` or a `bst checkout` so
    I can run it in my VM or on some actual hardware on my desk.

    It's just more convenient for the user to have a "do everything"
    build command.


The way I see this is essentially:

  * The behavior of BuildStream should not change unexpectedly, just
    because remote execution is enabled does not mean you expect
    the results of running BuildStream to differ.


I don't think it is behavior that users necessarily need to notice. 
Apart from the going-offline case, which I think is ok to special
case.


I've been using and testing BuildStream a lot, lets ignore the CI cases
because these hardly ever really bite me or annoy me as a user, the CI
might have some burden to setup but once it's running it "just works".

My local command line use cases have always been edit/compile/test
cycles.

This usually revolves around two variants, usually with strict mode
disabled:

  * I modify a library like webkit or glib, and I build a biggish
    frontend app like epiphany.

    After building my intent is to run `bst shell` and test the
    application directly on my laptop.

    In this case of course I want this to happen as fast as possible.

    Obviously we still have some road ahead of us in order to tackle
    the integration commands and make `bst shell` more seamless here,
    but this doesnt detract from the utter annoyance I would have if
    suddenly I had to do an additional `bst pull` after my `bst build`
    just to go ahead and launch my shell.

    Consider that maybe even it was not my own user configuration
    which prompted this obnoxious extra `pull` step to present itself,
    but some project.conf recommendation which automatically opted me
    into a BuildGrid setup.

    Then my experience would be:
      - Every day my regular edit/compile/test cycle was build + shell
      - Now just because I pulled a new version of my BuildStream
        project, suddenly I am told that I have to pull after my build
      - Very annoyed user


  * I modify a ref of one of my bst files and I rebuild an entire
    system with a new version of something external, and then I want
    to `bst checkout` the resulting image and boot it in a VM.

    Basically all of the same points above apply to this case as well.


I have to say that in every case that I launch `bst build` on the
command line myself, excepting for cases where I am trying to reproduce
a bug - I am always interested in having the result to either shell
into, or to checkout and deploy to something, and I am having a very
hard time to fathom why a user would launch `bst build` on the command
line without being interested in having the result locally.


Note that I am definitely emphasizing the case of running BuildStream
directly on the command line as a user, because that is the focal point
of user frustration and that is where we generate "bad press", we want
the actual user experience to be as pleasant as possible and this is
why we always choose configuration defaults which are optimized for the
user, not optimized for the CI setting.
 

  * We always optimize default settings for those who run BuildStream
    manually on the command line, not for a CI autobuilding setup.


I don't think we are talking about the same thing here, I am thinking
of developer configurations as well as CI setups.


I think I've explained things pretty clearly above, but if you still
disagree, then please enlighten me as to:

  * What is a use case where a developer, who would run `bst build` on
    the command be, where the developer in question is not interested
    in having the result of the build locally after ?

  * If you are even able to come up with a use case (because I'm
    honestly having a hard time figuring one out), is it in any way
    possible that this could be the most popular use case ?

    i.e. do people run `bst build` on the command line without being
    interested in having the result locally so much; that it is worth
    making the default *not* having the result locally at the end ?


    This is because even if there are many more builds run in CI than
    those run by a developer on their laptop, those CI builds can be
    configured once - we should consider that there will always be
    many more separate developer installations than CI installations,
    and to minimize overall configuration pain; optimize defaults for
    developers because of this.

    I consider the case of running `bst build` and not being interested
    in having the results in your local cache to be the CI setting,


I would disagree on that.
 

    while the developer who runs `bst build` on their laptop/desktop
    almost certainly wants to do something with the artifact they
    built.


There is a huge piece of nuance here.  Which is: the developer is
almost certainly not interested in *all* of the intermediate
artifacts that were built as a result of running `bst build app.bst`.


Yes, this is a misunderstanding or miscommunication on my part which I
have rectified earlier in this mail.

Intermediate objects are already not part of the imperative, as they
are only ever built on demand; the expectation is only to have the
usable result/product.


The time where the interest in an element is actually known is at
`checkout` or `pull` time.


This I refute flatly and have explained this clearly earlier in this
email, when I `bst build epiphany.bst`, I clearly want to run epiphany
in the shell afterwards. I also don't want to waste any time pulling
after the build when I could have been doing it in parallel.
 

Even the artifact for app.bst might not see as much interest as you
expect.  I might just be interested in knowing that it `builds` for a
subset of my `bst build` invocations.


Ok here is a case.

  * The user is proposing a patch to a BuildStream project

  * The user modifies a bst file, perhaps updating a ref or tweaking
    a build instruction

  * The user just wants to have peace of mind that their patch "builds"
    before submitting the patch upstream.

I concede that this is one case where you might run `bst build` without
being interested in the result... there is a "but" coming.

BUT... I have to say that this is a user which is quite lacking in
diligence, I hope they are not on my team...

As a maintainer of any software, the last thing I want to hear from a
contributor is "Here is a patch, I tried it and I confirm that it
builds". Rather, I want to know that the contributor at least ran the
software and checked that it fixes the issue it is supposed to fix, and
that the developer at least didn't notice any negative side effects
while smoke testing the software, before submitting the patch to my
project.

For integrations of multi-component softwares, it has always been
particularly challenging for developers to do pre-integration smoke
testing. That said, we're really hoping to make a big difference with
the guarantees that BuildStream can offer developers, i.e. we're
finally placing the integrated product in the developers hands, and we
hope that they can start using that instead of just asserting that "it
builds".


  * If we are going to add an option to make having the build results
    in the local cache not an imperative of `bst build`, it should not
    only be done for remote execution.

    I.e. currently when we run builds in CI, those builds ensure that
    the CI instances download artifacts from remote artifact servers
    regardless if anything even needs to be built.


Removing the implicit pull even for the local build case?  That makes sense for consistency, yes.
 

    So this optimization where we can avoid steps because we are only
    interested in a remote artifact server having the results, but not
    the local cache, is just as relevant for pulled artifacts as it is
    for remote execution, this should probably be opt-in with the same
    configuration option.


Sneaky on the opt-in :).  Agreed that it can be similarly configurable.


I thought it was rather loud, front and center, rather than sneaky ;-)


In closing, I'd like to say that during this thread I have made some
comments about remote execution being "just an optimization", and I
understand that it is a very important feature to you. I don't mean to
try to sideline remote execution or turn it into a second class
citizen, and I think that this discussion might be partly emotionally
driven due to my poor selection of words.

That said, I do think we have to be careful to avoid the new feature
breaking the existing basic, local build scenarios, and I do feel
strongly that the user experience should be seamless regardless of
whether remote execution is enabled or not, which I think I have
outlined pretty clearly above.

Cheers,
    -Tristan
[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]