Re: Feature proposal: incremental build support

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Tue, 2017-10-24 at 19:38 +0100, Chandan Singh wrote:
> Hi,
>
> When building large projects, it is very desirable to have support for doing
> incremental builds so that when one makes changes to just a small subset of
> files in the project, not everything has to be rebuilt. So, I would like to
> propose adding a feature that would allow users to do incremental builds (or
> dirty builds).

Thanks for this Chandon,

So, I dont have mind share much right now at all for "future stuff",
but will try to summarize our IRC chat from late last night.

 o Lets take baby steps and talk about one implementable, digestable
   thing at a time

 o Incremental builds for workspaces is most pressing, lets address
   that before getting too fancy

 o Lets not talk about what Sander wants to talk about; caching build
   sandboxes - I really dont expect that to be 2017 material

 o Chandon has an approach, briefly discussed, doesnt take much code,
   but is disruptive inasmuch as it leaves build sandboxes
   (dependencies and all) behind, this is a severe change which I think
   leaves us open to a whole surface of edge cases where things can
   go wrong / behave unexpectedly

 o Suggest we try to just take the workspace code as is, not add any
   new user facing options or anything, but make them behave better.

   Workspaces are workspaces, if they get littered with .o files from
   a build, that's expected.

 o Instead of staging files from a workspace into a build directory,
   lets *mount* it directly at the expected location into a sandbox

   This leaves only a couple edge cases and "fits" into the general
   scheme of things.

     o bst shell --sysroot /path/to/buildsandbox

       Has to become workspace aware, so that it also mounts the
       workspace from the right place

     o Error reporting about failed builds need to be enhanced to
       tell the user the right thing about the build sandbox, like
       where to find it etc.

     o Might make sense to symlink from the build directory to
       the workspace when the sandbox is *not* in use, only after
       a failed build.

       This might obviate the previous, and allow the user to
       navigate ~/.cache/buildstrea/build/<sandbox>/root/.... in
       the regular ways

Cheers,
    -Tristan

> While the actual mechanics of how it would work are still open for discussion,
> I envision that it would allow users to indicate that they are doing a dirty
> build, by providing an option like `--dirty` to `bst build`. When this option
> is provided, BuildStream would preserve the contents of the build directory,
> including modification times. It is quite important that we preserve
> modification time as a lot of build systems (make, ninja etc) use it to
> determine if a file needs to be rebuilt.
>
> Current implementation
> ----------------------
>
> Currently, BuildStream does not support incremental builds at all. Compared to
> using the underlying build tools (make, ninja etc) directly, this results
> in an inferior workflow. When building outside the sandbox using those build
> tools directly, it is very natural to do incremental builds. But the same is
> not true when building in BuildStream as it starts fresh for each build,
> resulting in wasted time and resources while re-compiling the same files over
> and over again.
>
> While the option of doing a clean build is very useful when one is doing a
> production build, it slows down developers when one is iterating over a project
> locally.
>
> Proposed implementation
> -----------------------
>
> To support this functionality, there would be minimal user-facing changes:
>
>   - add `--dirty` option to `bst build` command
>   - provide a way to clear dirty build areas - this can be done as a separate
>     `bst` sub-command.
>
> To support dirty builds, we will need to change the way BuildStream stages
> sources and manages the build area. I am going to describe one way in which
> this can be implemented. When `--dirty` is not specified, the behavior of
> BuildStream will stay unchanged from what it is currently. When `--dirty` is
> supplied, I propose following changes:
>
>   - Don't override modification times of source files. This can be achieved by
>     omitting the call to `utils._set_deterministic_mtime()` here:
>     https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/element.py#L1389
>
>   - Name the build directories in a deterministic way so that it is possible to
>     use the same build directory in a future build. Currently, it is created as
>     a temporary directory but if this changed to use the name of the source,
>     we should be set.
>
>   - Preserve the contents of the build directory and do not clean it up after
>     the build finishes, regardless of whether or not the build succeeds.
>
>   - Instead of trying to create a new build directory for each build, try to
>     re-use the build directory with the same name if one already exists.
>
> If we add an option to allow dirty builds, it would also be reasonable to
> provide users a way to clean their dirty build areas. To support this, we
> can add something like `bst clean path-to-bst-file.bst`. Alternatively, a
> `--clean` option can be added to `bst build` as well.
>
> Please let us know your comments.
>
> I'm looking forward to hearing your ideas for implementation. My current idea
> is based on my quick attempt to create a proof-of-concept.
>
> Cheers,
> Chandan
>
> --
> Chandan Singh
> https://chandansingh.net
> _______________________________________________
> Buildstream-list mailing list
> Buildstream-list gnome org
> https://mail.gnome.org/mailman/listinfo/buildstream-list