On Tue, 2017-10-24 at 19:38 +0100, Chandan Singh wrote:
> Hi,
>
> When building large projects, it is very desirable to have support for doing
> incremental builds so that when one makes changes to just a small subset of
> files in the project, not everything has to be rebuilt. So, I would like to
> propose adding a feature that would allow users to do incremental builds (or
> dirty builds).
Thanks for this Chandon,
So, I dont have mind share much right now at all for "future stuff",
but will try to summarize our IRC chat from late last night.
Thanks for taking the time.
o Lets take baby steps and talk about one implementable, digestable
thing at a time
o Incremental builds for workspaces is most pressing, lets address
that before getting too fancy
o Lets not talk about what Sander wants to talk about; caching build
sandboxes - I really dont expect that to be 2017 material
:). I don't mind that at all, just wanted to bring it up as related as I seem to recall this was how we initially thought about implementation.
o Chandon has an approach, briefly discussed, doesnt take much code,
but is disruptive inasmuch as it leaves build sandboxes
(dependencies and all) behind, this is a severe change which I think
leaves us open to a whole surface of edge cases where things can
go wrong / behave unexpectedly
o Suggest we try to just take the workspace code as is, not add any
new user facing options or anything, but make them behave better.
Workspaces are workspaces, if they get littered with .o files from
a build, that's expected.
+1. I can see how a convenience `build --clean` could be of value, but that can also be accomplished by close/open workspace.
o Instead of staging files from a workspace into a build directory,
lets *mount* it directly at the expected location into a sandbox
This leaves only a couple edge cases and "fits" into the general
scheme of things.
o bst shell --sysroot /path/to/buildsandbox
Has to become workspace aware, so that it also mounts the
workspace from the right place
o Error reporting about failed builds need to be enhanced to
tell the user the right thing about the build sandbox, like
where to find it etc.
o Might make sense to symlink from the build directory to
the workspace when the sandbox is *not* in use, only after
a failed build.
So that you have a complete tree at the sandbox location? That makes sense.
I wonder if we could/should retain and reuse the sandbox in the case of workspaces. That is, if the inputs to the sandbox didn't change, do we need to restage everything? Might be a premature optimization, though it might also address the staging of large base environments in that way. Let's defer for now, given it adds more complexity.
This might obviate the previous, and allow the user to
navigate ~/.cache/buildstrea/build/<sandbox>/root/.... in
the regular ways
Cheers,
Sander
Cheers,
-Tristan
>
> While the actual mechanics of how it would work are still open for discussion,
> I envision that it would allow users to indicate that they are doing a dirty
> build, by providing an option like `--dirty` to `bst build`. When this option
> is provided, BuildStream would preserve the contents of the build directory,
> including modification times. It is quite important that we preserve
> modification time as a lot of build systems (make, ninja etc) use it to
> determine if a file needs to be rebuilt.
>
> Current implementation
> ----------------------
>
> Currently, BuildStream does not support incremental builds at all. Compared to
> using the underlying build tools (make, ninja etc) directly, this results
> in an inferior workflow. When building outside the sandbox using those build
> tools directly, it is very natural to do incremental builds. But the same is
> not true when building in BuildStream as it starts fresh for each build,
> resulting in wasted time and resources while re-compiling the same files over
> and over again.
>
> While the option of doing a clean build is very useful when one is doing a
> production build, it slows down developers when one is iterating over a project
> locally.
>
> Proposed implementation
> -----------------------
>
> To support this functionality, there would be minimal user-facing changes:
>
> - add `--dirty` option to `bst build` command
> - provide a way to clear dirty build areas - this can be done as a separate
> `bst` sub-command.
>
> To support dirty builds, we will need to change the way BuildStream stages
> sources and manages the build area. I am going to describe one way in which
> this can be implemented. When `--dirty` is not specified, the behavior of
> BuildStream will stay unchanged from what it is currently. When `--dirty` is
> supplied, I propose following changes:
>
> - Don't override modification times of source files. This can be achieved by
> omitting the call to `utils._set_deterministic_mtime()` here:
> https://gitlab.com/BuildStream/buildstream/blob/master/buildstream/element.py#L1389
>
> - Name the build directories in a deterministic way so that it is possible to
> use the same build directory in a future build. Currently, it is created as
> a temporary directory but if this changed to use the name of the source,
> we should be set.
>
> - Preserve the contents of the build directory and do not clean it up after
> the build finishes, regardless of whether or not the build succeeds.
>
> - Instead of trying to create a new build directory for each build, try to
> re-use the build directory with the same name if one already exists.
>
> If we add an option to allow dirty builds, it would also be reasonable to
> provide users a way to clean their dirty build areas. To support this, we
> can add something like `bst clean path-to-bst-file.bst`. Alternatively, a
> `--clean` option can be added to `bst build` as well.
>
> Please let us know your comments.
>
> I'm looking forward to hearing your ideas for implementation. My current idea
> is based on my quick attempt to create a proof-of-concept.
>
> Cheers,
> Chandan
>
> --
> Chandan Singh
> https://chandansingh.net
> _______________________________________________
> Buildstream-list mailing list
> Buildstream-list gnome org
> https://mail.gnome.org/mailman/listinfo/buildstream-list
>
_______________________________________________
Buildstream-list mailing list
Buildstream-list gnome org
https://mail.gnome.org/mailman/listinfo/buildstream-list
