Re: [BuildStream] Plans for workspaces and incremental builds

[William Salmon <will salmon codethink co uk>]

On 02/10/2019 12:13, Darius Makovsky via buildstream-list wrote:
> Recently I've been thinking about workspaces and how they currently work
> versus how they should work in the future. One of the main goals is to 
> facilitate
> remote execution (RE) builds of workspaced sources in addition to local 
> build support.
> I've had some initial thoughts about this.
[...]
> Consequently, the post-process wsdir key is identical to the pre-process 
> wsdir
> key and the concept of key stability can be removed: WS keys do not require
> resetting and post-process recalculation and meaningful keys are 
> obtained at
> staging.

Well done!

On 02/10/2019 12:13, Darius Makovsky via buildstream-list wrote:
> [...]. In summary:
>
> * remove unstable cache key concept

This seems like a huge win for the complexity of our code +1 !

> * do not reset or recalculate workspace cache keys
> * store source digest, dependency hash, and artifact ref for workspaces
> * introduce mechanism to diff and apply trees
> * add logic to decide to continue or abort incremental builds


Your summary seems to reflect what has been discussed at previous build 
meets.

Further however..

Using workspaces to avoid major rebuilding is a huge advantage to how 
workspaces work in bst1.2. A common use case for myself is iterating the 
install commands for elements that may take hours to build. Currently I 
spend 1hr+ building normally, it fails at a late stage then I create a 
workspace and have to wait for it to build again then if it fails then I 
get to take advantage of a incremental build. Once I have got my .bst 
file right then I can remove the workspace and do a final rebuild so 
that my artifact can be cached.

Could we remove some of the friction hear? I would like to be able to 
create a workspace from a failed build if I have cached buildtrees 
enabled. Also can the final build with out a workspace realize that it 
is the same as the already build workspace?

I see that the final step would only be valid if I have only been 
modifying the bst file or if I create a commit who's contents are 
identical to those in the build tree?

PS

I imaging you have thought of the following but just in case..

When calculating the diffs don't forget that there might not be any 
change to the source it might only be the bst file that changed. And 
don't forget to handle soft reset properly/better.


> [1] https://gitlab.com/BuildStream/buildstream/merge_requests/1563
> [2] 
> https://gitlab.com/BuildStream/buildstream/tree/traveltissues/benchmark-3
>
> Best Regards,
> Darius
> _______________________________________________
> buildstream-list mailing list
> buildstream-list gnome org
> https://mail.gnome.org/mailman/listinfo/buildstream-list