Re: Non-workspaced incremental builds

[Jürg Billeter <j bitron ch>]

Hi Charles,

On Mon, 2018-01-29 at 10:35 +0000, Charles Bailey wrote:
> This is a summary of a proposed feature that would make use of "build
> tree" caching (that is caching of the build-root after a completed
> build) to enable faster rebuilding of dependent projects in
> BuildStream.

Thanks for writing this up. It matches my understanding.

> Build tree timestamps:
>
> Currently, when sources are staged they are given a known mtime, if
> the same happens for staging a cached build tree then all files will
> have the same timestamp, rather than object files having a more recen
> time than the corresponding sources. This is not an issue for most
> build tools that rely on timestamps as an output having an identical
> timestamp to its sources is usually considered "up to date".

Yes, I expect this to work as long as we don't start incremental builds
using a build tree from a failed build attempt. Which should anyway be
the case for non-workspaced builds.

> Unsolved issues:
>
> - Locating a cached build-tree
>
>   [...]
>
>   These latter two are not easily found by a cache key lookup of the
>   build element to be built.
>
>   In theory, there may be many possible cached build trees that could be
>   used as the base for an incremental build. The best candidate would
>   depend on how different the source and dependency files of the
>   candidate are from the build element to be built. You could view this
>   as a metric space.

I suggest local metadata (similar to workspaces.yaml) to store the
cache key of the last successful build for each element. If we update
this also when we pull pre-built artifacts from a remote cache,
developers can benefit from incremental builds even if they've never
built the element locally.

Global metadata keeping track of the last successful build for each
element doesn't make sense to me as multiple developers or servers
might concurrently build completely different branches of the same
element, which would lead to less consistent incremental build times.

Storing build trees in the regular artifact cache is the most
convenient option. However, we need to be aware that this will require
a lot of storage space and high network bandwidth between clients and
remote artifact servers.

> - Dependency timestamps
>
>   This issue also affects workspaced incremental builds. If a new
>   version of a dependency is staged, all of its files will be given a
>   start-of-epoch timestamp. This means that any timestamp-based build
>   tool will assume that all outputs in the incremental build area are up
>   to date with respect to input files (e.g. header files) in the
>   dependency, even if they have actually changed since the previous
>   incremental build.

I've suggested a possible solution for this in #215. The same approach
should work for non-workspaced incremental builds as well.

Jürg