Re: Proposal for Remote Execution

[Jürg Billeter <j bitron ch>]

On Wed, 2018-04-18 at 11:58 +0100, Jim MacArthur wrote:
> On 11/04/18 21:37, Jürg Billeter wrote:
> > The CAS is similar to a OSTree repository, storing directories as Merkle
> > trees. However, unlike OSTree, CAS does not include support for refs.
>
> What's the material difference between a Merkle tree hash and an OSTree 
> ref (in the way BuildStream uses refs)? It's my understanding that our 
> OSTree refs are a unique identifier for the pre-build conditions, plus 
> the project and artifact name, while a Merkle hash would be a unique 
> identifier for the results of the build, but is there some other 
> difference I should know about?

This is correct. To elaborate a bit, both CAS and OSTree use Merkle
trees with SHA256 hashes for directories. OSTree adds commit objects on
top of this (similar to git), also hashed with SHA256. Each commit
object points to a Merkle tree (hash). And OSTree refs build on top of
commit objects (also similar to git). Each ref maps a custom name to a
commit object (hash).

As you correctly describe, BuildStream uses unique cache keys that
correspond to the pre-build conditions for refs. The functionality
provided by commit objects on top of Merkle trees (i.e., commit
parent/message) is not used by BuildStream. However, refs are essential
as otherwise we can't find the Merkle tree (and thus the artifact) when
we only have BuildStream's cache key.

Jürg