Re: Stop the train ! Caching build trees is going to be too big

[Sander Striker <s striker striker nl>]

Hi,

On Sat, Apr 28, 2018 at 2:31 PM Tristan Van Berkom <tristan vanberkom codethink co uk> wrote:

On Fri, 2018-04-27 at 11:25 +0000, Sander Striker wrote:
[...]
> > What I propose that we do, is the following:
> >
> >   * Split artifact keys in two:
> >
> >     * The regular artifact remains "${project}/${element}/${key}"
> >
> >     * The cached build tree is addressable as
> >       "${project}/${element}/${key}/build"
> >
> >     * Alternatively, we split the artifact into metadata, logs,
> >       output and build components, this remains to be discussed
> >       and analyzed.
>
> I would prefer us to take a slightly different approach than storing
> under different keys, and instead store a "BuildStreamArtifact"
> message under the key.  That can then be used to download
> the different elements of the artifact.
> This is a similar approach to the ActionResult stored in the
> BuildFarm ActionCache.

The only real problem I have with the approach you suggest is that it
sounds like it can only be implemented with the CAS artifact cache,
while implementing it the way I suggest would probably be transparent
when we change artifact cache implementations from OSTree -> CAS.

If what you are suggesting is that nowhere other than the '_artifactcache/ostreecache.py' implementation, we are going to be using

multiple keys, then I have no concern.

If however, we are looking to change the code to use the

ArtifactCache API to deal with a key per partial artifact, that seems like

we're no longer keeping the change localized to the artifactcache

implementation.

 

Really this is only a matter of making it easier for separate ongoing
developments to happen without friction or blocking on eachother, and
this also sounds like an implementation detail which we could change
without too much effort later on, at the cost of an artifact version
bump.

Does this make sense ?

I'll admit that I'm not sure :).  If we're truly talking implementation detail,

and we'll be churning a version anyway, then disregard my previous

remarks.

 

Cheers,
    -Tristan

Cheers,

Sander

 

--

Cheers,

Sander