Reply to self, to try and clarify some things.
Hi,
Hi Sander,
This one seems to keep falling through the cracks, picking it back up ;-)
It is one of those threads :)
On Wed, 2018-09-12 at 15:57 +0100, Sander Striker wrote:
> Hi,
>
> On Tue, Sep 11, 2018 at 12:26 PM Tristan Van Berkom <tristan vanberkom codethink co uk> wrote:
> > On Mon, 2018-09-10 at 17:01 +0200, Sander Striker wrote
>
> [...]
> > > I am assuming the $(cache_key) is actually unique enough on its own?
> > > And this name is purely symbolic, contextualized for a project?
> > > Nothing would require this under the hood? As in, I wouldn't expect
> > > a remote artifact cache to carry the $(project-name)/$(element-name)
> > > as part of its keys(?).
> >
> > Since the beginning, the refs we use to store / retrieve an artifact
> > have always been namespaced as:
> >
> > ${project-name}/${element-name}/${cache-key}
> >
> > Whether to change that is an interesting discussion indeed.
>
> I'm merely thinking about potential reuse; and whether the
> namespacing is causing an artificial reason why cache results are not
> shareable.
Not an artificial reason really no.
Consider two elements from different projects, which purport to have
the same inputs, these will produce the same `${cache-key}` but we
cannot go so far as to say that the output for these two inputs can be
the same.
That would be unfortunate. So if you include e.g. webkit in two projects that share an ArtifactCache, you'll be building it twice?
That would only be a good assumption for the `files/` subdirectory of
the artifact, i.e. the output that a build actually generated.
Right... I think we're getting to the core of the issue :). This part of the artifact is the expensive part to produce. And any opportunity we have not to redo work here, we should take IMO.
Depending on what kind of provenance information we encode in files in
the `meta/` directory, and what kind of information is included in the
build logs, this will be incorrect.
If for instance, we want to encode information about from which project
the artifact came from in it's metadata; then reusing an artifact
generated by one project in another project will certainly be
incorrect; even if the `files/` output stored in that artifact are the
same.
It makes me think that we may need an indirect mapping here. One where we can share the binaries between the Artifacts and maybe fill out the "information about from which project
the artifact came from in it's metadata" when we build in the context of said project.
I'm expecting that, when using remote execution, we actually reuse much of the builds, due to the fact that Actions care about the source input tree, and not where the source input tree came from. That concept could maybe be extended to an implementation for local builds as well?
There is another angle to think about here too, which is the aliases
which belong to the project.conf which produced an element: If the
artifact is not produced within at least it's project's namespace; then
we don't have any assurances at all that the aliases used to produce
the final URLs used in Sources, are in fact the URLs we expect.
Interestingly this again touches on issue 569 which I raised but is not
really related: By not binding the outputs of a project into a
namespace, we are weakening the statement that `refs` can in fact be
unique for the URLs we declare them for (because now the URLs fall out
of the equation, as those are only defined by the project.conf).
I see your point.
So if we can have cache key map to a source key (the root of a merkle tree representing the staged sources), then you can create a cache key function that substitutes all of the sources with the source key, and it will be correct without the element and project names.
We could leverage SourceCache (
#440) for this, which is essentially the mapping of $cachekey to Directory digest. Where the directory is representing the staged sources for the element.
That gives us a second chance to consider whether there is anything to build. In short:
1) verify if $cachekey is in ArtifactCache, if not then
2) verify if $cachekey is in SourceCache, if
a) yes, then calculate a new $cachekey_s that uses the directory digest instead of source alias/ref, and again check if $cachekey_s is in ArtifactCache
b) no, then stage the sources, and calculate the digest of the staged sources. Then calculate a new $cachekey_s and check if $cachekey_s is in ArtifactCache
If we do encounter a hit on $cachekey_s, we now have an oportunity to reuse the actual built artifacts, and we can create a new artifact with appropiate metadata (which may include the fact of reuse).
Only when we do not encounter either $cachekey or $cachekey_s should we build anything. And then obviously put entries under both cachekeys on completion.
I've left out how the SourceCache gets populated, because I think we've already had previous discussions on that.
Now, that said, with the above in mind, it clearly doesn't matter if $cachekey contains project and element, because there is a second line in place for build avoidance. I think that is a good trade off.
This doesn't make refactoring or renaming a project prohibitively expensive. And it also does the right thing (read: dpesn't waste time and resources) when people copy a .bst (or part of a project) rather than using a junction.
[...trimmed a whole lot of good conversation...]
Cheers,
Sander
