Re: Manifests and metadata

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Mon, 2017-11-06 at 18:02 +0000, Sam Thursfield wrote:
> Hi all
>
> I had a go at building a minimal VM image with BuildStream and I found
> that my split rules aren't working particularly well -- lots of junk
> ends up in the image that shouldn't be there.
>
> I also found we don't have any good way of debugging the split rules. I
> can `bst checkout` my 'compose' element and see what it contains, but I
> can't easily reference a given file back to the artifact that produced
> it or the split-rules domain that led to it being there.  There needs to
> be some way of getting hold of a manifest that contains this
> information.

Ok so, all the information which you want to deduce exists with a
loaded pipeline, this is the kind of information which make it possible
for example, to give better warnings on file overlaps:

   https://gitlab.com/BuildStream/buildstream/issues/114

Instead of adding metadata anywhere, try to follow that road and use
what we have instead, how can buildstream help diagnose what happened
with an already build compose element ? Is it possible that this could
be deduced from a log file, after all it seems that all you want as an
end game, is something that you can read as a human and diagnose /
debug with.

I firstly dont want to accept anything which adds metadata which can be
deduced and is not otherwise vital - the manifest and attributes are
*probably* vital for signing and verifying a checked out artifact, and
we're not sure it will optimize things yet - so it's not implemented
yet.

Secondly, your proposal suggests that in a compose element's output,
the artifact metadata be special somehow; I refute this, the output
artifact of an element, is the output artifact of an element, it is in
no way special because of its kind.

Especially, the provenance of files contained in a compose element, or
any element which is strict about having *only build dependencies* is
intentionally cut off at that point. There *should* be no knowledge in
reverse dependencies of whence came the content of this compose
element. So if you want to reason about that after the fact, the only
way to do so should be to have the originating project on hand to
deduce it for you again.

For your use case anyway, I think what you want is just enhanced
logging in the compose element (not in the master log, in the log of
the actual compose element).

Cheers,
    -Tristan