Re: [BuildStream] Proposal: BuildStream manifest file generation

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Tue, 2018-08-14 at 13:21 +0100, Sander Striker wrote:
> > On Tue, Aug 14, 2018 at 12:54 PM Tristan Van Berkom via BuildStream-list <buildstream-list gnome org> 
> > wrote:
> [...]
> > Ummm, really not what I was shooting for.
> >
> > Rather the question is, why should BuildStream have any added
> > configuration, or any knowledge of "What a manifest is" at all ?
> >
> > We need to provide ways to access and extract the data that you need,
> > there is no doubt about that; there are other requests already in queue
> > for these, and a solution needs to be thought out to better provide
> > introspection about sources and their refs; which probably need backing
> > support from Plugins to work at all.
> >
> > For a project that wants to create some kind of post build summary, I
> > would expect them to do something like this, which took me all of 20
> > minutes to write:
> >
> >   https://gitlab.com/BuildStream/buildstream/snippets/1744606
> >
> > In short, I think that given BuildStream should be very well
> > scriptable, any concept of a "manifest" in BuildStream proper is just
> > scope creep.
> >
> > Can we please instead focus on the `bst artifact` command subgroup, and
> > ensure there is a proposal to allow similar introspection for Sources
> > in some way (maybe a `bst source-show` command or similar ?) - and in
> > this way, ensure that people have the tools they need to accomplish the
> > tasks they want to achieve ?
>
> I actually still have a draft reply open to the artifact command
> subgroup :).  I do think it makes sense to add something there to the
> effect you describe above.
>
> Maybe
>   bst artifact origin <cachekey>
> or 
>   bst artifact genesis <cachekey>
>
> Which would return enough information to know what went into
> producing the artifact.  That would enable answering the question
> "For a given artifact, where did it come from?"
>
> Is that in line with what you were thinking?  Are you proposing to do
> mappings like this outside of BuildStream proper?

Basically yes. I think we already have some issues we clearly need to
tackle in order to provide tools which extract isolated pieces of
information for the user, for a variety of purposes and use cases.

Using these tools, it should be very easy to aggregate the pieces of
information that you want into a single file, in the shape and form
that you want, without having to bake any such aggregation directly
into BuildStream itself.

Also, it should be fairly efficient given that:

* You only really need to load the post-build project state once in
  order to derive the cache keys.

* Operations which only deal with artifacts should not have to load
  the project; as they only need to extract information from a specific
  artifact.

Cheers,
    -Tristan