Re: [BuildStream] Proposal: BuildStream manifest file generation

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

On Tue, 2018-08-14 at 11:02 +0100, Josh Smith wrote:
> Hi Tristan, 
> Thanks for the input, I've addressed points below and had a few thoughts about improving this proposal.
> Proposed Solution 2.0
> ~~~~~~~~~~~~~~
> Project configuration will provide a `manifest:` configuration for
> selecting what data will be included in the manifest. (The 'manifest'
> may need to be renamed for clarity)
>
>     manifest:
>         elements:
>             junction:
>             - options
>         sources:
>             git:
>             - ref
>             - url
>             tar:
>             - url
>             - ref
> This configuration would enable all git and tar sources to include ref and url in the manifest as well as 
> having junctions state their options. 
> Regarding source names, I think it is best to identify sources by index and kind since there is no field 
> for identifying sources.
> Potentially the ability to optionally name sources could be quite useful to this feature.
>
> Providing a manifest configuration will invoke a manifest to be produced during a build (maybe checkout?). 
> The manifest can most likely also be 
> generated through other means such as providing `--build-manifest` to commands such as show.
> Element() and Source() will get buildstream private functions to build their own manifest section using the 
> manifest configuration available through their associated project. 
>
> Side Note: Currently, config validation is performed within Configure because each Plugin knows which 
> config keys are valid.
> If we could make this a constant that is defined by the plugins, then validation can be performed for both 
> manifest configuration
> and before we call `Configure()` (We could avoid changes to backwards compatibility by only performing 
> these validation checks 
> when the constant is actually populated)

Ummm, really not what I was shooting for.

Rather the question is, why should BuildStream have any added
configuration, or any knowledge of "What a manifest is" at all ?

We need to provide ways to access and extract the data that you need,
there is no doubt about that; there are other requests already in queue
for these, and a solution needs to be thought out to better provide
introspection about sources and their refs; which probably need backing
support from Plugins to work at all.

For a project that wants to create some kind of post build summary, I
would expect them to do something like this, which took me all of 20
minutes to write:

  https://gitlab.com/BuildStream/buildstream/snippets/1744606

In short, I think that given BuildStream should be very well
scriptable, any concept of a "manifest" in BuildStream proper is just
scope creep.

Can we please instead focus on the `bst artifact` command subgroup, and
ensure there is a proposal to allow similar introspection for Sources
in some way (maybe a `bst source-show` command or similar ?) - and in
this way, ensure that people have the tools they need to accomplish the
tasks they want to achieve ?

Cheers,
    -Tristan