Re: [BuildStream] Protect against plugin modifications of artifacts

[William Salmon <will salmon codethink co uk>]

On 26/06/2020 12:18, Tristan Van Berkom wrote:
[...]
> Summary
> =======
> I think that for the summary of plugins which currently exploit the
> directory API to write data to the sandbox, things fall into a few
> categories:
>
>    A) Stuff that we can totally do in the sandbox, but people didnt
>       do because they were lazy or wanted something a bit more
>       convenient to work with than BuildStream.
>
>       E.g. oci, docker, tar - these are all expected to be implemented
>       by staging the required tooling into "/" and processing in the
>       sandbox.

When I wrote a first go at a genimage plugin I found that while running 
genimage in side the sandbox was pretty easy, having some basic things 
in element.bst' yaml as switches or option and then the plugin translate 
that in to a genimage config was a great way to keep the load of 
learning genimage off the user.

The script elements that valentine created drawing inspiration from my 
plugin in FD-SDK and GBM are far harder to maintain, share or create and 
i would like to supper seed them at some point with a revamped genimage 
plugin. Various people have had issue junctioning FD and copying script 
elements and having incompatibility issues when updating FD and thus 
genimage but not there scripts.

Its interesting to note that the plugin that i wrote did no use the api 
you are removing but a variation that you have expressed your dis 
approval with, and i think the api you are proposing to remove would 
have been a better bet and would have worked better than what i came up 
with.

There for I think that the comments that you have made for `C` also 
apply here.

>    B) Stuff that cannot be implemented at all in BuildStream.
>
>       E.g. collect_manifest, this is just a plain hack which cannot be
>       supported at all without designing some way for BuildStream to
>       deliver the data which `collect_manifest` desires.

I actually think a plugin is a really neat way to do this class of 
problem, as bst checks if it needs updating for you to some degree. If 
the cache key is the same bst knows not to redo the ordit. But i realize 
a external thing could implement yet another caching mechanism based on 
our cache keys.

That said allowing bst to be imported and using bst as a pyton lib to 
inspect a project would also make it easier to do such tasks. Did some 
one mention this recently?

Collect_manifest and the other auditing work is a important part of 
FD-SDK so saying we are going to write off this class of issue for bst2 
seems rather sweeping to me. Unless we can demonstrate that `bst show 
--thing` can give the required information. It seems important to me to 
make sure that the required information is rendered in a way that is not 
too onerous for a down stream thing to use, is there a `--json` option 
for bst show?.

>    C) Possibly metadata for package managers like rpm or dpkg.
>
>       In this case we might be able to create some helper facility to
>       safely create a file based on a %{variable}, in order to avoid
>       having to use a shell command like `cat > file <<EOF ...` and
>       the escaping which this entails.

I think this may be quite a tricky piece of discussion to get the 
flexibility required for the long list of plugins that do this sort of 
thing while still addressing your concerns although i think it will be 
well worth it, and I would imaging the implementation not being too 
difficult.

> Cheers,
>      -Tristan