Re: [BuildStream] Protect against plugin modifications of artifacts

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Hi Gökçen,

On Fri, 2020-06-26 at 11:18 +0100, Gökçen Nurlu via buildstream-list wrote:
> Hi all,
[...]
> > All the plugins above have a common point: they treat their dependencies as
> > their input, as other elements would treat their sources. As a brief summary,
> > here is the list of operations they are doing:
> >
> > - Archiving / checksuming files and writing that in the sandbox
> > - Writing a manifest of some kind
>
> Thanks for the investigation, +1. This part is actually my motivation
> to chime in. Correct me if I'm wrong, all the plugins above except
> dpkg_* ones at least, they are specifically designed to use their
> "artifact" outside of the buildstream universe. dpkg_* ones might be,
> too. I was just too lazy to read the code in detail..
>
> Here is my 2cs: what about enabling this Directory API
> (write/read/etc) only for the kinds of Elements that mark themselves
> as "leaf-level" and we forbid them to be used in a different bst file?
> Sorry if this has been discussed and rejected before.
>
> At least for the current plugins, they could just add a
> `LEAF_LEVEL=True` as a class attr, similar to other flags we have.
> For other elements that are actually in the middle of the build plan
> and don't have this flag, we can stop exposing
> `sandbox.get_virtual_directory()` at all, somehow. I find this easier
> to grasp and explain to plugin devs, there'd be less places to modify
> in bst codebase and minimal changes for (element) plugin codes.
> Citation is needed for the last two though, it's been some time since
> I checked the codebase :)

This is close to something which came up many times in the early months
of BuildStream development, essentially what you are calling "leaf"
elements is something we were calling "deployment" elements.

We drew a hard line on this and I maintain it was the right decision.

Basically, there is a need to wrap up bootable images and suchlike,
where you might want to encode a custom hostname or give it some custom
certificates and such.

This should all be done post-checkout, because anything that is a
BuildStream artifact, has a cache key, and should be reproducible.

If you need variance and you want host tools, there is no value to be
doing it in BuildStream anyway, and there is always the expectation
that some post processing may be needed on content which extracted by
`bst artifact checkout` (there are plenty of valid use cases for this).

Cheers,
    -Tristan