Re: [BuildStream] Proposal: OCI image source plugin

[Daniel Silverstone <daniel silverstone codethink co uk>]

On Wed, Oct 10, 2018 at 13:12:00 -0400, Chandan Singh wrote:
> I think it is a great idea to add support for loading OCI images so
> definitely +1 on the proposal.

Thanks.

> I too noticed the announcement about OCI registry specification being
> included in the next version but personally, I won't count on that
> happening in the short term. So, we might have to support both OCI and
> Docker for now. Although in future, once there is enough
> standardization around the registry API, we can definitely consider
> merging them into a single plugin.

Yeah I figured that might be a risk point.

> Even if the OCI specification comes soon-ish, I am not convinced that
> popular implementations (like Docker) will support it fast enough. For
> example, this issue [0] has been open for over 2 years to have `docker
> load` support OCI image format but is not fixed yet.
> [0]: https://github.com/moby/moby/issues/25779

While this is indeed sad, it does validate my assumption that OCI images were
ultimately gzip compresed tar files :-)

> So, I think the Docker and OCI plugin can coexist for some time. The
> Docker plugin should probably move to BuildStream proper sometime soon
> but that is a separate topic.

This is definitely something for another discussion :-)

> That said, I definitely think there is some scope for reuse of some
> logic between the two plugins, somewhat similar to how we have
> `_downloadablefilesource` that is shared by several source plugins.

Indeed.  Docker images are layered similarly to OCI images, though how the
content is acquired from a registry differs from how it'd be acquired from
a file.  Perhaps short-term, using the docker source as a reference and then
reimplementing in the OCI plugin would be the best approach, but writing that
with a view to later either extending the OCI plugin to also support docker
registries and images, or else refactoring to share content when importing the
docker plugin into the main codebase.

> Daniel, I have some experience with OCI images so let me know if you
> need any help :)

Thanks Chandan.  I'm unlikely to be the engineer in my team who actually does
the implementation work, but I'll make sure whoever does contacts you.

We'll certainly want some *SMALL* OCI images (e.g. two or three simple files
in a number of layers to validate whiteouts etc) as part of the test suite so
that we can ensure the source plugin works properly.  If you can help to define
and manufacture those images, that'd be super-helpful.

D.

-- 
Daniel Silverstone                          https://www.codethink.co.uk/
Solutions Architect               GPG 4096/R Key Id: 3CCE BABE 206C 3B69