Re: OSTree and OCI images

[Giuseppe Scrivano <gscrivan redhat com>]

Alexander Larsson <alexl redhat com> writes:

> For flatpak, these are the OCI operations we currently support:
>
>  * Export a (non-layered) branch (flatpak app) as an OCI image in an
>    OCI directory layout. This is "flatpak build-bundle --oci" in the
>    current code. In addition to just dumping the ostree content
>    as a tarball layer this adds enough ostree specific commit metadata
>    to the image that the ostree commit can be reconstructed with the
>    same commit id at a later point.
>
> *  Import an OCI image from an local directory into a (non-
>    layered) ostree branch, merging multiple layers in the
>    source image, and also applying any optional ostree specific 
>    metadata in the image. This is "flatpak build-import-bundle --oci"
>    in the current code.
>
> With this in place we can do manual conversions to/from oci images on
> the server side, which is useful if you're distributing the end result
> via ostree, but your build system produces OCI, or if you use OCI as
> an intermediate transport somewhere.
>
> The next step for flatpak is to support installing an app directly from
> an OCI source. This means adding support for remote OCI repos to the
> oci-import code. I want the end result to be a merged commit in the
> local repo.
>
> A natural extension of the above would be to also store the individual
> OCI layers in the ostree repo (in addition to the merged layer), so
> that reconstructing imported OCI images is possible, and so that you
> can avoid downloading base layers multiple times. This implies work on
> OCI layer id -> commit mapping, and pruning support for these.
>
> The question is, what part of this goes into ostree? I can see several
> approaches:
>
> * Just the basic primitives:
>   - apply layers during commit
>   - apply layers during checkout
>   - map OCI image ids => commit & handle pruning of these in some way
>   Then everything else can be done at a higher level.
>
>   Of course, this means a lot of code will not be shared. Maybe that
>   is not a huge deal though. Also, such sharing could happen in a 
>   higher level library.
>
> * Support in ostree for parsing the OCI image layout and json files 
>   so you can feed it an oci directory to import/export a single image. 
>   Something like:
>    ostree export --oci=DIR COMMIT
>   and
>    ostree commit --tree=oci-image=DIR:image:tag
>
>   One issue with this approach is that its harder to support remote
>   OCI repos without having the commit operation do a bunch of network
>   i/o, which seems like a bad fit.
>
> * Support for named "OCI remotes" in the ostree config so you can do 
>   something like:
>
>    ostree remote add --oci my-oci-remote URI
>    ostree pull my-oci-remote image:tag
>
>   One issue with this is that there is no natural mapping from the
>   ostree branch names to the OCI image names and tags. The ostree
>   branch names are very free-form, but the OCI image names are more
>   limited (can't have slashes in the for one).
>
> What approach would the atomic project be interested in?

We had problems to store a FQN name into an ostree branch.

The branch name must satisfy this regex:

#define OSTREE_REF_FRAGMENT_REGEXP "[-._\\w\\d]+"
#define OSTREE_REF_REGEXP "(?:" OSTREE_REF_FRAGMENT_REGEXP "/)*" OSTREE_REF_FRAGMENT_REGEXP

This is the encoding we used for it (function _encode_to_ostree_ref):

https://github.com/projectatomic/atomic/blob/master/Atomic/syscontainers.py#L1197

A problem I see with the second approach, IIUIC, would be to add/keep in sync the
docker registries in ostree as well, while now we are using Skopeo for
resolving an image using the registries configured for Docker.

As the second approach seems to depend on the basic primitives anyway,
we could probably start from the primitives and move more stuff into
ostree as it makes sense.

And we could teach Skopeo to write to OSTree so that we won't need to
deal with the "OCI remotes" case.  Is Flatpak using/going to use Skopeo?

Regards,
Giuseppe