Re: OSTree and OCI images

[Alexander Larsson <alexl redhat com>]

On tor, 2016-11-24 at 11:39 +0100, Giuseppe Scrivano wrote:
> Alexander Larsson <alexl redhat com> writes:
>
> > For flatpak, these are the OCI operations we currently support:
> >
> >  * Export a (non-layered) branch (flatpak app) as an OCI image in
> > an
> >    OCI directory layout. This is "flatpak build-bundle --oci" in
> > the
> >    current code. In addition to just dumping the ostree content
> >    as a tarball layer this adds enough ostree specific commit
> > metadata
> >    to the image that the ostree commit can be reconstructed with
> > the
> >    same commit id at a later point.
> >
> > *  Import an OCI image from an local directory into a (non-
> >    layered) ostree branch, merging multiple layers in the
> >    source image, and also applying any optional ostree specific 
> >    metadata in the image. This is "flatpak build-import-bundle --
> > oci"
> >    in the current code.
> >
> > With this in place we can do manual conversions to/from oci images
> > on
> > the server side, which is useful if you're distributing the end
> > result
> > via ostree, but your build system produces OCI, or if you use OCI
> > as
> > an intermediate transport somewhere.
> >
> > The next step for flatpak is to support installing an app directly
> > from
> > an OCI source. This means adding support for remote OCI repos to
> > the
> > oci-import code. I want the end result to be a merged commit in the
> > local repo.
> >
> > A natural extension of the above would be to also store the
> > individual
> > OCI layers in the ostree repo (in addition to the merged layer), so
> > that reconstructing imported OCI images is possible, and so that
> > you
> > can avoid downloading base layers multiple times. This implies work
> > on
> > OCI layer id -> commit mapping, and pruning support for these.
> >
> > The question is, what part of this goes into ostree? I can see
> > several
> > approaches:
> >
> > * Just the basic primitives:
> >   - apply layers during commit
> >   - apply layers during checkout
> >   - map OCI image ids => commit & handle pruning of these in some
> > way
> >   Then everything else can be done at a higher level.
> >
> >   Of course, this means a lot of code will not be shared. Maybe
> > that
> >   is not a huge deal though. Also, such sharing could happen in a 
> >   higher level library.
> >
> > * Support in ostree for parsing the OCI image layout and json
> > files 
> >   so you can feed it an oci directory to import/export a single
> > image. 
> >   Something like:
> >    ostree export --oci=DIR COMMIT
> >   and
> >    ostree commit --tree=oci-image=DIR:image:tag
> >
> >   One issue with this approach is that its harder to support remote
> >   OCI repos without having the commit operation do a bunch of
> > network
> >   i/o, which seems like a bad fit.
> >
> > * Support for named "OCI remotes" in the ostree config so you can
> > do 
> >   something like:
> >
> >    ostree remote add --oci my-oci-remote URI
> >    ostree pull my-oci-remote image:tag
> >
> >   One issue with this is that there is no natural mapping from the
> >   ostree branch names to the OCI image names and tags. The ostree
> >   branch names are very free-form, but the OCI image names are more
> >   limited (can't have slashes in the for one).
> >
> > What approach would the atomic project be interested in?
>
> We had problems to store a FQN name into an ostree branch.

Ok, so the problems go both ways then. Some oci image names don't map
to valid ostree branch names, and some ostree branch names don't map to
valid oci image names.

> The branch name must satisfy this regex:
>
> #define OSTREE_REF_FRAGMENT_REGEXP "[-._\\w\\d]+"
> #define OSTREE_REF_REGEXP "(?:" OSTREE_REF_FRAGMENT_REGEXP "/)*"
> OSTREE_REF_FRAGMENT_REGEXP
>
> This is the encoding we used for it (function _encode_to_ostree_ref):
>
> https://github.com/projectatomic/atomic/blob/master/Atomic/syscontain
> ers.py#L1197

That is the reverse problem compared to what i have. I.e. you're
pulling an arbirarily (validly) named oci image and need to store it in
ostree. I have a valid ostree branch (a flatpak app ref like
"app/org.gnome.gedit/x86_64/stable" and need to convert it to an OCI
image name.

> A problem I see with the second approach, IIUIC, would be to add/keep
> in sync the
> docker registries in ostree as well, while now we are using Skopeo
> for
> resolving an image using the registries configured for Docker.

Yeah, I don't think it makes sense to add some super generic image/repo
handling to ostree, just minimal OCI support. So, for the "atomic
install" case where you have to pull from docker registires and whatnot
you probably wouldn't use this anyway.

> As the second approach seems to depend on the basic primitives
> anyway,
> we could probably start from the primitives and move more stuff into
> ostree as it makes sense.

Yeah, that was my approach with the mtree layer-merging stuff.
And given that, the next thing to figure out is probably how we can
share the oci to ostree layer id mapping.

> And we could teach Skopeo to write to OSTree so that we won't need to
> deal with the "OCI remotes" case.  Is Flatpak using/going to use
> Skopeo?

Probably not, I have some code for handling local OCI dirs, and
extending it to pull from remote ones should be easy. Shelling out to
skopeo sounds like a major pain to make robust, when called from e.g.
gnome-software. How would you do things like proper progress or
localized error reporting? Screen-scrape the skopeo output?

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                            Red Hat, Inc 
       alexl redhat com            alexander larsson gmail com 
He's a scrappy small-town card sharp from the 'hood. She's a provocative 
tomboy Valkyrie with her own daytime radio talk show. They fight crime!