Re: OSTree and OCI images

[Alexander Larsson <alexl redhat com>]

On tis, 2016-11-22 at 21:23 -0500, Colin Walters wrote:
> On Fri, Nov 18, 2016, at 04:38 AM, Alexander Larsson wrote:
>
> > However, the code you link to above is not about pruning commits,
> > it is
> > about pruning branches (which then refer to the above commits).
> > "ostree
> > prune" will never remove a branch like that. Of course, when
> > pulling a
> > layer we need to know what ostree commit id the layer id maps to,
> > so we
> > need the branch to be there, and thus you need some custom branch
> > pruning. (Unless we store the oci layer id => ostree commit id some
> > other way.)
>
> Right, conceptually there needs to be two levels of prune logic, one
> for
> oci -> ostree, and ostree.  The atomic command's OCI support does
> the first part.
>
> This distinction is all mirrored in rpm-ostree BTW, which currently
> uses a separate repo in /ostree/repo/extensions/rpmostree/pkgcache,
> as well as its own strong refs on the "base" commits.
>
> And that leads to pointing out there has always been custom refs
> in the OSTree-for-host case like "ostree/1/0/0" because of the
> distinction between deployments and the origin refs.
>
> Anyways what I'm trying to get at here is - what do we see as the
> near term scope of this OCI-in-ostree work?  Should we take
> most of the core logic for managing the refs -> layers from
> the atomic command down into ostree in addition to the importing?

I'm not fully clear on what I think should be in ostree itself, and
what could be done on a higher level. Some primitives (like the merge-
layers-into-mtree stuff) are hard to do outside of ostree, so it makes
a lot of sense to add them. For the higher level stuff things are not
so clear cut, but if multiple projects (i.e. both atomic and flatpak)
are using something then maybe it makes sense for ostree proper to do
it.

For flatpak, these are the OCI operations we currently support:

 * Export a (non-layered) branch (flatpak app) as an OCI image in an
   OCI directory layout. This is "flatpak build-bundle --oci" in the
   current code. In addition to just dumping the ostree content
   as a tarball layer this adds enough ostree specific commit metadata
   to the image that the ostree commit can be reconstructed with the
   same commit id at a later point.

*  Import an OCI image from an local directory into a (non-
   layered) ostree branch, merging multiple layers in the
   source image, and also applying any optional ostree specific 
   metadata in the image. This is "flatpak build-import-bundle --oci"
   in the current code.

With this in place we can do manual conversions to/from oci images on
the server side, which is useful if you're distributing the end result
via ostree, but your build system produces OCI, or if you use OCI as
an intermediate transport somewhere.

The next step for flatpak is to support installing an app directly from
an OCI source. This means adding support for remote OCI repos to the
oci-import code. I want the end result to be a merged commit in the
local repo.

A natural extension of the above would be to also store the individual
OCI layers in the ostree repo (in addition to the merged layer), so
that reconstructing imported OCI images is possible, and so that you
can avoid downloading base layers multiple times. This implies work on
OCI layer id -> commit mapping, and pruning support for these.

The question is, what part of this goes into ostree? I can see several
approaches:

* Just the basic primitives:
  - apply layers during commit
  - apply layers during checkout
  - map OCI image ids => commit & handle pruning of these in some way
  Then everything else can be done at a higher level.

  Of course, this means a lot of code will not be shared. Maybe that
  is not a huge deal though. Also, such sharing could happen in a 
  higher level library.

* Support in ostree for parsing the OCI image layout and json files 
  so you can feed it an oci directory to import/export a single image. 
  Something like:
   ostree export --oci=DIR COMMIT
  and
   ostree commit --tree=oci-image=DIR:image:tag

  One issue with this approach is that its harder to support remote
  OCI repos without having the commit operation do a bunch of network
  i/o, which seems like a bad fit.

* Support for named "OCI remotes" in the ostree config so you can do 
  something like:

   ostree remote add --oci my-oci-remote URI
   ostree pull my-oci-remote image:tag

  One issue with this is that there is no natural mapping from the
  ostree branch names to the OCI image names and tags. The ostree
  branch names are very free-form, but the OCI image names are more
  limited (can't have slashes in the for one).

What approach would the atomic project be interested in?

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                            Red Hat, Inc 
       alexl redhat com            alexander larsson gmail com 
He's a scrappy hunchbacked card sharp on the edge. She's a cosmopolitan 
communist snake charmer with a knack for trouble. They fight crime!