Re: OSTree and OCI images

[Alexander Larsson <alexl redhat com>]

On tor, 2016-11-24 at 11:39 +0100, Giuseppe Scrivano wrote:
>  
> A problem I see with the second approach, IIUIC, would be to add/keep
> in sync the
> docker registries in ostree as well, while now we are using Skopeo
> for
> resolving an image using the registries configured for Docker.
>
> As the second approach seems to depend on the basic primitives
> anyway,
> we could probably start from the primitives and move more stuff into
> ostree as it makes sense.

Here is a straw man set of basic primitives we could add to OSTree
which I think would help higher levels handle OCI images, as well as
other kinds of layered formats.

* Make prune respect related commits
 
 Each OSTree commit already have a list of "related commits ids". This 
 is not currently used at all. So, lets make it possible to set this 
 when you commit, and then we change ostree prune to respect this as a 
 reference. In other words, once some live commit is referencing a 
 related commit the related commit is also live (ie. will not be 
 pruned).

* Add weak references

 Make it possible to create a ref (i.e. a branch) that is marked as 
 "weak". Exactly how this would look can be decided elsewhere[1], but 
 the end result of a weak reference is that the commit it points to 
 won't be considered live by this ref, and if a prune operation 
 removes that commit the *reference* would be pruned instead.

* Make it easy to create new commit based on layers

 This is essentially pr #578, where you can use oci/docker style trees 
 by applying an mtree (which you can create from a ref, or from an 
 archive) to an existing mtree as if it was a layer.

With these simple primitives its possible to support layered images in
several ways.

For example, you could make an image by just commiting the image
manifest itself as a commit with a strong ref, referencing the layers
by commit ids in the related commit, and storing the mapping from image
layers to ostree commit ids as weak refs. This way the base layers and
their mapping will stay around until no image ref needs it anymore, and
you can apply the layers on checkout.

Another alternative is to store the image pre-composed, with the layers
references by related-commits + weak refs. This is similar to the
above, except you need to do less work during checkout, at the cost of
some minor duplication of directory metadata for the combined commit.
You could store the manifest in the commit metadata which would let you
fully reproduce the original layered image anyway.

A third alternative is to store the image pre-composed, but *not* refer
to the layers via related commit ids. This is essentially a "squash"
operation, which makes it impossible to reconstruct the layers, or re-
use the layers for downloading other images. The advantage is that any
files that were removed/replaced in an layer no longer has to be stored
on the client machine, which may lead to smaller disk-space use.


[1] Weak references could be implemented as regular branches with some
prefix, like "." or ".weak.". This would be very easy to add and pretty
safe in terms of backwards compatibility. Alternatively they could be
considered their own strong "type" of ref in ostree, similar to how git
splits branches and tags. I.e. we would have a new directory in the
ostree repo called refs/weak/, next to the refs/head. This would let us
handle things like mirroring and pulling differently for weak refs.

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
 Alexander Larsson                                            Red Hat, Inc 
       alexl redhat com            alexander larsson gmail com 
He's a witless day-dreaming cat burglar with acid for blood. She's a foxy 
communist pearl diver who can talk to animals. They fight crime!