Re: [atomic-devel] Proposing a new way to deliver Atomic systems: rpm-ostree jigdo ♲📦

[Colin Walters <walters verbum org>]

On Wed, Dec 6, 2017, at 10:59 PM, Petros Angelatos wrote:
> Hi all,
>
> The docker delta situation might not be as unsolvable as it seems.
> I've been working on Balena[1], a moby based container engine focused
> on IoT usecases. 

Ah, interesting, submittted a PR to add it to our related projects list:
https://github.com/ostreedev/ostree/pull/1369

BTW this part of the discussion may be better on
atomic-devel@ but briefly the containers side of PA has been working
on libraries for this stuff rather than having it all inside the engine, e.g.:
https://github.com/containers/image

And OK wow, this is the first time I looked at Resin.

> One of its main features is the ability to pull
> images using binary deltas in a way that produces bit-identical copies
> of layers and image metadata. The deltas are based on librsync, which
> I ported to Go, but other algorithms could be added in the future like
> bsdiff. It currently achieves 10x-71x reduction in payload size
> compared to docker pull[2].

librsync over the whole image?  One of the good things in the ostree static
delta design is they are "chunked" at a configurable size, so it's possible
to process incrementally/resumably.   The libostree deltas are also strongly
influenced by the Chromium ones:
http://dev.chromium.org/chromium-os/chromiumos-design-docs/autoupdate-details
Basically "The protobuf is a series of instructions that the client must perform in order. "
is a powerful idea (although it also introduces versioning issues).

Is there any more information about the delta part of Balena?  I briefly
tried to look; found commits like
https://github.com/resin-os/balena/commit/ba0cbdd22c9fd3ba665de1d60d37619071fa5747
but it seems like a lot of the code is very entangled into the Moby fork.

Anyways much bigger picture: if you make any effort to propose this
as a standard upstream in OCI please keep me in the loop!

> This means one can ignore the compressed
> checksums completely and work with the image id, which is the hash of
> all uncompressed layers plus metadata. This is how we plan to verify
> delta pulled images.

Yeah, that  makes sense.  (I have my doubts about notary:
https://mail.gnome.org/archives/ostree-list/2016-October/msg00002.html
but let's not go too far afield here)
 
> Where I'm getting with this needs another piece of information that is
> not on the balena website, but we've basically implemented a bootable
> container system with it used in resinOS. It's very similar to OSTree
> in that the sysroot only contains a docker-like directory and some
> metadata about what is the active sysroot. Then, a special statically
> linked PID 1 will load the container sysroot, do a pivot root, and
> exec to /sbin/init of the new root. This means signed, atomic updates
> with binary deltas using docker images.

Right, OK.
 
> To conclude my thoughts, an alignment with OCI could be using OSTree
> as a storage driver for balena. More concretely, I see the current
> balena deltas providing bandwidth efficiency and OSTree providing
> storage efficiency using the hardlink farm approach.

See also
https://github.com/containers/storage/pull/137
 
> I'd love to hear your thoughts on this. It looks like we're moving
> towards the same goals and it would be great if we could work on
> something together.

There are a whole lot of angles to this.  First again let me repeat
that if you guys decide to make an effort to push deltas "upstream"
to OCI, that'd be a very interesting thing for us (with my "Project Atomic"
hat on) to collaborate on.

It'd also be interesting to see whether the libostree backend
for containers/storage linked above is interesting for your host OS.

That said though I think we're going to be in co-opetition here since
the rpm-ostree vision is quite different.   (And the point of libostree
is that e.g. a similar thing could be built using apt/deb:
https://github.com/hughsie/PackageKit/pull/225)

For example, on a CentOS Atomic Host system *today* (and
for quite a while now), one can:

rpm-ostree install libvirt

It Just Works.  We didn't have to change the libvirt RPM at all;
it's the same RPM that works out of the box with yum.
The whole ecosystem of tools that works today on top of libvirt
also Just Works.  But now my system has fully atomic upgrades,
and does both containers and virt.  I actually use this on my home server.

Similar considerations apply for my laptop where I'm typing
this email, which uses Fedora Atomic Workstation.  I have
a bunch of layered packages, like
` krb5-workstation opensc origin-clients pcsc-lite-ccid powerline`
which all also Just Work.

(Now I do do most my development in a "dev/pet" container which
 uses Docker...need to write all this up more)

Anyways this hybrid ability means rpm-ostree can potentially capture a huge
array of use cases, from servers both big and small, to laptops, and
IoT cases.   

To rephase, the point is that the "image" side is always there, *and*
the package side is always there.  For example, a case we've always
had in Project Atomic is: what if I want to use a different version
of Docker, or I don't want  Docker at all?

Now with the latest rpm-ostree, one can:
`rpm-ostree override replace ./docker-{,common-,rhel-push-plugin-}1.13*.rpm`
or whatever, and that also works.  The package side comes into play.

I happened to see:
https://github.com/resin-os/meta-resin/blob/508ea99ba600c5328fc2d8d9654cd6f35e0c998e/meta-resin-common/conf/distro/include/resin-os.inc#L81

I'm sure you guys have hit these types of cases; do you also offer the ipkg
binaries and have a workflow for temporarily/permanently changing
the package components?

So I personally am going to continue to spend most of
my time on rpm-ostree - basically continuing to augment/fix the existing
RPM ecosystem, while also keeping things like bootloader handling,
state management, etc. in libostree so it can be shared with other ecosystems.
And also supporting libostree for non-host cases like flatpak and BuildStream.

But there are definitely potential areas of alignment as I mentioned above.
And thanks for posting about Resin/Balena here - very interesting!