Re: API question & Rust API bindings

[Colin Walters <walters verbum org>]

Hi Felix,

In a word: awesome!

On Fri, Oct 19, 2018, at 6:17 PM, Felix Krull wrote:
> I've been trying to build my own version of ostree-push [1], mostly for
> learning purposes. I'm pulling objects from the source repo using
> ostree_repo_load_object_stream, but there doesn't seem to be an equivalent
> function for writing objects. I thought ostree_repo_write_content would be
> that, but it appears to only understand file objects, feeding it metadata
> objects just makes it error. I suspect I might have to use
> ostree_repo_write_metadata for, well, metadata (I haven't had a chance to
> try yet). But that seems fiddly; is there something I'm missing here?

The type of an object is carried externally; stored in the repository it's
the filename suffix.  Similarly, the checksum is stored externally, and
so is the size (just encoded in the Unix stat buffer). This means that any
protocol serializing an object also needs to carry at least that 3-tuple of
(objtype, checksum, size).

One thing I'd like to enable better is augmenting tar slightly and supporting
that as a serialization format.  If the server doesn't trust the client (i.e. it
will be recomputing checksums), I don't think there's a huge advantage to
sending native ostree object formats rather than just having the server re-synthesize
them.  One just needs to send the commit object out-of-band; the dirtree/dirmeta
and content objects get re-synthesized.

(Well, one disadvantage is that if you have a heavily hardlinked object,
 today the libostree export-to-tar code doesn't generate hardlinks in tar,
 so each copy of the object will need to be parsed/checksummed N times;
 but we could fix that)

This intersects heavily with the ostree/OCI discussions.  Today e.g. flatpak
sticks the ostree commit object as JSON into the OCI metadata, you can
see the code here:
https://github.com/flatpak/flatpak/blob/master/common/flatpak-oci-registry.c

Since the OSTree format is reproducible, if you pass the commit sha256
you know you got the same thing at the other end.

That said a simple protocol is also sane.

> And, semi-unrelated to that, I built some Rust bindings for libostree,
> automatically generated from the gobject-introspection metadata. They're
> pretty incomplete because I only needed Repo so that's all I added for now.
> The code lives here: https://gitlab.com/fkrull/rust-libostree There's also
> a crates.io release: https://crates.io/crates/libostree

This is *awesome*!  I looked over your code some, and it looks
good to me overall, though I have to admit to not (yet) being
very familiar with the gtk-rs binding infrastructure[1].

It's probably about time we added a "language bindings"
section to the docs, since we also have the golang ones.

I'd also be interested in using the Rust bindings (and potentially the golang ones)
for *test cases* in libostree itself.  This has a few advantages; one is that we're
testing the bindings as we change the library, and second that we can use
a higher level language for tests.  (Though this is all somewhat nicer with
dynamic languages, as there one doesn't need to rebuild the bindings
when adding an API)

But again: awesome!

I'm also interested in what you're using libostree for, if you care to share?
(But you don't have to of course!)

[1]  Honestly I am still learning Rust and while we now hard require it in rpm-ostree,
I have been hesitating to take that leap into depending on gtk-rs for a
few reasons; one is that the libostree-using code in rpm-ostree all works
generally fine and any rewriting carries risks, plus there's a whole new
layer of FFI to understand and that could carry bugs.  But we'll almost
certainly dip our toes into this area soon, and if we do we'll
almost certainly build on your work and help contribute to it.