Re: Proposing new ostreedev/ Rust repositories: ostree-ext and ostree-container

["Colin Walters" <walters verbum org>]

On Mon, Apr 5, 2021, at 5:15 PM, Will Manley wrote:
> >  - github.com/ostreedev/ostree: introspectable C library exists as it 
> > does today in ostreedev/ostree (there's also the /usr/bin/ostree CLI; 
> > will get back to that in a bit)
> >  - crates.io/crates/ostree: Rust bindings for libostree as it exists 
> > today (at https://gitlab.com/fkrull/ostree-rs)
> >  - github.com/ostreedev/ostree-ext: A new repository moved from 
> > https://github.com/cgwalters/ostree-ext/
> >  - github.com/cgwalters/ostree-container: A new repository migrated 
> > from https://github.com/cgwalters/ostree-container
>
> I'd support this proposal, but I think it would be better if all the 
> code were in the same repo.  I think it makes communication easier.  
> There would be a single place to raise issues and watch and discuss 
> PRs.  It also means that you can have single PRs that touch multiple 
> areas atomically - for example updating some GI annotation and some 
> manual rust binding at the same time.  This keeps all the context of a 
> change together.

Yeah...I have already hit this around incorrectly bound functions; e.g. the whole cycle for 
https://github.com/ostreedev/ostree/pull/2319 is:

- Land in ostreedev/ostree git master
- Wait for new upstream release
- Wait for crates.io/ostree to pick it up and release
- Finally profit

I just ended up shortcutting all of that with e.g. 
https://github.com/coreos/rpm-ostree/commit/ce20267b2d61bdb844369f44b8774077f6d262aa

In thinking about this after sending the mail, I think you're clearly right that the functionality from 
ostree-container should be part of "ostree-ext", just gated under an optional feature flag.

> Having an atomic git commit is also much more convenient than having to 
> make a release in a dependency so you can refer to the new version in 
> your dependant project - and much more "correct" than figuring it out 
> manually based on commit dates and messages.  This becomes particularly 
> important when trying to apply automated testing across the project.

Right, as of recently we use Rust for some tests in the core repo, but there's indeed no ability to test 
newly added APIs in the same commit without adding a wholly out-of-band binding.

(A long time ago, this was one of the motivations for gobject-introspection with *dynamic* languages; it just 
works out of the box to write tests for a C library in python/gjs etc. because the bindings are generated 
dynamically from the typelib.  But...at a certain scale one starts wanting static languages for the tests)

> Rust has built in support for this in the form of Cargo workspaces. You 
> have a single repo with subdirectories for each of your crates, and 
> they can refer to each other using { path="../ostree" } references.  In 
> our case it would be like that, but one of the crates wouldn't be a 
> crate - it would be the current ostree project.  The build system for 
> ostree wouldn't have to change.  In terms of Debian packaging it would 
> be a single source package with multiple binary packages built from it.

Right.  But...trying to stick all of this in the core repository is going to make it a lot harder to keep the 
Rust dependency chain optional.  And it's not just embedded, flatpak currently wants to keep the dependency 
chain small so flatpak can be used even on older host distributions.   A CI context that does a "minimal C 
only" build could work so that we're not breaking that of course but...ultimately it's a much more invasive 
proposal.

A more aggressive version is (and this would require Felix's buy-in) merging crates.io/crates/ostree with 
ostree-ext and ostree-container so we have

github.com/ostreedev/ostree (as it exists today)
github.com/ostreedev/ostree-rs

This also makes more obvious the github vs gitlab issue.  It's...well, I guess not out of the question for me 
to say that this effort should glom on to the existing Gitlab setup at https://gitlab.com/fkrull/ostree-rs 
somehow; I am conflicted often about that (xref https://blog.verbum.org/2020/12/03/still-on-github/ ).

But my instinct here is to not try to rework everything right now; I'll try to polish up ostree-ext and fold 
ostree-container into it in the meantime as a feature gate, and start on a 0.1.0 release, we can continue 
discussion.