Re: slow creation of ostrees using pungi/koji

[Colin Walters <walters verbum org>]

On Fri, Aug 18, 2017, at 09:31 AM, Dusty Mabe wrote:
> I've noticed that it can be really slow to create ostrees in Fedora using 
> pungi/koji. I think most of the time is spent in the 'committing'
> phase and believe this probably is related to the fact that we are
> writing a ton of small files.
>
> Is there anything we can do to improve this? We are essentially accessing
> the ostree repo via an nfs mount (so it's just a mountpoint on the local system).
> I wonder if it would be better to work in a temporary directory and then have some
> sort of 'ostree push' that would be more efficient.

Previous threads:

https://mail.gnome.org/archives/ostree-list/2015-October/msg00007.html
https://bugzilla.gnome.org/show_bug.cgi?id=756540

One thing that might be an issue here (though I'm just guessing without
profiling) is that today the `ostree commit` logic is synchronous.  On local
filesystems it's OK, but on NFS all the latency probably adds up.  However,
`ostree pull-local` (which is a way to do "push") is multithreaded/async.

One best practice is to commit to a `bare-user` repository first.  This
is mentioned in the docs:
https://ostree.readthedocs.io/en/latest/manual/buildsystem-and-repos/

So something like this:

ostree --repo=build-repo init --mode=bare-user
# Clone the latest commit from upstream
ostree --repo=build-repo pull-local /path/to/nfs/repo ${ref}
# Generate new commit
rpm-ostree --repo=build-repo compose tree --touch-if-changed=changed.stamp ..
if test -f changed.stamp; then
  ostree --repo=/path/to/nfs/repo pull-local build-repo ${ref}
else
  echo no changes
fi

Some code that's similar to this lives here:
https://pagure.io/fedora-atomic-host-continuous/blob/master/f/centos-ci/run-treecompose
Except it uses rsync rather than NFS.

Now, things can be a lot more efficient if the `build-repo` is treated
as a persistent cache and isn't regenerate each run.  Say you're running
rpm-ostree in a privileged kube container with a Kubernetes `emptyDir`
volume.