Re: Dilemma using ostree as regular user, losing permission bits

[Tristan Van Berkom <tristan vanberkom codethink co uk>]

Thanks for your prompt reply Colin,

On Mon, 2017-06-05 at 10:36 -0400, Colin Walters wrote:
> On Mon, Jun 5, 2017, at 05:49 AM, Tristan Van Berkom wrote:
[...]
> > Using bare repositories works fine for both requirements (A) and
> > (B),
> > assuming that the same user is committing and checking out files, but
> > then will fail for requirement (C), as far as I can see things will
> > break down as soon as a user on another computer attempts to checkout
> > from a cloned repository (in the case that their local uid differs from
> > the original committing user's uid).
>
> I'll be honest, I have never had anything like requirement (C) in mind.
> What specific technology is in use here?  NFS?  sshfs?
>
> I don't understand how it would work, since one can't create a hardlink
> to file one doesn't own.

Sorry I should have been more clear, by (C) I really only mean the
typical upstream/downstream workflow, where we have a central, but
separate repository which we use to share build artifacts between
peers.

And indeed in this case it makes sense to use an archive-z2 repo on the
artifact server.

What I was alluding to in the above is that if we were to use "bare"
repositories on a build machine, anything committed by the regular user
can be checked out by that same user without any loss; *but* that
regular user's UID/GID would be owner of any committed files, and so
"bare" checkouts of branches pulled from a central repository would end
up failing on other machines (unless other build machines accidentally
happen to use the same UID for their user, most commonly UID 1000).

> Maybe you want something more like a NFS mount that acts as a read-only
> mirror, and then each client has a bare-user mirror?  I did *kind of* have
> something like this in mind with the "parent repo" code that is in libostree
> today, except it's more about reading, we won't do "copy down" automatically
> if you want to hardlink.
>
> We could indeed add some logic to libostree to enhance this use case.
> But I'd like to know a bit more about the specific approach you're using.

Right, so to attempt to be brief, the setup with BuildStream is that:

  o Every build happens in an isolated bubblewrap sandbox (no host
    tools available there, ever).

  o Every individual build result (make DESTDIR=${directory} install),
    is stored as an "artifact" in a local users ostree repository,
    these build results are stored in separate named ostree branches.

  o Build sandboxes are constructed by staging various separate
    artifacts (ostree checkouts) into a temporary directory with a
    deterministic staging order (dependency ordered, e.g. glibc first).

    Note: We do not checkout multiple ostree branches into the same
          directory, not sure if that might work but we need to
          handle the cases of file overlaps and symlinked directories
          specially.

  o Source code is staged in ${sandbox}/buildstream/build

  o bubblewrap is used to run a series of commands in that build
    sandbox, and eventually the build results end up in
    ${sandbox}/buildstream/install

  o Finally we create a new commit to the local artifact cache
    by collecting the results from ${sandbox}/buildstream/install

For build artifact sharing between peers:

  o We make attempts to download (pull) artifacts from the artifact
   
server before attempting to perform builds locally in case we
    dont
have to.

  o If the user is authorized (with ssh), then we also push the
    artifacts to an artifact server after builds complete and are
    cached in the local repository.

> > What is the rationale behind forcing 0755 on all regular files in user
> > mode repos and clearing of setuid/setgid bits in user mode checkouts ?
> > (what software depends on this lossy behavior, and why ?)
>
> The rationale for clearing suid bits is that you *really* don't want to
> create suid binaries owned by your build user - any
> other user on the system could execute them to gain *your* uid's
> privileges.
>
> Another way to say this is that bare-user is designed for *container*
> use cases, and builds should always be in a container that runs
> as non-root.  And IMO, you shouldn't use suid binaries in containers.
> There's absolutely no use case for having e.g. a suid `/usr/bin/sudo`
> in your build containers.  Nor for any of the PAM binaries, etc.

Ok so I think I understand the why: "bare-user" is basically about
*running* software in a container, in which case you dont normally (or
ever) have any need for suid anyway.

Our use case differs slightly inasmuch as we want to *build* software
in a container (or bubblewrap) as a regular user, but the software that
we build will not necessarily run inside a container (deploying a
container image is only one possibility).

In order for the regular user inside a bubblewrap sandbox to deploy
what they have built to a bootable disk image or a "package" of sorts,
we need the checkouts to have exactly what we previously put into them.

Otherwise we end up creating bootable disk images with all regular
files OR'ed with 0755 and stripped of suid bits. These images still
"boot" but the situation is far from ideal.

> (bubblewrap basically enforces this by using PR_SET_NO_NEW_PRIVS)
>
> As far as 755...hmm, I am not sure why we're doing that.  Is this
> an aesthetic concern that every file is executable?   
>
> > Perhaps it would be more interesting to use "bare" repo mode for
> > regular users, but provide an option for overriding the uid/gid of
> > every file at ostree pull time ? This way a central repository could
> > store files as uid/gid 0, but regular users could pull to their own
> > mirrors while overriding the uid/gid in their local clones ?
>
> You can do that today with "central repository" being an "archive"
> repository accessible via HTTP, and users pull down to "bare-user"
> repos right?

Which is what we currently do, except that:

 A.) If we checkout without user-mode checkouts, then we fail because
     the active user is not root, and does not share the same UID of
     the original committer.

 B.) If we checkout with user-mode checkouts, then we lose the precious
     permission bits, so we can't really use this checked out data to
     create a bootable system disk image (we can, but we've lost some
     important information).

On Mon, 2017-06-05 at 10:48 -0400, Colin Walters wrote:
> On Mon, Jun 5, 2017, at 10:36 AM, Colin Walters wrote:
>
> > You can do that today with "central repository" being an "archive"
> > repository accessible via HTTP, and users pull down to "bare-user"
> > repos right?
>
> To follow up on this, if we had a config option where one could make
> a repo (of any mode I guess) act as a "pull-through" cache of an upstream
> repository, would that satisfy your use case?

This does sound like an interesting feature, currently we're looking
into using the summary file to determine (when a build is launched),
just what exactly needs to be built and what is available in the remote
cache (would allow us to optimize the build plan a bit better).

Maybe this pass-through mode could make all of this easier; however
we're not using that summary yet because there are concerns with
concurrent updates to the summary file at "ostree-push" time.

However this does not of course address my primary problem, which is:

 User A: wants to make a commit, with any file permissions and possibly suid bits
 User A: wants to checkout exactly what they checked in without loss
 User A: wants to ostree-push that to a shared repo
 User B: wants to pull and checkout that commit which User A uploaded, and have
         exactly the same thing (except that whatever User B checks out, belongs
         to User B's local UID)


Hope this was brief enough :)


Best Regards,
    -Tristan