Feature proposal: artifact signing

[Sam Thursfield <sam thursfield codethink co uk>]

Hello

This is a proposal to add a way of signing BuildStream artifacts with
one or more private keys, and verifying signatures using the public
parts.

We propose using GnuPG, via the GPGME library, to calculate and
verify signatures for artifacts, based on a checksum of the artifact's
contents.

Storing the signatures should be done in a backend specific way, but
with a single signing and verification codepath shared by all backends.
Signing an artifact should not change its identity -- a signature should
function as 'out of band' data for the artifact.

We propose to keep the process entirely manual from a user point of
view. This would mean adding new `bst sign` and `bst verify` commands.
Automatic signing and verification can be implemented later -- first we
should get the foundations right.

The use cases for this feature are:

  * Verifying that artifacts come from a trusted builder. [This could
    also be done by ensuring that only the builder has push access
    to a cache, but that may not always be practical].

  * Allowing QA teams to sign off on already-built artifacts after
    testing them.

A possible future use would be to implement something like the Debian
chain of trust for uploads (see:
https://manpages.debian.org/jessie/apt/apt-secure.8.en.html).


Existing implementations
------------------------

BuildStream doesn't support this at all right now, but some related
tools already do.

OSTree provides an `ostree gpg-sign` command which signs the checksum
of a commit against a GPG private key that's available in the
filesystem. An arbitrary number of signatures can be stored with
different keys.

To verify that a commit is signed, you can run `ostree show ${ref}`,
which gets you output like this:

    commit 4fdc67500455ff8fe7a20ec098979ccb5973308f48f3bc8c24d6c376f1a92d74
    Date:  2017-10-10 15:28:24 +0000
    (no subject)

    Found 2 signatures:

    Signature made Tue 10 Oct 2017 16:42:21 BST using RSA key ID 
7FCA23D8472CDAFA
    Good signature from "Ostree Tester <test test com>"

    Signature made Tue 10 Oct 2017 16:47:39 BST using RSA key ID 
46218ABC1359FFC1
    Can't check signature: public key not found

Flatpak Builder wraps this in a simple way: you can pass `--gpg-key` and
optionally `--gpg-homedir` when building so that the build output is
signed with that key when exported to an OSTree repo.

The `flatpak install` command refuses to install a runtime or app that
has no valid GPG signatures, unless the `--no-gpg-verify` option has
been set for the given remote.


Proposed implementation
-----------------------

We want the following user-facing changes:

  * add a `bst sign` command which attaches a signature to an
    existing local artifact, given a GPG private key
  * add a `bst verify` command which verifies that all the signatures
    on an artifact are valid using keys in the local GPG keyring
  * add `bst verify --key ID|PATH` which verifies that at least one
    of the valid signatures on the artifact is from the given GPG key

We don't think there's a "one size fits all" approach to storing
signatures in artifacts. However, it's important to have a single
codepath for doing signing and verification. This reduces the risk
of security holes in lesser-used backends. If we are careful about
how checksums are calculated it would also makes it possible to convert
from one artifact format to another without invaliding signatures.

Both artifact cache backends will be extended to provide:

  * a way to calculate the checksum of an artifact's contents
  * a way to store and retrieve signatures for an artifact

The generic codepath will have:

  * a sign() method that asks the backend to checksum an artifact's
    contents, signs the checksum with the given private key, and
    asks the backend to store the signature
  * a verify() method that asks the backend to checksum an artifact's
    contents, asks for a list of signatures on the artifact, and
    verifies the signatures against the given public keys

In the OSTreeCache backend, artifacts are stored as OSTree commits. We
will use OSTree's 'commit metadata' feature to attach our signatures.
(It would be nice if we can do this in such a way that OSTree's own GPG
signing functionality recognises our signatures, but that's not
required). The checksum will be the tree ID of the artifact's contents.
This can be easily found from the commit, no extra calculations would be
needed.

In the the TarCache backend, artifacts are stored as tar files. We will
store each signature as a file in the tarball in a `signatures/`
directory. The checksum will be a SHA256 of the tarball's contents
excluding the `signatures/` directory, which could be slow to calculate.
We should make sure that we use the same checksum algorithm as OSTree
here so that signatures are compatible.

We will need to check that `bst push` and `bst pull` transfer any new
OSTree commit metadata objects for existing OSTree commits.

GPG keys can be set to expire at at a certain time. If a key was valid
when an artifact was signed but has now expired, we should consider the
signature as valid.

GPG keys can be revoked with revocation certificates. We consider this
out of scope for now -- users will need to manually remove keys that
they don't trust.

That's all!
Please let me know your comments.
Sam
--
Sam Thursfield, Codethink Ltd.
Office telephone: +44 161 236 5575