Re: ostree.sizes metadata issues?

[Dan Nicholson <nicholson endlessm com>]

On Mon, Aug 24, 2020 at 4:33 PM Dan Nicholson <nicholson endlessm com> wrote:
> On Mon, Aug 24, 2020 at 7:48 AM Alexander Larsson <alexl redhat com> wrote:
> > I don't remember the exact details, but OSTree has this:
> >
> > /**
> >  * OSTREE_MAX_METADATA_SIZE:
> >  *
> >  * Default limit for maximum permitted size in bytes of metadata objects fetched
> >  * over HTTP (including repo/config files, refs, and commit/dirtree/dirmeta
> >  * objects). This is an arbitrary number intended to mitigate disk space
> >  * exhaustion attacks.
> >  */
> > #define OSTREE_MAX_METADATA_SIZE (10 * 1024 * 1024)
> >
> > And I remember endless running into this at some point back when it was originally using ostree.sizes.
>
> As far as I know, it's never happened in the 5 or so years that I've
> been around this code. However, very early on I believe the object
> checksums were stored as hex strings, which would certainly bloat the
> size. In upstream it's simply a byte array. I think that the hex
> string format was only in our fork because I don't see any history of
> it upstream.

Sorry, I do remember this now and it did have to do with the
formatting. Originally, the code we had stored the object sizes in a
separate "sizes" object file that used hex strings for the checksums
and we did have to add a hack to avoid this limit:

https://github.com/endlessm/ostree/commit/749cf7ba0f8371cdcf0e32a8a5311f4f04a35d3f

However, the format that's in upstream in the commit metadata has
never had this issue as far as I know.