Re: [Tracker] embedded code copy in tracker is problematic

[Carlos Garnacho <carlosg gnome org>]

Hey Michael,

On Tue, Mar 1, 2016 at 6:48 PM, Michael Biebl <mbiebl gmail com> wrote:
> 2016-03-01 18:00 GMT+01:00 Carlos Garnacho <carlosg gnome org>:
> > Hi Michael,
> >
> > On Tue, Mar 1, 2016 at 4:52 PM, Michael Biebl <mbiebl gmail com> wrote:
> > > Hi everyone,
> > >
> > > I just noticed that the new tracker 1.6.2 contains a code copy of
> > > sqlite and no longer allows one to use the system sqlite library.
> > > This is problematic for various reasons and distros like Debian [1]
> > > and Fedora strongly discourage such code copies.
> > >
> > > Would it be possible to re-add the ability to link against the system
> > > sqlite and only fall back to the embedded copy if the system library
> > > doesn't meet the requirements of tracker (and output a big fat warning
> > > in this case)?
> >
> > Not sure if you missed the action caused by sqlite 3.11. From that
> > version on, they've hidden by default a sql function that's
> > indispensable for us.
> >
> > https://web.nvd.nist.gov/view/vuln/detail?vulnId=CVE-2015-7036
>
> Seems I missed that, indeed. Most likely because of:
>
> sqlite3 (3.11.0-2) unstable; urgency=low
>
>   * Compile with SQLITE_ENABLE_FTS3_TOKENIZER for backwards compatibility
>     (closes: #815499).
>   * Update Standards-Version to 3.9.7 .
>
>  -- Laszlo Boszormenyi (GCS) <gcs debian org>  Tue, 23 Feb 2016 21:31:39 +0100
>
> So this particular change was reverted in Debian.
>
>
> > Tracker itself is not hit by this cve, but we've evidently become
> > colateral damage since this is removed by default.
> >
> > The embedded copy solution has only been done on current stable
> > releases (1.4 and 1.6). It's not one I'm too happy with. But it's
> > surely better than requiring -DSQLITE_ENABLE_FTS3_TOKENIZER
> > system-wide (partly why I just went for always using the embedded
> > copy, this is something distros don't want enabled). For master (and
> > upcoming 1.8), I've opted for using FTS5 (which doesn't have this
> > problem), and still rely on the system sqlite library.
> >
> > I understand and share your concerns, but this is kind of a rough spot
> > we're on :).
>
> Has there been any discussion with sqlite upstream to solve that
> differently? I mean breaking consumers of the sqlite APIs can't be the
> proper fix for that.

I talked with them (Richard Hipp and Dan Kennedy) through private
email. The solutions basically seemed to be:
- Including a static sqlite copy wherever fts3_tokenizer() is needed
- Using FTS5, which offers a way to customize FTS tokenizing that are
not affected by this vulnerability
- Adding such a similar way to FTS3

Basically, the vulnerability is completely intrinsic to the
fts3_tokenizer() call with 2 arguments, they can't both fix the cve
and keep offering it unchanged. Of those three options, all three
require changes in the users of this call, plus for the third we'd
have to wait for an hypothetical change, and wouldn't erase 3.11 from
earth either...

So I took solutions 1 and 2 wherever they apply. I also considered
backporting the FTS5 changes to stable branches, but it's too many
changes and too bleeding edge for me to be comfortable with it...

Sorry about this situation, I know it will also be a pain to you.
There's a lot to grumble about it, we can do that in unison :)

Cheers,
  Carlos