Re: [Tracker] support for other database backends?

[Martyn Russell <martyn lanedo com>]

On 05/11/2012 02:49 PM, Jussi Laako wrote:
> Hi Ivan,

Hello Jussi,

We toyed with this idea for some months a while back. We would love the 
opportunity to try this out if someone is paying ;)

In principal it should be possible and likely we could improve on some 
of the existing cross user issues too (like sharing data not in $HOME 
like documentation or applications installed).

> > For example, libtracker-sparql is using direct access to sqlite for
> > reading and the tracker-store daemon via dbus for writing. Postgres
> > has a client-server architecture itself... that means a heavy
> > rewritting of libtracker-sparql internals or losing performance in an
> > extra IPC roundtrip.
>
> Of course idea would be to drop dbus and a separate daemon altogether
> from the picture.

You're talking about tracker-store here right, not tracker-miner-fs or 
the other miners?

D-Bus isn't so bad now we pass file descriptors around :) Also, we don't 
use it in the majority of cases where we're reading from the database, 
we use direct access.

Jürg any input on this?

> > Do you have any specific reason for postgres? Some people tested that
> > (for mobile devices) and the conclusion was that sqlite is ok: we are
> > in the "comfort zone" of sqlite, so postgres wouldn't give us any
> > sensibly better results.

Out of interest, what are the priorities for your use case here? With 
the existing tracker, we prioritise queries over updates so naturally 
they're faster. Last time I checked (a while back), MySQL was faster 
than PostgreSQL for querying. Just wondering why you've thought of 
PosgreSQL over MySQL really.

> It provides really flexible event notification straight from the SQL
> statements (see LISTEN/NOTIFY). Also dbus signals can arrive
> out-of-order compared to database transaction ordering. Postgre nicely

I wasn't aware of the D-Bus ordering issue? Though I believe the store 
does collapse duplicate events. It also has been designed to work on 
embedded systems with less resources, so we have various optimisations 
to avoid spamming clients with events which are useless.

> optimizes also writes, caching (no need to sync/re-read from the fs
> layer to get updated data after writes), etc.

Indeed. With the store, we really needed to be sure data was written in 
cases where you had instant power outage (e.g. you drop your phone down 
the toilet). So while caches are good, syncing is needed to some extent.

Perhaps there is better flexibility for Tracker with PostgreSQL and SQLite?

> So the nice part is that postgre provides all needed parts in one
> optimized package:
> - Multi-process shared database access
> - All needed IPC
> - Change notifications triggered straight from the database transaction

PostgreSQL is nicer to work with than MySQL from what I recall. We would 
gain a lot from a bigger database, but likely loose a lot on embedded 
systems and the efficiencies we made there. That's partly why we wanted 
to do some testing on this some months ago, to figure if the switch was 
worth it.

--
Regards,
Martyn

Founder and CEO of Lanedo GmbH.