Re: Proposed module: tracker

[Jamie McCracken <jamiemcc blueyonder co uk>]

Joe Shaw wrote:
> Hi,
>
> On Wed, 2007-01-10 at 16:26 +0000, Jamie McCracken wrote:
> > Joe Shaw wrote:
> > > [snipped my concerns about Tracker]
> > all these also apply to EDS too yet no one complains?
>
> Some of them apply.  EDS isn't a general data store; it has APIs and
> backends that are very specific to the types of data it stores.  Also,
> very few applications (none other than Evolution, I believe) actually
> store info in EDS.  They're all read only, consuming EDS's data.

it does have the ability to write to but unlike text files its actually 
safe to have multiple writers with a db

> > Tracker is object to object with key values as properties so it can do 
> > more complex stuff (like a playlist object is a collection of music 
> > files). 
>
> Sorry, I don't understand what you mean by "object to object with key
> values as properties."

It can store relational hierarchy - its not just a flat key-value 
storage mechanism. So object to object relationships can be managed 
(Email->EmailAttachemnt, album->track, history->bookmarks etc)

> For your playlist example, I have a hard time believing that Tracker can
> offer a more optimized storage strategy than one that is tailored for
> the application.

trackers one would be both extensible and shareable - more flexible than 
something set in stone or that only one process could write to at a time


> > If we ever want first class objects or semantic web style functionality 
> > then stores like tracker/EDS are needed.
>
> I agree that things like EDS are needed.  For a music player, it should
> absolutely expose its data to the desktop; I said exactly that in my
> last email.

And same for photos, documents etc

> > They are not slower - in many cases they result in far fewer disk seeks. 
> > The key is to get stuff in bulk (like get all the properties in one call 
> >   as opposed to getting each individual ones separately). This 
> > eliminates the IPC overhead (something GConf needs too)
>
> You either have IPC round-trip overhead, or you have memory overhead in
> sending these large messages.  If you're sending them through the D-Bus
> bus, message passing overhead is very high, because it validates every
> message.

its a question of getting the balance right - you would request chunks 
of data but the optimal size of the chunk would be subject to 
experimentation.

If you get close in the above, the overheads become negligible - X is a 
good example here too of how it manages to offset the IPC overhead

> It's not uncommon for music libraries to have in excess of 10,000 songs.
> Since this is often thrown around as a potential use case, I think it's
> basically a requirement to see this implemented (even prototyped) using
> Tracker before its suitability can be determined.

smart apps download data as they need them (this is the whole point of 
client/server database apps)

However we are planning to support persistent live queries where speed 
is essential and the client wishes to get all results. In these cases 
the result set is held in a separate sqlite db (its persistent because 
it will last across sessions and survive system shutdowns). This 
query/db will be a flattened table in the most optimised format 
available so I am pretty confident we can match or even beat any other 
system in performance and apps can read it in-process if they wish (its 
like a snapshot but with added benefit of having live updates). EG it 
could be a query for all music files with a list of metadata.

> > And one API is easier to learn than a multitude for different metadata 
> > and keyword systems (honestly how many systems do we need to do tagging 
> > and metadata?).
>
> Again, no one has articulated what the "metadata problem" is.  Tagging
> is a pretty easy one to solve, and we can solve it with one API.
> Leaftag could be that API, or something else could be that API.  It
> doesn't really matter.

Tagging in fspot does not use leaftag afaik. Their metadata system 
appears to be based around something called semweb. Nautilus metadata is 
all xml based and not really shareable system wide.

THeres no question we need a metadata server to sort out all this mess 
so apps can interoperate with it.

> > Some examples of how tracker can improve the desktop :
>
> These are fine ideas, but let's see them prototyped first, so we can
> make an informed decision about whether it's practical and works, rather
> than accepting it prima facie.

thats why I am not asking for "tracker the database" to be considered at 
this time. We do need these implemented and I will get round to them.




--
Mr Jamie McCracken
http://jamiemcc.livejournal.com/