[Tracker] The tracker-store and tracker-store-queue branches

[Philip Van Hoof <spam pvanhoof be>]

Hi guys!

Passionate observers of the project have probably noticed that JÃrg and
me have started working on several new branches. 

We started out with calling them architecture-something followed by
branches called tracker-store-something. Right now we have tracker-store
and tracker-store-queue.

These branches are about a rearchitecture that we are working on.

The rearchitecuture's main ideas are:

1) The daemon

Transforming trackerd into tracker-store. This means that tracker-store
wont have indexing nor monitoring code. It'll be your SPARQL update and
query servant that knows about the Nepomuk ontologies.

I like to refer to this as the separation of church and code: it removes
all inotify from the SPARQL service and moves that to the indexer.

This makes it for example more easy to replace the indexer, to have
different kinds of metadata servicing processes. It also gives
integrators the opportunity to have a SPARQL+Nepomuk store and query
service without an indexer.


2) The indexer

Transforming the indexer into a completely standalone entity. This means
that the indexer will do the monitoring too. Right now the indexer
doesn't do this file monitoring, trackerd does it.

Among the ideas the guys want to split the indexer up into multiple
components. Most of this work depends on finishing tracker-0.6 as it'll
be mostly the same people who are now working on that branch who'll
passionately do most of the work on the rearchitecture of the indexer.

Their plans all sound very cool to me.

3) The extractor

Making the communication between the big three (store, indexer and
extractor) more efficient is also among the plans.

Right now you can compare the communication with a bunch of roundtrips
(in both directions) between all three processes. We are envisioning a
better structure where the communication is more like a triangle.

  1. The indexer will monitor and find work for the extractor to do
  2. The extractor gets instructed to extract metadata and ...
  3. It sends it to tracker-store, which stores it

4) Your project's metadata delivery component 

The new architecture is also for all those crazy ideas floating around
in the community: Zeitgeist, GeoClue, etc. They'll be telling us about
metadata "news": as soon as they know more about something, they'll just
us. And we'll store it as Nepomuk and make it accessible over SPARQL.

- Zeitgeist is a good example of this: its backend implements all sorts
  of funky algorithms for translating user behaviour into computable and
  meaningful metadata information. Like history of document usage.
  Linking up relevance based on what the user was focusing on while
  working with a specific document. I don't know (it all sounds
  crazy-cool to me, just do it guys!).

- There's a GSoc for collecting your "online metadata". For example from
  Facebook, your favorite Youtube movies, etc. Cooool! And the student
  is a really clever guy I noticed. So this will eventually just work.
  You'll see! Awesome idea, I like this very much.

x) Development methodology

We plan to do this rearchitecture step by step. Taking small steps at
the time. This means that you'll see certain crazy decisions being made
within branches. Choices that might not always work well.

It's of course not the idea that we'll also leave it that way. The idea
is of those that we want to focus on a certain aspect of the architec-
ture before we replace something else that we have put in place quickly
to enable ourselves focusing.

We will work as much as possible using branches. Each such branch can be
considered, for our contributors, as such an area of focus. We plan to
increase communication about the plans we have soon: We're still in a
bit of experimental phase, but the overall idea has been solidifying
lately.

I think that with tracker-0.6 being prepped, and being almost finished,
that sooner or later there's going to be more and more focus on the
tracker-store branches.

Here's a blog item that I wrote about these rearch plans earlier:

http://pvanhoof.be/blog/index.php/2009/04/24/tracker-our-near-future-plans


We hope that our community contributors will somehow find a way to join
us in our efforts. We're trying really hard to keep you guys involved.
But being a team of people, we're also working pretty fast nowadays.

Huge ideas combined with a great team of developers equals huge changes.

Understanding communication is key, I'll try to continue increasing it.


Good, now back to your keyboards and start hacking!

Cheers

-- 
Philip Van Hoof, freelance software developer
home: me at pvanhoof dot be 
gnome: pvanhoof at gnome dot org 
http://pvanhoof.be/blog
http://codeminded.be