[Tracker] Reviving the project, a first attempt

[Philip Van Hoof <philip codeminded be>]

Guys,

Buried under its own weight of complexity the project is stifled. Why do
I think this?

What we tried to achieve during the Maemo Fremantle and MeeGo Harmattan
years at Nokia was so complicated to sell and complicated to develop
that most of the lead developers who got involved in the development of
the project from vstore to master became what I call "expensive people" 

* The vstore branch, find this in the mailing list:
                              From: 
JÃrg Billeter <j bitron ch>
                           Subject: 
Re: [Tracker] vstore to master
                              Date: 
Thu, 16 Apr 2009 18:15:27 +0200

I always wondered, JÃrg, where the name vstore came from? But it was a
fantastic branch and piece of work that you did. It clearly steered the
project in the direction of SPARQL and Nepomuk as ontology. Thanks.

With expensive people I mean that the people who held a lead role in the
Tracker project since that branch got merged to master, are people for
whom it's hard to commit to a full time maintainer role for a project
that has no funding in the form of either a salary or rate.

The SPARQL and Miner components both became complex and at the same time
intertwined. I regret them being intertwined and I always wanted the
SPARQL endpoint tracker-store to be a different project than the Miner
projects and the Extractor project:

        Although I don't like the overkill design of Nepomuk-KDE, I like
        how Jos van den Oever kept libstreamanalyzer and strigi and
        separate from the Nepomuk-KDE world.

Opinions in the team admittedly differed on that. I never raised my
voice on this issue because I thought that the team had to stay together
to suffer, as a team, the burden of delivering what we delivered for
Harmattan MeeGo: The adhesion between Lanedo, Codethink, Codeminded and
for the Qt libraries like qtcontacts-tracker OpenIsmus was too important
for delivering, that splitting up the project wasn't a worthwhile risk.

This is not the case anymore. And I heard from developers of a new phone
OS being developed that Tracker is again used, that it was again a hard
sell internally, but I didn't expect anything less as our project was an
extremely hard sell within the Harmattan team at Nokia too, and that
they aren't dissatisfied with it. Even satisfied. Wow!

I mean. yes. Wow. Our contacts who where all insanely good developers at
Nokia from the Harmattan time have kept their word and said to whoever
wanted to hear the truth that Tracker, although a hard sell, solves a
truly important and hard problem: inter-app information sharing on a
high level using a inter-app defined and shared ontology with a query
language suitable for the purpose, SPARQL, combined with a efficient
file system meta data indexer. We delivered that and proof that its high
quality software exists in the fact that a few million N9 users, yes a
few million, are using it right now and yet there are no massive
protests on forums about corrupted data: Tracker's SPARQL endpoint is
today tested and robust high quality software. I salute JÃrg for that.

I would also like to thank our top contributors and the people who
worked on Qt based libraries built on top of libtracker-sparql for
spreading the truth about our team and Tracker. You guys know who you
are, I don't have to name you ;-)

And oh my God I'm writing so much text just to make a simple point ..

The API libtracker-extract's tracker_extract_client_get_metadata is not
public enough because the Tracker is relying too heavy on the file
system miner. Today it is time to change this.

Phone builders want to rid themselves of file system mining. Instead
they want to let MTP daemons, who deal with incoming files, do the
processing and extraction of file meta data. They don't want to
configure with DConf or a GKeyFile to point to a directory where the MTP
daemon will write files, at all.

Instead they want their MTP daemon to use a simple API that will trigger
tracker-extract into extracting the file and then writing the SPARQL
INSERT to the SPARQL endpoint.

I think we mis designed this because we thought too much that the entire
world of Nepomuk, inter app data sharing and meta data in general
necessarily depends on file system indexing. This is just not true.

Tomorrow's phone builders might not even use a file system. Why would
inter app data sharing then necessarily depend on file system indexing?!

File system indexing is of course important, but only for users who need
it. Like a desktop. A desktop needs it. A phone might not need it. And
if it does, they understandably want to limit its use.

I would like to propose to start with adapting libtracker-extract to be
fully documented, to change tracker_extract_client_get_metadata's API in
such a way that it is truly obvious for a platform builder, integrator
or app developer of for example a MTP daemon to call it in order to get
the file's meta data to be inserted into tracker-store before the MTP
daemon had to write the file itself.

To make it possible to call this on a .tmp-XYZ file for a file that will
later be renamed to Girlfriend.JPEG in the DCIM folder of the phone.
Right now this ain't possible, because libtracker-extract is too focused
on being "just a tool library for the filesystem miner".

To make language bindings for it like for JS, Dalvik, MonoTouch, Qt.

It ought to be a library for all application developers, just like how
libtracker-sparql is such a library: obvious in API, well documented,
suitable for wrapping it with for example a Qt layer and all that stuff.

I think whoever starts with improving libtracker-extract in this
direction, perhaps by renaming, copying or refactoring to a new library
the API tracker_extract_client_get_metadata, will revive the project to
its original glory.

Kind regards,

Philip


-- 


Philip Van Hoof
Software developer
Codeminded BVBA - http://codeminded.be