Proposed future for Tracker - Smaller modules

[Martyn Russell <martyn lanedo com>]

Hello all,

So for some time now, I've been thinking that Tracker should be split 
into smaller parts. This has been suggested before and I've mentioned 
this a few times recently but not formally discussed it on the mailing 
list. So here we are :)


What is Tracker?
----------------
Well, put simply it's:

a) a two GUIs (tracker-needle, tracker-preferences)
b) a daemon to update the DB (tracker-store and ontology)
c) a daemon to 'first-pass' index the file system (tracker-miner-fs)
d) a daemon to 'second-pass' index the file system (tracker-extract)
e) a database schema configuration - i.e. ontology
f) a bunch of command line utilities:
   tracker-control
   tracker-info
   tracker-import
   tracker-search
   tracker-sparql
   tracker-stats
   tracker-tag
g) a series of libraries (only a few of which are public):
   libstemmer
   libtracker-bus
   libtracker-common
   libtracker-control (public)
   libtracker-data
   libtracker-direct
   libtracker-extract
   libtracker-fts
   libtracker-miner (public)
   libtracker-sparql (public)
   libtracker-sparql-backend
h) a series of plugins:
   evolution (broken)
   firefox
   nautilus
   thunderbird

There are a lot of other things packaged too, like functional tests, 
examples, unit tests, documentation, man pages, some utilities (e.g. 
sandboxing), etc.


What is the goal?
-----------------
What I would like to achieve is:

1. Make it easier to build just what's needed (e.g. just
   tracker-store), i.e. more modular.
2. Make development and maintenance easier by moving components to
   their own projects so development can be focused
3. Make a clear distinction between *core* functionality and "nice to
   have stuff".
4. I would like to unify the command line utilities a bit more
   similarly to how git operates, so instead of 'tracker-control -r',
   'tracker control -r', etc.


WHY?
----
In a lot of cases, all binaries do not need to be re-built each time a 
feature is being developed, yet right now it is. If I want to test a 
feature branch for tracker-miner-fs, I have to build and install the 
whole of tracker-store, ontologies, and often (depending on build 
options), that includes unit tests, documentation for each component 
too. This takes up time when switching between various feature branches 
and most of the time there is no reason to re-build half of the code again.

It's obvious that tracker-store is the core and so is the ontology it 
uses. Some of the libraries are too, but things like tracker-miner-fs is 
optional depending on what you're using Tracker for. This becomes even 
more apparent for some binaries like tracker-needle.

I recently added a ./configure option --enable-minimal to only build 
tracker-store and this would be easier if it was just in a separate 
repository.

Finally, making releases to fix something like the ontology or libraries 
should be much quicker, painless and easier for everyone involved to 
work with.


How would this work?
--------------------
This is not entirely clear to me yet. I've started working on a branch 
to group data in the tree together for subcomponents, but it's not even 
close to being complete:

  https://git.gnome.org/browse/tracker/log/?h=data-in-binary-dirs

I would like to hear any suggestions the community has about the best 
way to achieve this or what they would like from such a "break up". 
Right now the hardest part to work out is the (internal) libraries 
because some (like libtracker-common) are used by ALL components (pretty 
much).

Here is an example of what we could (ideally) have (right now, I've not 
fully checked this is possible...):

Module = tracker-store
  - ontology (database schema)
  - libstemmer (stemming library used by libtracker-fts)
  - libtracker-common (type and various utilities)
  - libtracker-data (database access and implementation)
  - libtracker-fts (SQLite module for tokenising)
  - tracker-store (daemon providing DBus interface)

Module = libtracker-sparql
  - libtracker-bus (DBus and read/write access to the DB)
  - libtracker-direct (readonly access to the DB)
  - libtracker-sparql-backend (internal implementation)
  - libtracker-sparql (public facing library to the DB)

Module = libtracker-miner
  - libtracker-miner (public facing library for miner base operations)

Module = libtracker-control
  - libtracker-control (public facing library to control data miners)

Module = tracker-data-miners
  - libtracker-extract (internal and used by tracker-extract)
  - tracker-miner-fs (1st pass indexing)
  - tracker-extract (2nd pass indexing)
  - tracker-miner-apps (indexes desktop files)
  - tracker-miner-rss (indexes RSS feeds)
  - tracker-miner-user-guides (indexes HTML and user documentation)
  - firefox (bookmarks)
  - thunderbird (emails)

Module = tracker-preferences
  - tracker-preferences (configure options), mainly for tracker-miner-fs
    (NOTE: This is not really in line with GNOME's search configuration)

Module = tracker-needle
  - tracker-needle (desktop search application and show case)
    (It should be added, both tracker-preferences and tracker-needle
     are written in Vala and it would be quite easy to merge these two)

Module = nautilus?
  - nautilus plugin (provides tagging widget)



Thoughts?

--
Regards,
Martyn

Founder & Director @ Lanedo GmbH.
http://www.linkedin.com/in/martynrussell