Re: [Tracker] Review request : Bridge manager subsystem

[Ivan Frade <ivan frade gmail com>]

Hi Adrien!

> as part of my gsoc project, I implemented a system to allow Tracker to
> index online resources. It's basically split into two parts :
> 1. The bridges are small programs which connect to a remote webservice
> and import data into Tracker via its SPARQL interface. They are
> standalone processes and all expose a common DBus interface. They are
> started by DBus activation.

Great! BTW, we call those processes "miners".

> 2. A bridge manager, in charge of calling the bridges to ask them to
> pull the data. Any program can talk to the bridges, here I implemented
> a bridge manager as a Tracker subsytem.

Uhm, the original idea was that each bridge/miner had its own
configuration and it is an independient entity; No control at all from
tracker itself.

Martyn was working in a generic configuration super-class, so you dont
need to write from scratch the configuration management for each
miner.

> Basically, what it does is
> list the available bridges, and call the synchronization method
> (Pull()) at a given interval. The default interval is 300 seconds, and
> is currently shared by all the bridges. It also exposes a DBus
> interface which allows to set the Pull () interval and to force a call
> to Pull() on one or all of the bridges.

I prefer completely autonomous bridges/miners than a "poll" solution;
besides, the poll code should be in tracker-store and it doesn't fit
there. Ttracker-store is just a SparlQL DB with a dbus interface; no
extra logic (DB and sparQL handling and backup stuff, that's all in
the store).

> Details :
> The object basically lists the available bridges (they all have a
> .desktop file in /usr/share/tracker/bridges), and keeps the list in
> memory.

This part is really interesting, because we need a tracker-applet that
receive information from all those bridges. The idea i talk with
martyn was something like the network manager applet, but instead of
listing networks, listing miners and their status.

When you click on the applet you would see a list like:

Filesystem OK
RSS           Updating
Flickr          Paused

Maybe a pause/play button to pause/run the miners. To do this we need
all miners present in DBus, so maybe we can use your code to implement
this.

> Where to get the code :
> git://git.mymadcat.com/tracker , branch tracker-bridges
> Then, if you want to get some bridges to play with, first install
> git://git.mymadcat.com/vapi
> git://git.mymadcat.com/libtrackerbridge
> git://git.mymadcat.com/bridge-manager
> git://git.mymadcat.com/bridge-facebook
> git://git.mymadcat.com/bridge-flickr
> git://git.mymadcat.com/bridge-twitter
> git://git.mymadcat.com/bridge-gdata

 WOW, i want to try this!

> Unknown issues :
> I'm sure there are many... Please report them to me !

 A thing that is common for almost all miners is the
"connection_status" functions to know whether we are online, or
whether the connection is good to retrieve massive data. It would be
great to do it with an interface and multiple implementations and in a
library. How about create a libtracker-miner library? (Something
similar to the old tracker-module library that Carlos created in 0.6?)

> Cheers

 Cheers, and congratulations for the good work!

Ivan