Re: [Tracker] Discussing the API of libtracker-miner

[Adrien Bustany <madcat mymadcat com>]

El 18/08/09 07:00, Martyn Russell escribió:
> On 17/08/09 19:00, Adrien Bustany wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > Hi all,
>
> Hi :)
>
> > currently, we have a branch of tracker named libtracker-namer, which
> > includes a lib to be shared among all miners (if I got it right).
>
> Yep.
>
> > Currently, this lib includes a miner skeleton, and monitoring
> > functions. The idea would be to include more features in this lib,
> > merging the work I did for my gsoc project.
>
> That sounds great.
>
> I should add, I have taken a look at your work and it looks good, but
> I think we will be writing the miner API in C and can probably provide
> a vala binding for it. We want people to be able to write miners in
> whatever language they want and C is most portable for this. Also, we
> have a large portion of the work done in C already of course.
I'm won't copy Philip's answer here, but yes if it's to extend an
existing codebase I don't mind coding in C at all (I just code in Vala
'cause it's faster, and using DBus from C is sometimes painful).
> > The miners I coded all connect to remote web services to pull data
> > like twitter status, flickr albums etc. They thus need to know about
> > the network status before initiating a sync (not mandatory, but it
> > prevents failing miserably). Also, they all need to store a token or
> > user/pass tuple to authenticate against the remote web service.
>
> I think the authentication part will be the hardest part to figure out
> for cross platform working and also for the Maemo platform which
> doesn't use GnomeKeyring either (AFAIK).
Yep, that's why from day 0 I hided the gnome-keyring implementation
behind an interface... Mikkel's link is pretty interesting, although
maybe a bit overkill for us. To me, the model where you have an
identifier associated with a secret + a set of attributes is pretty
simple, and should work everywhere. But yes, maybe we should discuss it
more.
> > Therefore, I propose the additions of the following services to the
> > lib :
> >
> > 1. Network status
> > Could use NetworkManager if available (we're not making a hard
> > dependency here) to know if the computer is connected to internet or
> > not. This would be abstracted with an interface to allow other network
> > providers (I think OpenSolaris for example uses something else).
> > NetworkStatus get_network_status ();
> > with NetworkStatus being an enum with 2 values, CONNECTED and
> > DISCONNECTED (better to put an enum here if ever we want to add more
> > values, for example to handle cases like 3G connections where data
> > transfers might be expensive for the user).
>
> Yes, I have used this in Gossip before too, it is fairly straight
> forward, but I don't think we can use NetworkManager in Maemo and I
> don't know if KDE use it. We can probably use some other low level
> tricks to know if we have a connection. Any suggestions?
PackageKit had the same problem... I don't exactly remember how they
solved it, but I can have a look at the codebase. KDE uses
NetworkManager (via another interface called Solid IIRC), and on maemo
we can use libconic (and connman on Intel's distro). Again, as long as
we abstract things behind an interface, I think we solve the problem
(but maybe I'm too optimistic ? :) ).
> > 2. Tracker constants
> > Currently the fs miner hardcodes Tracker's DBus path. Wouldn't it be
> > simpler to put it in a shared header ?
>
> Yes. We have that in libtracker-miner. There is a common DBus API
> header there which is unfinished at this point (I will come on to that
> later).
>
> > 3. Credentials storage
> > Note : this might not belong to libtracker-miner... Discussion is open!
> > To connect to the webservice, you generally need a token or a
> > user/pass tuple. Sometimes, you also need to store additionnal data,
> > so a simple user/pass API is not enough. Again, things should be
> > hidden by an interface.
> > I currently use something like (vala syntax) :
> > string get_password (string token_name, out HashTable<string, string>?
> > user_data) throws GLib.Error;
> > void forget_password (string token_name);
> > void store_password (string token_name, string password,
> > HashTable<string, string>? user_data) throws GLib.Error;
>
> Again, this is problematic and needs some consideration. Ivan do you
> have any ideas about how this would work on the Maemo platform?
http://maemo.org/development/documentation/apis/4-1/ doesn't mention
anything about password management,
http://maemo.org/development/sdks/maemo_5_api_documentation/ either at a
first sight. Needs confirmation though.
> > 4. Configuration storage
> > It'd be nice for all the miners to have an easy API to store key/value
> > tuples. I've been told Martyn is working on it...
>
> Well, we have tracker-config-file in libtracker-common and
> tracker-keyfile-object with some convenience APIs, but it is all in C,
> not vala. You can see how we have done tracker-config in
> tracker-miner-fs and how we use those APIs there.
Yep, actually I used this for the tracker-subsystem, which is in vala (I
wrote a very simple vapi file). To me that system is powerful, but
adding a new setting is not that simple (needs to modify the private
structure, the object properties, write getters and setters...). I'd
rather advocate for a GKeyFile like API.
> > Thank you for your feedback !
>
> No problem.
>
> I should continue here with our current plans for the miner APIs:
>
> Methods:
>
>  * gchar * get_status()          - any string
human readable ?
>  * gdouble get_progress()        - 0->1 or -1 if not supported
ok, I have this in my current interface although no bridge^Wminer uses it.
>  * void    pause(gchar *reason)
>  * void    resume()
>  * void    start()
>  * void    stop()
looks fine to me
> Signals:
>
>  * started()
>  * stopped(gboolean interrupted)
>  * paused(gchar *reason)
>  * resumed()
>  * progress(gchar *status, gdouble progress)
>  * error(gint code, gchar *reason)
>
> These are the basic control and information APIs we intend for miners.
> If there is anything missing here, please let me know.
it's more or less what I use now, so I'm happy with it :)
> Also, I was thinking about writing the new applet in vala later on.
> But before I do that, we will probably add an API in libtracker-miner
> which will list all miners available, all miners running and basically
> service all the needs to query miners from tracker-status. The
> information shown in the applet should also be available from
> tracker-status about each miner.
in my solution, all miners install a .desktop file in
$(datadir)/tracker/bridges (should be changed to miners) containing the
name of the miner, as well as its dbus name and path (and some other
information only useful for web miners). I use this solution because it
allows a single process to have several miners, useful for example for
the twitter one (which does identi.ca as well) and the google one (does
only picasa, but could do other things).
> Did I miss anything?
if we sort out all the issues raised in this thread, then this will be a
good progress :)

Cheers

Adrien