Re: Proposing Tracker for inclusion into GNOME 2.18

[Jamie McCracken <jamiemcc blueyonder co uk>]

Ross Burton wrote:
> On Tue, 2006-10-24 at 21:33 +0100, Jamie McCracken wrote:
> > > > Next up we would define the basic metadata or properties of the object 
> > > > (EG Contact.FirstName, Contact.Surname, Contact.EmailAddress etc along 
> > > > with their types) and add these to the metadata types table
> > > $ grep EVC_ e-vcard.h| wc -l
> > > 61
> > >
> > > Two of those are false positives, but that means that there are
> > > currently 59 defined keys in EDS.  The version in Maemo adds another
> > > three keys, but vCards allow arbitrary keys.  Does Tracker handle this
> > > programatically?  If I add a vCard that has a number of custom fields,
> > > can they be stored and indexed?
> > you would need to define each metadata field individually that you want 
> > indexed
> >
> > if fields dont need to be indexed then they can be stored as blobs 
> > either separately or as one big blob
>
> This limitation means that I'm not sure Tracker is suitable for storing
> contacts in a general manner.  The Nokia 770 addressbook application
> adds several custom fields and parameters to fields to the contacts,
> which is a no-op as far as EDS is concerned as the data store doesn't
> care.  Evolution also adds some custom fields and parameters over the
> vCard specification.  
>
> Then again, as it doesn't store vCards, if Tracker were used as a
> backend for EDS it could examine the vCards, check all of the used
> fields are registered with Tracker, registering the ones that are not as
> required before populating object.  How efficient this turns out to be
> remains to be seen.

yes thats possible but surely only the interesting fields should be 
extracted?

you can also blob up custom or uninteresting fields if you dont know 
anything about them or they are not important

> > > > Metadata is stored in a separate table according to its data type (IE 
> > > > string, numeric, Date etc). We use a composite primary key (ServiceID, 
> > > > MetaDataID) and the value itself is indexed.
>
> Is there a list type?  Some vCard types are list-of-string and this
> needs to be handled to be searched correctly.

csv format would be recommended for these (store as a blob or string)

we can do substring searches on non-indexed fields so its not necessary 
to index everything.

rarely populated or rarely searched fields should never be indexed in 
general. They can of course still be searched and they certainly wont be 
much slower if they are rare or number of contacts is < 1000

> > > Would I be able to search for all contacts that have an X-JABBER field
> > > where TYPE contains HOME (so FOOHOMEBAR won't match)?   A real-world
> > > example from the Nokia 770: give me all contacts with an X-JABBER field
> > > where the X-OSSO-BOUND parameter is "ross burtonini com".
> > You would need to completely flatten the metadata as 
> > Contact.HomeJabberID, Contact.WorkJabberID etc so that all metadata is 
> > mapped 1:1
>
> Eek.  Explosion of metadata.

yes but not all will be used by all contacts?


> Oh, can a single metadata type have multiple occurrences in an object?
> Can a Contact object have multiple Contact.JabberID fields?  The Contact
> object representing me would have three email addresses, four Jabber
> accounts, and two web pages.
>
> > It would be nice as tracker is a freedesktop thingie to avoid evolution 
> > specific naming and also to make it clean and easy for other software to 
> > use. Contacts also have relevance to Emails and instant messaging so 
> > that kind of software might wanna use these contact objects too.
>
> Yes, of course.  The only Evolution-specific values currently used in
> Evolution-generated vCards are very Evolution-specific (such as what of
> the four combos an email address appears in).
>
> > A contact can only have one metadata value per type (this is a primary 
> > key constraint)
>
> Ah, you've just answered my question above.  What is the solution for my
> case?

denormalise it into Contact.WorkEmailAddress, Contact.HomeEmailAddress

Each metadata item needs to be unique so that you can get/set its value 
individually as well as search it. Theres no way round this in any 
metadata DB.

> > > Is all of this possible with Tracker, or can tracker not efficiently
> > > store vCard data?
> > I think it should be okay
>
> I'm not so sure.
>
> > So basically what I would need from you is a spec for the contact object 
> >   detailing what methods you need and what metadata (all flattened out 
> > although non searchable stuff can be kept as a single blob type) they 
> > need then I can look at implementing it.
>
> Basically if you open e-vcard.h and look at the long list of EVC_*
> defines at the top, you have the set of official vCard fields (N, ADR,
> PHONE, etc) and a set of commonly used extension fields (X-JABBER, etc).
> 99% of these are strings, although some might be binary (e.g.
> PHOTO;ENCODING=b;TYPE=PNG:[base64-encoded PNG] compared to
> PHOTO;TYPE=URL:http://burtonini.com/head.png).
>
> As you can see storing vCard-like contacts is a tricky proposition.

yes - theres a lot of crud in the vCard spec.

> Storing a custom subset that isn't as flexbily as vCard is possible, but
> as vCards are a standard format, is it worth the cost of loosing data on
> import?

generally I would only extract the interesting fields and have a custom 
blob for the remainder (EG Contact.vCard) so you can recreate the 
original vCard if you need to.


--
Mr Jamie McCracken
http://jamiemcc.livejournal.com/