[Evolution-hackers] Teaching the addressbook to sort contacts (and offer a cursor API)

[Tristan Van Berkom <tristanvb openismus com>]

Hi all,
   I've been preparing this work for some time now and
just realized that feature freeze is upon us in a couple
of weeks.

In the hope that it's not too late to consider landing
this new feature set, let's present the new cursor API
(details below).

There really are a lot of details to mention here so
it's possible I miss some things, please don't hesitate
to point out any gaps.

I've created a staging branch where I've squashed all
of this work into some meaningful patches, but still
there is a large number of patches (17 actually):

https://git.gnome.org/browse/evolution-data-server/log/?h=cursor-staging

You are encouraged to build the branch and run the example
contact browser in ${evolution-data-server}/tests/cursor-example/ ;-)

I've also opened a bug to track this here:
    https://bugzilla.gnome.org/show_bug.cgi?id=705178

Note that this branch assumes that the patches adding
asynchronous APIs for creating DRA book clients are applied:
    https://bugzilla.gnome.org/show_bug.cgi?id=701260
(or rather, the branch starts with the patches on that bug).

I know this is a big change to digest, I would really
like to be able to land this for 3.10 (and it has had
considerable testing so far) but if it's too late
in the cycle to consider this feature addition then
we can consider it for 3.12.

Cheers,
    -Tristan


Below is a summary of changes I wrote up today, the built
documentation (linked to below) shows the actual API and
includes a lot of detail on implementing and using cursors.

What is EBookClientCursor ?
~~~~~~~~~~~~~~~~~~~~~~~~~~~
EBookClientCursor is the user facing API to interact with
cursors (much like EBookClientView, EBookClientCursor is a
client side object with a dedicated server side object
backing it). An EBookClientCursor can also be filtered with
a search expression.

The cursor API allows the following use cases:

  o Browse a potentially large sorted list of
    contacts without needing to load more contacts
    than those which you need to display on screen.

  o Specify EContactFields as sort key(s), multiple
    keys can be specified.

  o Rich locale sensitive sorting (phonebook sort
    order is preferred over dictionary sort order
    in locales which provide "phonebook" style sorting
    rules)

  o Interaction with the user's alphabet

    - Fetch an array of letters representing
      the user's alphabet

    - Receive notifications of active alphabet
      changes (in the case of a system wide
      dynamic locale change).

    - Position the cursor directly before
      any contact which starts with a given
      letter (a letter in the user's alphabet).

    - Derive which letter in the user's alphabet
      that a contact should sort under

  o Provide feedback on current total contact and
    cursor position

  o Change the search expression at will, without
    losing the cursor position

More details on the cursor API in the documentation[0].

Limitations on EContactFields
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Currently only EContactFields which are configured to
be in the summary can be used as sort keys or referred
to in search expressions.

In other words, the SQLite cursor implementation code
does not have fallbacks to scan the whole table and filter
the result list, manually sort contacts or anything of
the sort. Only SQLite queries are used to implement
the cursor API.

This is something we can perhaps improve on in the
future, however I've left out these fallbacks since
it would be very expensive (in terms of computing
resources) to run these fallback code paths anyway.

Localized Sorting and Alphabets
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Localized sorting, along with the machinery needed
to leverage the locale specific alphabet, is brought
to you by ICU libraries.

These patches make ICU (4.8) a hard dependency, it
would be possible to "fake it" if ICU libraries are
not found but I wouldn't recommend that (ICU is
already quite widely distributed, I doubt it's
worth working around this dependency).

EDS's toolbox also gains an interesting tool with
which all these ICU features can be leveraged;
the ECollator[1].

Locales and Locale Changes
~~~~~~~~~~~~~~~~~~~~~~~~~~
Locales can now be stored as an addressbook setting
onto backends which support locale settings, backends
need to implement the EBookBackend->get_locale() and
EBookBackend->set_locale() methods if they need to
manipulate locale sensitive data.

Locale change notifications are delivered from the
"org.freedesktop.locale1" D-Bus interface[2], EDataBook
listens to that interface for locale changes and then
instructs the backend to store the new locale setting.

EBookClient also now gains a "locale" property, this
property is kept in sync with the locale setting stored
by the backend (as a read-only D-Bus object property
on the org.gnome.evolution.AddressBook interface).

Server Side Object Layout
~~~~~~~~~~~~~~~~~~~~~~~~~
On the server side, EDS grows a new abstract class
called the EDataBookCursor[3]. The EDataBookCursor offers
an API which can be used to navigate the cursor, and
it can also be placed on the session bus along side
the EDataBook which created the cursor.

The EDataBookCursor API is designed to be a single
API entry point to access cursor logic while being
usable both as an object on the bus (server side)
or usable directly by a process with an open
EBookBackend (direct read access mode).

The EDataBookCursor is created using the new
EBookBackend->create_cursor() method at the request of
the EDataBook (if the backend does not implement cursors
then no cursor is created and an appropriate error is reported).

Backends are expected to provide their own
EDataBookCursor implementation, the virtual methods
of EDataBookCursor are (hopefully) well documented
for that reason.

However, there is also an EDataBookCursorSqlite[4] object
which can be used to implement cursors for any backend
which stores it's contacts using EBookBackendSqliteDB.

Direct Read Access
~~~~~~~~~~~~~~~~~~
An EBookClientCursor will either own a D-Bus proxy
to a serverside EDataBookCursor, or in the case of
Direct Read Access mode, the EBookClientCursor will
own an EDataBookCursor directly.

At all times, the EBookClientCursor keeps a copy
of the addressbook revision. EDataBookCursor calls
are protected from races by passing in the expected
revision string, the cursor calls bail out with
an "out of sync" condition if the revision string
does not match.

For backends which support direct read access, the
revision string must always be committed along with
any other data in any addressbook modification
atomically (and the cursor implementation must
be able to perform atomic reads, where the data
being read is the same data at the time which
the revision string was read and compared).

The EBookBackendFile ensures this by nesting
any revision changes along with any other modifications
into a single SQLite transaction.

Example Code
~~~~~~~~~~~~
This branch also comes with a new example program which
can serve as a sort of reference implementation for a
locale aware contact browser (my intention is to also
link that example code into the documentation but I have
not gotten there yet).

The example code can be found at:
  $(evolution-data-server)/tests/cursor-example/

And the program can be run in an environment where the said
build of EDS is installed. The example takes an arguments
which is the path to a directory containing vcard files
(must have the .vcf suffix)... (I might push the 400 contacts
which I've been using into EDS git, improving the demo
experience, if that's appropriate).

Unfortunately the current version of the example program uses
the new GtkBuilder templates feature from GTK+ master,...
as Matthew and I discussed on IRC, I will probably need
to port that back to GTK+ 3.8 or 3.6 for the example to be
included.


[0]https://people.gnome.org/~tvb/libebook/EBookClientCursor.html
[1]https://people.gnome.org/~tvb/libedataserver/libedataserver-ECollator.html
[2]http://www.freedesktop.org/wiki/Software/systemd/localed/
[3]https://people.gnome.org/~tvb/libedata-book/EDataBookCursor.html
[4]https://people.gnome.org/~tvb/libedata-book/EDataBookCursorSqlite.html