Re: [Evolution-hackers] PIM server synchronization and Evolution online/offline state

[Milan Crha <mcrha redhat com>]

On Tue, 2012-05-08 at 10:56 +0200, Christian Hilberg wrote:
> @Milan: Do you think you could post your API work here at e-h list?
> That would give us something to base our discussion on. Even if no
> GSoC student picks up the topic, your work should not be lost.

	Hi,
sure, the initial draft is attached. I'm not attaching our conversation
around it, as it was long, even it might clarify certain things. As a
starter, let's call it EBackendOfflineCache, not as the initial draft
calls the base structure EBackendCache.

As far as I can tell, it is capable to satisfy all needs from Kolab, I
tried to draft it in an extendable way.
	Bye,
	Milan

The idea is to provide generic EBackendCache structure, create summary of stored objects
in an SQLite database, with all the common work being done in this structure, which will
be reusable on both addressbook and calendar backends.
The cache object will allow storing respective objects either to the summary itself,
or to a separate files, through EBackendCacheStorageInterface (there will be two
implementations availabe for it, the interface for file will also allow definition
of a default extension to be used on the objects).
The cache will take care of proper marking of offline state for each object and provide
functions to get all changes being done on an object while in offline.

/***************************************************************************************/

struct _EBackendCacheStorageInterface
{
	GTypeInterface parent_interface;

	/* adds list of extra fields to add into summary when creating
	   the database. Key is field name (may check it doesn't clash
	   with other fields) and Value is field type. */
	void (* add_required_fields) (GHashTable *fields);

	/* Generic function prototypes to read/write objects;
	   fields corresponds to current DB row being read from/write to */
	gboolean (* read_object) (EBackendCache *cache,
				  const gchar *uid,
				  GHashTable *fields,
				  gchar **out_str_object,
				  GCancellable *cancellable,
				  GError **error);

	gboolean (* write_object) (EBackendCache *cache,
				   const gchar *uid,
				   GHashTable *fields,
				   const gchar *str_object,
				   GCancellable *cancellable,
				   GError **error);

	/* optional function which is called before the object is removed
	   from database */
	void (* remove_object) (EBackendCache *cache,
				const gchar *uid,
				GCancellable *cancellable,
				GError **error);
};

/***************************************************************************************/

/* name of the field where object is stored in DB */
#define E_BACKEND_CACHE_DB_STORAGE_FIELD_OBJECT "db-storage-object"

/* to store objects in summary itself */
struct _EBackendCacheDBStorageInterface
{
	EBackendCacheStorageInterface parent_interface;
};

/***************************************************************************************/

/* to store objects in separate files; descendant can implement
   get_extension() - default is none; the actual filename is
   base_path/objects/uid.ext
*/
struct _EBackendCacheFileStorageInterface
{
	EBackendCacheStorageInterface parent_interface;

	const gchar * (* get_extension) (void);
};

/***************************************************************************************/

typedef void (* EBackendCacheSearchCustomFunc) (EBackendCache *cache,
						const gchar *uid,
						GHashTable *fields,
						gpointer user_data);

typedef void (* EBackendCacheSearchCustomTableFunc) (EBackendCache *cache,
						     const gchar *object_uid,
						     const gchar *rowid,
						     GHashTable *fields,
						     gpointer user_data);

/* summary fields used by EBackendCache itself */
#define E_BACKEND_CACHE_FIELD_OBJECT_UID	"object_uid"
#define E_BACKEND_CACHE_FIELD_OBJECT_REV	"object_rev" /* object's revision - value defined by descendant */
#define E_BACKEND_CACHE_FIELD_SYNC_STATE	"sync_state" /* one of EBackendCacheOfflineState */

typedef enum EBackendCacheOfflineState
{
	E_BACKEND_CACHE_OFFLINE_STATE_SYNCED = 0,
	E_BACKEND_CACHE_OFFLINE_STATE_LOCALLY_CREATED,
	E_BACKEND_CACHE_OFFLINE_STATE_LOCALLY_MODIFIED,
	E_BACKEND_CACHE_OFFLINE_STATE_LOCALLY_DELETED
};

struct _EBackendCacheClass
{
	GObjectClass parent_class;

	/* virtual methods */

	/* adds list of extra fields to add into table when creating
	   the table 'table_name'. 'table_name' is NULL when called for
	   actual summary table, otherwise contains custom table name.
	   Key in fields is field name (may check it doesn't clash with
	   other fields) and Value is field type.
	   Always call parent's method, to get all required fields.
	*/
	gboolean	(* add_required_fields) 	(EBackendCache *cache,
							 const gchar *table_name,
							 GHashTable *fields,
							 GError **error);

	/* allows to define custom tables used by the descendant. Each table_name
	   is passed to 'add_required_fields' and such table is created (or updated)
	   on cache load; the list is freed with g_slist_free_full (table_names, g_free)
	   when no longer needed by the caller. */
	void		(* get_custom_tables)		(EBackendCache *cache,
							 GSList **table_names);

	/* this is called on write, within e_backend_cache_put() call,
	   to update descendant-supplied fields based on the object changes.
	   'fields' is map from field name to field value, both newly allocated
	   strings which will be freed when no longer needed. The 'fields' are
	   not populated with currently stored values */
	gboolean	(* update_object_fields)	(EBackendCache *cache,
							 const gchar *uid,
							 GObject *in_object,
							 GHashTable *fields,
							 GError **error);

	/* generic functions to convert objects from/to string */
	GObject 	(* string_to_object)		(EBackendCache *cache,
							 const gchar *uid,
							 const gchar *in_str_object,
							 GError **error);
	gchar *		(* object_to_string)		(EBackendCache *cache,
							 const gchar *uid,
							 GObject *in_object,
							 GError **error);

	/* converts sexp string into sexp structure;
	   returned pointer is later freed with free_sexp();
	   the descendant can alternatively set sql_where_part to WHERE part
	   of an SQLite statement for fine-grained searching - this
	   is the only place which unhides the usage of SQL, but for
	   a good reason */
	gpointer	(* prepare_sexp)		(EBackendCache *cache,
							 const gchar *sexp_str,
							 gchar **sql_where_part,
							 GError **error);

	/* frees structure previously allocated by prepare_sexp() */
	void		(* free_sexp)			(EBackendCache *cache,
							 gpointer sexp);

	/* returns whether given object, identified by uid and fields,
	   satisfies criteria described by sexp; call
	   e_backend_cache_get_object_from_fields() to get actual object
	   instead of fields. */
	gboolean	(* check_object_with_sexp)	(EBackendCache *cache,
							 gpointer sexp,
							 const gchar *uid,
							 GHashTable *fields,
							 GCancellable *cancellable,
							 GError **error);
};

/* where is the cache stored - all data are in this folder;
   it is set only on cache creation; each folder can contain only one
   cache
*/
const gchar *	e_backend_cache_get_base_path		(EBackendCache *cache);

/* builds path with filename where to store attachment for the given
   object's uid, using suggested filename for it. The resulting path
   with filename may not correspond to the passed in, due to sanitize
   for usage on the file system. Free returned pointer with g_free(). */
gchar *		e_backend_cache_build_attachment_path	(EBackendCache *cache,
							 const gchar *object_uid,
							 const gchar *filename);

/* online/offline mode, in which the cache currently operates */
gboolean	e_backend_cache_get_online		(EBackendCache *cache);
void		e_backend_cache_set_online		(EBackendCache *cache,
							 gboolean is_online);

/* predefined persistent properties, optionally used by backends */
gboolean	e_backend_cache_get_populated		(EBackendCache *cache);
void		e_backend_cache_set_populated		(EBackendCache *cache,
							 gboolean is_populated);

gchar *		e_backend_cache_get_sync_data		(EBackendCache *cache);
void		e_backend_cache_set_sync_data		(EBackendCache *cache,
							 const gchar *sync_data);

/* custom keys, used by backends */
gchar *		e_backend_cache_get_key			(EBackendCache *cache,
							 const gchar *key,
							 GError **error);

const gchar *	e_backend_cache_peek_key		(EBackendCache *cache,
							 const gchar *key,
							 GError **error);

gboolean	e_backend_cache_set_key			(EBackendCache *cache,
							 const gchar *key,
							 const gchar *value,
							 GError **error);

/* whether cache changed since opening or last save; this includes also
   persistent properties changes */
gboolean	e_backend_cache_is_dirty		(EBackendCache *cache);

/* locks/unlocks saving of changes to file - when doing mass changes, then avoids
   often disk writes; the changes are saved either on last unlock or immediately
   after each change, if updates are not locked. Each call of lock should have
   called its corresponding unlock. */
gboolean	e_backend_cache_lock_updates		(EBackendCache *cache,
							 GError **error);
gboolean	e_backend_cache_unlock_updates		(EBackendCache *ebsdb,
							 gboolean do_commit,
							 GError **error);

/* put stores object into the cache under given UID; if such objects exists, then it's replaced */
gboolean	e_backend_cache_put			(EBackendCache *cache,
							 const gchar *uid,
							 GObject *object,
							 GError **error);

/* removes object with given 'uid' from cache; returns false if not found */
gboolean	e_backend_cache_remove			(EBackendCache *cache,
							 const gchar *uid,
							 GError **error);

/* removes all objects from cache, making in effectively empty */
gboolean	e_backend_cache_clear			(EBackendCache *cache,
							 GCancellable *cancellable,
							 GError *error);

/* returns total count of currently stored objects in the cache */
guint		e_backend_cache_count_objects		(EBackendCache *cache,
							 GError **error);

gboolean	e_backend_cache_has_object		(EBackendCache *cache,
							 const gchar *uid,
							 gboolean **found,
							 GCancellable *cancellable,
							 GError **error);

/* free returned object with g_object_unref() */
gboolean	e_backend_cache_get_object		(EBackendCache *cache,
							 const gchar *uid,
							 GObject **object,
							 GCancellable *cancellable,
							 GError **error);

/* free returned string with g_free() */
gboolean	e_backend_cache_get_object_string	(EBackendCache *cache,
							 const gchar *uid,
							 gchar **str_object,
							 GCancellable *cancellable,
							 GError **error);

/* Transforms fields to actual object. If the object is already loaded, then
   that one is returned, instead of repeated parsing. Free returned pointer
   with g_object_unref() */
gboolean	e_backend_cache_get_object_from_fields	(EBackendCache *cache,
							 const gchar *uid,
							 GHashTable *fields,
							 GObject **object,
							 GCancellable *cancellable,
							 GError **error);

/* gets all stored objects; the returned 'objects' contains GObject-s and
   should be freed with g_slist_free_full (objects, g_object_unref); */
gboolean	e_backend_cache_get_objects		(EBackendCache *cache,
							 GSList **objects,
							 GCancellable *cancellable,
							 GError **error);

/* gets all stored uids; the returned 'uids' contains gchar *-s and
   should be freed with g_slist_free_full (objects, g_free); */
gboolean	e_backend_cache_get_uids		(EBackendCache *cache,
							 GSList **uids,
							 GCancellable *cancellable,
							 GError **error);

/* gets all objects which had been changed in offline mode;
   note the obejcts are marked as changed in offline till
   its state is changed either by e_backend_cache_update_offline_state()
   or by e_backend_cache_clear_offline_changes().
   Returned "changes" hash table consists of uid-to-EBackendCacheOfflineState mapping;
   if there was no change done on the object then NULL is returned. Returned hash
   table should be freed by g_hash_table_destroy. */
gboolean	e_backend_cache_get_offline_changes	(EBackendCache *cache,
							 GHashTable **changes,
							 GCancellable *cancellable,
							 GError **error);

/* clears locally stored data with offline changes and marks all left
   objects as being synchronized with the server */
void		e_backend_cache_clear_offline_changes	(EBackendCache *cache,
							 GCancellable *cancellable,
							 GError **error);

/* sets offline state for particular object. This is usually used only for unsetting
   particular objects from offline changes, the rest does the cache transparently. */
gboolean	e_backend_cache_set_offline_state	(EBackendCache *cache,
							 const gchar *uid,
							 EBackendCacheOfflineState state,
							 GError **error);

/* returns currently assigned offline state on a given object */
EBackendCacheOfflineState
		e_backend_cache_get_offline_state	(EBackendCache *cache,
							 const gchar *uid,
							 GError **error);

/* searches cache based on the given 'sexp_str' and returns list
   of uids, which satisfies 'sexp_str' query. Returned list should
   be freed with g_slist_free_full (uids, g_free). */
gboolean	e_backend_cache_search_uids		(EBackendCache *cache,
							 const gchar *sexp_str,
							 GSList **uids,
							 GCancellable *cancellable,
							 GError **error);

/* searches cache based on the given 'sexp_str' and returns list
   of objects, which satisfies 'sexp_str' query. Returned list should
   be freed with g_slist_free_full (objects, g_object_unref). */
gboolean	e_backend_cache_search			(EBackendCache *cache,
							 const gchar *sexp_str,
							 GSList **objects,
							 GCancellable *cancellable,
							 GError **error);

/* searches cache based on the given 'sexp_str' and calls 'func'
   for each object which satisfies 'sexp_str' query. 'user_data' are
   passed into 'func'. Use e_backend_cache_get_object_from_fields()
   when the actual object is needed. */
gboolean	e_backend_cache_search_custom		(EBackendCache *cache,
							 const gchar *sexp_str,
							 EBackendCacheSearchCustomFunc func,
							 gpointer user_data,
							 GCancellable *cancellable,
							 GError **error);

/* adds a new attachment for object 'object_id'; fails if an attachment
   with given path already exists. Caller can use e_backend_cache_build_attachment_path()
   to build */
gboolean	e_backend_cache_add_attachment		(EBackendCache *cache,
							 const gchar *object_uid,
							 const gchar *path,
							 GError **error);

/* updates attachment of the given path for the given object;
   fails if such attachment or object doesn't exist. This has
   effect of only marking assigned object as locally modified,
   if currently in offline mode*/
gboolean	e_backend_cache_update_attachment	(EBackendCache *cache,
							 const gchar *object_uid,
							 const gchar *path,
							 GError **error);

/* removes attachment from list of attachments for given object,
   same as files from the cache, if they are stored under cache's directory;
   fails if such attachment or object doesn't exist */
gboolean	e_backend_cache_remove_attachment	(EBackendCache *cache,
							 const gchar *object_uid,
							 const gchar *path,
							 GError **error);

/* Same as e_backend_cache_remove_attachment(), only removes
   all attachments for given object in once call;
   does nothing if none exists */
gboolean	e_backend_cache_remove_all_attachments	(EBackendCache *cache,
							 const gchar *object_uid,
							 GError **error);

/* gets list of attachment paths for given object; 'paths' should
   be freed with g_slist_free_full (paths, g_free). */
gboolean	e_backend_cache_get_attachments		(EBackendCache *cache,
							 const gchar *object_uid,
							 GSList **paths,
							 GCancellable *cancellable,
							 GError **error);

/* erases all locally stored data from disk. The only left
   operation after this call is to free the 'cache' object,
   because anything else will result in an error. */
gboolean	e_backend_cache_erase			(EBackendCache *cache,
							 GError *error);

/* Adds or rewrites current values in custom table 'table_name'
   associated with 'object_uid' and identified by 'rowid'.
   The 'rowid' is completely managed by the caller and together with
   'object_uid' identifies row in the custom_table.
   The 'fields' is a map from field name to field value stored as string.
   Note the 'object_uid' can be empty string or an unexistent object. */
gboolean	e_backend_cache_custom_table_put	(EBackendCache *cache,
							 const gchar *table_name,
							 const gchar *object_uid,
							 const gchar *rowid,
							 GHashTable *fields,
							 GCancellable *cancellable,
							 GError **error);

/* Removes row from a given custom table identified by object_uid and rowid. */
gboolean	e_backend_cache_custom_table_remove	(EBackendCache *cache,
							 const gchar *table_name,
							 const gchar *object_uid,
							 const gchar *rowid,
							 GCancellable *cancellable,
							 GError **error);

/* Removes all rows associated with given object_uid. */
gboolean	e_backend_cache_custom_table_remove_all	(EBackendCache *cache,
							 const gchar *table_name,
							 const gchar *object_uid,
							 GCancellable *cancellable,
							 GError **error);

/* Reads fields from custom table 'table_name' identified by object_id and rowid.
   The 'fields' is map from field names to field's value stored as string, which
   shopuld be freed with g_hash_table_destroy(). */
gboolean	e_backend_cache_custom_table_get	(EBackendCache *cache,
							 const gchar *table_name,
							 const gchar *object_uid,
							 const gchar *rowid,
							 GHashTable *fields,
							 GCancellable *cancellable,
							 GError **error);

/* checks whether give custom table 'table_name' contains any row associated to
   given object_id. Note the function returns TRUE even when not found. */
gboolean	e_backend_cache_custom_table_has_uid	(EBackendCache *cache,
							 const gchar *table_name,
							 const gchar *object_uid,
							 gboolean *found,
							 GCancellable *cancellable,
							 GError **error);

/* checks whether give custom table 'table_name' contains given 'rowid' associated
   to given object_id. Note the function returns TRUE even when not found. */
gboolean	e_backend_cache_custom_table_has_rowid	(EBackendCache *cache,
							 const gchar *table_name,
							 const gchar *object_uid,
							 const gchar *rowid,
							 gboolean *found,
							 GCancellable *cancellable,
							 GError **error);

/* gets list of all object_uids (as string) stored in given custom table.
   Returned pointer should be freed with g_slist_free_full (object_uids, g_free). */
gboolean	e_backend_cache_custom_table_get_uids	(EBackendCache *cache,
							 const gchar *table_name,
							 GSList **object_uids,
							 GCancellable *cancellable,
							 GError **error);

/* gets list of all rowids (as string) stored in given custom table.
   Returned pointer should be freed with g_slist_free_full (rowids, g_free). */
gboolean	e_backend_cache_custom_table_get_rowids	(EBackendCache *cache,
							 const gchar *table_name,
							 const gchar *object_uid,
							 GSList **rowids,
							 GCancellable *cancellable,
							 GError **error);

/* calls 'func' for each row associated to 'object_uid' in the given
   custom table 'table_name'. */
gboolean	e_backend_cache_custom_table_foreach	(EBackendCache *cache,
							 const gchar *table_name,
							 const gchar *object_uid,
							 ECalBackendCacheCustomTableSearchFunc func,
							 gpointer user_data,
							 GCancellable *cancellable,
							 GError **error);

/***************************************************************************************/

Then there will be two descendants, one for book and one for cal backends,
unfortunately named EBookBackendOfflineCache and ECalBackendOfflineCache.
I would prefer to not using the 'Offline' word in their names, but both names
are occupied currently, thus no way. Using EBackendOfflineCache seems
to me as good for consistency, but awful for the readability. If you can
come with better names for descendants then I'm all for it.
	
The EBookBackendOfflineCache API will be pretty much similar to
EBookBackendSqliteDB, from where most of the SQLite code will be taken.
Note the current EBookBackendSqliteDB behaves differently in some aspects,
but I do not see an issue with that. The book descendant will not
use attachments for now, because they are not supported by eds.

On the other hand the ECalBackendOfflineCache supports attachments,
and will also use custom tables to store timezones in the cache.
These will use empty 'object_uid' for it, and 'rowid' will be TZID.
Also the UID in calendar consists of its uid and rid, which the descendant
itself will hide to the caller.