Proposal for bookmarks/history database

[Peter Harvey <peter a harvey gmail com>]

Hi all,

I think people are generally getting tired of EphyNode and some of it's
issues. In many ways the basic *idea* is nice, but it's just become too
complicated. A proposal to replace the bookmarks/history databases
(which are EphyNode collections) with SQLite was floated. I'm going to
take a shot at defining the functions of an SQLite-backed database for
history/bookmarks.

First, the concept of topics and other 'set'-like things is well
established. We like to have our bookmarks arranged in related sets. Our
history items are arranged by site. etc. EphyNode captured this, and I'd
like any replacement to do the same.

Second, memory usage can be a pain. I believe we currently keep in a
copy of all the information for each bookmark, and *then* make a copy
for each widget which uses that information. This is a GTK+ thing, and
unavoidable.

Database layout (warning: I am not a database person).
------------------------------------------------------

Each 'node' of information is identified by a unique integer. We store
the statement "node X belongs to a set Y" as "node X is a child of node
Y".

table 'set'
  int parent      # index on this field
  int child

table 'pages'
  int id          # primary key
  string url      # url
  string title    # title from last visit
  string icon     # icon from last visit
  string btitle   # NULL if not bookmarked otherwise it's bookmark title
  string bicon    # NULL if should default to icon
  date visit      # date of visit

table 'sites'
  int id          # primary key
  string name     # things like 'slashdot.org' or 'abc.com'

table 'topics'
  int id          # primary key
  string title    # title of the topic





Basic query mechanisms
----------------------

We have a number of predefined sets, given numbers below 128. We use
callbacks rather than return sets when executing queries. This allows us
to, for example, put the history information directly into a
GtkListStore or similar. Basically just query-and-store.

Various parameters all have the same meaning
  data  - a bitmask of the fields from the respective table
  group - a bitmask of the fields to group by
          useful for grouping by url to avoid multiple visits
  user  - user data provided to the callback

------

// Standard predefined 'sets'
enum {
 EPHY_DB_PAGES,
 EPHY_DB_BOOKMARKS,
 EPHY_DB_TOPICS,
 ...
};

// Data fields for each node
enum {
 EPHY_DB_PAGE_URL,
 EPHY_DB_PAGE_TITLE,
 EPHY_DB_PAGE_ICON,
 EPHY_DB_PAGE_BTITLE,
 EPHY_DB_PAGE_BICON,
 EPHY_DB_PAGE_VISIT,
 EPHY_DB_SITE_NAME,
 EPHY_DB_TOPIC_TITLE
); 

typedef void (PagesCallback*)(gint id, const char *url, const char *title,
                              const char *icon, const char *btitle,
                              const char *bicon, long visit, void *user);

typedef void (TopicsCallback*)(gint id, const char *title, void *user);

typedef void (SitesCallback*)(gint id, const char *title, void *user);

ephy_db_query_pages  (gint parent, gint data, gint group,
                      PagesCallback pg, void *user_data);

ephy_db_query_topics (gint parent, gint data, gint group,
                      TopicsCallback pg, void *user_data);

ephy_db_query_sites  (gint parent, gint data, gint group,
                      SitesCallback pg, void *user_data);

ephy_db_query_page   (gint child, gint data,
                      PagesCallback pg, void *user_data);

ephy_db_query_topic  (gint child, gint data,
                      TopicsCallback pg, void *user_data);

ephy_db_query_site   (gint child, gint data,
                      SitesCallback pg, void *user_data);

------

An example query for ephy_db_query_sites is:
  SELECT <fields from data bitmask> FROM
     (SELECT DISTINCT child FROM set WHERE parent=<parent>) set,
     pages
  WHERE pages.id = set.child
  GROUP BY <fields from group bitmask>;




Basic data mechanisms
---------------------

A simple caching scheme will allow multiple ephy_db_query_page calls
(made in quick session to the same node) to be performed very fast. A
simple getting, setting and signaling scheme will also make life easy.

-----

ephy_db_set (gint child, ...);

ephy_db_get (gint child, ...);

ephy_db_delete (gint child);

gint ephy_db_new_child ();

ephy_db_add_child (gint parent, gint child);

ephy_db_remove_child (gint parent, gint child);

ephy_db_connect_child_* (gint child, some callback, user data);

ephy_db_connect_parent_* (gint child, some callback, user data);

-----

Now, the callbacks are called whenever a node or the child of a parent
node changes (respectively). The callback is passed a bitmask of the
data fields that changed. 

To know what parents to signal we will need to execute a query on the
database. This is slow, but very rare, and will be very fast if we just
cache information for the most recently touched/created node.

We may need some specialist functions, but they would become trivial to
implement. In particular a "this child was added to a parent" would be a
nice signal and data combo.






OK, that's it. Let me know what you think.

Regards,
Peter.