Re: Proposal for bookmarks/history database

[Peter Harvey <pah06 uow edu au>]

> Ah, yes, we have completely different mentalities! I was thinking of
> turning Epiphany into a web browser which uses SQL all over the place.
> 
> So, if I understand correctly, you're using SQLite to benefit from its
> efficiency as a persistent hashtable with a fixed upper-bound memory
> footprint.

Yes. It also gives us an opportunity to scrap EphyNode and use something
else better suited to large databases.

I need to learn to explain my thinking better. :)


> > d. An extension would just have to register a new 'set of fields' which
> >    we would then turn into a table. It may also register a 'global' node
> >    if it so desires so that sets of nodes it creates can be easily 
> >    found. This isn't required though. A 'bookmark thumbnail browser'
> >    extension might just add a 'thumbnail' fields and then set those 
> >    fields for nodes which are children of the 'bookmarks' node in the 
> >    globals table.
> 
> I recommend putting all metadata into a single table (as galeon-sqlite
> does), since it's WAY, WAY simpler.

I'm mainly concerned about speed in that system though. Will it really
be fast? This is where my naivety wrt databases comes in. I don't trust
tables that involve millions of strings. :)


> As for more complex stuff: Imagine being me, an SQL-proficient extension
> writer. I see this API and I think, "okay, I can add/remove
> bookmarks/topics, link them to each other, and query history". I want to
> do something more complex: say, make another table and join Epiphany's
> bookmarks list against it.
> 
> I can either:
> 
> (1) learn the EphyDb API, or
> (2) create my own database in
> ~/.gnome2/epiphany/extensions/my-extension.db. Then the smallest amount
> of EphyDb code I'd need to get data from Epiphany that I can use to
> query my own database.
> 
> Then I can use SQL to do anything -- for instance, NATURAL JOINs with
> other tables in my database. Whenever I want to JOIN against something
> in Epiphany, I'll just query Epiphany, and use the IDs it gives me to
> query against my own database.
> 
> What can (1) do for me that (2) can't? Nothing. In some cases (2) will
> even be faster, and it will certainly be easier for me to program my
> extension, since I already understand all the concepts. The only problem
> I have is learning enough of the EphyDb API to circumvent it.
> 
> I just don't see what EphyDb has to offer me, the extension author. The
> more time you spend developing an API for me, the more time I'll spend
> figuring out the workaround.

I think that for *most* extensions, and especially for most of Epiphany,
it'd be nice to avoid having SQLite calls and SQL statements littered
about.

I think the easiest solution to what you've described is to have ephy_db
provide the SQLite handle to extensions that really want to do something
more. Those extensions will have to be cautious that they don't break
the database, and if the table schemas change they'll have to change
too. Extensions which don't need such advanced SQL can work through
ephy_db and be safe that their data is managed appropriately.


> > Most extensions would need only basic queries I believe.
> 
> Depends on the extension author, I suppose. I thought you were using
> SQLite for the "SQL" part, instead of the "ite" :).

Well, see above.


> I'd guess that's because you're not comfortable with SQL; it seems like
> a lot less work to me. But to each his own.

Hehe, no I'm not a fan of SQL. The lack of event mechanisms really
bothers me. I've avoided every database subject at uni. Took logic
instead.


> To be fair, EphyDb *could* make it simple to attach a field or two to
> history or bookmark items, and that could work okay. But when you start
> talking about adding 'globals' and making new tables, I know I'll be
> looking for a workaround when I write my extensions, because I see no
> reason to learn that API.

The API would be something like:

  ephy_db_register_fields (char *setname, char *fields[], int length);
     - 'setname' becomes the table name.
     - 'fields' array become the data fields.
     - we will just add an 'id' column to the table as well.

If we want to use GQuarks (fast!) to identify fields we can use:

  GQuark ephy_db_get_quark (const char *setname, const char *field);

It would just append the two strings separated by a '/', generate the
quark, and return.


> In the long run, I can see that we started off with vastly different
> mentalities. Neither is right or wrong; I don't know how much I can help
> you with your implementation, beyond suggesting that you shouldn't
> really worry about extensions too much.

When it all falls to bits then I'll come begging for help. :)
Particularly performance-wise where I have no idea. I don't know the
basics like 'inner join' and the like. I just think of databases as big
arrays with some hashing, and think of the most efficient steps in which
to do things. I remember now that the wayward DISTINCT in one of my
previous SQL statements was intentional - it was to hint to SQLite in
subsequent operations that there were no duplicates so it could use an
optimised code path if any.

Anyway, we really should open the floor to other devs to find out
whether they would prefer more SQL throughout Epiphany, or something
more abstract.

Thanks,
Peter.