[Tracker] Response to a question of a company about SPARQL watching

[Philip Van Hoof <philip codeminded be>]

Hi team,

I received a series of questions from a company in automotive industry 
about implementing full SPARQL watching in Tracker's store. They told me 
they have something that does periodic requering of queries executed by 
the SPARQL clients.

We of course discussed this during our Nokia sprints a few years ago. 
Because I think the community should also know about our conclusions and 
ideas of back then I'll copy paste my response to the people who sent me 
the questions.

https://live.gnome.org/Tracker/Documentation/SignalsOnChanges

At some point we also wanted to implement full SPARQL query watching or 
triggering or SPARQL views, but due to this being too complex to monitor 
the complete graph of a Nepomuk tree we decided to go for class signals 
instead and let the clients do the remainder themselves (by requering or 
having a more clever update mechanism).

Note the complexity that every change to a relation or literal property 
(with relation I mean a property that is a reference to another object) 
in the root of the Nepomuk tree of your data, can possibly be a trigger 
for any such SPARQL view (or SPARQL watching, however you want to call 
it). This is far more complicated than traditional triggers in most 
database systems, as those triggers usually operate on a single table. 
The complexity is more comparable to query views in products like SQL 
Server or Oracle, on queries that are very complicated (subqueries, 
joins and rather hard expressions on the literals like regular 
expressions in some cases on a decomposed denormalized schema -> not fun 
to implement query views for that). Of course can we make certain 
assumptions about RDF while inserting of statements happens, in relation 
to SPARQL queries. It's probably not impossible because of those 
assumptions that can be made. HOWEVER :-)

The idea behind class signals is that we'll signal the clients that they 
might have to check the results of the query they are showing to the 
user. They can then rerun their own SPARQL query themselves to achieve 
full SPARQL watching (themselves). Such signalling systems aren't 
uncommon in MVVM designs where the view-model will translate the 
incomplete event that came from the model to complete updates on its 
view. (view would in qt be QML, view-model would be a QAbstractItemModel 
(and QSparql has one which you could use to decorate), and the model 
would be for example a QList of the results you get from QSparql or the 
QAbstractItemModel it supports itself).

A reason for that is that we didn't want to make tracker-store stateful 
with relation to the clients.

We didn't want to make it stateful with relation to clients because one 
idea is to one day have a libtracker-sparql that, just like libsqlite 
does for SQL and DB schemas, provides a way for any application to 
provide it with an ontology and then (just) start using SPARQL INSERT 
and SPARQL on that ontology. In this scenario the GNOME desktop (or the 
KDE desktop) would provide the Nepomuk ontology and the GNOME filesystem 
miner would use libtracker-sparql to fill it up. That there is a service 
like tracker-store, activated by D-Bus, is in that design scenario just 
an implementation detail of libtracker-sparql. That then there exists 
something like a semantic desktop is not really because of 
libtracker-sparql but because of the desktop having chosen the same 
Nepomuk based ontology for all of its libtracker-sparql clients 
(including the miners).

In other words is libtracker-sparql just a library that helps you build 
a semantic desktop, instead of "Tracker" providing you with "the" 
semantic desktop.

That means that this tracker-store (in libexec) would be provided by 
libtracker-sparql as a D-Bus activated 'tool' or 'implementation detail' 
that shuts itself down after a period of no-write activity. You can see 
how this can't work well together with full SPARQL watching: unless we 
have a very quick way to know in a stateless (with relation to the 
libtracker-sparql clients) way what results it has for what queries (of 
all the clients) and whether the just arrived SPARQL INSERT influences 
them at startup of the 'tool' / 'implementation detail'. Obviously can't 
this tool's startup time be long, as this would influence badly the 
SPARQL INSERT performance for the libtracker-sparql client doing the 
write query.

Because of that is the class signal signal stateless. And because of 
that we prefer all query state to remain in the libtracker-sparql 
clients themselves.

Meaning that they are responsible for reexecuting the query, and not the 
store. The store will, however, provide hints. Hints based on classes. 
Hence the name class signals.

That DOESN'T mean that the reexecuting can't be done transparently by 
libtracker-sparql (so developers who use of libtracker-sparql must not 
necessarily be aware when using such a higher level feature). Just that 
the executing-of the query, in a full SPARQL watching design, happens in 
the process of the libtracker-sparql client and not in the tracker-store 
process (for aforementioned state reasons).

In fact is QSparql's QAbstractItemModel developed that way already. 
There's just no similar support for something like that in 
libtracker-sparql. Mostly because we didn't want libtracker-sparql to 
depend on Gtk+ for it to provide a GtkTreeModel that does the same thing 
as QSparql's QAbstractItemModel. We could, however, develop a 
libtracker-sparql-gtk+ that builds on top of libtracker-sparql and 
provides a GtkTreeModel that can be used as a view-model in a MVVM 
architecture.

Such a GtkTreeModel and the QSparql QAbstractItemModel are in a MVVM 
design the view-models. If you prefer .NET Prism4 terminology the models 
that I have in mind are in C# called ObservableCollection<T>.  It 
implements INotifyCollectionChanged and INotifyPropertyChanged which 
makes it bindable to a XAML WPF view. XAML WPF views are comparable to 
QML in Qt (and in a oldfashioned way a Glade file in Gtk+).

Note to people on the public mailing list that I gave the .NET's Prism4 
for WPF examples as design principle examples and not because I am in 
any way interested in religious hate opinions on Mono and .NET or 
(seriously) whatever. I also wrote about Qt's QML and QAbstractItemModel 
and about Gtk+.

Kind regards,

Philip