Re: [Tracker] domain-ontologies, continued

[Carlos Garnacho <carlosg gnome org>]

Hey :)

A general update for everyone, I do consider this ready for merging
and I think I'll do early next week unless I see glaring bugs. All
tests pass, tracker does still behave as it ever used to by default,
there's some documentation, and I put the branch to practice in:

https://people.gnome.org/~carlosg/gnome-news.flatpak

If you want to try just do:
flatpak --user install --bundle ./gnome-news.flatpak
flatpak run org.gnome.News

You will see in ps:
$ ps -ef |grep tracker
carlos   26673  1404  0 18:08 ?        00:00:00 bwrap --args 13
/app/libexec/tracker-miner-rss -d org.gnome.News
carlos   26680 26673  0 18:08 ?        00:00:00 bwrap --args 13
/app/libexec/tracker-miner-rss -d org.gnome.News
carlos   26681 26680  0 18:08 ?        00:00:00
/app/libexec/tracker-miner-rss -d org.gnome.News
carlos   26687  1404  0 18:08 ?        00:00:00 bwrap --args 13
/app/libexec/tracker-store -d org.gnome.News
carlos   26694 26687  0 18:08 ?        00:00:00 bwrap --args 13
/app/libexec/tracker-store -d org.gnome.News
carlos   26695 26694  0 18:08 ?        00:00:00
/app/libexec/tracker-store -d org.gnome.News
...

which are the sandboxed tracker processes spawned to handle the
gnome-news data. gnome-news requirements outside the sandbox can be
made to be quite minimal, eg. no r/w access to the user homedir
whatsoever.


On Sun, Jun 11, 2017 at 7:19 PM, Philip Van Hoof <philip codeminded be> wrote:
> On Tue, 2017-06-06 at 20:42 +0200, Carlos Garnacho wrote:
> > Hey hey,
> >
> > During the past week I've been continuing the stuff that Philip
> > started on wip/domain-ontologies. I pushed my current progress on
> > wip/carlosg/domain-ontologies:
> >
> > https://git.gnome.org//browse/tracker/log/?h=wip/carlosg/domain-ontologies
>
> Great! I'll keep an eye on the upcoming changing. Maybe I can soon help
> out here and there.
>
> > So far it's shaping up nicely, private databases are possible there
> > through the public tracker_sparql_connection_local_new(_async) calls,
> > xsd/dc/rdfs/nrl/nao are the are loaded from GResource and are the base
> > ontology, the local connection will run a dedicated thread for
> > updates, in a very similar fashion to tracker-store itself.
>
> Wow, nice progression!

Thanks :), there was enough peer pressure already :p, flatpak has been
a thing already for a couple of years and Tracker was an important
missing piece. People just has been allowing sandboxed apps to talk to
org.freedesktop.Tracker1 which is far too coarse.

> > My topmost
> > items in the todo now are:
>
> > - TrackerDataManager (and many other subsystems) is still a singleton,
> > which doesn't play nicely if you can now create multiple connections
> > that require it differently. I'm halfway into having it be an
> > object/struct so each connection can get its own instance.
>
> Aha, I was at first thinking or planning to simply start multiple
> tracker-store processes. But maybe when TrackerDataManager isn't a
> singleton anymore then one tracker-store could host multiple databases.

Yeah, I've actually gone that way wrt tracker-store. You can get
separate tracker-store processes each managing its own locations. It
is indeed feasible with this work to have a single tracker-store
process exporting several dbus objects, it's just a combination I
don't see many gains with. You have to teach everything else to talk
with multiple objects, and doesn't bring many benefits to multiprocess
approach.

But FWIW I actually finished this item. I don't think it's as
important from the service pov as it is from the API one, mainly for
not having it limited to one single global TrackerSparqlConnection for
everything. Clients or plugins may feasibly create multiple local r/w
connections, although the traditional tracker-store backed one is
still unique.

> And, more importantly, a single client's process could connect to
> multiple different metadata graph stores.
>
> Because it's probably inexcusable to have to expect from a client
> developer to split code up in multiple processes, to connect to multiple
> graph stores.
>
> > - Lots of documentation need to be written around this: how to create
> > new ontologies, data migration concerns, dos and don'ts, ...
>
> Yup. But that's good. Lot's of work for the open source youth ;-)
>
> > - Even if some apps could take advantage of private databases, for
> > some scenarios we do need to make it possible running a standalone set
> > of tracker dbus services for private use. I'm still unclear on how to
> > make it most transparent to apps, probably through libtracker-control
> > API.
>
> nod. Encode a per-application and/or per domain UUID in the DBus path of
> the objects?

Everything fell more in place with the separate processes aproach,
tracker processes take over the org.example.ExampleApp.Tracker1
namespace. Teaching the libtracker-bus side about this was minimal,
and flatpak allows for autostarting (and talking to) services within
the same namespace than the app.

> > There's of course more items for the longer term, but all tests pass
> > with no functional changes, so seems good enough for an update :).
> >
> > And btw, I still think it makes sense to tag tracker-next as 2.0, and
> > use the opportunity to switch to semver, I do hope it plays out and
> > reduces some maintenance burden in maintaining multiple versions.
>
> Wow, great news. Yeah, semver also has this special rule for 0.y.z
> releases, where you can more freely change y and z during a 0.y.z
> 'milestone' release, than after a 1.y.z release. I guess you could do a
> similar thing going from 1.y.z to 2.y.z.
>
> And indeed, the kind of changes involved in domain-ontologies sound like
> worthy of a major increment. Although it could probably be done backward
> compatible (meaning only a minor increment would be needed).

True :), and the libtracker-sparql bits indeed are. Still worth a bump
IMHO, domain ontologies and code split involved.

Cheers,
  Carlos