Re: [Tracker] Indexing removable devices on demand

[Cosimo Cecchi <cosimoc gnome org>]

Hey Martyn and Felipe,

Martyn Russel wrote:
> > Besides, my mentor has pointed me out that we need to ensure that:
>
> Who is your mentor by-the-way? Anyone I know? :)

I am mentoring Felipe, so yeah :)

> >   * the mount stops being tracked as soon as applications are not
> >     interested in results from it anymore
>
> This sounds like updating tracker:available for resources of that mount 
> point in the DB or did you mean that we don't have inotify event 
> monitoring on that location? I presume the former.
>
> We have a mechanism already to remove old mount point data, usually in 
> terms of days not on demand. We would have to disable that for these 
> mount points.

I am leaning towards the latter; basically I don't want Tracker
hammering the CPU trying to index files on a removable device as long as
no application is explicitly interested in those results - more on this
below.

> >   * that the mount doesn't get automatically indexed the next times
> it's
> >     plugged
> >   * results from that mount should be kept in the database cache
> >     indefinitely, even after the mount goes away so that we wouldn't
> >     need to start fresh the next time
> >
> > Do you suggest any new/better approach?
>
> No, it seems sane enough.
>
> We usually don't keep results in the database indefinitely for
> removable 
> devices because it can just clutter the DB with not benefit (consider 
> your friend brings their USB device round and you never see it on your 
> machine again, that data is then stored and never used again).

I think the idea is to use Tracker as a cache layer for data coming from
those devices, and letting the application decide completely on the
device availability; since Tracker can store e.g. the device UUID and
recognize it when it's later plugged in again, when mining is first
requested for a device, it's created in the store, and data starts
getting processed.
When the application is not interested in data from that device, mining
is interrupted; next time mining for the same device is requested, data
processing resumes from where it left, and old entries are
modified/removed from what's in the store already (having a policy that
cleans up devices that haven't been used for more than X days might be a
good optimization, if needed).
This also might require the application to track devices it's interested
in itself - since the mechanism is generic an application could
potentially even request a network mount to be temporarily scanned for
files, and it seems a little weird to ask Tracker a list of devices GIO
already knows better about.

Of course, for this to work properly, the application needs to be smart
in its queries and not just blindly display e.g. every nfo:Document in
the database. E.g. in the designs we have for Documents, items from
removable devices are not listed in the main overview, but they're
hidden behind an item representing the device itself [1].

Thanks,
Cosimo

[1]
https://github.com/gnome-design-team/gnome-mockups/raw/master/documents/documents-notifications.png