Re: [Tracker] Indexing removable devices on demand

[Martyn Russell <martyn lanedo com>]

On 05/18/2012 05:30 PM, Cosimo Cecchi wrote:
> Hey Martyn and Felipe,
>
> Martyn Russel wrote:
> > > Besides, my mentor has pointed me out that we need to ensure that:
> >
> > Who is your mentor by-the-way? Anyone I know? :)
>
> I am mentoring Felipe, so yeah :)

Ah cool! :)

> I am leaning towards the latter; basically I don't want Tracker
> hammering the CPU trying to index files on a removable device as long as
> no application is explicitly interested in those results - more on this
> below.

This is fine, one consideration though, is that if files do change there 
while the application is not used (which might be unlikely), Tracker 
won't know about it until the next time the application is launched and 
asked to index that removable device again. Depending on the changes 
between the application's use of the data, this could be quick or a long 
wait. For example, you close application X and then format a 4Gb 
partition on the removable device which was indexed. This scenario isn't 
much different to what could happen now though (i.e. you plug the device 
into another machine and heavily change the data on it, then bring it 
back to your machine with Tracker running).

> > >    * that the mount doesn't get automatically indexed the next times
> > it's
> > >      plugged
> > >    * results from that mount should be kept in the database cache
> > >      indefinitely, even after the mount goes away so that we wouldn't
> > >      need to start fresh the next time
> > >
> > > Do you suggest any new/better approach?
> >
> > No, it seems sane enough.
> >
> > We usually don't keep results in the database indefinitely for
> > removable
> > devices because it can just clutter the DB with not benefit (consider
> > your friend brings their USB device round and you never see it on your
> > machine again, that data is then stored and never used again).
>
> I think the idea is to use Tracker as a cache layer for data coming from
> those devices, and letting the application decide completely on the
> device availability; since Tracker can store e.g. the device UUID and
> recognize it when it's later plugged in again, when mining is first
> requested for a device, it's created in the store, and data starts
> getting processed.
> When the application is not interested in data from that device, mining
> is interrupted; next time mining for the same device is requested, data
> processing resumes from where it left, and old entries are
> modified/removed from what's in the store already (having a policy that
> cleans up devices that haven't been used for more than X days might be a
> good optimization, if needed).

All of those points should be working in the existing Tracker packages 
being shipped today already (other than letting applications decide of 
course).

> This also might require the application to track devices it's interested
> in itself - since the mechanism is generic an application could
> potentially even request a network mount to be temporarily scanned for
> files, and it seems a little weird to ask Tracker a list of devices GIO
> already knows better about.
>
> Of course, for this to work properly, the application needs to be smart
> in its queries and not just blindly display e.g. every nfo:Document in
> the database. E.g. in the designs we have for Documents, items from
> removable devices are not listed in the main overview, but they're
> hidden behind an item representing the device itself [1].

Indeed.

--
Regards,
Martyn

Founder and CEO of Lanedo GmbH.