Re: [Tracker] tracker-extract not being restarted

[Ralph Böhme <rb netafp com>]

Hi Martyn,

just back from SambaXP and rereading your great reply. Thanks again for taking time!

Am 12.05.2014 um 16:18 schrieb Martyn Russell <martyn lanedo com>:
> > Hm. But then, how made tracker-extract made aware of filesystem
> > event, ie new files? I though tracker-miner-fs sets up the watches
>
> So:
>
> 1. tracker-miner-fs uses inotify (and other backends), it is signalled on a new file.
>
> 2. tracker-miner-fs processes that file adding basic information (file size, etc).
>
> 3. tracker-extract is started on login and listens for GraphUpdated, a signal from tracker-store. The 
> tracker-store process is the only one that can insert data. tracker-extract also queries for a list of IDs 
> of resources on start up of the ones that it won't be notified of and which have no nie:data-source set. 
> These are the files it knows it needs to process.
>
> 4. tracker-extract then runs with that list of files and also queues any new files from the GraphUpdated 
> signal.
>
> > and notifies other processes (tracker-extract) via DBUS IPC ?
>
> We used to use DBus and that would instantiate processes for us, but now we have this "passive" approach. 
> It has advantages and disadvantages. The advantage is, tracker-miner-fs is simpler and doesn't have to deal 
> with the mess of broken files crashing tracker-extract and timeouts, etc. Also it means the file basic 
> information is added to Tracker quicker than it was previously because we have no delay on tracker-extract. 
> Another advantage half realised (bits can be improved still), is that we can group process files of the 
> same time and prioritise files of particular types. We have mime type priorities currently I think. 
> Disadvantages include having to wait for the second pass indexing to complete :)
>
> tracker-extract has _ALWAYS_ crashed because of the nature of the files and libraries we're using being the 
> weakest point.

Yeah, I know, been there. :)

> With this in mind, it's quite bad that we don't have some sort of watchdog or way to keep it up and 
> running. We could easily do this in tracker-miner-fs. Part of me thinks we should actually have a 
> tracker-control --daemon which watches all process and keeps some sort of journal / log going and also 
> maintains processes like tracker-extract keep running. I should add, this is something that nearly EVERY 
> embedded solution does itself with some script which keeps miner-fs running (not even tracker-extract), 
> because there is no other way to guarantee Tracker is there to catch all events.

Ok, thanks. So taking up on this approach I wrote two Solaris SMF service manifests (resembles systemd) for 
tracker-miner-fs and tracker-extract. This works in that it restarts crashing tracker-extract and then 
tracker-extract continues indexing.

Unfortunately this doesn't work for files that crash it reproducibly as you mention below:

> The only downsides I see are repeated attempts causing a circular problem with tracker-extract.

Definitely. For uses with possibly many files causing crashes this renders Tracker useless.

> But that's easily remedied, we had to do something like this in tracker-miner-fs before already.

So I'm taking you on this here, that you can easily fix this please! :)

If you don't feel an apparent need to get this fixed for your own usecases, please let me know, so I can try 
to bring affected parties and their pockets together next week. :)

Thanks again!
-Ralph