Re: [Tracker] more issues with indexer-split

[Martyn Russell <martyn imendio com>]

Jamie McCracken wrote:
> > > trunk only checks directories (If a file in a directory is modified then
> > > the directories mtime is also altered so no need to check every file)
> > > hence startup is much faster.

Note: the mtime of the parent directory ONLY is updated. This is not
recursive. So if you have /foo/bar/baz/sliff.txt, the mtime of baz/ is
updated not for bar/ and foo/.

This means you _HAVE_ to go into every directory to see if it has a
subdirectory with an mtime that has updated.

> > We can do this. Can you guarantee that on EVERY file system type the
> > parent directory mtime is updated when a file changes? I am not 100%
> > sure this is the case.
>
> on all major platforms yes (*nix and windows)

Hmm. This wories me. How mtime is used across file systems tends to vary
slightly and this might come back to bite us.

> it is for me - its in the order of 3x slower than trunk at startup 

What exactly is 3x slower? The crawling?

I have been thinking about this. The best solution here to me is to send
ALL files/directories to the indexer and let the indexer check the mtime
of a directories before deciding to process the files it holds. This
should dramatically reduce the DB lookups on startup. But if the
slowness is NOT in the indexer, then there is little you can do except
increase the throttle. Have you tested it again recently since I made
throttle mandatory whenever it is called (i.e. it is 5+config value).
This made a lot of difference for me.

-- 
Regards,
Martyn