Re: [Tracker] Enhancement proposal: Automatic reindexing when adding new extractors


On mar, 2010-02-09 at 21:31 +0100, Adrien Bustany wrote:
Hi all, 

this proposal is about automatically reindexing a mime type when a new
extractor is added/updated. There's already a "Reindex" call in

I think we should take several use cases into account here.

      * A new tracker-extractor module is installed
      * As mbiebl pointed out, a tracker-extractor module has been added
        some new capability.
      * The library a tracker extractor relies on gains new capabilities
        (i.e. GStreamer, poppler)

IMHO the trickiest one is the 3rd, which either requires integration
from packagers, or some way for extractors to probe the file types
supported. The second would largely depend on whether the library is
able to tells us that, which I don't think happens often, so we might be
just forced to the first option.

For the second usecase, we clearly need some way to version the
extractors, so it is known when to re-extract. The keyfile with version
info approach looks quite sane to me, we should provide some command
line tool to bump the version number for a given extractor.

Philip told me he'd like to keep tracker-extract as stupid as possible,
so the logic here would be implemented in tracker-miner-fs, at init time.
All extractors modules provide a function to know the mime types they can
index, but we want to avoid loading all the modules at start. Therefore,
a solution using desktop files is favoured. The desktop files would be
installed in ${datadir}/tracker/extractors and would have the format

This makes much sense to me, since the extractor could not run at all if
tracker-miner-fs thinks everything is up-to-date. Certainly, having
complete info about the mimetypes the extractors knows about in the
miner would avoid queries about unsupported files to the extractor, as
it happens right now.

Name=Foobar extractor

What I propose is, having the mimetypes info separated from the
versioning info, I think this way we can provide reasonable support for
the 3 usecases mentioned above:

      * If tracker-extractor-foo 0.15 provides better information than
        the previous release 0.14, the version file installed by the
        package bumps its number, tracker-miner-fs notices the version
        bump and does its job.
      * if some gstreamer package is added, package scripts use some CLI
        tracker tool if available to add new mimetypes for the gstreamer
        extract module, tracker-miner-fs notices the changes in
        mimetypes supported and does its job. The version number isn't
        actually changed, since the extract module hasn't changed.
      * A new extractor is installed, tracker-miner-fs notices no prior
        info about it and does it's job, more or less like a version

So, I guess there should be some $(datadir)/tracker/extractors/ with
version info and a $(datadir)/tracker/extractor-mimetypes/ with a
mimetype->extractor mapping. The main caveat I see here is how would the
initial mimetype mapping be done for certain modules (gstreamer yet
again in mind :), this could require yet again packagers help.

Besides, we also need to take into account restarting tracker-extract if
it's alive at the time of the update, and making things persistent so if
tracker-miner-fs shuts down pending file checks wouldn't be lost. This
is going to be tricky :)


at startup, the FS miner loads all the description files and checks if a
new extractor has been added, removed, or changed its mtime. If
so, it calls the reindex method with the appropriate mime type.
To detect a change in a desktop file, a list of each desktop file with
modification time is kept in cache by the FS miner.

Ideas :
- Describe several extractors in one file
  That makes is much more difficult to detect a change, since only one
  in a file listing 10 might have changed when the file modification time
- Adding a version number in the desktop file, to avoid relying only on
the mtime
  of the desktop file.

Please tell me your thoughts

tracker-list mailing list
tracker-list gnome org

[Date Prev][Date Next]   [Thread Prev][Thread Next]   [Thread Index] [Date Index] [Author Index]