[Tracker] Reviving the libstreamanalyzer based extractor module

[Philip Van Hoof <philip codeminded be>]

Hi team,

During a Tracker/Nepomuk/SPARQL training I gave at one of my customers I 
noted the interest in extractors that can dive into archives and 
document types that have a tree of other documents (like MIME documents).

Right now a Tracker extractor module can't extract an MP3 that is 
attached in an E-mail located in a Maildir or stored in a tar.gz file. 
My recommendation for E-mail client authors is and will probably always 
be to store MIME parts, that have disposition set to attachment or not, 
Base64 decoded on the filesystem. So that means that the cache of an 
E-mail client stores an MP3 attachment of an E-mail ... as an MP3. And 
not as a blob of Base64 encoded quote plain text unquote (which is 
stupid and not useful). That way writing a miner for such an E-mail 
client would mean to configure the FS miner to just index the cache 
directory (and perhaps tweak the nie:url value and add nmo rdf:type 
qualifiers).

That or libtracker-extract should allow a stream or buffer based 
extraction, and/or a file descriptor based one (in which case we could 
pass the extractor modules, the ones now only used by tracker-extract, a 
by pipe created FD from the E-mail client, and write the Base64 decoded 
data to the pipe FD - or something). Unfortunately is tracker-extract 
right now entirely FILE based (not really FD based, nor stream based).

Also do some use-cases of Tracker's FS miner want files in archives to 
be indexed.

Tracker's native extractors can't do any of this. That's because they 
are open/seek/read/close based and not stream based. The 
libstreamanalyzer library aims to implement extraction of metadata in a 
stream based way, with support for diving into archives and MIME documents.

It was for that reason that I once wrote tracker-topanalyzer.cpp in the 
src/tracker-extract directory. It's unmaintained nowadays.

I think it would be a great first addition if the tracker-extract .rule 
file based environment would be adapted to have two levels of matching: 
first on container and then on MimeType. The first level would for all 
of its native extractors be "Just File", and for the libstreamanalyzer's 
be "MIMEDocument" and "Archive". The second level would be the same as 
now. Ideally this level system could also be used for multimedia files 
(videos have first a MIME type and then a codec type, for example).

Then would it start being possible for a extractor module like 
tracker-topanalyzer.cpp to get kicked into action for diving into 
archive files and MIME documents (and the native ones would still 
operate on native file types).

Also should the tracker-topanalyzer.cpp be fixed. It has been a long 
time that it was last tested and I don't expect it to still work. And 
for it to work it would probably be needed that libstreamanalyzer gets 
adapted to follow Tracker's Nepomuk adaptations (right now 
libstreamanalyzer doesn't know about the nmm ontology, afaik).


Kind regards,

Philip