Re: [Tracker] Reviving the libstreamanalyzer based extractor module

[Philip Van Hoof <philip codeminded be>]

Op 14/06/2013 19:36, Ivan Frade schreef:

Hi Ivan,

During a Tracker/Nepomuk/SPARQL training I gave at one of my customers I noted the interest in extractors that can dive into archives and document types that have a tree of other documents (like MIME documents).

 Just today another message in this mailing list was mentioning it :)

Yep. I noticed it.

That or libtracker-extract should allow a stream or buffer based extraction, and/or a file descriptor based one (in which case we could pass the extractor modules, the ones now only used by tracker-extract, a by pipe created FD from the E-mail client, and write the Base64 decoded data to the pipe FD - or something). Unfortunately is tracker-extract right now entirely FILE based (not really FD based, nor stream based).

 FD passing and buffered extraction are both good ideas. They are also independent. We could implement any of them without the other.

A problem that I see with the FD passing using pipe is that we can't know for sure whether a library that we depend on for metadata extraction wont use seek() and assume they got a real file's fd. I'm even afraid that most do. The others are probably buffer or mmap based. Meaning that libstreamanalyzer's way of just rewriting all extractors to be stream based is probably the only way to end up with a consistent and sensible solution for in-archive metadata extraction.

I think it would be a great first addition if the tracker-extract .rule file based environment would be adapted to have two levels of matching: first on container and then on MimeType. The first level would for all of its native extractors be "Just File", and for the libstreamanalyzer's be "MIMEDocument" and "Archive". The second level would be the same as now. Ideally this level system could also be used for multimedia files (videos have first a MIME type and then a codec type, for example).

Is this two level matching really needed? at the end we recognize the containers with mime-types (e.g. application/x-tgz). With the current .rules files, we can assign those "container mime-types" to the topanalyzer.

You're right if there is only one such kind of extractor. As soon as you want to select libstreamanalyzer for one kind of archive-mime-type combination and another extractor for another archive-mime-type combination, this wont work. But I agree that we could have a tracker-extract-container.c and .rule for application/x-tgz among other container types that then splits it out to tracker-extract-container-streamanalyzer.cpp and tracker-extract-container-somethingelse.c based on logic defined not in the top .rule system but on what tracker-extract-container.c itself does. And at first this can simply be to throw them all to a streamanalyzer.cpp one (which will likely look a lot like what tracker-topanalyzer.cpp is now).

Note that there's no reason to keep tracker-topanalyzer.cpp's filename. With the .rule based system the filename topanalyzer.cpp makes no sense anymore.

Then would it start being possible for a extractor module like tracker-topanalyzer.cpp to get kicked into action for diving into archive files and MIME documents (and the native ones would still operate on native file types).

Also should the tracker-topanalyzer.cpp be fixed. It has been a long time that it was last tested and I don't expect it to still work. And for it to work it would probably be needed that libstreamanalyzer gets adapted to follow Tracker's Nepomuk adaptations (right now libstreamanalyzer doesn't know about the nmm ontology, afaik).

I wonder if Jos is still working on it. We could bring back to life that topanalyzer extractor, use it for compressed files and move on from there.

If Jos isn't working on it anymore, surely we can look into it ourselves. I doubt that Jos would reject patches that conditionally make libstreamanalyzer spit out a better ontology than the broken upstream Nepomuk ontologies for multimedia. A bit of stream and decorator in C++ will do good to us.

Kind regards,

Philip