Re: on crawling upnp contentdirectories...

[Jussi Kukkonen <jku linux intel com>]

On 01/27/2011 11:10 AM, Juan A. Suárez Romero wrote:
> On Tue, 2011-01-25 at 13:38 +0200, Jussi Kukkonen wrote:
>>> did you consider disk-caching using a lightway database (something like
>>> sqlite)? I think that would address at least the 2 last bullets.
>>
>> Briefly, yes. It's just that it feels wrong to implement something that
>> other components seem to have been designed to do... I could be wrong
>> about this of course:,I'm just looking at tracker more closely to see if
>> it is a better fit.
> 
> I agree here that caching could be an optional services provided by
> Grilo which plugins could use or not, similar to the one we have about
> GrlNet, which plugins can use or not.
> 

Ok, cool. Just so there's no misunderstanding: You guys are of course
free to take all of the code on the linked gitorious project and go with
that even if I'm still looking at other options -- it is LGPL. It does
the caching and should handle concurrent query and crawl. The searches
perform fairly well, but the memory use can be fairly high, it always
crawls on startup and the searches are currently done on "title" only.
There are still some problems with duplicate data -- and those might
never be fully resolvable -- example duplication can be seen if you
crawl rygel that uses tracker as backend.

I'll keep you posted on how the situation looks from here...

>>>> So, I'm currently looking at implementing this feature in a better place
>>>> in the infrastructure. Tracker seems like an obvious choice, even though
>>>> that probably means quite a bit more work than working with your nice
>>>> little upnp plugin.
>>>
>>> Mmm... So your current approach is to have tracker using grilo to get
>>> the crawl the upnp servers and store it in its local database?
>>> And then, when selecting a upnp server for a search operation, use the
>>> tracker plugin instead to retrieve the content?
>>
>> Well, I'm a relative newbie to tracker so I can't say exactly how that
>> would work yet. My first impression is that there's no reason to use
>> grilo in tracker. There should probably be a tracker upnp miner that
>> works similarly to how the fs-miner handles removable media: cache
>> things for a period of days and only show search results when the source
>> is actually available (your plugin could be useful as a starting point
>> for that though).
>>
> 
> I'm not very sure about this approach. The point is that using Tracker
> to store that kind of information is beyond the Tracker purposal. In
> fact, I think that this discussion happened several times in Tracker
> lists, and they were very clear about the purpose of Tracker: it is not
> a database where everything fits.

I understand the scope-creep danger tracker has but I don't personally
see the problem in this particular case: UPnP media servers on local
network are as much personal data storage as removable media is --
sometimes they may be someone elses data but the ones you most often see
are yours. Still, it's definitely a possibility that I'm wrong about
that. I will have that discussion when I'm familiar enough with tracker
internals to actually participate...

Jussi