Re: [Tracker] Video thumbnailing

[Sam Thursfield <ssssam gmail com>]

On Tue, Sep 13, 2011 at 10:43 AM, Ivan Frade <ivan frade gmail com> wrote:
> Hi,
>
> On Mon, Sep 12, 2011 at 9:38 PM, Jens Georg <jensg openismus com> wrote:
> > On Mo, 2011-09-12 at 18:56 +0100, Sam Thursfield wrote:
> > > Hi all,
> > >
> > > It would be nice if Tracker could provide video thumbnails, in the
> > > same way that it currently extracts album art, for media centre apps
> > > etc. to use.
>
>  Take into account that there are two different things here: 1.
> choosing an image for the file (E.G. album-art) and 2. creating a
> thumbnail of an image.
>
>  For audio files the things are very clear: step 1 is responsibility
> of the extractor, step 2 is implemented in tumbler (i.e. in the
> Thumbnail DBus API).
>
>  With audio files, the extractor tries to discover the image (either
> reading the image embedded in the file, a cover.png in the same folder
> or an heuristic about files and directories) and saves it under
> .media-art (following the MediaArt spec) [step 1]. The first time
> anybody needs a thumbnail for that music file, they check if they can
> find its representation under .media-art and then request a thumbnail
> [step 2]. If there is no media-art then they will show the default
> icon.
>
>  Note that is not tracker but the application who triggers step 2. We
> don't like to pre-generate thumbnails because it is a heavy IO
> operation and it harms seriously the performance of mobile devices.
>
>  For video files, if they have an image embedded, everything should be
> the same as with audio. You just need to put a bit of code in the
> gstreamer extractor because it is not implemented AFAIK.
>
>  If a video doesn't have an embedded image, the coherent solution
> would be to extract a frame from the stream and save it under
> .media-art in the extractor... BUT taking that frame is also heavy IO
> operation and slows down *a lot* the extraction. So we moved that
> functionality into a plugin in tumbler. If you request a thumbnail of
> a video, you get that famous frame.

Thanks, I after thinking a while I agree with all you just said.
Finding a sidecar poster.jpg file and putting it in .media-art is
definitely best done by tracker-miner-fs; but I can see a distinction
between that and a thumbnail, and imagine - a HDD is connected with
500 films, and Tracker immediately starts generating thumbnails for
them all: that is not going to be great for performance :)

>  So the things are not symmetric. An ambitious idea would be to move
> all this business of "object representation" into tumbler: the client
> goes there, ask for an image for a URI and it gets an image back
> (thumbnail for an image, frame or poster for a movie, avatar for a
> contact...).
>
> Since we have to open video files anyway to get their
> > > tags and stream info, it makes sense to reuse that pipeline for
> > > thumbnailing - some video rips also have an obvious
> > > folder.jpg/poster.jpg image that we can identify and use instead.
>
>  The first is a thumbnail, the second is "album-art"... and each one
> happens in a different place.

It does seem a good idea to keep media-art (posters, covers) as a
separate concept to thumbnails. For the former we could perhaps add an
nie:image property that would associate an album / video with its
cover image(s) in media-art, associate contacts with avatars, etc.

Clients could then choose whether they wanted a poster/cover (via
Tracker / Grilo) or an actual video/image/document thumbnail (via
Tumbler / thumbnailer API).

> > > Does anyone have comments on whether this is a good/bad idea?
>
>  I would start implementing the extraction of poster/album-art if it
> is embedded in the video. That should solve your problem for some
> files. From there, let see where to get posters from, and how to
> integrate that in the current Tracker-Tumbler mix.

I've not managed to find a single example of a video with an embedded
poster, do you have any? If none turn up I'll add the code to detect
poster/folder.jpg but nothing more.

Sam