Re: [Tracker] Video thumbnailing

[Ivan Frade <ivan frade gmail com>]

Hi,

On Mon, Sep 12, 2011 at 9:38 PM, Jens Georg <jensg openismus com> wrote:
> On Mo, 2011-09-12 at 18:56 +0100, Sam Thursfield wrote:
> > Hi all,
> >
> > It would be nice if Tracker could provide video thumbnails, in the
> > same way that it currently extracts album art, for media centre apps
> > etc. to use.

 Take into account that there are two different things here: 1.
choosing an image for the file (E.G. album-art) and 2. creating a
thumbnail of an image.

 For audio files the things are very clear: step 1 is responsibility
of the extractor, step 2 is implemented in tumbler (i.e. in the
Thumbnail DBus API).

 With audio files, the extractor tries to discover the image (either
reading the image embedded in the file, a cover.png in the same folder
or an heuristic about files and directories) and saves it under
.media-art (following the MediaArt spec) [step 1]. The first time
anybody needs a thumbnail for that music file, they check if they can
find its representation under .media-art and then request a thumbnail
[step 2]. If there is no media-art then they will show the default
icon.

 Note that is not tracker but the application who triggers step 2. We
don't like to pre-generate thumbnails because it is a heavy IO
operation and it harms seriously the performance of mobile devices.

 For video files, if they have an image embedded, everything should be
the same as with audio. You just need to put a bit of code in the
gstreamer extractor because it is not implemented AFAIK.

 If a video doesn't have an embedded image, the coherent solution
would be to extract a frame from the stream and save it under
.media-art in the extractor... BUT taking that frame is also heavy IO
operation and slows down *a lot* the extraction. So we moved that
functionality into a plugin in tumbler. If you request a thumbnail of
a video, you get that famous frame.

 So the things are not symmetric. An ambitious idea would be to move
all this business of "object representation" into tumbler: the client
goes there, ask for an image for a URI and it gets an image back
(thumbnail for an image, frame or poster for a movie, avatar for a
contact...).

Since we have to open video files anyway to get their
> > tags and stream info, it makes sense to reuse that pipeline for
> > thumbnailing - some video rips also have an obvious
> > folder.jpg/poster.jpg image that we can identify and use instead.

 The first is a thumbnail, the second is "album-art"... and each one
happens in a different place.

> Associating folders/posters with the videos sounds like a great idea.

 Definitely. An imagine we could take them from http://www.themoviedb.org/ !!!

> > I propose storing them according to the media art spec[2] (I'm sure it
> > can be easily extended for videos), and we could also generate small
> > thumbnails following the thumbnail spec[3] to save Nautilus the job.

 Yes, that is the right place to put those images.

> I don't quite see why you would add them to the media-art spec _and_
> the .thumbnails? Is it because you want to retain the original?

 The media-art will have multiple sizes. Thumbnail is an image in a
standard size... take is as a cache to load things faster in the UI.

> > Does anyone have comments on whether this is a good/bad idea?

 I would start implementing the extraction of poster/album-art if it
is embedded in the video. That should solve your problem for some
files. From there, let see where to get posters from, and how to
integrate that in the current Tracker-Tumbler mix.

 Hope this helps,

Ivan