Re: Beagle CPU usage (was Proposed module: tracker)

[Bastien Nocera <hadess hadess net>]

On Tue, 2007-03-27 at 00:32 +0100, Bastien Nocera wrote:
> Hey Federico,
> 
> On Mon, 2007-01-15 at 19:02 -0600, Federico Mena Quintero wrote:
> > El lun, 15-01-2007 a las 16:31 +0000, Bastien Nocera escribió:
> > > On Mon, 2007-01-15 at 10:15 -0600, Federico Mena Quintero wrote:
> > > <snip>
> > > > We put a call to setrlimit() in gst-office-thumbnailer, so that the
> > > > helper convert(1) process is limited to 256 MB of memory and 5 seconds
> > > > of CPU time.  Some files do *not* get thumbnailed, of course, but this
> > > > is better than making the machine unusable.
> > > 
> > > Neat. I'll try and do something similar for Totem's thumbnailer (it
> > > currently uses a 30 sec time-limit, so it would be nice to also set a
> > > certain amount of CPU).
> > 
> > The right fix is to put this in libgnomeui/gnome-thumbnail.c, so that
> > all thumbnailers get the same treatment.  We just did this for
> > gsf-office-thumbnailer because it was super-urgent (patch attached).  I
> > guess you could use a child_setup_func for g_spawn*() in gnome-thumbnail
> > to set up the resource limits :)
> 
> I'll mention this code is busted as far as the memory usage is
> concerned. From the setrlimit man page:
>        RLIMIT_AS
>               This is the maximum size of a process’ total available
> memory, in bytes. If this limit is exceeded, the malloc() and mmap()
> functions shall fail with errno set to [ENOMEM]. In addition, the
> automatic stack growth fails with  the  effects
>               outlined above.
> 
> I implemented your code some time ago in Totem's unstable branch, and I
> was wondering why movies weren't getting thumbnailed anymore. I couldn't
> mmap files larger than 256 megs.
> 
> This code shouldn't go in the platform.

FYI, I've modified Totem's code to take into account the input file size
when thumbnailing a file, so the mmap won't fail, and I get 256 megs of
real memory (hopefully) to process the file.

Other thumbnailers that need to map large files might have different
needs (eg. mmap'ing large fonts, or images). The thumbnailers have the
knowledge to do that, not the libgnomeui code.

-- 
Bastien Nocera <hadess hadess net>