Re: live.gnome.org down

[Olav Vitters <olav vitters nl>]

On Tue, Sep 06, 2011 at 02:07:36PM -0400, Owen Taylor wrote:
> On Fri, 2011-09-02 at 19:17 -0400, Owen Taylor wrote:
> > label.gnome.org ran out of memory this morning, it was looping hard in
> > the OOM killer. I got Red Hat IT to power cycle the machine a few hours
> > ago, but when I just looked the load average was at 24 and the machine
> > looked like it was heading for another OOM death.
> > 
> > So, I stopped httpd, so ldap would stay up and we wouldn't have to get
> > the machine rebooted again. I also stopped puppet, since I think that
> > would start httpd when run.
> > 
> > If anybody wants to dig in and try to figure out what is going on, that
> > probably would be a good idea.
> 
> What was going on is that a user accidentally (I think) uploaded a 97M
> binary file as the content of a fairly frequently accessed page.
> 
> When anybody tried to access that page, it would spin forever eating
> enormous amounts of memory, which would eventually take the server down.
> 
> I reverted the page change, and removed the offending revisions.

Damn.. I like your investigation skills. How were you able to figure
this out? I noticed the heavy memory usage and CPU, but couldn't do much
more :-(

could you also setup 25GB or space on the wiki VM? I wanted to migrate
it but don't have knowledge on VMs. live.gnome.org + other smaller sites
take up 9.7GB, so 25GB should be ok.

-- 
Regards,
Olav