Re: live.gnome.org down

[Owen Taylor <otaylor redhat com>]

On Fri, 2011-09-02 at 19:17 -0400, Owen Taylor wrote:
> label.gnome.org ran out of memory this morning, it was looping hard in
> the OOM killer. I got Red Hat IT to power cycle the machine a few hours
> ago, but when I just looked the load average was at 24 and the machine
> looked like it was heading for another OOM death.
> 
> So, I stopped httpd, so ldap would stay up and we wouldn't have to get
> the machine rebooted again. I also stopped puppet, since I think that
> would start httpd when run.
> 
> If anybody wants to dig in and try to figure out what is going on, that
> probably would be a good idea.

What was going on is that a user accidentally (I think) uploaded a 97M
binary file as the content of a fairly frequently accessed page.

When anybody tried to access that page, it would spin forever eating
enormous amounts of memory, which would eventually take the server down.

I reverted the page change, and removed the offending revisions.

- Owen