Re: httpd on label

[Owen Taylor <otaylor redhat com>]

On Mon, 2010-04-19 at 14:40 +0200, Olav Vitters wrote:
> I stopped it as load keeps increasing. Cannot easily figure out what is
> wrong.
> 
> Seems more stuff has been added to label:
>  * crm
>  * webstats
> 
> This while Bugzilla had to be moved off label due to the problems it
> caused. Please do NOT add stuff there. Window is supposed to be our
> website machine.
> 
> Can someone please
> 1. Move those additional things *off* label (new VM)
> 2. Figure out what is wrong with httpd?
> 
> I lowered maxservers etc, isn't helping. No time to check further.
> 
> FWIW, load was at a consistent 260. When I stopped it after a while
> (15-20min) load was back at 35-50 or so.

Bit of a false trail here - I thought this was related:

http://mail.gnome.org/archives/gnome-infrastructure/2009-August/msg00059.html

But instead of involving the event logs, it was about the edit log -
though the edit log is smaller, it was still possible to get to the
point where accumualating the statistics for the edit log takes longer
than the cache interval.

The long-running processes I straced were all reading through the edit
log. 

I moved the edit log out of the way to:

  /usr/local/www/moin/live.gnome.org/data/edit-log.bak

Which means that:

 http://live.gnome.org/RecentChanges

Is now nearly empty. Someone probably should try to merge in a few
months of history from the saved edit log to the current edit log that
is being written now.

Truncating the edit log is probably a good thing, *but* doens't seem to
have fixed the load problems.

So far it's looking like label just can't handle the load of incoming
traffic on live.gnome.org - maybe the decrease in IO capability from
from RAID-1 to degraded RAID-1 (machine currently has one drive going
bad and needs to be replaced) was enough to put it over the edge.

I don't see any evidence that webstats (or especially CRM) has anything
to do with it.

Though it's said that a few thousand page views per hour would be enough
to stress any machine. Moin seems pretty darn slow. 

Options:

 - Try switching to mod_wsgi?
 - Move live.gnome.org to a VM on combobox and put a ton more horsepower
   behind it.

- Owen