Moin event stats

[Owen Taylor <otaylor redhat com>]

So, this morning [old] bugzilla was glacial and the load on label was
around 60.

The load was all httpd processes. I straced several of them and they
were all reading through:

 /usr/local/www/moin/live.gnome.org/data/event-log

A 4GB file. I moved that file away (renamed it to .bak) and restarted
httpd and the load went down to reasonable 1-5.

My interpretation is that is that that file just keeps getting appended
to. And then there are public wiki pages under:

 http://live.gnome.org/EventStats

that read from it and print summaries. There is caching of the
accumulated stats, but at some point the statistics accumulation just
never finishes so the caching is not effective and you get a bunch
of processes all trying to accumulate the stats at once.

So, one or more of the following need to be done:

 * Set up rotation on event-log

 * Make sure that the event stat macros are never run.
   I'm pretty sure that the relevant macro that occurs in /EventStats
   also occurs in  translated pages, and could probably even be added to
   random pages, so blocking at the http level will not be effective.

Also delete the event-log.bak file I created (unless it is needed for
some reason)

Can somebody *please* look into this? My rough estimate from 30 minutes
of accumulation is that it will take 8000 or so hours before the
event-log gets to 4GB again, but if we don't take care of it now, that
year will be up quick.

Other conclusions:

 - Moving bugzilla off of label will be good for bugzilla, but
   it may mean that stupid problems with live.gnome.org just aren't
   noticed. Especially if we move ldap off as well.

 - We really need some nice simple page where we have load average
   graphs of all the servers over long time periods. Then on every
   sysadmin team meeting you could take a look at that page and say
   "woah, label has been running at a load of 20 for the last week,
   who is going to look at that?"

- Owen