GNOME sysadmin summary - August 2005

[Ross Golder <ross golder org>]

GNOME Sysadmin Summary - August 2005
====================================

On Sun 7th, we discovered that the mysql authentication tables had
disappeared, and various mysql-dependent services were having problems.
Ross and Owen rebuilt the authentication tables from information
gathered from various sources.

On Mon 8th, Ross upgraded mySQL on button to v4.1.11 (based on a FC4
SRPM) for Olav Vitters, who had reported that bugzilla was having a
minor problem with the older version.

On Fri 19th, at around 4am (EDT) button went down. Jeff Waugh reported
it first on #sysadmin. Ross couldn't log on to investigate, so phoned
Owen Taylor, who contacted Matthew Galgoci, who reported it to the
Phoenix colo and managed the ticket. It turned out to be some kind of
hardware failure. Normal service was restored around mid-day.

On Mon 22nd, Ross added LDAP alias lookups to all the gnome.org servers.
This solved one problem, but created a few others. The automatic website
updates and bugzilla e-mails stopped working until Wed 24th, when it got
fixed.

On Wed 24th, Ross (with advice and help from Owen) set up a new
anonymous CVS server on window, using an NFS mount of the CVS master
from container. This provides a lag-free anoncvs mirror. Documented at
<http://live.gnome.org/Infrastructure>.

On Thu 25th, around 4:30 am US EDT (-0400), window.gnome.org ran out of
memory for unknown reasons. As on Fri 19th, it was reported on #sysadmin
and Owen was alerted by phone. After some delays, we got a colo
technician there about 11:00 am and rebooted the system; came up fine.
No clues into the syslog as to what caused the EOM situation.

On Fri 26th, it was noticed that anonymous CVS wasn't working after
window's crash/reboot. Owen found the problem, and added the necessary
bind mount of /etc/gnome/cvs to the startup scripts.

On Fri 26nd, Toni switched the 'mailman gnome org' alias to direct
traffic to a 'mailman' queue on the new request tracker.

--
Ross