Re: Mass Reboot: Sunday 20th, 22:00 - 23:30 PM GMT+2

[Andrea Veri <av gnome org>]

Status update:

1. bugzilla and clipboard are back up again and the services were restored just fine.

2. the manegement cards have been finally configured. (this allows us to reboot, power cycle, check the status of the hardware remotely)

3. mysqld's replication has been configured and will be finalized in the next few hours.

4. a bugzilla-backup node will be set up to avoid any downtime in the case vbox (the machine hosting bugzilla) will get in trouble.

The GNOME Sysadmin team worked as much hard as it could to restore the services but most of all we would like to thank everyone for the usual patience and help you gave us reporting issues and contacting us when a problem occurred.

Have an awesome day,

2013/10/21 Andrea Veri <av gnome org>

Hi,

all the servers rebooted successfully except two machines (seems they are stuck at "Rebooting machine") that are actually going to be power-cycled as soon as the on-site personnel will phisically reach the machines given that (unfortunately) the PDUs for the two machines appear to be non-responsive to our logins. (a ticket has been opened to have this fixed asap as well)

The machines affected are:

1. bugzilla

2. clipboard (ns-master.gnome.org and ldap.gnome.org, all the DNS requests are currently being served by our slave while SSSD is luckily caching all the logins to git.gnome.org, you might be experiecing slow git pushes though given sssd tries to ping ldap.gnome.org anyway)

We'll be making sure that the machines will be power cycled as soon as possible to restore all the services. 

Thanks for your patience and as always keep an eye on [1].

[1] https://status.gnome.org

2013/10/16 Andrea Veri <av gnome org>

Hi,

we'll be performing a mass reboot of all our machines this upcoming sunday to apply all the kernel and security updates. All the services will be affected for a few minutes until the machines reboot, please stay tuned for more updates from the Sysadmin Team and most of all keep an eye at [1], we'll make sure to update it as things move forward.

Have an awesome day,

[1] https://status.gnome.org

--
Cheers,

Andrea

Debian Developer,
Fedora / EPEL packager,
GNOME Sysadmin,
GNOME Foundation Membership & Elections Committee Chairman

Homepage: http://www.gnome.org/~av

--
Cheers,

Andrea

Debian Developer,
Fedora / EPEL packager,
GNOME Sysadmin,
GNOME Foundation Membership & Elections Committee Chairman

Homepage: http://www.gnome.org/~av

--
Cheers,

Andrea

Debian Developer,
Fedora / EPEL packager,
GNOME Sysadmin,
GNOME Foundation Membership & Elections Committee Chairman

Homepage: http://www.gnome.org/~av