Detailed move timeline

[Owen Taylor <otaylor redhat com>]

Here's an attempt to write down with some detail what we need to do on 
our side for the server move.

Things I could use help with:

 * Creating a hold page on socket.gnome.org; when socket.gnome.org is
   accessed as www.gnome.org, live.gnome.org, planet.gnome.org,
   or bugzilla.gnome.org (any URL), it should respond with a temporary
   redirect to a page that describes the situation.

   Without any evidence, I think these are the 4 most important URLs;
   I don't want to worry about the 20 or more other sites we host.)

   Text can be stolen from my gnome-announce-list mail or written
   from scratch.
  
 * Verification and fixing when things start coming up; I don't know
   exactly when this will be, I'd say check in on #sysadmin on
   Sat evening if you are in the US or Sun morning if you are in EU
   or points east. 

Most of the rest of the stuff here is neither hard nor parallelizable.

- Owen

=============

December 11:

 * To make sure that we can get edit DNS as soon as possible,
   verify that menubar.gnome.org can be logged into by sysadmins:
   - Without LDAP running
   - With /home/users unmounted
   This may require reconfiguring the NSS configuration.

 * Create a hold page on socket.gnome.org

December 12:

05:00 UTC - Backup jobs are normally complete
            Verify:
            - MySQL backup scripts completed successfully
              on button and drawable
            - All backup rsyncs completed successfully
            - Spot check on backup server changes in last day for:
                git.gnome.org
                live.gnome.org
                home directories
             - Make sure that the rsync'ed dump for bugzilla
               has the right date and gunzip's successfully
               to /dev/null

11:00 UTC - Update IP addresses in zone files to point to new IPs,
            except for www.gnome.org, planet.gnome.org, 
            bugzilla.gnome.org, and live.gnome.org, which will be 
            pointed to a hold page on socket.gnome.org.

12:00 UTC - Halt all servers
            Change registrar information to point to new 
            ns-master.gnome.org IP.

12:30 UTC - Red Hat IT begins migration

   * disconnecting systems from power
   * physical transport of the systems, including insurance coverage
   * re-racking systems in new location
   * reconfiguring primary IP address of each system
   * connecting and (re)configuring remote management mechanisms
     (serial, KVM, RSA/IMM/DRAC-type devices)

This might be complete in as little as 12 hours; it is expected to be
complete by the end of the UTC day on Sun Dec 13.

After completion:

 * Verify that menubar.gnome.org is responding to DNS queries for
   gnome.org. (externally, not just from the gnome.org servers)

 * Verify that SLAPD on label is responding to requests from other
   gnome.org servers.

 * Verify that container.gnome.org is exporting /home/user and
   /home/admin to other gnome.org servers. Manually mount these
   directories on any servers that got started before container.

 * Start VMs on vbox.gnome.org, log in to them via the console, update
   their IPs.

 * Verify www.gnome.org is working (requires /etc/hosts entry),
   change DNS to point back at the real site rather than the hold page

 * Verify planet.gnome.org is working (requires /etc/hosts entry),
   change DNS to point back at the real site rather than the hold page

 * Verify that mail is working

 * Verify that mysql is running on drawable.gnome.org.

 * Verify bugzilla.gnome.org is working, change DNS to point back at the
   real site rather than the hold page

 * Verify live.gnome.org is working (may require /etc/hosts entry),
   change DNS to point back at the real site rather than the hold page

 * Verify git.gnome.org is working

 * Keep on verifying, respond to any reported problems, go through
   cron spam, etc.