Quick notes on database restoration

[Owen Taylor <otaylor redhat com>]

* Setting up mysql on combobox was really, really easy thanks
  to puppet. I tried to tune the config to use ~16GB of ram.
  (32GB machine, using 8GB for existing vm's)

* Hit problems with the undump of the binary attach_data table
  with messages like "ERROR at line 1031: Unknown command \0"
  google revealed various issues in the mysql bug tracker, and
  people reporting success with using --hex-blob option to
  mysqldump. We should adjust our backup scripts accordingly
  if we keep backing up using mysqldump.

  I was (oddly) able to get over this problem by splitting the
  attachment data dump into small pieces and undumping the
  pieces individually. (I was just trying to narrow down to
  the problematic parts.)

* Big bottleneck was waiting for the index on bugs_fulltext to be
  recreated. With waiting for that, the entire undump took about
  5-6 hours for the bugs database, which is really too long
  to be a good backup. We should consider using the mysql replication
  capability to keep a live hot replica of our databases so we
  can switch over immediately. The hot replica could also be
  stopped during backups allowing avoiding using mysqldump.

* Restored services are:

   bugzilla.gnome.org
   blogs.gnome.org
   shell-perf.gnome.org

  Not yet restored:

   art.gnome.org (db restored, couldn't figure out how to repoint)
   CivicCRM (db restored, didn't look at repointing)
   piwik (DB in the process of restoring)

Help with repointing the later three definitely appreciated, I need to
switch over to GNOME 3 work now.

- Owen