Re: GNOME Servers were down

[Jeff Schroeder <jeffschroed gmail com>]

On Thu, Apr 29, 2010 at 1:51 AM, Ray Wang <raywang gnome org> wrote:
> Hi Owen,
>
> Today, we experienced many gnome servers were down, and Olav figure
> out that label or other servers were rebooted, that lead to many
> services were not available, for example, git, bugzilla, puppet0, wiki
> and NFS etc.
>
> Olav guided me that bugzilla is on vbox as a VM, and I added him on
> vbox, we figured out that those VMs will not able to start when
> domain0 is rebooted, the configuration files are not in /etc/xen/auto,
> we are not sure you have done with these on vbox, but please notice
> that and make sure those necessary services can start when host
> restart?  We left other services to you, and please decide which VM
> should be started, thanks a lot! :)
>
> I just wonder what VMs on combobox, are they KVM based? how can we
> start them if the host is down?

The steps to restart the test wiki vm on combox were:
    - vgscan; vgchange -ay # Just to be paranoid and make sure
    - cd /etc/libvirt/qemu; virsh define wiki.xml
    - chcon  -t virt_image_t /dev/mapper/VolGroup00-wiki--data
    - virsh start wiki

This volume shouldn't need to be relabelled after every machine reboot
should it? If so lets setup something so we can fix the selinux labels
and autostart the vms.

> For the record, label has a fault hard drive, so that raid1 is failed.
>
> I may not state the situation completely, feel free to add more
> details, Olav. :)
>
> Thanks both of you for spending your time. :)
>
> --
> Ray Wang
>  - Free As In Freedom

Does anyone know why the servers rebooted? Did the rack lose power and
was this scheduled maintenance of some sort? If it was scheduled
perhaps we could get redhat to send notifications. Any ideas Owen?

-- 
Jeff Schroeder

Don't drink and derive, alcohol and analysis don't mix.
http://www.digitalprognosis.com