Re: GNOME Servers were down

[Owen Taylor <otaylor redhat com>]

On Thu, 2010-04-29 at 05:57 -0700, Jeff Schroeder wrote:

> > I just wonder what VMs on combobox, are they KVM based? how can we
> > start them if the host is down?
> 
> The steps to restart the test wiki vm on combox were:
>     - vgscan; vgchange -ay # Just to be paranoid and make sure
>     - cd /etc/libvirt/qemu; virsh define wiki.xml
>     - chcon  -t virt_image_t /dev/mapper/VolGroup00-wiki--data
>     - virsh start wiki
> 
> This volume shouldn't need to be relabelled after every machine reboot
> should it? If so lets setup something so we can fix the selinux labels
> and autostart the vms.

Thanks for tracking this down. I had to set that label explicitly on the
partition to get it initially mounted but I expected it to stick. So the
question is why it stuck for /dev/mapper/VolGroup00-wiki--disk0, and not
for the --data partition.

Looking into, libvirt created a selinux rule for the 'disk0' partition,
but I didn't do the same for the manually created --data partition and
just ran chcon manually. So, probably the SELinux policy was upgraded
at some point and the filesystem got relabeled.

I've now run:
 
semanage fcontext -a -t virt_image_t /dev/mapper/VolGroup00-wiki--data

And I think that should get the label to stick.

> > For the record, label has a fault hard drive, so that raid1 is failed.
> >
> > I may not state the situation completely, feel free to add more
> > details, Olav. :)
> >
> > Thanks both of you for spending your time. :)
> >
> > --
> > Ray Wang
> >  - Free As In Freedom
> 
> Does anyone know why the servers rebooted? Did the rack lose power and
> was this scheduled maintenance of some sort? If it was scheduled
> perhaps we could get redhat to send notifications. Any ideas Owen?

Unscheduled power outage for the whole data center.... Red Hat IT has
been scrambling all morning to get our internal systems back up.
I don't have any more detailed information as to the cause.

- Owen