CentOS - Mar 2012 - CentOS 6.2 software raid 10 with LVM - Need help with degraded drive and only one MBR

CentOS Community,

I have a dedicated server with 4 hard drives in a RAID 10 software 
configuration running LVM. My OS is CentOS 6.2. Earlier today, I 
rebooted my system and my system did not come back online. I opened a 
ticket with my datacenter who informed me that one of my hard drives is 
no longer recognized by the bios and has failed. I was told that an OS 
reinstall was needed.

I don't understand why a reinstall would be necessary when the drives 
are in RAID 10. Apparently when the datacenter did the initial OS 
install, they ONLY installed the MBR on one drive instead of all 4 
leaving the other 3 drives unbootable.

Is this a way to salvage this with a liveCD without having to reload the 
OS? This server is a very important mail server running OpenLDAP and 
MySQL. I figured maybe I could install the MBR using a liveCD which may 
fix the system.

If an OS reload is the ONLY option, is there a way to reload it without 
touching the /var or /opt filesystems? (yes they were created as a 
seperate partition) however I am not sure if OpenLDAP or MySQL installs 
anything to /usr which I would be completely screwed...

Please help

Essentially I guess the short question is, How do I make a non bootable 
drive bootable if the original MBR is no longer available?

On 3/2/2012 9:03 AM, Jonathan Vomacka wrote:
> CentOS Community,
>
> I have a dedicated server with 4 hard drives in a RAID 10 software
> configuration running LVM. My OS is CentOS 6.2. Earlier today, I
> rebooted my system and my system did not come back online. I opened a
> ticket with my datacenter who informed me that one of my hard drives is
> no longer recognized by the bios and has failed. I was told that an OS
> reinstall was needed.
>
> I don't understand why a reinstall would be necessary when the drives
> are in RAID 10. Apparently when the datacenter did the initial OS
> install, they ONLY installed the MBR on one drive instead of all 4
> leaving the other 3 drives unbootable.
>
> Is this a way to salvage this with a liveCD without having to reload the
> OS? This server is a very important mail server running OpenLDAP and
> MySQL. I figured maybe I could install the MBR using a liveCD which may
> fix the system.
>
> If an OS reload is the ONLY option, is there a way to reload it without
> touching the /var or /opt filesystems? (yes they were created as a
> seperate partition) however I am not sure if OpenLDAP or MySQL installs
> anything to /usr which I would be completely screwed...
>
> Please help

Jonathan Vomacka wrote:
> CentOS Community,
>
> I have a dedicated server with 4 hard drives in a RAID 10 software
> configuration running LVM. My OS is CentOS 6.2. Earlier today, I
> rebooted my system and my system did not come back online. I opened a
> ticket with my datacenter who informed me that one of my hard drives is
> no longer recognized by the bios and has failed. I was told that an OS
> reinstall was needed.
>
> I don't understand why a reinstall would be necessary when the drives
> are in RAID 10. Apparently when the datacenter did the initial OS
> install, they ONLY installed the MBR on one drive instead of all 4
> leaving the other 3 drives unbootable.
<snip>
Whatever the outcome, I would strongly recommend escalating this to a
manager, since the staff installed it incorrectly - perhaps they don't
understand what a RAID is? But they need to make this right - you're
paying for it.

Besides, it's not Windows....

         mark

On 03/02/2012 09:03 AM, Jonathan Vomacka wrote:
> CentOS Community,
> 
> I have a dedicated server with 4 hard drives in a RAID 10 software 
> configuration running LVM. My OS is CentOS 6.2. Earlier today, I 
> rebooted my system and my system did not come back online. I opened a 
> ticket with my datacenter who informed me that one of my hard drives is 
> no longer recognized by the bios and has failed. I was told that an OS 
> reinstall was needed.
> 
> I don't understand why a reinstall would be necessary when the drives 
> are in RAID 10. Apparently when the datacenter did the initial OS 
> install, they ONLY installed the MBR on one drive instead of all 4 
> leaving the other 3 drives unbootable.
> 
> Is this a way to salvage this with a liveCD without having to reload the 
> OS? This server is a very important mail server running OpenLDAP and 
> MySQL. I figured maybe I could install the MBR using a liveCD which may 
> fix the system.
> 
> If an OS reload is the ONLY option, is there a way to reload it without 
> touching the /var or /opt filesystems? (yes they were created as a 
> seperate partition) however I am not sure if OpenLDAP or MySQL installs 
> anything to /usr which I would be completely screwed...
> 
> Please help
Advice provided as-is.

Boot from a live CD using the CentOS 6.2 install media. Once booted:

<bash># grub
<grub> root (hd0,0)
<grub> setup (hd0)
<grub> root (hd1,0)
<grub> setup (hd1)
<grub> root (hd2,0)
<grub> setup (hd2)
<grub> quit
<bash># reboot

This assumes that grub sees the drives at '0, 1 and 2' and the boot
partition is the first on each drive. If it is, when you type 'root
(hdX,0)' it should report that a file system was found. The 'setup
(hdX)' will tell grub to write the MBR to the specified disk.

-- 
Digimer
E-Mail:              digimer at alteeve.com
Papers and Projects: https://alteeve.com

Putting an MBR on all disks right after an OS install, as previously
mentioned, is of course the best option (although it's too late for that
in this instance).  Others have talked about using the live CD to
recover from your situation, which is good.

Other, "less good" options that might be available to you are:
  - change the boot device in the BIOS
  - if the server isn't capable of doing that, have them
    flip the cables.

I, also, would recommend having a little chat with their manager ...

Devin

On Fri, Mar 02, 2012 at 09:03:44AM -0500, Jonathan Vomacka
wrote:
> CentOS Community,
> 
> I have a dedicated server with 4 hard drives in a RAID 10 software 
> configuration running LVM. My OS is CentOS 6.2. Earlier today, I 
> rebooted my system and my system did not come back online. I opened a 
> ticket with my datacenter who informed me that one of my hard drives is 
> no longer recognized by the bios and has failed. I was told that an OS 
> reinstall was needed.
how was your disks partitions? and how was the lvm setup?
was /boot made of a mdadm raid1 array?

there is a dracut issue on raid1 array (fixed by now, if you updated
to the latest dracut release and rebuild your initramfs):
http://bugs.centos.org/view.php?id=5400

Btw the anaconda installer does install grub on the MBR of both raid1 members.
No idea if it does so for lvm over raid10 array.

Tru

-- 
Tru Huynh (mirrors, CentOS i386/x86_64 Package Maintenance)
http://pgp.mit.edu:11371/pks/lookup?op=get&search=0xBEFA581B
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
URL:
<http://lists.centos.org/pipermail/centos/attachments/20120302/d32de490/attachment-0004.sig>

On 3/2/2012 3:38 PM, Tru Huynh wrote:
> On Fri, Mar 02, 2012 at 09:03:44AM -0500, Jonathan Vomacka wrote:
>> CentOS Community,
>>
>> I have a dedicated server with 4 hard drives in a RAID 10 software
>> configuration running LVM. My OS is CentOS 6.2. Earlier today, I
>> rebooted my system and my system did not come back online. I opened a
>> ticket with my datacenter who informed me that one of my hard drives is
>> no longer recognized by the bios and has failed. I was told that an OS
>> reinstall was needed.
>
> how was your disks partitions? and how was the lvm setup?
> was /boot made of a mdadm raid1 array?
>
> there is a dracut issue on raid1 array (fixed by now, if you updated
> to the latest dracut release and rebuild your initramfs):
> http://bugs.centos.org/view.php?id=5400
>
> Btw the anaconda installer does install grub on the MBR of both raid1
members.
> No idea if it does so for lvm over raid10 array.
>
> Tru
>
>
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
Tru,

This was a raid 10, would the same apply? If I do a fresh reinstall how 
can I guarantee that the MBR is across all drives?

On Friday, March 02, 2012 04:18:48 PM Digimer wrote:
> 10 (one-zero) == a mirror of two stripped arrays. You can lose up to two
> drives, so long as they are both from the same strip set.
You really don't want a mirror of two striped arrays; you want a striped
array of mirrors.

A striped array of mirrors is much more resilient to multiple drive failures
than a mirrored set of striped arrays.

See
http://www.linux-mag.com/id/7928/?hq_e=el&hq_m=1151565&hq_l=36&hq_v=3fa9646c7f

What you're describing is RAID01, not RAID10.