CentOS - Apr 2017 - Possible bug with latest 7.3 installer, md RAID1, and SATADOM.

I'm seeing a problem that I think maybe a bug with the mdraid software on
the latest CentOS installer. I have a couple of new supermicro servers and each
system has two innodisk 32GB SATADOM's that are experiencing the same issue.
I used the latest CentOS-7-x86_64-1611 to install to the two SATADOM's a
simple RAID1 for the root. The install goes just fine but when I boot off the
new install I see one of two behaviours. It either hangs at boot, or boots fine
but I start getting errors when using the system. For example it will give me 
the following error if I try to run yum update.

error: rpmdb: damaged header #6 retrieved -- skipping.

It will just hang giving that error over and over. I have to use a different
login session to kill it or reboot. It doesn't even log anything to
journelctl or /var/log/messages. At first I thought either the hardware was the
issue(sata port, controller, SATADOM, etc). However, I do not see any issues if
I don't try to raid the disks. Setting either of the SATADOM's up as a
single system drive works just fine. It does not matter if I choose xfs or ext4
for the filesystem when I try to RAID them either. Making an md RAID1 out of the
two disks with 7.3 installer is the only combination I see this issue with. If I
use the previous 7.2 installer(CentOS-7-x86_64-1511) I don't see the problem
at all. I can run yum update, reboot, and everything is still ok. I should also
point out that I tested the CentOS 7.3 installer creating a md RAID1 system
drive using two regular spinning hard drives and that worked just fine. I was
wondering if anyone else has seen something similar or can confirm this problem
before I submit it as a real bug..

TL/DR. Two different supermicro servers, both using innodisk 32GB SATADOM's
and latest CentOS 7.3 installer to create a RAID1 system results in freezes and
weird errors. Using the CentOS 7.2 installer works fine.

David Miller.

Is there a reason you are not using the built in controller to RAID the
SATADOMs?

As I remember on SuperMicro there are two controllers. One for the SATADOMs
and another for the conventional disks.

Cameron

On Fri, Apr 14, 2017 at 2:28 PM, David C. Miller <millerdc at
fusion.gat.com>
wrote:
> I'm seeing a problem that I think maybe a bug with the mdraid software
on
> the latest CentOS installer. I have a couple of new supermicro servers and
> each system has two innodisk 32GB SATADOM's that are experiencing the
same
> issue. I used the latest CentOS-7-x86_64-1611 to install to the two
> SATADOM's a simple RAID1 for the root. The install goes just fine but
when
> I boot off the new install I see one of two behaviours. It either hangs at
> boot, or boots fine but I start getting errors when using the system. For
> example it will give me  the following error if I try to run yum update.
>
> error: rpmdb: damaged header #6 retrieved -- skipping.
>
> It will just hang giving that error over and over. I have to use a
> different login session to kill it or reboot. It doesn't even log
anything
> to journelctl or /var/log/messages. At first I thought either the hardware
> was the issue(sata port, controller, SATADOM, etc). However, I do not see
> any issues if I don't try to raid the disks. Setting either of the
> SATADOM's up as a single system drive works just fine. It does not
matter
> if I choose xfs or ext4 for the filesystem when I try to RAID them either.
> Making an md RAID1 out of the two disks with 7.3 installer is the only
> combination I see this issue with. If I use the previous 7.2
> installer(CentOS-7-x86_64-1511) I don't see the problem at all. I can
run
> yum update, reboot, and everything is still ok. I should also point out
> that I tested the CentOS 7.3 installer creating a md RAID1 system drive
> using two regular spinning hard drives and that worked just fine. I was
> wondering if anyone else has seen something similar or can confirm th
>  is problem before I submit it as a real bug..
>
> TL/DR. Two different supermicro servers, both using innodisk 32GB
> SATADOM's and latest CentOS 7.3 installer to create a RAID1 system
results
> in freezes and weird errors. Using the CentOS 7.2 installer works fine.
>
> David Miller.
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos
>

>> On Fri, Apr 14, 2017 at 2:28 PM, David C. Miller <millerdc at
fusion.gat.com>
>> wrote:
>> 
>> I'm seeing a problem that I think maybe a bug with the mdraid
software on
>> the latest CentOS installer. I have a couple of new supermicro servers
and
>> each system has two innodisk 32GB SATADOM's that are experiencing
the same
>> issue. I used the latest CentOS-7-x86_64-1611 to install to the two
>> SATADOM's a simple RAID1 for the root. The install goes just fine
but when
>> I boot off the new install I see one of two behaviours. It either hangs
at
>> boot, or boots fine but I start getting errors when using the system.
For
>> example it will give me  the following error if I try to run yum
update.
>>
>> error: rpmdb: damaged header #6 retrieved -- skipping.
>>
>> It will just hang giving that error over and over. I have to use a
>> different login session to kill it or reboot. It doesn't even log
anything
>> to journelctl or /var/log/messages. At first I thought either the
hardware
>> was the issue(sata port, controller, SATADOM, etc). However, I do not
see
>> any issues if I don't try to raid the disks. Setting either of the
>> SATADOM's up as a single system drive works just fine. It does not
matter
>> if I choose xfs or ext4 for the filesystem when I try to RAID them
either.
>> Making an md RAID1 out of the two disks with 7.3 installer is the only
>> combination I see this issue with. If I use the previous 7.2
>> installer(CentOS-7-x86_64-1511) I don't see the problem at all. I
can run
>> yum update, reboot, and everything is still ok. I should also point out
>> that I tested the CentOS 7.3 installer creating a md RAID1 system drive
>> using two regular spinning hard drives and that worked just fine. I was
>> wondering if anyone else has seen something similar or can confirm th
>>  is problem before I submit it as a real bug..
>>
>> TL/DR. Two different supermicro servers, both using innodisk 32GB
>> SATADOM's and latest CentOS 7.3 installer to create a RAID1 system
results
>> in freezes and weird errors. Using the CentOS 7.2 installer works fine.
>>
>> David Miller.
>> _______________________________________________
> From: "Cameron Smith" <cameron at networkredux.com>
> To: "CentOS mailing list" <centos at centos.org>
> Sent: Friday, April 14, 2017 2:48:56 PM
> Subject: Re: [CentOS] Possible bug with latest 7.3 installer, md RAID1,	and
SATADOM.
> Is there a reason you are not using the built in controller to RAID the
> SATADOMs?
> 
> As I remember on SuperMicro there are two controllers. One for the SATADOMs
> and another for the conventional disks.
> 
> Cameron
It requires additional software to monitor the hardware RAID. CentOS can monitor
the health of the drives and the mdRAID. It is trivial to setup postfix to relay
through my mail gateway so both smartd and md will send me an email as soon as
it sees an issue. Relying on a hardware raid card is just one more point of
failure. I only get HBA cards and let Linux handle it. On top of that I can move
the drives to any other system and it will still work.

David Miller.