CentOS - Dec 2011 - odd mdadm behavior

Allo esteemed Centos-ers,

Noticed something funny with an mdadm mirror based raid the other day.

So I had a system disk set to mirror via mdadm.

One of the disks went south at a remote office and since there was no  
one available to swap out the disk, I thought to leave it for later.

Well, due to work being what is it, this later became a year.

During one of the servers many reboots, much of the data on the system  
went missing.  The logs even showed a gap from the day the disk went  
bad to the day this reboot occurred.

So picture a gap in the logs of about a year or so.

I first suspected foul play but soon discovered that the previous bad  
disk in the mirror became functional again and that its data basically  
put the server back to the time period being a year ago or so.

Not a big deal as I do 2 types of backups every day so recovery was  
fine.

Has any one seen this before and what could I have done to prevent this?

Thanks in advance,

- aurf

Vreme: 12/19/2011 09:29 PM, aurfalien at gmail.com pi?e:
> Allo esteemed Centos-ers,
>
> Noticed something funny with an mdadm mirror based raid the other day.
>
> So I had a system disk set to mirror via mdadm.
>
> One of the disks went south at a remote office and since there was no
> one available to swap out the disk, I thought to leave it for later.
>
> Well, due to work being what is it, this later became a year.
>
> During one of the servers many reboots, much of the data on the system
> went missing.  The logs even showed a gap from the day the disk went
> bad to the day this reboot occurred.
>
> So picture a gap in the logs of about a year or so.
>
> I first suspected foul play but soon discovered that the previous bad
> disk in the mirror became functional again and that its data basically
> put the server back to the time period being a year ago or so.
>
> Not a big deal as I do 2 types of backups every day so recovery was
> fine.
>
> Has any one seen this before and what could I have done to prevent this?
>
> Thanks in advance,
>
> - aurf
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
>
I had similar thing happening to me just few weeks ago. What happened to 
me was that raid broke and data was being written only to master disk, 
that started acting up. So reboot 1 month after the raid breakage 
brought 1 month old data since that other disk came online and raid master.

I fixed the problem and will better watch out for bad signs. And I am 
going to reinstall it from 5.6 (now 5.7) to 6.2.

-- 

Ljubomir Ljubojevic
(Love is in the Air)
PL Computers
Serbia, Europe

Google is the Mother, Google is the Father, and traceroute is your
trusty Spiderman...
StarOS, Mikrotik and CentOS/RHEL/Linux consultant

On Mon, 19 Dec 2011, aurfalien at gmail.com wrote:
> Allo esteemed Centos-ers,
>
> Noticed something funny with an mdadm mirror based raid the other 
> day.
>
> So I had a system disk set to mirror via mdadm.
>
> One of the disks went south at a remote office and since there was 
> no one available to swap out the disk, I thought to leave it for 
> later.
I'm interested to know if you used mdadm to fail and remove the bad 
disk from the array when it first started acting up.

-- 
Paul Heinlein <> heinlein at madboa.com <> http://www.madboa.com/