CentOS - Dec 2018 - Accidentally nuked my system - any suggestions ?

Le 04/12/2018 ? 23:50, Stephen John Smoogen a ?crit?:
> In the rescue mode, recreate the partition table which was on the sdb
> by copying over what is on sda
> 
> 
> sfdisk ?d /dev/sda | sfdisk /dev/sdb
> 
> This will give the kernel enough to know it has things to do on
> rebuilding parts.
Once I made sure I retrieved all my data, I followed your suggestion,
and it looks like I'm making big progress. The system booted again,
though it feels a bit sluggish. Here's the current state of things.

[root at alphamule:~] # cat /proc/mdstat
Personalities : [raid1]
md125 : active raid1 sdb2[1] sda2[0]
      512960 blocks super 1.0 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md126 : inactive sda1[0](S)
      16777216 blocks super 1.2

md127 : active raid1 sda3[0]
      959323136 blocks super 1.2 [2/1] [U_]
      bitmap: 8/8 pages [32KB], 65536KB chunk

unused devices: <none>

Now how can I make my RAID array whole again? For the record, /dev/sda
is intact, and /dev/sdb is the faulty disk. How can I force
synchronization with /dev/sda?

Cheers,

Niki

-- 
Microlinux - Solutions informatiques durables
7, place de l'?glise - 30730 Montpezat
Site : https://www.microlinux.fr
Blog : https://blog.microlinux.fr
Mail : info at microlinux.fr
T?l. : 04 66 63 10 32

On 05/12/2018 05:37, Nicolas Kovacs wrote:
> Le 04/12/2018 ? 23:50, Stephen John Smoogen a ?crit?:
>> In the rescue mode, recreate the partition table which was on the sdb
>> by copying over what is on sda
>>
>>
>> sfdisk ?d /dev/sda | sfdisk /dev/sdb
>>
>> This will give the kernel enough to know it has things to do on
>> rebuilding parts.
> 
> Once I made sure I retrieved all my data, I followed your suggestion,
> and it looks like I'm making big progress. The system booted again,
> though it feels a bit sluggish. Here's the current state of things.
> 
> [root at alphamule:~] # cat /proc/mdstat
> Personalities : [raid1]
> md125 : active raid1 sdb2[1] sda2[0]
>        512960 blocks super 1.0 [2/2] [UU]
>        bitmap: 0/1 pages [0KB], 65536KB chunk
> 
> md126 : inactive sda1[0](S)
>        16777216 blocks super 1.2
> 
> md127 : active raid1 sda3[0]
>        959323136 blocks super 1.2 [2/1] [U_]
>        bitmap: 8/8 pages [32KB], 65536KB chunk
> 
> unused devices: <none>
> 
> Now how can I make my RAID array whole again? For the record, /dev/sda
> is intact, and /dev/sdb is the faulty disk. How can I force
> synchronization with /dev/sda?
> 
> Cheers,
> 
> Niki
> 
If you are confident in the state of sda, I would remove sdb from the 
array, copy the partition table from sda to sdb as Stephen suggested 
earlier, then add sdb back to the array and allow the data to be synced:

For example:

mdadm --fail /dev/md125 /dev/sdb2
mdadm --remove /dev/md125 /dev/sdb2

mdadm --fail /dev/md126 /dev/sdb1
mdadm --remove /dev/md126 /dev/sdb1

mdadm --fail /dev/md127 /dev/sdb3
mdadm --remove /dev/md127 /dev/sdb3

sfdisk ?d /dev/sda | sfdisk /dev/sdb

then add them back and watch then rebuild:

mdadm --add /dev/md125 /dev/sdb2
mdadm --add /dev/md126 /dev/sdb1
mdadm --add /dev/md127 /dev/sdb3

After they have all resynced, I would flush the device buffers for good 
measure. For example:

blockdev --flushbufs /dev/sdb1
...

Lastly, don't forget to reinstall grub to sdb:

grub2-install --recheck /dev/sdb

On Wed, 5 Dec 2018 at 00:36, Nicolas Kovacs <info at microlinux.fr>
wrote:
>
> Le 04/12/2018 ? 23:50, Stephen John Smoogen a ?crit :
> > In the rescue mode, recreate the partition table which was on the sdb
> > by copying over what is on sda
> >
> >
> > sfdisk ?d /dev/sda | sfdisk /dev/sdb
> >
> > This will give the kernel enough to know it has things to do on
> > rebuilding parts.
>
> Once I made sure I retrieved all my data, I followed your suggestion,
> and it looks like I'm making big progress. The system booted again,
> though it feels a bit sluggish. Here's the current state of things.
>
It will because you have 1/2 the bandwidth and there can be a tiny bit
of 'write to 2 disks.. nope. read from disk b, nope switch to a'.
> [root at alphamule:~] # cat /proc/mdstat
> Personalities : [raid1]
> md125 : active raid1 sdb2[1] sda2[0]
>       512960 blocks super 1.0 [2/2] [UU]
>       bitmap: 0/1 pages [0KB], 65536KB chunk
>
> md126 : inactive sda1[0](S)
>       16777216 blocks super 1.2
>
> md127 : active raid1 sda3[0]
>       959323136 blocks super 1.2 [2/1] [U_]
>       bitmap: 8/8 pages [32KB], 65536KB chunk
>
> unused devices: <none>
>
> Now how can I make my RAID array whole again? For the record, /dev/sda
> is intact, and /dev/sdb is the faulty disk. How can I force
> synchronization with /dev/sda?
>
> Cheers,
>
Phil Perry posted all the things in a better email than I could have (pperry++)
> Niki
>
> --
> Microlinux - Solutions informatiques durables
> 7, place de l'?glise - 30730 Montpezat
> Site : https://www.microlinux.fr
> Blog : https://blog.microlinux.fr
> Mail : info at microlinux.fr
> T?l. : 04 66 63 10 32
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> https://lists.centos.org/mailman/listinfo/centos


-- 
Stephen J Smoogen.

Le 05/12/2018 ? 08:31, Phil Perry a ?crit?:
> If you are confident in the state of sda, I would remove sdb from the
> array, copy the partition table from sda to sdb as Stephen suggested
> earlier, then add sdb back to the array and allow the data to be synced:
> 
> For example:
> 
> mdadm --fail /dev/md125 /dev/sdb2
> mdadm --remove /dev/md125 /dev/sdb2
> 
> mdadm --fail /dev/md126 /dev/sdb1
> mdadm --remove /dev/md126 /dev/sdb1
> 
> mdadm --fail /dev/md127 /dev/sdb3
> mdadm --remove /dev/md127 /dev/sdb3
> 
> sfdisk ?d /dev/sda | sfdisk /dev/sdb
> 
> then add them back and watch then rebuild:
> 
> mdadm --add /dev/md125 /dev/sdb2
> mdadm --add /dev/md126 /dev/sdb1
> mdadm --add /dev/md127 /dev/sdb3
> 
> After they have all resynced, I would flush the device buffers for good
> measure. For example:
> 
> blockdev --flushbufs /dev/sdb1
> ...
> 
> Lastly, don't forget to reinstall grub to sdb:
> 
> grub2-install --recheck /dev/sdb
Thanks very much for the detailed answer. I'll probably give this a spin
next week, since right now I have an urgent job to finish, and I'm happy
to be able to work on a usable system even though it's a bit sluggish.
As soon as the stress is over, I'll try it out.

cheers,

Niki

-- 
Microlinux - Solutions informatiques durables
7, place de l'?glise - 30730 Montpezat
Site : https://www.microlinux.fr
Blog : https://blog.microlinux.fr
Mail : info at microlinux.fr
T?l. : 04 66 63 10 32