thr3ads.net - CentOS - [CentOS] (C8) root on mdraid [Nov 2020]

If this information is useful, please help other people find it:
Share via:

Łukasz Posadowski

2020-Nov-15 11:32 UTC

[CentOS] (C8) root on mdraid

Hello everyone. 

I'm trying to install CentOS 8 with root and swap partitions on
software raid. The plan is:
- create md0 raid level 1 with 2 hard drives: /dev/sda and /dev/sdb,
using Linux Rscue CD,
- install CentOS 8 with Virtual Box on my laptop,
- rsync CentOS 8 root partition on /dev/md0p1,
- chroot in CentOS 8 root partition,
- configure /etc/mdadm.conf, grub.cfg, initramfs, install bootloader on
both sda and sdb drives.

I think I can do first four of the above, but my CentOS installation
acts strange after rebooting the server. It recognizes the raid, but
boots randomly with root on /dev/sda1 (and recognizes raid
with /dev/sdb disk), or with root on /dev/sdb1 (and recognizes raid
with /dev/sda disk). When booting from Linux Rescue CD, the raid with
two disk is recognized.



Creating md0 wth Linux Rescue CD.

# lsblk
NAME  MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0   7:0    0 819.5M  1 loop /run/archiso/sfs/airootfs
sda     8:0    0 232.9G  0 disk 
sdb     8:16   0 232.9G  0 disk 
sr0    11:0    1   871M  0 rom  

# mdadm --create /dev/md0 --level=1 --raid-devices=2 
--metadata=0.9 /dev/sda /dev/sdb
mdadm: array /dev/md0 started.

# mdadm --detail --scan --verbose
ARRAY /dev/md0 level=raid1 num-devices=2 metadata=0.90 UUID=fef2f68b:
1a462c80:c44c77eb:7ee19756 devices=/dev/sda,/dev/sdb

# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb[1] sda[0]
      244198464 blocks [2/2] [UU]
      [>....................]  resync =  1.9% (4682944/244198464)
finish=52.2min speed=76356K/sec bitmap: 2/2 pages [8KB], 65536KB chunk

unused devices: <none>

(1 hour later)

# cat /proc/mdstat 
Personalities : [raid1] 
md0 : active raid1 sdb[1] sda[0]
      244198464 blocks [2/2] [UU]
      bitmap: 0/2 pages [0KB], 65536KB chunk

unused devices: <none>

# lsblk
NAME  MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop0   7:0    0 819.5M  1 loop  /run/archiso/sfs/airootfs
sda     8:0    0 232.9G  0 disk  
??md0   9:0    0 232.9G  0 raid1 
sdb     8:16   0 232.9G  0 disk  
??md0   9:0    0 232.9G  0 raid1 
sr0    11:0    1   871M  0 rom  

---------------------------------

So far so good. Now partitions:

# fdisk /dev/md0

Welcome to fdisk (util-linux 2.33.1).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.

Device does not contain a recognized partition table.
Created a new DOS disklabel with disk identifier 0x64998720.

Command (m for help): o
Created a new DOS disklabel with disk identifier 0xfe784f57.

Command (m for help): n
Partition type
   p   primary (0 primary, 0 extended, 4 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (1-4, default 1): 
First sector (2048-488396927, default 2048): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (2048-488396927, default
488396927): +220G

Created a new partition 1 of type 'Linux' and of size 220 GiB.

Command (m for help): n
Partition type
   p   primary (1 primary, 0 extended, 3 free)
   e   extended (container for logical partitions)
Select (default p): p
Partition number (2-4, default 2): 
First sector (461375488-488396927, default 461375488): 
Last sector, +/-sectors or +/-size{K,M,G,T,P} (461375488-488396927,
default 488396927): +4G

Created a new partition 2 of type 'Linux' and of size 4 GiB.

Command (m for help): w
The partition table has been altered.
Calling ioctl() to re-read partition table.
Syncing disks.


# lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
loop0       7:0    0 819.5M  1 loop  /run/archiso/sfs/airootfs
sda         8:0    0 232.9G  0 disk  
??md0       9:0    0 232.9G  0 raid1 
  ??md0p1 259:0    0   220G  0 part  
  ??md0p2 259:1    0     4G  0 part  
sdb         8:16   0 232.9G  0 disk  
??md0       9:0    0 232.9G  0 raid1 
  ??md0p1 259:0    0   220G  0 part  
  ??md0p2 259:1    0     4G  0 part  
sr0        11:0    1   871M  0 rom   


# mkfs -t ext4 /dev/md0p1
mke2fs 1.44.5 (15-Dec-2018)
Creating filesystem with 57671680 4k blocks and 14417920 inodes
Filesystem UUID: 511d0d07-4d55-4b6c-939e-d23ae2da1ce0
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632,
2654208, 4096000, 7962624, 11239424, 20480000, 23887872

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done     

# mkswap /dev/md0p2 
Setting up swapspace version 1, size = 4 GiB (4294963200 bytes)
no label, UUID=5667d007-0d3e-4b54-8a72-6ade52609adc

--------------------------------------

Mount would be root partition:

# mkdir /mnt/centos8
# mount -t ext4 /dev/md0p1 /mnt/centos8/

--------------------------------------

OK, I have what I need. Now log into virtualised CentOS 8 in Virtual
Box.

# rpm -q rsync
rsync-3.1.3-7.el8.x86_64
# rpm -q mdadm
mdadm-4.1-13.el8.x86_64

# rsync -vaxAXHSz --delete / 192.168.111.167:/mnt/centos8/
The authenticity of host '192.168.111.167 (192.168.111.167)' can't
be
established. ECDSA key fingerprint is
SHA256:HukswAQlarthYqbpVHIHm80JaWHKX12AxgOn8UuKQxk. Are you sure you
want to continue connecting (yes/no/[fingerprint])? yes Warning:
Permanently added '192.168.111.167' (ECDSA) to the list of known hosts.
root at 192.168.111.167's password: sending incremental file list
./
bin -> usr/bin
lib -> usr/lib
lib64 -> usr/lib64
sbin -> usr/sbin
[lots and lots of files...]
var/yp/

sent 694,980,592 bytes  received 745,802 bytes  1,795,422.95 bytes/sec
total size is 1,558,934,091  speedup is 2.24

--------------------------------------

Chrooting time:

# mount --types proc /proc /mnt/centos8/proc
# mount --rbind /sys /mnt/centos8/sys
# mount --make-rslave /mnt/centos8/sys
# mount --rbind /dev /mnt/centos8/dev
# mount --make-rslave /mnt/centos8/dev
# chroot /mnt/centos8 /usr/bin/bash
# source /etc/profile


Configure raid: 

# mdadm --detail --scan --verbose
ARRAY /dev/md0 level=raid1 num-devices=2 metadata=0.90 UUID=fef2f68b:
1a462c80:c44c77eb:7ee19756 devices=/dev/sda,/dev/sdb
# mdadm --detail --scan --verbose > /etc/mdadm.conf


Edit /etc/fstab:

# cat /etc/fstab 

#
# /etc/fstab
# Created by anaconda on Sun Nov  8 10:05:31 2020
#
# Accessible filesystems, by reference, are maintained under
# '/dev/disk/'. See man pages fstab(5), findfs(8), mount(8) and/or
# blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update
# systemd units generated from this file.
#
#UUID=36c5a887-79c3-4b6a-b433-0e4d7574b1e4 /
#ext4    defaults        1 1
/dev/md0p1 / ext4 defaults 1 1


Configure and install bootloader:

# cat /etc/default/grub 
GRUB_TIMEOUT=8
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g'
/etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rhgb quiet"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

# grub2-mkconfig -o /boot/grub2/grub.cfg 
Generating grub configuration file ...
done

# grub2-install /dev/sda
Installing for i386-pc platform.
Installation finished. No error reported.
# grub2-install /dev/sdb
Installing for i386-pc platform.
Installation finished. No error reported.


Configure initramfs:

# dracut -f /boot/initramfs-4.18.0-193.28.1.el8_2.x86_64.img
# 4.18.0-193.28.1.el8_2.x86_64

--------------------------------

Reboot and login to my new CentOS 8 installation:

# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 232.9G  0 disk 
sdb      8:16   0 232.9G  0 disk 
??sdb1   8:17   0   220G  0 part /
??sdb2   8:18   0     4G  0 part 
sr0     11:0    1   871M  0 rom  

# mdadm --detail --scan --verbose
INACTIVE-ARRAY /dev/md127 num-devices=1 metadata=0.90 UUID=fef2f68b:
1a462c80:c44c77eb:7ee19756 devices=/dev/sda

# cat /proc/mdstat 
Personalities : 
md127 : inactive sda[0](S)
      244198464 blocks
       
unused devices: <none>


I can live with md127, because it happens constantly on multiple
distributions, but why root is on /dev/sdb1 (and sometimes
it's /dev/sda1)? I'm running out of spells now. :) Do anyone can
suggest what else I forgot to do?

thanks

-- 
?ukasz Posadowski

Gordon Messmer

2020-Nov-15 22:16 UTC

head link

[CentOS] (C8) root on mdraid

On 11/15/20 3:32 AM, ?ukasz Posadowski wrote:> Do anyone can suggest what else I forgot to do?

Use metadata version 1.2 instead of 0.9.

You need for the filesystem to be not visible until after the RAID is 
assembled, and the easiest way to do that is to put the metadata at the 
beginning of the drive and the partition table inside the RAID volume.

With metadata version 0.9, the partition table is technically inside the 
volume, but it's at the same offset that it would be for a disk with no 
RAID volume, so it can be recognized before the volume is assembled.

Łukasz Posadowski

2020-Nov-16 06:40 UTC

head link

[CentOS] (C8) root on mdraid

Sun, 15 Nov 2020 14:16:48 -0800
Gordon Messmer <gordon.messmer at gmail.com>:
> On 11/15/20 3:32 AM, ?ukasz Posadowski wrote:
> > Do anyone can suggest what else I forgot to do?
> 
> 
> Use metadata version 1.2 instead of 0.9.
> 
> You need for the filesystem to be not visible until after the RAID is 
> assembled, and the easiest way to do that is to put the metadata at
> the beginning of the drive and the partition table inside the RAID
> volume.
> 
> With metadata version 0.9, the partition table is technically inside
> the volume, but it's at the same offset that it would be for a disk
> with no RAID volume, so it can be recognized before the volume is
> assembled.
Thanks, I'll try that. I'm use to metadata 0.9, because GRUB have
(had?) some issue with the newer ones.

-- 
?ukasz Posadowski

Tony Mountifield

2020-Nov-16 10:29 UTC

head link

[CentOS] (C8) root on mdraid

In article <20201115123245.db62b8248e1f248afe02844a at
lukaszposadowski.pl>,
Lukasz Posadowski <mail at lukaszposadowski.pl>
wrote:> 
> Hello everyone. 
> 
> I'm trying to install CentOS 8 with root and swap partitions on
> software raid. The plan is:
> - create md0 raid level 1 with 2 hard drives: /dev/sda and /dev/sdb,
> using Linux Rscue CD,
> - install CentOS 8 with Virtual Box on my laptop,
> - rsync CentOS 8 root partition on /dev/md0p1,
> - chroot in CentOS 8 root partition,
> - configure /etc/mdadm.conf, grub.cfg, initramfs, install bootloader on
> both sda and sdb drives.
> 
> I think I can do first four of the above, but my CentOS installation
> acts strange after rebooting the server. It recognizes the raid, but
> boots randomly with root on /dev/sda1 (and recognizes raid
> with /dev/sdb disk), or with root on /dev/sdb1 (and recognizes raid
> with /dev/sda disk). When booting from Linux Rescue CD, the raid with
> two disk is recognized.
I thought it was much more usual to partition both disks to give sda1,2,3
and sdb1,2,3, and then create /dev/md0 from sda1/sdb1, /dev/md1 from sda2/sdb3,
and so on.

That's the way I have always done it, and have never had any problems.
Never seen an attempt to partition an md device before. In that case,
how would the kernel and initrd be found in order to assemble the RAID?

Cheers
Tony
-- 
Tony Mountifield
Work: tony at softins.co.uk - http://www.softins.co.uk
Play: tony at mountifield.org - http://tony.mountifield.org

John Pierce

2020-Nov-16 14:12 UTC

head link

[CentOS] (C8) root on mdraid

On Mon, Nov 16, 2020, 2:29 AM Tony Mountifield <tony at softins.co.uk>
wrote:
>
> I thought it was much more usual to partition both disks to give sda1,2,3
> and sdb1,2,3, and then create /dev/md0 from sda1/sdb1, /dev/md1 from
> sda2/sdb3,
> and so on.
>
What I always did was to mdraid a single full disk partition then use lvm
for any file systems.    Boot disks did need a separate /boot partition.
>

Łukasz Posadowski

2020-Nov-18 14:56 UTC

head link

[CentOS] (C8) root on mdraid

Mon, 16 Nov 2020 10:29:17 +0000
Tony Mountifield <tony at softins.co.uk>:
> In article
> <20201115123245.db62b8248e1f248afe02844a at lukaszposadowski.pl>,
Lukasz
> Posadowski <mail at lukaszposadowski.pl> wrote:
> > 
> > Hello everyone. 
> > 
> > I'm trying to install CentOS 8 with root and swap partitions on
> > software raid. The plan is:
> > - create md0 raid level 1 with 2 hard drives: /dev/sda and /dev/sdb,
> > using Linux Rscue CD,
> > - install CentOS 8 with Virtual Box on my laptop,
> > - rsync CentOS 8 root partition on /dev/md0p1,
> > - chroot in CentOS 8 root partition,
> > - configure /etc/mdadm.conf, grub.cfg, initramfs, install
> > bootloader on both sda and sdb drives.
> > 
> > I think I can do first four of the above, but my CentOS installation
> > acts strange after rebooting the server. It recognizes the raid, but
> > boots randomly with root on /dev/sda1 (and recognizes raid
> > with /dev/sdb disk), or with root on /dev/sdb1 (and recognizes raid
> > with /dev/sda disk). When booting from Linux Rescue CD, the raid
> > with two disk is recognized.
> 
> I thought it was much more usual to partition both disks to give
> sda1,2,3 and sdb1,2,3, and then create /dev/md0 from
> sda1/sdb1, /dev/md1 from sda2/sdb3, and so on.
> 
> That's the way I have always done it, and have never had any problems.
> Never seen an attempt to partition an md device before. In that case,
> how would the kernel and initrd be found in order to assemble the
> RAID?
It totaly works either way - raid on partitions and raid on disks. I'm
use to mirroring whole disks.

-- 
?ukasz Posadowski

Reasonably Related Threads

Search for more apparently analagous threads

CentOS - Nov 2020 - (C8) root on mdraid

[CentOS] (C8) root on mdraid

[CentOS] (C8) root on mdraid

[CentOS] (C8) root on mdraid

[CentOS] (C8) root on mdraid

[CentOS] (C8) root on mdraid

[CentOS] (C8) root on mdraid

Reasonably Related Threads