Niki Kovacs
2015-Feb-18 07:09 UTC
[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?
Hi,
I just replaced Slackware64 14.1 running on my office's HP Proliant
Microserver with a fresh installation of CentOS 7.
The server has 4 x 250 GB disks.
Every disk is configured like this :
* 200 MB /dev/sdX1 for /boot
* 4 GB /dev/sdX2 for swap
* 248 GB /dev/sdX3 for /
There are supposed to be no spare devices.
/boot and swap are all supposed to be assembled in RAID level 1 across 4
disks.
The / partition is supposed to be assembled in RAID level 5 across 4 disks.
With Slackware I created the arrays manually like this:
# mdadm --create /dev/md1 --level=1 --raid-devices=4 --metadata=0.90
/dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
# mdadm --create /dev/md2 --level=1 --raid-devices=4 --metadata=0.90
/dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2
# mdadm --create /dev/md3 --level=5 --raid-devices=4 --metadata=0.90
/dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3
Using this setup, I had 650 MB of disk space on /dev/md3.
Now I tried to do the same thing with CentOS 7. Everything seemed to
work at first, but here's what I got now:
[root at nestor:~] # df -h
Sys. de fichiers Taille Utilis? Dispo Uti% Mont? sur
/dev/md127 226G 1,1G 213G 1% /
devtmpfs 1,4G 0 1,4G 0% /dev
tmpfs 1,4G 0 1,4G 0% /dev/shm
tmpfs 1,4G 8,5M 1,4G 1% /run
tmpfs 1,4G 0 1,4G 0% /sys/fs/cgroup
/dev/md125 194M 80M 101M 45% /boot
/dev/sde1 917G 88M 871G 1% /mnt
The root partition (/dev/md127) only shows 226 G of space. So where has
everything gone?
[root at nestor:~] # cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : active raid1 sdc2[2] sdd2[3] sdb2[1] sda2[0]
204736 blocks super 1.0 [4/4] [UUUU]
md126 : active raid1 sdd1[3] sdc1[2] sdb1[1] sda1[0]
4095936 blocks super 1.2 [4/4] [UUUU]
md127 : active raid5 sdc3[2] sdb3[1] sdd3[4] sda3[0]
240087552 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4]
[UUUU]
bitmap: 0/1 pages [0KB], 65536KB chunk
unused devices: <none>
[root at nestor:~] # mdadm -D /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Wed Feb 18 06:49:01 2015
Raid Level : raid5
Array Size : 240087552 (228.97 GiB 245.85 GB)
Used Dev Size : 80029184 (76.32 GiB 81.95 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Feb 18 08:04:26 2015
State : active
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : localhost:root
UUID : cfc13fe9:8fa811d8:85649402:58c4846e
Events : 4703
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 8 19 1 active sync /dev/sdb3
2 8 35 2 active sync /dev/sdc3
4 8 51 3 active sync /dev/sdd3
Apparently no spare devices have been created. So why do I only have 226
GB of disk space under CentOS, when I had roughly 650 GB under Slackware?
I'm a bit lost here. Any suggestions?
Cheers,
Niki
--
Microlinux - Solutions informatiques 100% Linux et logiciels libres
7, place de l'?glise - 30730 Montpezat
Web : http://www.microlinux.fr
Mail : info at microlinux.fr
T?l. : 04 66 63 10 32
Niki Kovacs
2015-Feb-18 07:23 UTC
[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?
Le 18/02/2015 08:09, Niki Kovacs a ?crit :> > Apparently no spare devices have been created. So why do I only have 226 > GB of disk space under CentOS, when I had roughly 650 GB under Slackware? >An idea just crossed my mind. Could it be that 'df' is reporting a wrong partition size on the RAID 5 array? And how can I check if this is the case? -- Microlinux - Solutions informatiques 100% Linux et logiciels libres 7, place de l'?glise - 30730 Montpezat Web : http://www.microlinux.fr Mail : info at microlinux.fr T?l. : 04 66 63 10 32
Michael Volz
2015-Feb-18 08:24 UTC
[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?
Hi Niki,
md127 apparently only uses 81.95GB per disk. Maybe one of the partitions has the
wrong size. What's the output of lsblk?
Regards
Michael
----- Urspr?ngliche Mail -----
Von: "Niki Kovacs" <info at microlinux.fr>
An: "CentOS mailing list" <CentOS at centos.org>
Gesendet: Mittwoch, 18. Februar 2015 08:09:13
Betreff: [CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?
Hi,
I just replaced Slackware64 14.1 running on my office's HP Proliant
Microserver with a fresh installation of CentOS 7.
The server has 4 x 250 GB disks.
Every disk is configured like this :
* 200 MB /dev/sdX1 for /boot
* 4 GB /dev/sdX2 for swap
* 248 GB /dev/sdX3 for /
There are supposed to be no spare devices.
/boot and swap are all supposed to be assembled in RAID level 1 across 4
disks.
The / partition is supposed to be assembled in RAID level 5 across 4 disks.
With Slackware I created the arrays manually like this:
# mdadm --create /dev/md1 --level=1 --raid-devices=4 --metadata=0.90
/dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
# mdadm --create /dev/md2 --level=1 --raid-devices=4 --metadata=0.90
/dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2
# mdadm --create /dev/md3 --level=5 --raid-devices=4 --metadata=0.90
/dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3
Using this setup, I had 650 MB of disk space on /dev/md3.
Now I tried to do the same thing with CentOS 7. Everything seemed to
work at first, but here's what I got now:
[root at nestor:~] # df -h
Sys. de fichiers Taille Utilis? Dispo Uti% Mont? sur
/dev/md127 226G 1,1G 213G 1% /
devtmpfs 1,4G 0 1,4G 0% /dev
tmpfs 1,4G 0 1,4G 0% /dev/shm
tmpfs 1,4G 8,5M 1,4G 1% /run
tmpfs 1,4G 0 1,4G 0% /sys/fs/cgroup
/dev/md125 194M 80M 101M 45% /boot
/dev/sde1 917G 88M 871G 1% /mnt
The root partition (/dev/md127) only shows 226 G of space. So where has
everything gone?
[root at nestor:~] # cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : active raid1 sdc2[2] sdd2[3] sdb2[1] sda2[0]
204736 blocks super 1.0 [4/4] [UUUU]
md126 : active raid1 sdd1[3] sdc1[2] sdb1[1] sda1[0]
4095936 blocks super 1.2 [4/4] [UUUU]
md127 : active raid5 sdc3[2] sdb3[1] sdd3[4] sda3[0]
240087552 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4]
[UUUU]
bitmap: 0/1 pages [0KB], 65536KB chunk
unused devices: <none>
[root at nestor:~] # mdadm -D /dev/md127
/dev/md127:
Version : 1.2
Creation Time : Wed Feb 18 06:49:01 2015
Raid Level : raid5
Array Size : 240087552 (228.97 GiB 245.85 GB)
Used Dev Size : 80029184 (76.32 GiB 81.95 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Wed Feb 18 08:04:26 2015
State : active
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 512K
Name : localhost:root
UUID : cfc13fe9:8fa811d8:85649402:58c4846e
Events : 4703
Number Major Minor RaidDevice State
0 8 3 0 active sync /dev/sda3
1 8 19 1 active sync /dev/sdb3
2 8 35 2 active sync /dev/sdc3
4 8 51 3 active sync /dev/sdd3
Apparently no spare devices have been created. So why do I only have 226
GB of disk space under CentOS, when I had roughly 650 GB under Slackware?
I'm a bit lost here. Any suggestions?
Cheers,
Niki
--
Microlinux - Solutions informatiques 100% Linux et logiciels libres
7, place de l'?glise - 30730 Montpezat
Web : http://www.microlinux.fr
Mail : info at microlinux.fr
T?l. : 04 66 63 10 32
_______________________________________________
CentOS mailing list
CentOS at centos.org
http://lists.centos.org/mailman/listinfo/centos
Niki Kovacs
2015-Feb-18 08:59 UTC
[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?
Le 18/02/2015 09:24, Michael Volz a ?crit :> Hi Niki, > > md127 apparently only uses 81.95GB per disk. Maybe one of the partitions has the wrong size. What's the output of lsblk?[root at nestor:~] # lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 232,9G 0 disk ??sda1 8:1 0 3,9G 0 part ? ??md126 9:126 0 3,9G 0 raid1 [SWAP] ??sda2 8:2 0 200M 0 part ? ??md125 9:125 0 200M 0 raid1 /boot ??sda3 8:3 0 76,4G 0 part ??md127 9:127 0 229G 0 raid5 / sdb 8:16 0 232,9G 0 disk ??sdb1 8:17 0 3,9G 0 part ? ??md126 9:126 0 3,9G 0 raid1 [SWAP] ??sdb2 8:18 0 200M 0 part ? ??md125 9:125 0 200M 0 raid1 /boot ??sdb3 8:19 0 76,4G 0 part ??md127 9:127 0 229G 0 raid5 / sdc 8:32 0 232,9G 0 disk ??sdc1 8:33 0 3,9G 0 part ? ??md126 9:126 0 3,9G 0 raid1 [SWAP] ??sdc2 8:34 0 200M 0 part ? ??md125 9:125 0 200M 0 raid1 /boot ??sdc3 8:35 0 76,4G 0 part ??md127 9:127 0 229G 0 raid5 / sdd 8:48 0 232,9G 0 disk ??sdd1 8:49 0 3,9G 0 part ? ??md126 9:126 0 3,9G 0 raid1 [SWAP] ??sdd2 8:50 0 200M 0 part ? ??md125 9:125 0 200M 0 raid1 /boot ??sdd3 8:51 0 76,4G 0 part ??md127 9:127 0 229G 0 raid5 / Any idea what's going on ? -- Microlinux - Solutions informatiques 100% Linux et logiciels libres 7, place de l'?glise - 30730 Montpezat Web : http://www.microlinux.fr Mail : info at microlinux.fr T?l. : 04 66 63 10 32
g
2015-Feb-18 09:07 UTC
[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?
On 02/18/2015 01:23 AM, Niki Kovacs wrote:> Le 18/02/2015 08:09, Niki Kovacs a ?crit : >> >> Apparently no spare devices have been created. So why do I only >> have 226 GB of disk space under CentOS, when I had roughly 650 GB >> under Slackware? > > An idea just crossed my mind. Could it be that 'df' is reporting a > wrong partition size on the RAID 5 array? And how can I check if > this is the case?i have not built any raid system in linux, but from reading, i saw that there is a little difference from unix. also, between linux flavors, there can be a lot of difference. in a way, case of... who wrote the book and how whoever is reading it. ie, Slackware and CentOS. looking at question of 'reporting', 'df' has various ways of reporting size and might/may/could be what is causing difference. so, until an exact reason/cause is replied... besides '-h', what other arguments for 'df' did you try? df --block-size=1000 df --block-size=1024 df --block-size=K df --block-size=M df --block-size=G df --si df -T instead of reading man df have a look at info coreutils 'df invocation' you can also use 'lsblk', which i find it to be off a bit due to how it rounds of sizes, except when using '-b' 'disk utility', 'system monitor', 'kde info center', 'gparted', are other ways of viewing allocation. much luck finding solution. -- peace out. in a world with out fences, who needs gates. CentOS GNU/Linux 6.6 tc,hago. g .
Niki Kovacs
2015-Feb-18 20:21 UTC
[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?
Le 18/02/2015 09:24, Michael Volz a ?crit :> md127 apparently only uses 81.95GB per disk. Maybe one of the partitions has the wrong size. What's the output of lsblk?I just spent a few hours experimenting with the CentOS 7 installer in a VirtualBox guest with four virtual hard disks. I can now confirm this is a very stupid bug in the (very stupid) installer. Or at least one more random weirdness. Here goes. The new installer is organized around mount points, which have to be defined first. OK, so I first define my mountpoint /boot, set it to 200 MB (which is enough), define it to be RAID level 1 across four disks with an ext2 filesystem. So far so good. Next step is similar, swap mountpoint is 2 GB, also RAID level 1 across four disks. Finally, the / (root partition) mountpoint is supposed to take up the full amount of remaining disk space. In my virtual guest, I defined 4 X 40 GB to fiddle with. The installer shows me something like 38.6 GB, which looks like the remaining space on each disk's partition. Now I define RAID level 5 across four disks... ... and here it comes. Once RAID level 5 is defined, I have to REDEFINE the maximum disk space by putting in a random large number, for example 4 X 40 GB = 160 GB. Because what is meant here is THE TOTAL RESULTING AMOUNT OF DISK SPACE IN THE RAID 5 ARRAY, AND NOT THE MAXIMUM SIZE OF A DISK PARTITION. So once I fill that field with 160 GB, the installer "automagically" sets it to 106.8 GB, which is in effect the maximum available disk space using RAID 5. Usability anyone? Cheers from the sunny South of France, Niki Kovacs -- Microlinux - Solutions informatiques 100% Linux et logiciels libres 7, place de l'?glise - 30730 Montpezat Web : http://www.microlinux.fr Mail : info at microlinux.fr T?l. : 04 66 63 10 32