thr3ads.net - CentOS - [CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares? [Feb 2015]

If this information is useful, please help other people find it:
Share via:

Niki Kovacs

2015-Feb-18 07:09 UTC

[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?

Hi,

I just replaced Slackware64 14.1 running on my office's HP Proliant 
Microserver with a fresh installation of CentOS 7.

The server has 4 x 250 GB disks.

Every disk is configured like this :

	* 200 MB /dev/sdX1 for /boot
	* 4 GB /dev/sdX2 for swap
	* 248 GB /dev/sdX3 for /

There are supposed to be no spare devices.

/boot and swap are all supposed to be assembled in RAID level 1 across 4 
disks.

The / partition is supposed to be assembled in RAID level 5 across 4 disks.

With Slackware I created the arrays manually like this:

   # mdadm --create /dev/md1 --level=1 --raid-devices=4 --metadata=0.90
     /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
   # mdadm --create /dev/md2 --level=1 --raid-devices=4 --metadata=0.90
     /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2
   # mdadm --create /dev/md3 --level=5 --raid-devices=4 --metadata=0.90
     /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3

Using this setup, I had 650 MB of disk space on /dev/md3.

Now I tried to do the same thing with CentOS 7. Everything seemed to 
work at first, but here's what I got now:

[root at nestor:~] # df -h
Sys. de fichiers Taille Utilis? Dispo Uti% Mont? sur
/dev/md127         226G    1,1G  213G   1% /
devtmpfs           1,4G       0  1,4G   0% /dev
tmpfs              1,4G       0  1,4G   0% /dev/shm
tmpfs              1,4G    8,5M  1,4G   1% /run
tmpfs              1,4G       0  1,4G   0% /sys/fs/cgroup
/dev/md125         194M     80M  101M  45% /boot
/dev/sde1          917G     88M  871G   1% /mnt

The root partition (/dev/md127) only shows 226 G of space. So where has 
everything gone?

[root at nestor:~] # cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : active raid1 sdc2[2] sdd2[3] sdb2[1] sda2[0]
       204736 blocks super 1.0 [4/4] [UUUU]

md126 : active raid1 sdd1[3] sdc1[2] sdb1[1] sda1[0]
       4095936 blocks super 1.2 [4/4] [UUUU]

md127 : active raid5 sdc3[2] sdb3[1] sdd3[4] sda3[0]
       240087552 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] 
[UUUU]
       bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>

[root at nestor:~] # mdadm -D /dev/md127
/dev/md127:
         Version : 1.2
   Creation Time : Wed Feb 18 06:49:01 2015
      Raid Level : raid5
      Array Size : 240087552 (228.97 GiB 245.85 GB)
   Used Dev Size : 80029184 (76.32 GiB 81.95 GB)
    Raid Devices : 4
   Total Devices : 4
     Persistence : Superblock is persistent

   Intent Bitmap : Internal

     Update Time : Wed Feb 18 08:04:26 2015
           State : active
  Active Devices : 4
Working Devices : 4
  Failed Devices : 0
   Spare Devices : 0

          Layout : left-symmetric
      Chunk Size : 512K

            Name : localhost:root
            UUID : cfc13fe9:8fa811d8:85649402:58c4846e
          Events : 4703

     Number   Major   Minor   RaidDevice State
        0       8        3        0      active sync   /dev/sda3
        1       8       19        1      active sync   /dev/sdb3
        2       8       35        2      active sync   /dev/sdc3
        4       8       51        3      active sync   /dev/sdd3

Apparently no spare devices have been created. So why do I only have 226 
GB of disk space under CentOS, when I had roughly 650 GB under Slackware?

I'm a bit lost here. Any suggestions?

Cheers,

Niki

-- 
Microlinux - Solutions informatiques 100% Linux et logiciels libres
7, place de l'?glise - 30730 Montpezat
Web  : http://www.microlinux.fr
Mail : info at microlinux.fr
T?l. : 04 66 63 10 32

Niki Kovacs

2015-Feb-18 07:23 UTC

head link

[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?

Le 18/02/2015 08:09, Niki Kovacs a ?crit :
>
> Apparently no spare devices have been created. So why do I only have 226
> GB of disk space under CentOS, when I had roughly 650 GB under Slackware?
>
An idea just crossed my mind. Could it be that 'df' is reporting a wrong
partition size on the RAID 5 array? And how can I check if this is the case?


-- 
Microlinux - Solutions informatiques 100% Linux et logiciels libres
7, place de l'?glise - 30730 Montpezat
Web  : http://www.microlinux.fr
Mail : info at microlinux.fr
T?l. : 04 66 63 10 32

Michael Volz

2015-Feb-18 08:24 UTC

head link

[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?

Hi Niki,

md127 apparently only uses 81.95GB per disk. Maybe one of the partitions has the
wrong size. What's the output of lsblk?

Regards
Michael

----- Urspr?ngliche Mail -----
Von: "Niki Kovacs" <info at microlinux.fr>
An: "CentOS mailing list" <CentOS at centos.org>
Gesendet: Mittwoch, 18. Februar 2015 08:09:13
Betreff: [CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?

Hi,

I just replaced Slackware64 14.1 running on my office's HP Proliant 
Microserver with a fresh installation of CentOS 7.

The server has 4 x 250 GB disks.

Every disk is configured like this :

	* 200 MB /dev/sdX1 for /boot
	* 4 GB /dev/sdX2 for swap
	* 248 GB /dev/sdX3 for /

There are supposed to be no spare devices.

/boot and swap are all supposed to be assembled in RAID level 1 across 4 
disks.

The / partition is supposed to be assembled in RAID level 5 across 4 disks.

With Slackware I created the arrays manually like this:

   # mdadm --create /dev/md1 --level=1 --raid-devices=4 --metadata=0.90
     /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1
   # mdadm --create /dev/md2 --level=1 --raid-devices=4 --metadata=0.90
     /dev/sda2 /dev/sdb2 /dev/sdc2 /dev/sdd2
   # mdadm --create /dev/md3 --level=5 --raid-devices=4 --metadata=0.90
     /dev/sda3 /dev/sdb3 /dev/sdc3 /dev/sdd3

Using this setup, I had 650 MB of disk space on /dev/md3.

Now I tried to do the same thing with CentOS 7. Everything seemed to 
work at first, but here's what I got now:

[root at nestor:~] # df -h
Sys. de fichiers Taille Utilis? Dispo Uti% Mont? sur
/dev/md127         226G    1,1G  213G   1% /
devtmpfs           1,4G       0  1,4G   0% /dev
tmpfs              1,4G       0  1,4G   0% /dev/shm
tmpfs              1,4G    8,5M  1,4G   1% /run
tmpfs              1,4G       0  1,4G   0% /sys/fs/cgroup
/dev/md125         194M     80M  101M  45% /boot
/dev/sde1          917G     88M  871G   1% /mnt

The root partition (/dev/md127) only shows 226 G of space. So where has 
everything gone?

[root at nestor:~] # cat /proc/mdstat
Personalities : [raid1] [raid6] [raid5] [raid4]
md125 : active raid1 sdc2[2] sdd2[3] sdb2[1] sda2[0]
       204736 blocks super 1.0 [4/4] [UUUU]

md126 : active raid1 sdd1[3] sdc1[2] sdb1[1] sda1[0]
       4095936 blocks super 1.2 [4/4] [UUUU]

md127 : active raid5 sdc3[2] sdb3[1] sdd3[4] sda3[0]
       240087552 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] 
[UUUU]
       bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>

[root at nestor:~] # mdadm -D /dev/md127
/dev/md127:
         Version : 1.2
   Creation Time : Wed Feb 18 06:49:01 2015
      Raid Level : raid5
      Array Size : 240087552 (228.97 GiB 245.85 GB)
   Used Dev Size : 80029184 (76.32 GiB 81.95 GB)
    Raid Devices : 4
   Total Devices : 4
     Persistence : Superblock is persistent

   Intent Bitmap : Internal

     Update Time : Wed Feb 18 08:04:26 2015
           State : active
  Active Devices : 4
Working Devices : 4
  Failed Devices : 0
   Spare Devices : 0

          Layout : left-symmetric
      Chunk Size : 512K

            Name : localhost:root
            UUID : cfc13fe9:8fa811d8:85649402:58c4846e
          Events : 4703

     Number   Major   Minor   RaidDevice State
        0       8        3        0      active sync   /dev/sda3
        1       8       19        1      active sync   /dev/sdb3
        2       8       35        2      active sync   /dev/sdc3
        4       8       51        3      active sync   /dev/sdd3

Apparently no spare devices have been created. So why do I only have 226 
GB of disk space under CentOS, when I had roughly 650 GB under Slackware?

I'm a bit lost here. Any suggestions?

Cheers,

Niki

-- 
Microlinux - Solutions informatiques 100% Linux et logiciels libres
7, place de l'?glise - 30730 Montpezat
Web  : http://www.microlinux.fr
Mail : info at microlinux.fr
T?l. : 04 66 63 10 32
_______________________________________________
CentOS mailing list
CentOS at centos.org
http://lists.centos.org/mailman/listinfo/centos

Niki Kovacs

2015-Feb-18 08:59 UTC

head link

[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?

Le 18/02/2015 09:24, Michael Volz a ?crit :> Hi Niki,
>
> md127 apparently only uses 81.95GB per disk. Maybe one of the partitions
has the wrong size. What's the output of lsblk?
[root at nestor:~] # lsblk
NAME      MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda         8:0    0 232,9G  0 disk
??sda1      8:1    0   3,9G  0 part
? ??md126   9:126  0   3,9G  0 raid1 [SWAP]
??sda2      8:2    0   200M  0 part
? ??md125   9:125  0   200M  0 raid1 /boot
??sda3      8:3    0  76,4G  0 part
   ??md127   9:127  0   229G  0 raid5 /
sdb         8:16   0 232,9G  0 disk
??sdb1      8:17   0   3,9G  0 part
? ??md126   9:126  0   3,9G  0 raid1 [SWAP]
??sdb2      8:18   0   200M  0 part
? ??md125   9:125  0   200M  0 raid1 /boot
??sdb3      8:19   0  76,4G  0 part
   ??md127   9:127  0   229G  0 raid5 /
sdc         8:32   0 232,9G  0 disk
??sdc1      8:33   0   3,9G  0 part
? ??md126   9:126  0   3,9G  0 raid1 [SWAP]
??sdc2      8:34   0   200M  0 part
? ??md125   9:125  0   200M  0 raid1 /boot
??sdc3      8:35   0  76,4G  0 part
   ??md127   9:127  0   229G  0 raid5 /
sdd         8:48   0 232,9G  0 disk
??sdd1      8:49   0   3,9G  0 part
? ??md126   9:126  0   3,9G  0 raid1 [SWAP]
??sdd2      8:50   0   200M  0 part
? ??md125   9:125  0   200M  0 raid1 /boot
??sdd3      8:51   0  76,4G  0 part
   ??md127   9:127  0   229G  0 raid5 /

Any idea what's going on ?

-- 
Microlinux - Solutions informatiques 100% Linux et logiciels libres
7, place de l'?glise - 30730 Montpezat
Web  : http://www.microlinux.fr
Mail : info at microlinux.fr
T?l. : 04 66 63 10 32

2015-Feb-18 09:07 UTC

head link

[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?

On 02/18/2015 01:23 AM, Niki Kovacs wrote:> Le 18/02/2015 08:09, Niki Kovacs a ?crit :
>>
>> Apparently no spare devices have been created. So why do I only
>> have 226 GB of disk space under CentOS, when I had roughly 650 GB
>> under Slackware?
>
> An idea just crossed my mind. Could it be that 'df' is reporting a
> wrong partition size on the RAID 5 array? And how can I check if
> this is the case?
i have not built any raid system in linux, but from reading, i saw
that there is a little difference from unix.

also, between linux flavors, there can be a lot of difference.

in a way, case of...

  who wrote the book and how whoever is reading it.

ie, Slackware and CentOS.

looking at question of 'reporting', 'df' has various ways of
reporting
size and might/may/could be what is causing difference.

so, until an exact reason/cause is replied...

besides '-h', what other arguments for 'df' did you try?

  df --block-size=1000
  df --block-size=1024
  df --block-size=K
  df --block-size=M
  df --block-size=G
  df --si
  df -T

instead of reading

  man df

have a look at

  info coreutils 'df invocation'

you can also use 'lsblk', which i find it to be off a bit due to
how it rounds of sizes, except when using '-b'

'disk utility', 'system monitor', 'kde info center',
'gparted', are
other ways of viewing allocation.

much luck finding solution.

-- 

peace out.

in a world with out fences, who needs gates.

CentOS GNU/Linux 6.6

tc,hago.

g
.

Niki Kovacs

2015-Feb-18 20:21 UTC

head link

[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?

Le 18/02/2015 09:24, Michael Volz a ?crit :> md127 apparently only uses 81.95GB per disk. Maybe one of the partitions
has the wrong size. What's the output of lsblk?
I just spent a few hours experimenting with the CentOS 7 installer in a
VirtualBox guest with four virtual hard disks. I can now confirm this is
a very stupid bug in the (very stupid) installer. Or at least one more
random weirdness. Here goes.

The new installer is organized around mount points, which have to be
defined first. OK, so I first define my mountpoint /boot, set it to 200
MB (which is enough), define it to be RAID level 1 across four disks
with an ext2 filesystem. So far so good.

Next step is similar, swap mountpoint is 2 GB, also RAID level 1 across
four disks.

Finally, the / (root partition) mountpoint is supposed to take up the
full amount of remaining disk space. In my virtual guest, I defined 4 X
40 GB to fiddle with. The installer shows me something like 38.6 GB,
which looks like the remaining space on each disk's partition.

Now I define RAID level 5 across four disks...

... and here it comes.

Once RAID level 5 is defined, I have to REDEFINE the maximum disk space
by putting in a random large number, for example 4 X 40 GB = 160 GB.
Because what is meant here is THE TOTAL RESULTING AMOUNT OF DISK SPACE
IN THE RAID 5 ARRAY, AND NOT THE MAXIMUM SIZE OF A DISK PARTITION. So
once I fill that field with 160 GB, the installer "automagically" sets
it to 106.8 GB, which is in effect the maximum available disk space
using RAID 5.

Usability anyone?

Cheers from the sunny South of France,

Niki Kovacs
--
Microlinux - Solutions informatiques 100% Linux et logiciels libres
7, place de l'?glise - 30730 Montpezat
Web : http://www.microlinux.fr
Mail : info at microlinux.fr
T?l. : 04 66 63 10 32

Possibly Parallel Threads

Search for more apparently analagous threads

CentOS - Feb 2015 - CentOS 7: software RAID 5 array with 4 disks and no spares?

[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?

[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?

[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?

[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?

[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?

[CentOS] CentOS 7: software RAID 5 array with 4 disks and no spares?

Possibly Parallel Threads