CentOS - Sep 2012 - Advice on partitioning a Dell MD1200 disk array

Hi,

I've just got possession of a Dell PE R720 with 2 MD1200 disk 
enclosures.

Both MD1200 are fully populated with 12 x 3 TB disks

The system will basically be a student file-server running CentOS 6.x  
serving various size files from small c programs to multi gigabyte 
audio and video files over GB ethernet.

The first MD1200 will be configured as the NFS disk. The requirements 
are for 6 fixed equally sized partitions, one for each cohort of 
students. For this I was thinking of splitting the MD1200 into 2 RAID5 
arrays with a hot spare each. Then partitioning each into 3 ext4 
partitions.

The second MD1200 will be used to backup the first, using BackupPC and 
for other storage purposes.

As I won't know the storage requirements for the "backup
partition"
and they will probably change over time anyway. I was thinking of 
using LVM for it. So how to partition the MD1200 for LVM. I don't want 
to put all 12 disks in  a RAID5 and put a LVM volume on it. Can I 
split it into 2 RAID5 and have a LVM volume spanning both.

Any suggestions. 

Just remember I'm due to retire at the end of this month so this will 
be my last big job for the Dept. And due to financial constraints I 
will not be replaced. So I will be handing this machine over to a co-
worker who is basically a Windoze admin with only a basic knowledge of 
Linux so nothing too fancy.  ;-)

Thanks,

Tony


--

Tony Molloy

CTO, Dept. of Comp. Sci.
University of Limerick
Limerick.
Ireland

Hi Tony,

because I suggest just something very general I post off list :)

From my POV as I'm currently facing similar setups with different
hardware rolling back from fine granular setups to simple 'bigger' less
complex configurations. (we do have 6 iscsi storages from 2TB (sun ZFS)
up to 32 TB)

	keep it small and simple! :)

I think you are very familiar with the general problems of big HW raids
and big filesystems like rebuild or check times, but splitting up and
adding more complex layers like multiple raids joining in lvm etc. makes
debugging and general handling very hard.

On the other hand, I checked and read a lot about filesystems the last
days being faced with serving user windows samba profiles with lot of
small files and big video/audio data etc.

Long story short:

I usually do one raidvolume per hardware raid box; e.g. we use 16*1TB
drives. Raid6 or Raid5 with spare. I did not notice big performance
differences.

I use LVM to make partitions or I prefer using just one big partition.

I tried xfs and ext4 and will go with ext4 as some test went better for
my setup and from what I read it looks not bad :)

I think you can combine block level devices (like multiple raid boxes)
by LVM into one bigger LV.

And last but not least: The CPU/RAM/Network of the host serving the
files is also very important! :)

I noticed, that the same iscsi storage got about 70MB/s on a new server
(xeon multicore), while on the old fileserver it just got up to 40MB/s.



	my2cents :) regards . G?tz



May be worth reading:

http://www.techforce.com.br/news/linux_blog/lvm_raid_xfs_ext3_tuning_for_small_files_parallel_i_o_on_debian#.UEPSI1RqYso

http://monolight.cc/2011/02/linux-filesystems-small-file-performance-on-hdds/

http://unix.stackexchange.com/questions/28756/what-is-the-most-high-performance-linux-filesystem-for-storing-a-lot-of-small-fi

https://wiki.archlinux.org/index.php/Ext3

https://wiki.archlinux.org/index.php/Ext4#Tips_and_tricks



Am 04.09.12 13:10, schrieb Tony Molloy:
> 
> Hi,
> 
> I've just got possession of a Dell PE R720 with 2 MD1200 disk 
> enclosures.
> 
> Both MD1200 are fully populated with 12 x 3 TB disks
> 
> The system will basically be a student file-server running CentOS 6.x  
> serving various size files from small c programs to multi gigabyte 
> audio and video files over GB ethernet.
> 
> The first MD1200 will be configured as the NFS disk. The requirements 
> are for 6 fixed equally sized partitions, one for each cohort of 
> students. For this I was thinking of splitting the MD1200 into 2 RAID5 
> arrays with a hot spare each. Then partitioning each into 3 ext4 
> partitions.
> 
> The second MD1200 will be used to backup the first, using BackupPC and 
> for other storage purposes.
> 
> As I won't know the storage requirements for the "backup
partition"
> and they will probably change over time anyway. I was thinking of 
> using LVM for it. So how to partition the MD1200 for LVM. I don't want 
> to put all 12 disks in  a RAID5 and put a LVM volume on it. Can I 
> split it into 2 RAID5 and have a LVM volume spanning both.
> 
> Any suggestions. 
> 
> Just remember I'm due to retire at the end of this month so this will 
> be my last big job for the Dept. And due to financial constraints I 
> will not be replaced. So I will be handing this machine over to a co-
> worker who is basically a Windoze admin with only a basic knowledge of 
> Linux so nothing too fancy.  ;-)
> 
> Thanks,
> 
> Tony
> 
> 
> --
> 
> Tony Molloy
> 
> CTO, Dept. of Comp. Sci.
> University of Limerick
> Limerick.
> Ireland 
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
> 

-- 
G?tz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
Fax  +49 7141 969 55 420
E-Mail goetz.reinicke at filmakademie.de

Filmakademie Baden-W?rttemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: J?rgen Walter MdL
Staatssekret?r im Ministerium f?r Wissenschaft,
Forschung und Kunst Baden-W?rttemberg

Gesch?ftsf?hrer: Prof. Thomas Schadt

Tony Molloy wrote the following on 9/4/2012 6:10 AM:
> Hi,
>
> I've just got possession of a Dell PE R720 with 2 MD1200 disk
> enclosures.
>
> Both MD1200 are fully populated with 12 x 3 TB disks
>
> The system will basically be a student file-server running CentOS 6.x
> serving various size files from small c programs to multi gigabyte
> audio and video files over GB ethernet.
>
> The first MD1200 will be configured as the NFS disk. The requirements
> are for 6 fixed equally sized partitions, one for each cohort of
> students. For this I was thinking of splitting the MD1200 into 2 RAID5
> arrays with a hot spare each. Then partitioning each into 3 ext4
> partitions.
I think that sounds like a balanced approach. With 12 SATA drives you're 
bound to run into failures during the life of these devices (3-5 yrs). 
By splitting the disks into 2 arrays, a failure of one disk will only 
degrade 1 array, affecting less users. Rebuild times will be reduced if 
there is a failure and performance for multi-user workloads should be 
increased by separating into multiple arrays.
>
> The second MD1200 will be used to backup the first, using BackupPC and
> for other storage purposes.
>
> As I won't know the storage requirements for the "backup
partition"
> and they will probably change over time anyway. I was thinking of
> using LVM for it. So how to partition the MD1200 for LVM. I don't want
> to put all 12 disks in  a RAID5 and put a LVM volume on it. Can I
> split it into 2 RAID5 and have a LVM volume spanning both.
I've never found any value in LVM. To me, it adds another layer of 
complexity or room for bugs to creep in. Arrays can be resized, GPT or 
MSDOS partitions can be resized, file systems can be resized, your 
hardware can do the spanning that LVM provides, why would you want to 
add another layer of abstraction? For similar reasons, I would stick to 
a file system that comes stock with your OS, ext4 or XFS should be fine, 
as they are both well supported in CentOS.
> Any suggestions.
>
> Just remember I'm due to retire at the end of this month so this will
> be my last big job for the Dept. And due to financial constraints I
> will not be replaced. So I will be handing this machine over to a co-
> worker who is basically a Windoze admin with only a basic knowledge of
> Linux so nothing too fancy.  ;-)
>
At first, I was going to recommend 1 hot spare, but if you think the new 
admin may not have as much time, or you want things a little more 
simple, then making two identical arrays is probably going to be easier. 
If he's not as familiar with your hardware, then I would print out a 
cheat sheet on how to rebuild the array, check the status (MegaCLI?, 
OpenManage?), etc. As always, document what you have, how you put it 
together, and who to call or where to look if he's stuck.

--Blake

>
> The second MD1200 will be used to backup the first, using BackupPC and
> for other storage purposes.
>
> As I won't know the storage requirements for the "backup
partition"
> and they will probably change over time anyway. I was thinking of
> using LVM for it. So how to partition the MD1200 for LVM. I don't want
> to put all 12 disks in  a RAID5 and put a LVM volume on it. Can I
> split it into 2 RAID5 and have a LVM volume spanning both.
>
Just a thought about the backup volume.

Because it going to be used for backups and your retirement (congrats!) I
would configure one partition and no volume management on top.  Why? 
Because backups are like insurance policies.  That are not fun and no one
reviews them, but when you need them, you really hope that everything is
in order.  So you don't want the next admin to find out several months or
years down the road that the backup software ran out of space and has been
failing to back anything up for the previous few months.

On Sep 4, 2012, at 5:10 AM, Tony Molloy <tony.molloy at ul.ie> wrote:
> Just remember I'm due to retire at the end of this month so this will 
> be my last big job for the Dept. And due to financial constraints I 
> will not be replaced. So I will be handing this machine over to a co-
> worker who is basically a Windoze admin with only a basic knowledge of 
> Linux so nothing too fancy.  ;-)
Hand him the machine and tell him to load Windows on it or whatever he wants to
maintain for the next X years, relax for 30 days and enjoy retirement.  It's
his problem now.  :-) :-) ;-)

Nate