CentOS - Oct 2011 - You suggestion for 'big' filesystem management Best Practice?

Hi,

we plan to set up a big file storage for media files like uncompressed
movies from student film projects, dvd images etc.

It should be some sort of archive and will not bee accessed by more than
may be 5 people at the same time.

The iSCSI RAID we have is about 26TB netto and I'm again faced with the
question: How many partitions, which filesystem, which mount options etc.

For the User it would be the most simpel thing, to have one big
filesystem she/he could fill with all the data and dont has to search
e.g. on multiple volumes.

On the other hand, if one big filesystem crashes or has do be checked it
will destroy a lot of data or the check will take hours ...


Any suggestions pro or cons are welcome! :-)

My favourite for now is 3 to 4 filesystems with the default ext4
settings. (Redhat EL 5.7, may be soon 6.1)

Thanks and best regards. G?tz
-- 
G?tz Reinicke
IT-Koordinator

Tel. +49 7141 969 420
Fax  +49 7141 969 55 420
E-Mail goetz.reinicke at filmakademie.de

Filmakademie Baden-W?rttemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats:
J?rgen Walter MdL
Staatssekret?r im Ministerium f?r Wissenschaft,
Forschung und Kunst Baden-W?rttemberg

Gesch?ftsf?hrer:
Prof. Thomas Schadt

Greetings,

2011/10/28 G?tz Reinicke <goetz.reinicke at
filmakademie.de>:
> Hi,
>
>
> The iSCSI RAID we have is about 26TB netto and I'm again faced with the
> question: How many partitions, which filesystem, which mount options etc.
>
> My favourite for now is 3 to 4 filesystems with the default ext4
> settings. (Redhat EL 5.7, may be soon 6.1)
>

You should seriously consider XFS (supported natively in RHEL and
hence Centos 6) for any single filesystem >16TB. (EXT4 supports only
upto 16TB)

If you are going to have an RHCS for HA, then GFS2. But I cant comment
on its performance for streaming applications etc.

-- 
Regards,

Rajagopal

From: G?tz Reinicke <goetz.reinicke at filmakademie.de>
> The iSCSI RAID we have is about 26TB netto and I'm again faced with the
> question: How many partitions, which filesystem, which mount options etc.
> For the User it would be the most simpel thing, to have one big
> filesystem she/he could fill with all the data and dont has to search
> e.g. on multiple volumes.
> On the other hand, if one big filesystem crashes or has do be checked it
> will destroy a lot of data or the check will take hours ...
Splitting the space, if you have the option, has advantages...
You already mentioned fsck.
You could assign different partitions to different groups of people.
You could, depending on your RAID level, create several arrays on 
different drives to limit disk contention (but you will lose either some 
space and/or some speed)...

JD

2011/10/28 G?tz Reinicke <goetz.reinicke at filmakademie.de>
> The iSCSI RAID we have is about 26TB netto and I'm again faced with the
> question: How many partitions, which filesystem, which mount options etc.
>
My vote goes to XFS, if only one server needs acces to the LUN's; and GPFS
(not GFS) if you need a cluster filesystem.

BR Bent (130 TB in one XFS installation, and 600 TB in one gpfs cluster)

> For the User it would be the most simpel thing, to have one big
> filesystem she/he could fill with all the data and dont has to search
> e.g. on multiple volumes.
>
> On the other hand, if one big filesystem crashes or has do be checked it
> will destroy a lot of data or the check will take hours ...
>
>
> Any suggestions pro or cons are welcome! :-)
>
> My favourite for now is 3 to 4 filesystems with the default ext4
> settings. (Redhat EL 5.7, may be soon 6.1)
>
> Thanks and best regards. G?tz
> --
> G?tz Reinicke
> IT-Koordinator
>
> Tel. +49 7141 969 420
> Fax  +49 7141 969 55 420
> E-Mail goetz.reinicke at filmakademie.de
>
> Filmakademie Baden-W?rttemberg GmbH
> Akademiehof 10
> 71638 Ludwigsburg
> www.filmakademie.de
>
> Eintragung Amtsgericht Stuttgart HRB 205016
>
> Vorsitzender des Aufsichtsrats:
> J?rgen Walter MdL
> Staatssekret?r im Ministerium f?r Wissenschaft,
> Forschung und Kunst Baden-W?rttemberg
>
> Gesch?ftsf?hrer:
> Prof. Thomas Schadt
>
>
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
>

On Fri, 2011-10-28 at 10:34 +0200, G?tz Reinicke wrote:
> we plan to set up a big file storage for media files like uncompressed
> movies from student film projects, dvd images etc.
> It should be some sort of archive and will not bee accessed by more than
> may be 5 people at the same time.
> The iSCSI RAID we have is about 26TB netto and I'm again faced with the
> question: How many partitions, which filesystem, which mount options
etc.> For the User it would be the most simpel thing, to have one big
> filesystem she/he could fill with all the data and dont has to search
> e.g. on multiple volumes.
Use LVM.

Divide it into numerous partitions. 
Initialize the partitions as Physical Volumes.
Create a Volume Group containing the Physical Volumes.
Then create the volumes you allocate storage as using Logical Volumes.
Your LV can span as many, or as few, PVs as you like.

To the SAN each PV is a partition, you can move it, delete it, etc..
In the OS you can move data between PVs, migrate LVs between PVs, etc..
Even connect to another SAN, attach PVs there to you volume group, and
migrate the LVs to those PVs - this migrating data from one SAN to
another.  Then drop the PVs on the old SAN from the VG.