CentOS - Feb 2014 - suggestions for large filesystem server setup (n * 100 TB)

Hi,

over time the requirements and possibilities regarding filesystems
changed for our users.

currently I'm faced with the question:

What might be a good way to provide one big filesystem for a few users
which could also be enlarged; backuping the data is not the question.

Big in that context is up to couple of 100 TB may be.

O.K. I could install one hardware raid with e.g. N big drives format
with xfs. And export one big share. Done.

On the other hand, e.g. using 60 4 TB Disks in one storage would be a
lot of space, but a nightmare in rebuilding on a disk crash.

Now if the share fills up, my users "complain", that they usually get
a
new share (what is a new raidbox).

From my POV I could e.g. use hardware raidboxes, and use LVM and
filesystem growth options to extend the final share, but what if one of
the boxes crash totally? The whole Filesystem would be gone.

hm.

So how do you handle big filesystems/storages/shares?

	Regards . G?tz

-- 
G?tz Reinicke
IT-Koordinator

Tel. +49 7141 969 82 420
E-Mail goetz.reinicke at filmakademie.de

Filmakademie Baden-W?rttemberg GmbH
Akademiehof 10
71638 Ludwigsburg
www.filmakademie.de

Eintragung Amtsgericht Stuttgart HRB 205016

Vorsitzender des Aufsichtsrats: J?rgen Walter MdL
Staatssekret?r im Ministerium f?r Wissenschaft,
Forschung und Kunst Baden-W?rttemberg

Gesch?ftsf?hrer: Prof. Thomas Schadt

I'd highly recommend getting a NetApp storage device for something that big.

It's more expensive up front, but the amount of heartache/time saved in the
long run is WELL worth it.




On Fri, Feb 28, 2014 at 8:15 AM, G?tz Reinicke - IT Koordinator <
goetz.reinicke at filmakademie.de> wrote:
> Hi,
>
> over time the requirements and possibilities regarding filesystems
> changed for our users.
>
> currently I'm faced with the question:
>
> What might be a good way to provide one big filesystem for a few users
> which could also be enlarged; backuping the data is not the question.
>
> Big in that context is up to couple of 100 TB may be.
>
> O.K. I could install one hardware raid with e.g. N big drives format
> with xfs. And export one big share. Done.
>
> On the other hand, e.g. using 60 4 TB Disks in one storage would be a
> lot of space, but a nightmare in rebuilding on a disk crash.
>
> Now if the share fills up, my users "complain", that they usually
get a
> new share (what is a new raidbox).
>
> From my POV I could e.g. use hardware raidboxes, and use LVM and
> filesystem growth options to extend the final share, but what if one of
> the boxes crash totally? The whole Filesystem would be gone.
>
> hm.
>
> So how do you handle big filesystems/storages/shares?
>
>         Regards . G?tz
>
> --
> G?tz Reinicke
> IT-Koordinator
>
> Tel. +49 7141 969 82 420
> E-Mail goetz.reinicke at filmakademie.de
>
> Filmakademie Baden-W?rttemberg GmbH
> Akademiehof 10
> 71638 Ludwigsburg
> www.filmakademie.de
>
> Eintragung Amtsgericht Stuttgart HRB 205016
>
> Vorsitzender des Aufsichtsrats: J?rgen Walter MdL
> Staatssekret?r im Ministerium f?r Wissenschaft,
> Forschung und Kunst Baden-W?rttemberg
>
> Gesch?ftsf?hrer: Prof. Thomas Schadt
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>
>

-- 
Matt Phelps
System Administrator, Computation Facility
Harvard - Smithsonian Center for Astrophysics
mphelps at cfa.harvard.edu, http://www.cfa.harvard.edu

G?tz Reinicke - IT Koordinator <goetz.reinicke at filmakademie.de> a
?crit?:
> Hi,
>
<snip>

If you are to have an ever-growing volume, I?d suggest some  
distributed FS, like glusterfs, moosefs, lustre?
You need more space ??Add a box.

We do use happily moosefs at work for a couple years (begun with a  
couple TB, now up to 250).

HTH,
Laurent.

On 02/28/2014 08:15 AM, G?tz Reinicke - IT Koordinator
wrote:
> ...
> Big in that context is up to couple of 100 TB may be.
>
> ...
>  From my POV I could e.g. use hardware raidboxes, and use LVM and
> filesystem growth options to extend the final share, but what if one of
> the boxes crash totally? The whole Filesystem would be gone.
>
> ...
> So how do you handle big filesystems/storages/shares?
>
We handle it with EMC Clariion fibre channel units, and LVM.  Always 
make your PV's relatively small, use RAID6 on the array, and mirror it, 
either at the SAN level with MirrorView or at the OS level, using LUNs 
as mdraid components for the PV's.  Today that would become a set of VNX 
systems with SAS on the backend and iSCSI on the front end, but, well, a 
SAN is a SAN.  Surplus FC HBA's are cheap; licenses won't be, nor will 
the array.  But how valuable is that data, or, more to the point, if you 
were to lose all of that 100TB what would it cost you to recreate it?

With this size of data, rolling-your-own should be the last resort, and 
only if you can't afford something properly engineered for high 
availability, like basically anything from NetApp, Nimble, or EMC (among 
others; those are the first three off the top of my head). The value-add 
with these three (among others) is the long track record of reliability 
and the software management tools that make life so much easier when a 
drive or other component inevitably fails.

An enterprise-grade SAN or NAS from a serious vendor is going to cost 
serious money, but you do get what you pay for, again primarily on the 
software side.  Our four Clariions (two CX3-10c's, one CX3-80, and a 
CX4-480) just simply don't go down, and upgrades are very easy and 
reliable, in my experience.  The two CX3-10c's have been online 
continually since mid-2007, and while they are way out of warranty, past 
the normal service life, even, they just run and run and run and run.  
(I even used the redundancy features in one of them to good effect while 
(slowly and carefully!) moving the array from one room to another.... 
long extension power cords, and long fiber jumpers worked to my 
advantage; of course, a stable rack on wheels made it possible.  The 
array stayed up, and no servers lost connectivity to the SAN during the 
move, not that I would recommend it for normal operations, but this 
wasn't a normal operation.)  The storage processor sends alerts when 
drives fault, and a drive fault is an easy hotswap with the DAE and the 
drive clearly identified at the front of the array.  Everything (drives, 
power supplies, storage processor modules, LCC's) except a whole DAE or 
storage processor enclosure is hotswap, and I haven't had a DAE fault 
yet that required pulling the whole DAE out of service.

If you do roll-your-own, do not use consumer-class drives.  One reason 
NetApp and the rest charge so much for drives is due to the extra 
testing and sometimes the custom firmware that goes into the drives (in 
a nutshell, you do NOT want the drive doing its own error recovery, 
that's the array storage processor's job!).

Those are my opinions and experience.  YMMV.

On Fri, Feb 28, 2014 at 7:15 AM, G?tz Reinicke - IT
Koordinator
>
> What might be a good way to provide one big filesystem for a few users
> which could also be enlarged; backuping the data is not the question.
Really?  Does that mean you already have a backup, don't care if you
lose it, or that you want something with redundancy built in (which
still isn't quite the same as having a backup).
> So how do you handle big filesystems/storages/shares?
The easy/expensive way is to buy an appliance like a NetApp and mount
it via nfs.  And use its own tools for snapshots, raid management,
etc.    I don't have any experience with it, but from what I've read I
would say that 'ceph' is the up-and-coming way of doing your own
distributed/redundant storage although I'm not sure I'd trust the
pieces that turn it into a filesystem yet.

-- 
  Les Mikesell
     lesmikesell at gmail.com

----- Original Message -----
| Hi,
| 
| over time the requirements and possibilities regarding filesystems
| changed for our users.
| 
| currently I'm faced with the question:
| 
| What might be a good way to provide one big filesystem for a few
| users
| which could also be enlarged; backuping the data is not the question.
| 
| Big in that context is up to couple of 100 TB may be.
| 
| O.K. I could install one hardware raid with e.g. N big drives format
| with xfs. And export one big share. Done.
| 
| On the other hand, e.g. using 60 4 TB Disks in one storage would be a
| lot of space, but a nightmare in rebuilding on a disk crash.
| 
| Now if the share fills up, my users "complain", that they usually
get
| a
| new share (what is a new raidbox).
| 
| From my POV I could e.g. use hardware raidboxes, and use LVM and
| filesystem growth options to extend the final share, but what if one
| of
| the boxes crash totally? The whole Filesystem would be gone.
| 
| hm.
| 
| So how do you handle big filesystems/storages/shares?
| 
| 	Regards . G?tz

My personal view is that you don't want any single machine to contain a
100TB file system.  You'd be best served using a distributed file system
such as GlusterFS or Lustre.  If you insist on having a single machine with a
100TB file system on it, make sure that you install at least 300GB of memory or
more if you think you'll ever have to perform a file system check on it. 
You're going to need it.

Note, it's that that difficult or expensive to build a supermicro box with
48 x 4TB drives to scale out the size that you need with GlusterFS, however,
building it is the easiest part.  It's maintaining it and troubleshooting it
when things go wrong.  Choosing a platform to support also depends on I/O access
patterns, number of clients, connectivity (IB vs Ethernet vs iSCSI/FC/AoE,etc).

Currently we're not using any clustered file system for our data access.  We
have a single NFS machine which is the "front-end" to the data.  It
contains a whole bunch of symlinks to other NFS servers (Dell R720XD/36TB each)
which the machines automount.  This is really simple to maintain and if we want
to do replication on a per volume level we can.  We are looking into GlusterFS
though for certain things.

-- 
James A. Peltier
Manager, IT Services - Research Computing Group
Simon Fraser University - Burnaby Campus
Phone   : 778-782-6573
Fax     : 778-782-3045
E-Mail  : jpeltier at sfu.ca
Website : http://www.sfu.ca/itservices

"Around here, however, we don?t look backwards for very long.  We KEEP
MOVING FORWARD, opening up new doors and doing things because we?re curious and
curiosity keeps leading us down new paths." - Walt Disney

On 28.02.2014 13:15, G?tz Reinicke - IT Koordinator
wrote:
> Hi,
> 
> over time the requirements and possibilities regarding filesystems
> changed for our users.
> 
> currently I'm faced with the question:
> 
> What might be a good way to provide one big filesystem for a few users
> which could also be enlarged; backuping the data is not the question.
I use GlusterFS. http://www.gluster.org/

HTH
Lucian

-- 
Sent from the Delta quadrant using Borg technology!

Nux!
www.nux.ro