Hello listmates, I would like to build a 12-15 TB RAID 5 data server to run under ContOS. Any recommendations as far as hardware, configuration, etc? Thanks. Boris.
On 4/13/2010 1:05 PM, Boris Epstein wrote:> Hello listmates, > > I would like to build a 12-15 TB RAID 5 data server to run under > ContOS. Any recommendations as far as hardware, configuration, etc? > > Thanks. > > Boris. > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos >Chassis - CSE-836A-R1200B Supermicro SC836 A-R1200B - Rack-mountable - 3U - SATA/SAS - hot-swap - power supply 1200 Watt RAID Card - 3ware 9650SE-16ML-SGL 9650SE-16ML-SGL RAID 0/1/5/6/10/50 16CH SATA II PCIE 256MB ECC DDR2 - PCI Express x8 - Up to 300MBps - 4 x SATA x4 Serial ATA/300 - Serial ATA BBU Module for RAID card - 3ware BBU-MODULE-03 Pick the cpu(s) and motherboard to fit the chassis. Obviously go with ECC ram and ONLY enterprise grade hard drives. To ensure compatibility check with 3ware to see which drives they recommend. Areca RAID cards will get you a little better performance but the module for the 9650SE series of 3ware cards is included with the Centos kernel. Getting the Areca driver going is a bit more work, but nothing that would be considered a huge hurdle for a competent sysadmin. Also, if you're looking for advice on Areca products call their Tekram contact in the USA. Their other distributors have been less than stellar on answering pre-sales questions. -- Ryan Manikowski ]] Devision Media Services LLC [[ www.devision.us ryan at devision.us | 716.771.2282 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://lists.centos.org/pipermail/centos/attachments/20100413/07ef1344/attachment-0002.html>
On Tue, Apr 13, 2010 at 1:14 PM, Ryan Manikowski <ryan at devision.us> wrote:> On 4/13/2010 1:05 PM, Boris Epstein wrote: > > Hello listmates, > > I would like to build a 12-15 TB RAID 5 data server to run under > ContOS. Any recommendations as far as hardware, configuration, etc? > > Thanks. > > Boris. > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos > > > Chassis - CSE-836A-R1200B?Supermicro SC836 A-R1200B - Rack-mountable - 3U - > SATA/SAS - hot-swap - power supply 1200 Watt > > RAID Card - 3ware 9650SE-16ML-SGL 9650SE-16ML-SGL RAID 0/1/5/6/10/50 16CH > SATA II PCIE 256MB ECC DDR2 - PCI Express x8 - Up to 300MBps - 4 x SATA x4 > Serial ATA/300 - Serial ATA > > BBU Module for RAID card - 3ware BBU-MODULE-03 > > > Pick the cpu(s) and motherboard to fit the chassis. Obviously go with ECC > ram and ONLY enterprise grade hard drives. To ensure compatibility check > with 3ware to see which drives they recommend. Areca RAID cards will get you > a little better performance but the module for the 9650SE series of 3ware > cards is included with the Centos kernel. Getting the Areca driver going is > a bit more work, but nothing that would be considered a huge hurdle for a > competent sysadmin. Also, if you're looking for advice on Areca products > call their Tekram contact in the USA. Their other distributors have been > less than stellar on answering pre-sales questions. > > > -- > Ryan Manikowski > > > ]] Devision Media Services LLC [[ > www.devision.us > ryan at devision.us | 716.771.2282 > > _______________________________________________ > CentOS mailing list > CentOS at centos.org > http://lists.centos.org/mailman/listinfo/centos > >Thanks Ryan! So, basically, I take it you've had positive experience with that Supermicro box. It does look good and seems fairly inexpensive, too. Boris.
>I would like to build a 12-15 TB RAID 5 data server to run under >ContOS. Any recommendations as far as hardware, configuration, etc?This comes up often:) Things to keep in mind when using large discs if you suffer a failure, while rebuilding in a degraded state it's not impossible or unlikely to drop another disc loosing it all. Rebuild times especially on busy arrays with large discs take lots of time...
2010/4/13 Boris Epstein <borepstein at gmail.com>:> Hello listmates, > > I would like to build a 12-15 TB RAID 5 data server to run under > ContOS. Any recommendations as far as hardware, configuration, etc?how about hitachi SMS 100 ? it is about that size and cost effective iscsi solution? -- Eero
On Tue, 13 Apr 2010, Boris Epstein wrote:> Hello listmates, > > I would like to build a 12-15 TB RAID 5 data server to run under > ContOS. Any recommendations as far as hardware, configuration, etc? > > Thanks. > > Boris.Smaller volumes is best, but really it depends on your I/O type as well. I have 15TB volumes loaded with medical imaging data that happily run and fsck just fine. We've had a couple of disk failures and the MD 3000 and MD1000 units handled this just fine taking around 26 hours to sync 1TB drives. The file system here is XFS On the other hand, I have natural language data sets which are millions of small files residing on a 4.5TB EXT4 file system. This file system has had a problem and to this day I still cannot perform a file system check to correct the errors because the e4fsck program chews up more than 42GB of memory and then dies. For details check out. https://bugzilla.redhat.com/show_bug.cgi?id=570639 What I'm trying to say is, understand your usage patterns. Large streaming files is far less intensive on the controller then millions of small files. Understand your hardware and what it is capable of in each configuration. RAID-0 vs 5 vs 6 vs 10. It is incredibly important. Understand your file system. Figure out what file system works best for your workload, how it functions and how the underlying hardware needs to be configured to maximize throughput. That is all for now. ;) -- James A. Peltier Systems Analyst (FASNet), VIVARIUM Technical Director HPC Coordinator Simon Fraser University - Burnaby Campus Phone : 778-782-6573 Fax : 778-782-3045 E-Mail : jpeltier at sfu.ca Website : http://www.fas.sfu.ca | http://vivarium.cs.sfu.ca http://blogs.sfu.ca/people/jpeltier MSN : subatomic_spam at hotmail.com TEAMWORK There's power in numbers. Learn to work together.
Just finished building a new server for inhouse use. 12.8TB Needed a nfs and samba server which could store 10TB and be reliable. Also wanted to replace out DHCP server and our internal DNS server. So needed to run dnsmasq plus ntp for time serving. This machine replace 3 older units. We used a Supermicro 3U case with 15 sata shuttles and 2N+1 760 Watt PS. On ebay for 300.00 Used a Supermicro H8DME-2 MB cause I had one sitting around. If I was going to buy one it would have been a Tyan S2932G2NR-SI for reliability (we have sold a bunch of these and they always work) Installed dual AMD 2382's and 16 GB DDR2-800 RECC plus 4 wire HSF's. Went to the 3ware online store and purchased a 9550SXU-16ML with the breakout cables for 340.00 Yes, I know its PCI-X. It was inexpensive and it runs almost as fast as the 9650 series at twice the price. Installed 15 pcs of the WD1001FALS disks. Very reliable and low cost even though they are desktop drives. Used these since we always have 3 on the shelf as spares and WD turns a bad one around in about 5 days. We have installed over 200 of these drives in raid arrays and have had 2 fail in the last 6 months. BTW, Initializing took <8 hours not days. Set it up as a Raid 5 array with one hot spare and autorebuild. Have tested it with both Centos 5.4 x86_64 and OpenSUSE 11.1 x86_64. Also installed an LSI SCSI card and a Sony AIT-5 Tape drive with Bacula for data backup and recovery. Again, all worked fine and the machine was able to do backups and keep a GigE connection saturated. Using an in house piece of code called disktest we are seeing 268 MB / sec on writes and 347 MB / sec on reads using 32 GB test files, 131K buffer sizes and single threading. If you would like to see a system that is almost identical, have a look at Coraid.com. They use a dual xeon and a BSD custom kernel. Hope this config helps...... Seth Bardash Integrated Solutions and Systems LLC seth at integratedsolutions.org Failure cannot survive knowledge and perseverance!