CentOS - Jan 2010 - 8-15 TB storage: any recommendations?

Hello everyone,

This is not directly related to CentOS but still: we are trying to set up
some storage servers to run under Linux - most likely CentOS. The storage
volume would be in the range specified: 8-15 TB. Any recommendations as far
as hardware?

Thanks.

Boris.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos/attachments/20100106/41b68c27/attachment-0002.html>

> recommendations as far as hardware?
Giving we have no clue what it is used for no:) Seriously,
it makes all the difference what this is backing, vm's exported
over nfs/iSCSI, samba, etc...

Boris Epstein wrote:
> This is not directly related to CentOS but still: we are trying to set 
> up some storage servers to run under Linux - most likely CentOS. The 
> storage volume would be in the range specified: 8-15 TB. Any 
> recommendations as far as hardware?
Why not just get a SAN appliance, and then attach it to your CentOS 
server with iSCSI. My company is getting ready to do the same. We have 
the hardware in place, just haven't had time to hook it all up and spin 
the thing up. We purchased an IBM SAN, and then we'll attach it to an 
older xSeries 235 server running CentOS.

Or are you looking for some cheaper solutions?

Regards,
Max

Boris Epstein wrote:
> Hello everyone,
> 
> This is not directly related to CentOS but still: we are trying to set
> up some storage servers to run under Linux - most likely CentOS. The
> storage volume would be in the range specified: 8-15 TB. Any
> recommendations as far as hardware?
Depends what you want to do ... i am satisfied with an via vb8001 with
nano cpu at 1.6 ghz (thou i still have problems with power scaling)
and an areca 1220 ... so i have 2 hdd in software raid 1 (motherboard) +
8 hdd in raid 6 from areca ... the drives are kept in 2x 5 hdd
supermicro racks ... this is a home samba and http server ...
Adrian
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3110 bytes
Desc: S/MIME Cryptographic Signature
URL:
<http://lists.centos.org/pipermail/centos/attachments/20100106/30b088d3/attachment-0002.bin>

From: Boris Epstein <borepstein at gmail.com>
>This is not directly related to CentOS but still: we are trying to set up
some storage servers to run under Linux - most likely CentOS. The storage volume
would be in the range specified: 8-15 TB. Any recommendations as far as
hardware?
Depends on your budget.
Here, we use HP DL180 servers (12 x 1TB disks in 2U)...
You can also check Sun Fire X**** servers; up to 48 x 1TB in 4U...

JD

On Thu, Jan 7, 2010 at 11:34 AM, John Doe <jdmls at yahoo.com>
wrote:
> From: Boris Epstein <borepstein at gmail.com>
>>This is not directly related to CentOS but still: we are trying to set
up some storage servers to run under Linux - most likely CentOS. The storage
volume would be in the range specified: 8-15 TB. Any recommendations as far as
hardware?
>
> Depends on your budget.
> Here, we use HP DL180 servers (12 x 1TB disks in 2U)...
> You can also check Sun Fire X**** servers; up to 48 x 1TB in 4U...
>
> JD
>
>
>
> _______________________________________________

SuperMicro also has good 1U, or 2U servers with plenty hard drive
space, and very affordable



-- 
Kind Regards
Rudi Ahlers
CEO, SoftDux Hosting
Web: http://www.SoftDux.com
Office: 087 805 9573
Cell: 082 554 7532

John Doe wrote:
> From: Boris Epstein <borepstein at gmail.com>
>> This is not directly related to CentOS but still: we are trying to set
up some storage servers to run under Linux - most likely CentOS. The storage
volume would be in the range specified: 8-15 TB. Any recommendations as far as
hardware?
> 
> Depends on your budget.
> Here, we use HP DL180 servers (12 x 1TB disks in 2U)...
> You can also check Sun Fire X**** servers; up to 48 x 1TB in 4U...
> 
Somebody said something about Sun servers being pricey and that quality 
was going downhill...something about cheap controllers...any comments on 
this?

BTW, the Sun X4540 can only be bought with all disks loaded. So it is 
not up to 48 but must be 48 in 4U.

Quoting Chan Chung Hang Christopher <christopher.chan at bradbury.edu.hk>:
> John Doe wrote:
>> From: Boris Epstein <borepstein at gmail.com>
>>> This is not directly related to CentOS but still: we are trying to
>>>  set up some storage servers to run under Linux - most likely   
>>> CentOS. The storage volume would be in the range specified: 8-15   
>>> TB. Any recommendations as far as hardware?
>>
>> Depends on your budget.
>> Here, we use HP DL180 servers (12 x 1TB disks in 2U)...
>> You can also check Sun Fire X**** servers; up to 48 x 1TB in 4U...
>>
>
> Somebody said something about Sun servers being pricey and that quality
> was going downhill...something about cheap controllers...any comments on
> this?
>
> BTW, the Sun X4540 can only be bought with all disks loaded. So it is
> not up to 48 but must be 48 in 4U.
Atleast old sun fire servers are using cheap ata controllers ...

My own solution is based on supermicro server with areca raid controller.

case: ATX Supermicro SC836TQ-R800V (16xSAS) with areca 16 port sata  
raid controller.(Areca ARC-1261ML)

You can easily buy cheap sata disks from 1 to 2TB * 16

--
Eero,
RHCE

On 01/06/2010 09:35 PM, Boris Epstein wrote:
> Hello everyone,
>
> This is not directly related to CentOS but still: we are trying to set
> up some storage servers to run under Linux - most likely CentOS. The
> storage volume would be in the range specified: 8-15 TB. Any
> recommendations as far as hardware?

I would recommend dont-homebrew. Get a vendor to build you a 3/4U box, 
get a couple of quad core cpu's in, and enough ram to do you in-use 
buffers. Also dont go over 1 TiB in storage per spindle if you want to 
get even relatively reasonable performance ( even when in use as a filer 
box ).

Not long back, I had the chance to do some performance metrics on a dual 
Areca-16xx hosted 24x1TiB disk setup - and we tested it for various 
loads, running CentOS-5.4/x86_64 and it consistently outperformed the 
sun thumper box's, coming in about 1/4th the price.

The other thing to keep in mind is to estimate and prove the cpu 
processing capability and network capability you are going to need out 
of this machine and dont skim on that. Dont just get overly focused on 
just the hba and disk metrics.

- KB

From: Karanbir Singh <mail-lists at karan.org>
> On 01/07/2010 02:30 PM, Boris Epstein wrote:
> > KB, thanks. When you say "dont go over 1 TiB in storage per
spindle"
> > what are you referring to as spindle?
> 
> disk. it boils down to how much data do you want to put under one 
> read/write stream.
> 
> the other thing is that these days 1.5TB disks are the best 
> bang-for-the-buck in terms of storage/cost. So maybe thats something to 
> consider, and limit disk usage down initially - expand later as you need.
> 
> Even better if your hba can support that, if not then mdadm ( have lots 
> of cpu right ? ), and make sure you understand recarving / reshaping 
> before you do the final design. Refactoring filers with large quantities 
> of data is no fun if you cant reshape and grow.
I also heard that disks above 1TB might have reliability issues.
Maybe it changed since then...

JD

On 1/6/2010 2:35 PM, Boris Epstein wrote:
>
> we are trying to set
> up some storage servers to run under Linux
You should also consider FreeBSD 8.0, which has the newest version of 
ZFS up and running stably on it.  I use Linux for most server tasks, but 
for big storage, Linux just doesn't have anything like this yet.

Yeah, yeah, btrfs someday, I know.  But today, ZFS is where it's at. 
It's the easiest way of managing large pools of storage I know of short 
of a Drobo, and Drobos have big problems of their own.

I'm not recommending OpenSolaris on purpose.  For the last few years, it 
was the only stable production-quality implementation of ZFS, but with 
FreeBSD 8.0, it just lost that advantage.  I think, as a Linux fan, you 
will be happier with FreeBSD than OpenSolaris.
> storage volume would be in the range specified: 8-15 TB.
That puts you right on the edge of workability with 32-bit hardware. 
ext3's limit on 32-bit is 8 TB, and you can push it to 16 TB by 
switching to XFS or JFS.  Best to use 64-bit hardware if you can.

I suggest you get a second-hand Sun X4500 if you're feeling cheap, 
http://www.sun.com/servers/x64/x4500/specs.xml. 48x 500G will do you 
nicely with some MD RAID.

Or you can go for the newer X4540 if you're feeling flush.

Regards, Iolaire

On 06/01/2010 22:35, Boris Epstein wrote:
> Hello everyone,
>
> This is not directly related to CentOS but still: we are trying to set 
> up some storage servers to run under Linux - most likely CentOS. The 
> storage volume would be in the range specified: 8-15 TB. Any 
> recommendations as far as hardware?
>
> Thanks.
>
> Boris.
>
>
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
>    
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos/attachments/20100108/f6d90aab/attachment-0002.html>

On Wed, Jan 6, 2010 at 10:35 PM, Boris Epstein <borepstein at gmail.com>
wrote:
> some storage servers to run under Linux - most likely CentOS. The storage
> volume would be in the range specified: 8-15 TB. Any recommendations as far
> as hardware?
I'm kind of partial to Areca raid controllers, you can get up to 24
ports, so that can be as much as 20 TB (that's real world terabytes,
not hardware manufacturer's) in a raid 6 with hot-spare using 1000 GB
drives.

BR Bent

On 1/11/2010 11:38 AM, John R Pierce wrote:
> Pasi K?rkk?inen wrote:
>> It seems X4500 (not available anymore) had Marvell SATA controllers,
that
>> are not supported with RHEL5.
>>
>
> And those marvell controllers caused major grief for Sun, especially
> when Solaris added support for NCQ somewhere in there. under heavy IO
> workloads, the controllers would just hang.   Some nasty bugs.  Driver
> software workarounds caused a big performance hit.
Is that a different chipset than 
http://www.newegg.com/Product/Product.aspx?Item=N82E16815121009 uses?  I 
replaced a Paradise and Adaptec card with one of these (or maybe the 
PCI-E version) and Centos recognized it and worked better than with the 
two different cards.

-- 
   Les Mikesell
    lesmikesell at gmail.com

Karanbir Singh wrote:
> On 01/08/2010 05:28 PM, R-Elists wrote:
>> what is wrong or what problems are you referring to with cciss please ?
>>
> 
> problems mostly centered around management and performance issues. the 
> world is littered with stores of cciss fail
> 
> 
Really? Man, I have been given this spanking new HP DL370 G6 and running 
Centos 5.4 on it...

Hi

Appologies I have not been following the thread here so?am just
wondering if you have a MSA, EVA, XP left hand san?or if this is just
storage that sits on the server with samba share? also what link is
between fc or ethernet.

Regards
Per Qvindesland

At Tisdag, 12-01-2010 on 11:57 "Chan Chung Hang Christopher"  wrote:

Karanbir Singh wrote:
> On 12/01/10 00:02, Christopher Chan wrote:
>>> problems mostly centered around management and performance issues.
the 
>>> world is littered with stores of cciss fail
>> Really? Man, I have been given this spanking new HP DL370 G6 and
running 
>> Centos 5.4 on it...
> 
> I've got a couple of DL380's at one setup and another 12
DL360's at
> another place. We have had enough problems with interfaces that all
the
> machines are now running off remote-storage. Our storage incident
rate
> has gone from 1/day average to under 2/month since then.
> 
> all of these machines are G4 and G5's running CentOS-5/x86_64
> 
Eeek! That thing will be hosting the school's vle. Looks like I better

memorize the after hours password for HP support.

What problems did you have? Do they occur mostly when the boxes are 
under high I/O load?

This is really new to me as I had no problems with a DL360 G3 box that

ran Windows 2000 and Exchange 2000 with regards to disk problems in my

previous job.
_______________________________________________
CentOS mailing list
CentOS at centos.org
http://lists.centos.org/mailman/listinfo/centos


-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos/attachments/20100112/347fd356/attachment-0001.html>

Per Qvindesland wrote:
> Hi
> 
> Appologies I have not been following the thread here so am just
> wondering if you have a MSA, EVA, XP left hand san or if this is just
> storage that sits on the server with samba share? also what link is
> between fc or ethernet.
If you are asking me, then there is no external direct attach storage. 
HQ (Bradbury School is an ESF school) provided the thing and it has two 
disks in mirror mode for a db and four disks in raid5 mode for system 
and the vle (customized moodle) with a 512 MB BBU module for the P410i 
controller. They added a 4-port Intel Gigabit adapter too but that is of 
no consequence with storage right now.
> 
> Regards
> Per Qvindesland
> 
> At Tisdag, 12-01-2010 on 11:57 "Chan Chung Hang Christopher" 
wrote:
> 
> Karanbir Singh wrote:
>> On 12/01/10 00:02, Christopher Chan wrote:
>>>> problems mostly centered around management and performance
issues.
> the 
>>>> world is littered with stores of cciss fail
>>> Really? Man, I have been given this spanking new HP DL370 G6 and
> running 
>>> Centos 5.4 on it...
>> I've got a couple of DL380's at one setup and another 12
DL360's at
>> another place. We have had enough problems with interfaces that all
> the
>> machines are now running off remote-storage. Our storage incident
> rate
>> has gone from 1/day average to under 2/month since then.
>>
>> all of these machines are G4 and G5's running CentOS-5/x86_64
>>
> 
> Eeek! That thing will be hosting the school's vle. Looks like I better
> 
> memorize the after hours password for HP support.
> 
> What problems did you have? Do they occur mostly when the boxes are 
> under high I/O load?
> 
> This is really new to me as I had no problems with a DL360 G3 box that
> 
> ran Windows 2000 and Exchange 2000 with regards to disk problems in my
> 
> previous job.
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos
> 
> 
> 
> 
> 
> ------------------------------------------------------------------------
> 
> _______________________________________________
> CentOS mailing list
> CentOS at centos.org
> http://lists.centos.org/mailman/listinfo/centos