zfs discuss - Jan 2010 - Building big cheap storage system. What hardware to use?

Hello.
We need big "cheap" storage.
Looking to Supermicro systems.
Something based on SC846E1-R900 case
http://www.supermicro.com/products/chassis/4U/846/SC846E1-R900.cfm with 24 disc
bays. This case with 3 GBit  LSI SASX36 expander.

But the problem with LSI based HBA timeouts really confuses me.

Should i get newer motherboard with 6GBit LSI SAS 2008 HDA like
http://www.supermicro.com/products/motherboard/QPI/5500/X8DT6-F.cfm?IPMI=Y&SAS=Y
or get older motherboard with LSI 1068 HBA?
Can anyone post good working configurations based on Supermicro hardware?

Planning to use 2Tb Hitachi SATA drives (any thoughts on HDD choose?).
-- 
This message posted from opensolaris.org

I have Supermicro 936E1 (X28 expander chip) and LSI 1068 HBA. I never got
timeout issue but I''m using Seagate 15K.7 SAS. SATA might be different
as it handles error and io timeout differently.

If you can wait, better wait for 6Gb SAS expander based product.

BTW. I''d get Supermicro X8DTH-6F motherboard as this gives enough
expansion slots.
-- 
This message posted from opensolaris.org

> I have Supermicro 936E1 (X28 expander chip) and LSI
> 1068 HBA. I never got timeout issue but I''m using
> Seagate 15K.7 SAS. SATA might be different as it
> handles error and io timeout differently. If you
> still want volume, you make take a look at 7200 RPM
> SAS version.
SAS disks more expensive. Besides, there is no 2Tb SAS 7200 drives on market
yet.
> If you can wait, better wait for 6Gb SAS expander
> based product.
Do you think it make sense if we will use SATA2 (300Gbit) disks?
I heard there were problems with SAS1 expanders in Supermicro chassis, after
they come out. Don''t want to debug new product.
> BTW. I''d get Supermicro X8DTH-6F motherboard as this
> gives enough expansion slots.
Thanks for the tip about motherboard.
-- 
This message posted from opensolaris.org

We use the following for our storage servers:

Chenbro 5U chassis (24 hot-swap drive bays)
1350 watt 4-way redundant PSU
Tyan h200M motherboard (S3992)
2x dual-core AMD Opteron 2200-series CPUs
8 GB ECC DDR2-SDRAM
4-port Intel PRO/1000MT NIC (PCIe)
3Ware 9550SXU PCI-X RAID controller (12-port, multi-lane)
3Ware 9650SE PCIe RAID controller (12-port, muli-lane)
24x 500 GB harddrive (either Seagate ES2 or Western Digital RE2)

Comes out to under $10,000 CDN, and gives 10 TB of disk space (3x 8-drive
raidz2).  If you use multiple 8-port SATA/SAS controllers instead of RAID
controller, and a 3-way PSU, it should come out to under $8,000 CDN.

Fully supportted by FreeBSD, so everything should work with OpenSolaris.
-- 
This message posted from opensolaris.org

On Wed, Jan 27, 2010 at 08:25:48PM -0800, borov wrote:
> SAS disks more expensive. Besides, there is no 2Tb SAS 7200 drives on
market yet.
Seagate released a 2 TB SAS drive last year.
http://www.seagate.com/ww/v/index.jsp?locale=en-US&vgnextoid=c7712f655373f110VgnVCM100000f5ee0a0aRCRD

-- 
Jason Fortezzo
fortezza at mechanicalism.net

>> SAS disks more expensive. Besides, there is no 2Tb SAS 7200 drives on
market yet.
>
> Seagate released a 2 TB SAS drive last year.
>
http://www.seagate.com/ww/v/index.jsp?locale=en-US&vgnextoid=c7712f655373f110VgnVCM100000f5ee0a0aRCRD
Yes, it was announced. But it is not available in Russia yet. I think it 
is not available anywhere.

Take some time to google model number. It is ST32000444SS for 2Tb version.

Freddie Cash <fjwcash at gmail.com> writes:
> We use the following for our storage servers:
> [...]
> 3Ware 9650SE PCIe RAID controller (12-port, muli-lane)
> [...]
> Fully supported by FreeBSD, so everything should work with
> OpenSolaris.
FWIW, I''ve used the 9650SE with 16 ports in OpenSolaris 2008.11 and
2009.06, and had problems with the driver just hanging after 4-5 days of
use.  iostat would report 100% busy on all drives connected to the card,
and even "uadmin 1 1" (low-level reboot command) was ineffective.  I
had
to break into the debugger and do the reboot from there.  I was using
the newest driver from AMCC.

-- 
Kjetil T. Homme
Redpill Linpro AS - Changing the game

Hello.
Thanks for config, but Chenbro badly widespread here, in Russia.
As for 3Ware RAID cards i think better get "dumb" HBA cards and let
ZFS
do all work.

That must be a combination of many things to make it happen.
ie. expander revision, SAS HBA revision, firmware, disk model, firmware, etc.

I didn''t see the problem on my system but I haven''t used SATA
disks with it so I can''t say.
-- 
This message posted from opensolaris.org

Yes, if I was to re-do the hardware config for these servers, using what I know
now, I would do things a little differently:
  - multiple 8-port SATA controllers
  - 1 vdev per controller
  - 6-drive raidz2 vdevs
  - find a case with more than 24 drive bays (any way to get a Thumper without
the extra hardware/software?) ;)
  - look into eSATA/mini-SAS and external drive enclosures (separate storage
from system)
-- 
This message posted from opensolaris.org

fjwcash at gmail.com said:
> Yes, if I was to re-do the hardware config for these servers, using what I
> know now, I would do things a little differently:
> . . .
>   - find a case with more than 24 drive bays (any way to get a Thumper
> without the extra hardware/software?) ;)
> . . .
It''s called the Sun Storage J4500 array.  Well, until Oracle gets
around to changing the name anyway....

Regards,

Marion

hello
my suggestion of a big and really cheap system
(use dev build or next release 03.2010)

mainboard
http://www.supermicro.com/xeon_3400/Motherboard/X8SIL.cfm
ecc, max 32 ram, intel 3420 server chipset, vga, 3x pci-e, 6 x sata
in germany about 180 euro

3 x sas controller lsi1068
http://www.supermicro.com/products/accessories/addon/AOC-USASLP-L8i.cfm
although uio (left-mounted), you can install them
in germany about 130 euro each

sas-II controller, based on lsi 2008
working (sucessfully installed with nexenta3 alpha and supermicro x8dth6),
-wait for hba (non-raid) pci-e models
-not available yet in germany 

=>
no need to use sas expander
use 3 controller and sas drives instead
  more simple and faster, no problems

see my reference hardware for my napp-it  zfs server:
http://www.napp-it.org/hardware/index.html

gea
-- 
This message posted from opensolaris.org