zfs discuss - Mar 2010 - Hardware for high-end ZFS NAS file server - 2010 March edition

Please recommend your up-to-date high-end hardware components for building a
highly fault-tolerant ZFS NAS file server.

I''ve seen various hardware lists online (and I''ve summarized
them at http://wiki.dandascalescu.com/reviews/storage.edit#Solutions), but
they''re on the cheapo side. I want to build a media server and be done
with with for a few years, (until the next generation storage media (holograms?
nanowires?) becomes commercially available. So please, knock yourselves out. The
bills for ZFS NAS boxes that I''ve seen run around $1k, and I''m
willing to invest up to $3k.

Requirements, in decreasing order of importance:
1. Extremely fault-tolerant.  I''d like to be able to lose two disks and
still be OK. I also want any silent hard disk read errors that are detected by
ZFS, to be reported somehow.
2. As quiet as it gets.
3. Able to easily extend storage
4. (low) If feasible, I''d like to be able to use a Blu-Ray drive with
the system.

I also have a few software requirements, which I think are pretty independent of
the hardware:
a. Secure ? I want to be able to tweak and control access at every level
b. Very fast network performance. The server should be able to stream 1080p
while doing a number of other tasks without issues.
c. Ability to serve all different types of hosts: NFS, SMB, SCP/SFTP
d. Flexible. I do a number of other things/experiments, and I?d like to be able
to use it for more than just serving files.

Really only #1 (reliable) and #2 (quiet) matter most. I''ve been mulling
over this server for too long and want to get it over with.

Looking forward to your recommendations,
Dan
-- 
This message posted from opensolaris.org

On 04/03/2010 09:46, Dan Dascalescu wrote:
> Please recommend your up-to-date high-end hardware components for building
a highly fault-tolerant ZFS NAS file server.
>
>    
2x M5000 + 4x EMC DMX

Sorry, I couldn''t resist :)

-- 
Robert Milkowski
http://milek.blogspot.com

Hello,

On 4 mar 2010, at 11.11, Robert Milkowski <milek at task.gda.pl> wrote:
> On 04/03/2010 09:46, Dan Dascalescu wrote:
>> Please recommend your up-to-date high-end hardware components for  
>> building a highly fault-tolerant ZFS NAS file server.
>>
>>
>
> 2x M5000 + 4x EMC DMX
>
> Sorry, I couldn''t resist :)
>
I would not recommend that, you can''t change boards in anything less  
than a M8000 so your service would have to switch nodes just to  
replace a CPU. ;)

Henrik
http://sparcv9.blogspot.com

On Thu, Mar 4, 2010 at 4:46 AM, Dan Dascalescu <
bigbang7+opensolaris at gmail.com <bigbang7%2Bopensolaris at
gmail.com>> wrote:
> Please recommend your up-to-date high-end hardware components for building
> a highly fault-tolerant ZFS NAS file server.
>
> I''ve seen various hardware lists online (and I''ve
summarized them at
> http://wiki.dandascalescu.com/reviews/storage.edit#Solutions), but
they''re
> on the cheapo side. I want to build a media server and be done with with
for
> a few years, (until the next generation storage media (holograms?
> nanowires?) becomes commercially available. So please, knock yourselves
out.
> The bills for ZFS NAS boxes that I''ve seen run around $1k, and
I''m willing
> to invest up to $3k.
>
> Requirements, in decreasing order of importance:
> 1. Extremely fault-tolerant.  I''d like to be able to lose two
disks and
> still be OK. I also want any silent hard disk read errors that are detected
> by ZFS, to be reported somehow.
> 2. As quiet as it gets.
> 3. Able to easily extend storage
> 4. (low) If feasible, I''d like to be able to use a Blu-Ray drive
with the
> system.
>
> I also have a few software requirements, which I think are pretty
> independent of the hardware:
> a. Secure ? I want to be able to tweak and control access at every level
> b. Very fast network performance. The server should be able to stream 1080p
> while doing a number of other tasks without issues.
> c. Ability to serve all different types of hosts: NFS, SMB, SCP/SFTP
> d. Flexible. I do a number of other things/experiments, and I?d like to be
> able to use it for more than just serving files.
>
> Really only #1 (reliable) and #2 (quiet) matter most. I''ve been
mulling
> over this server for too long and want to get it over with.
>
> Looking forward to your recommendations,
> Dan
>
>
What i did was this:

I got a norco 4020 (the 4220 is good too)

Both of those cost around 300-350 dolars.  That is a 4u case with 20 hot
swap bays.

Then i got a decent server board.  I used supermicro mbd-x7se because it has
4 pci-x slots.

I got 3 supermicro AOC-SAT2-mv8 cards  for the sata ports (each has 8)

20 1tb seagate drives, but you could use any size which fits your budget.

8 gb ddr2 800 ecc memory

3 64 gb ssd''s (2 for rpool mirror and 1 for l2arc)

Intel q9550 cpu.

This gives you a pretty beastly machine which has around 18-36 raw TB''s

I went with 3 raidz2 groups.

I plan to expland it with a sas expander and another norco case.  I hope
this gives you some ideas.
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100304/53bae777/attachment.html>

Thomas Burgess wrote:
>
>
> On Thu, Mar 4, 2010 at 4:46 AM, Dan Dascalescu 
> <bigbang7+opensolaris at gmail.com 
> <mailto:bigbang7%2Bopensolaris at gmail.com>> wrote:
>
>     Please recommend your up-to-date high-end hardware components for
>     building a highly fault-tolerant ZFS NAS file server.
>
>     I''ve seen various hardware lists online (and I''ve
summarized them
>     at http://wiki.dandascalescu.com/reviews/storage.edit#Solutions),
>     but they''re on the cheapo side. I want to build a media server
and
>     be done with with for a few years, (until the next generation
>     storage media (holograms? nanowires?) becomes commercially
>     available. So please, knock yourselves out. The bills for ZFS NAS
>     boxes that I''ve seen run around $1k, and I''m willing
to invest up
>     to $3k.
>
>     Requirements, in decreasing order of importance:
>     1. Extremely fault-tolerant.  I''d like to be able to lose two
>     disks and still be OK. I also want any silent hard disk read
>     errors that are detected by ZFS, to be reported somehow.
>     2. As quiet as it gets.
>     3. Able to easily extend storage
>     4. (low) If feasible, I''d like to be able to use a Blu-Ray
drive
>     with the system.
>
>     I also have a few software requirements, which I think are pretty
>     independent of the hardware:
>     a. Secure ? I want to be able to tweak and control access at every
>     level
>     b. Very fast network performance. The server should be able to
>     stream 1080p while doing a number of other tasks without issues.
>     c. Ability to serve all different types of hosts: NFS, SMB, SCP/SFTP
>     d. Flexible. I do a number of other things/experiments, and I?d
>     like to be able to use it for more than just serving files.
>
>     Really only #1 (reliable) and #2 (quiet) matter most. I''ve
been
>     mulling over this server for too long and want to get it over with.
>
>     Looking forward to your recommendations,
>     Dan
>
>
> What i did was this:
>
> I got a norco 4020 (the 4220 is good too)
>
> Both of those cost around 300-350 dolars.  That is a 4u case with 20 
> hot swap bays.
>
> Then i got a decent server board.  I used supermicro mbd-x7se because 
> it has 4 pci-x slots.
>
> I got 3 supermicro AOC-SAT2-mv8 cards  for the sata ports (each has 8)
>
> 20 1tb seagate drives, but you could use any size which fits your budget.
>
> 8 gb ddr2 800 ecc memory
>
> 3 64 gb ssd''s (2 for rpool mirror and 1 for l2arc)
>
> Intel q9550 cpu.
>
> This gives you a pretty beastly machine which has around 18-36 raw
TB''s
>
> I went with 3 raidz2 groups. 
>
> I plan to expland it with a sas expander and another norco case.  I 
> hope this gives you some ideas.
Simple. Cheat. Buy something someone else has already put the effort 
into making nicely.


http://computers.shop.ebay.com/i.html?_nkw=ibm+%28x3500%2C3500%29&_sacat=58058&_trksid=p3286.m270.l1313&_odkw=ibm+%28x3400%2Cx3500%2C3500%2C3400%29&_osacat=58058&bkBtn=1

IBM x3500
2 dual-core 5140 Xeons (2.3Ghz)
12 DIMM slots
3 8x PCI-E, 2 64-bit 133Mhz PCI-X, 1 32-bit PCI slots
2GB RAM
4x73GB SAS drives
Baseboard Management Controller and Remote Supervisor
IBM 8K raid controller with 256MB Battery-backed cache
< $600

Extra (redundant) power supply:   $50
8GB (4 x 2GB) more RAM:   $200
4 more IBM hot swap trays:   $100
Replace 4 x 73GB SAS with 8 x 1TB SATA drives:   < $800
4 x 2.5" Hot Swap in 5.25" HH bay chassis = $80
    
(http://www.supermicro.com/products/accessories/mobilerack/CSE-M14.cfm 
or similar)
multi-lane to 4 SATA breakout cable  = $15
2 x 100GB 2.5" 7200 RPM hard drives = $100
SSDs only if you want them (unlikely)

Total:  < $2000



It''s noisy on startup (when all the huge numbers of fans surge at max 
speed for 5 seconds), but then is very quiet.  If you''re careful, you 
can get one that''s still under some portion of IBM warranty.

It''s a monster - likely to out-perform anything else you''ll be
able to
get for anywhere near that cost. Plus, 2 Gbit ethernet controllers, AND 
a IP KVM/console.


Frankly, at this point, for $1000 (before disks), NOTHING can beat a 
used (or refurbed) X3500. 

-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA

On Thu, Mar 4, 2010 at 4:12 AM, Thomas Burgess <wonslung at gmail.com>
wrote:
> I got a norco 4020 (the 4220 is good too)
>
> Both of those cost around 300-350 dolars.? That is a 4u case with 20 hot
> swap bays.
Typically rackmounts are not designed for quiet. He said quietness is
#2 in his priorities...

Or does the Norco unit perform quietly or have the ability to be quieter?

its not quiet by default but it can be made somewhat more quiet by swapping
out the fans or going to larger fans.  Its still totally worth it.

I use smaller, silent htpc''s for the actual media and connect to the
norco
over gigabit.

My norco box is connected to the network with 2 link aggregated gigabit
ethernet cables.

It''s very nice.


On Thu, Mar 4, 2010 at 3:03 PM, Michael Shadle <mike503 at gmail.com>
wrote:
> On Thu, Mar 4, 2010 at 4:12 AM, Thomas Burgess <wonslung at
gmail.com> wrote:
>
> > I got a norco 4020 (the 4220 is good too)
> >
> > Both of those cost around 300-350 dolars.  That is a 4u case with 20
hot
> > swap bays.
>
> Typically rackmounts are not designed for quiet. He said quietness is
> #2 in his priorities...
>
> Or does the Norco unit perform quietly or have the ability to be quieter?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100304/724c7cac/attachment.html>

If I had a decently ventilated closet or space to do it in I wouldn''t
mind noise, but I don''t, that''s why I had to build my storage
machines
the way I did.

On Thu, Mar 4, 2010 at 12:23 PM, Thomas Burgess <wonslung at gmail.com>
wrote:
> its not quiet by default but it can be made somewhat more quiet by swapping
> out the fans or going to larger fans.? Its still totally worth it.
>
> I use smaller, silent htpc''s for the actual media and connect to
the norco
> over gigabit.
>
> My norco box is connected to the network with 2 link aggregated gigabit
> ethernet cables.
>
> It''s very nice.
>
>
> On Thu, Mar 4, 2010 at 3:03 PM, Michael Shadle <mike503 at gmail.com>
wrote:
>>
>> On Thu, Mar 4, 2010 at 4:12 AM, Thomas Burgess <wonslung at
gmail.com> wrote:
>>
>> > I got a norco 4020 (the 4220 is good too)
>> >
>> > Both of those cost around 300-350 dolars.? That is a 4u case with
20 hot
>> > swap bays.
>>
>> Typically rackmounts are not designed for quiet. He said quietness is
>> #2 in his priorities...
>>
>> Or does the Norco unit perform quietly or have the ability to be
quieter?
>
>

yah, i can dig it.  I''d be really upset if i couldn''t use my
rackmount
stuff.  I love my norco box.  I''m about to build a second one using a
sas
expander...but i can totally understand how noise would be a concern....

at the same time, it''s not NEARLY as loud as something like an ac
window
unit.



On Thu, Mar 4, 2010 at 3:27 PM, Michael Shadle <mike503 at gmail.com>
wrote:
> If I had a decently ventilated closet or space to do it in I
wouldn''t
> mind noise, but I don''t, that''s why I had to build my
storage machines
> the way I did.
>
> On Thu, Mar 4, 2010 at 12:23 PM, Thomas Burgess <wonslung at
gmail.com>
> wrote:
> > its not quiet by default but it can be made somewhat more quiet by
> swapping
> > out the fans or going to larger fans.  Its still totally worth it.
> >
> > I use smaller, silent htpc''s for the actual media and connect
to the
> norco
> > over gigabit.
> >
> > My norco box is connected to the network with 2 link aggregated
gigabit
> > ethernet cables.
> >
> > It''s very nice.
> >
> >
> > On Thu, Mar 4, 2010 at 3:03 PM, Michael Shadle <mike503 at
gmail.com>
> wrote:
> >>
> >> On Thu, Mar 4, 2010 at 4:12 AM, Thomas Burgess <wonslung at
gmail.com>
> wrote:
> >>
> >> > I got a norco 4020 (the 4220 is good too)
> >> >
> >> > Both of those cost around 300-350 dolars.  That is a 4u case
with 20
> hot
> >> > swap bays.
> >>
> >> Typically rackmounts are not designed for quiet. He said quietness
is
> >> #2 in his priorities...
> >>
> >> Or does the Norco unit perform quietly or have the ability to be
> quieter?
> >
> >
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100304/c9f35612/attachment.html>

Thanks for your suggestions.

In the meantime I had found this case and PSU - what do folks think?

Antec Twelve Hundred Gaming Case -
http://wiki.dandascalescu.com/reviews/gadgets/computers/cases#Antec_Twelve_Hundred_Gaming_Case_.E2.98.85

+ 12 5.25" externally-accessible bays, in which you can elastic-mount the
hard drives (http://www.silentpcreview.com/forums/viewtopic.php?t=8240)
+ under 28dBA SPL @ 1meter at 100% CPU fan


PSU: Nexus RX-8500, http://www.silentpcreview.com/article970-page7.html
It has a relatively constant noise profile 28 dBA at 300W, 33 dBA @ 700W

Not sure what wattage the system will use most of the time though?

If it''s under 400W, then the Antec CP-850 doesn''t exceed 14
dBA - almost silent.
-- 
This message posted from opensolaris.org

On Fri, Mar 5, 2010 at 8:41 PM, Dan Dascalescu <
bigbang7+opensolaris at gmail.com <bigbang7%2Bopensolaris at
gmail.com>> wrote:
> Thanks for your suggestions.
>
> In the meantime I had found this case and PSU - what do folks think?
>
> Antec Twelve Hundred Gaming Case -
>
http://wiki.dandascalescu.com/reviews/gadgets/computers/cases#Antec_Twelve_Hundred_Gaming_Case_.E2.98.85
>
> + 12 5.25" externally-accessible bays, in which you can elastic-mount
the
> hard drives (http://www.silentpcreview.com/forums/viewtopic.php?t=8240)
> + under 28dBA SPL @ 1meter at 100% CPU fan
>
>
> PSU: Nexus RX-8500, http://www.silentpcreview.com/article970-page7.html
> It has a relatively constant noise profile 28 dBA at 300W, 33 dBA @ 700W
>
> Not sure what wattage the system will use most of the time though?
>
> If it''s under 400W, then the Antec CP-850 doesn''t exceed
14 dBA - almost
> silent.
>
>
I think you''d be better off with a NORCO for a cheap storage server
case.

--Tim
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20100305/712777ef/attachment.html>

Tim Cook wrote:
>
>
> On Fri, Mar 5, 2010 at 8:41 PM, Dan Dascalescu 
> <bigbang7+opensolaris at gmail.com 
> <mailto:bigbang7%2Bopensolaris at gmail.com>> wrote:
>
>     Thanks for your suggestions.
>
>     In the meantime I had found this case and PSU - what do folks think?
>
>     Antec Twelve Hundred Gaming Case -
>    
http://wiki.dandascalescu.com/reviews/gadgets/computers/cases#Antec_Twelve_Hundred_Gaming_Case_.E2.98.85
>
>     + 12 5.25" externally-accessible bays, in which you can
>     elastic-mount the hard drives
>     (http://www.silentpcreview.com/forums/viewtopic.php?t=8240)
>     + under 28dBA SPL @ 1meter at 100% CPU fan
>
>
>     PSU: Nexus RX-8500,
>     http://www.silentpcreview.com/article970-page7.html
>     It has a relatively constant noise profile 28 dBA at 300W, 33 dBA
>     @ 700W
>
>     Not sure what wattage the system will use most of the time though?
>
>     If it''s under 400W, then the Antec CP-850 doesn''t
exceed 14 dBA -
>     almost silent.
>
>
> I think you''d be better off with a NORCO for a cheap storage
server case.
>
> --Tim
I guess it all boils down to exactly where this machine is intended to 
be placed, as to the tradeoffs between performance, cost, acoustics, and 
heat.

IMHO, server devices belong in the basement, which is most likely to 
provide the environment most conducive to the primary mission of a 
server - performance.   Basements are most likely to be free of the 
acoustic and heat constraints that having a server in a living space 
forces upon it.  Who cares if it shrieks like a banshee and puts out 
enough heat to melt ice if it''s parked over by the dryer?

If you have to have the server in a living space, well, you''re going to
pay in performance and cost.  At best, you can dampen acoustic concerns 
(no pun intended) by isolating the machine in a closet or other place 
where insulation of some time reduces sound. However, you pay for that 
in a severe heat increase.  And tricks to reduce noise are going to come 
at a significant performance penalty and monetary cost.

It''s that simple.   Personally, I''ve shifted to 2.5"
drives for a lot of
reasons (performance and per-spindle reliability, mainly), and they do 
cut down on noise and heat significantly. They''re more costly $/GB than
3.5" drives, though, and you''re going to have either buy a case
that
natively supports 2.5" hot-swap carriers, or invest in drive cages.

I''m a fan of the Supermicro 2U cases for 2.5" drives.  They come
with
excellent power supplies, and are quiet enough to use in a closet, but 
they''re hardly cheap ($700+, depending on model).  

http://www.supermicro.com/products/chassis/2U/?chs=213
http://www.supermicro.com/products/chassis/2U/?chs=216


In your case, I''d also consider a design which has EXTERNAL drive bays.
eSATA is the cheap way to connect them to a server, and this leaves you 
with better case choices for the machine itself - smaller, cheaper, and 
quieter.

http://www.newegg.com/Product/Product.aspx?Item=N82E16817332021&cm_re=venus-_-17-332-021-_-Product


-- 
Erik Trimble
Java System Support
Mailstop:  usca22-123
Phone:  x17195
Santa Clara, CA

2010/3/4 Michael Shadle <mike503 at gmail.com>:
> Typically rackmounts are not designed for quiet. He said quietness is
> #2 in his priorities...
I have a Supermicro 743 case, also 4U. The one I used is the "Super
Quiet" variant, which uses fewer & slower PWM fans. It''s got 8
hot
swap bays and an additional 3x 5.25" bays which you can put an
additional hot swap bay in.

It''s quiet enough to have in my home office without being a
distraction.

-B

-- 
Brandon High : bhigh at freaks.com