Hi folks, I''m in the market for a couple of JBODs. Up until now I''ve been relatively lucky with finding hardware that plays very nicely with ZFS. All my gear currently in production uses LSI SAS controllers (3801e, 9200-16e, 9211-8i) with backplanes powered by LSI SAS expanders (Sun x4250, Sun J4400, etc). But I''m in the market for SAS2 JBODs to support a large number 3.5inch SAS disks (60+ 3TB disks to start). I''m aware of potential issues with SATA drives/interposers and the whole SATA Tunneling Protocol (STP) nonsense, so I''m going to stick to a pure SAS setup. Also, since I''ve had trouble with in the past with daisy-chained SAS JBODs I''ll probably stick with one SAS 4x cable (SFF8088) per JBOD and unless there were a compelling reason for multi-pathing I''d probably stick to a single controller. If possible I''d rather buy 20 packs of enterprise SAS disks with 5yr warranties and have the JBOD come with empty trays, but would also consider buying disks with the JBOD if the price wasn''t too crazy. Does anyone have any positive/negative experiences with any of the following with ZFS: * SuperMicro SC826E16-R500LPB (2U 12 drives, dual 500w PS, single LSI SAS2X28 expander) * SuperMicro SC846BE16-R920B (4U 24 drives, dual 920w PS, single unknown expander) * Dell PowerVault MD 1200 (2U 12 drives, dual 600w PS, dual unknown expanders) * HP StorageWorks D2600 (2U 12 drives, dual 460w PS, single/dual unknown expanders) I''m leaning towards the SuperMicro stuff, but every time I order SuperMicro gear there''s always something missing or wrongly configured so some of the cost savings gets eaten up with my time figuring out where things went wrong and returning/ordering replacements. The Dell/HP gear I''m sure is fine, but buying disks from them gets pricey quick. The last time I looked they charged $150 extra per disk for when the only added value was a proprietary sled a shorter warranty (3yr vs 5yr). I''m open to other JBOD vendors too, was just really just curious what folks were using when they needed more than two dozen 3.5" SAS disks for use with ZFS. Thanks -Peter
Rocky Shek
2012-Nov-13 20:41 UTC
[zfs-discuss] [discuss] Hardware Recommendations: SAS2 JBODs
Peter, You may consider DataON JBOD. Lots of users are using DataON JBOD for ZFS storage. Yes, LSI SAS HBA is the best choice for ZFS DataON DNS-1600 4U 24 Bay 6G SAS JBOD http://dataonstorage.com/dns-1600 DataON DNS-1640 2U 24 Bay 6G SAS JBOD http://dataonstorage.com/dns-1640 DataON DNS-1660 4U 60 Bay 6G SAS JBOD http://dataonstorage.com/dns-1660 Rocky -----Original Message----- From: Peter Tripp [mailto:peter at psych.columbia.edu] Sent: Tuesday, November 13, 2012 12:08 PM To: discuss at lists.illumos.org; zfs-discuss at opensolaris.org Subject: [discuss] Hardware Recommendations: SAS2 JBODs Hi folks, I''m in the market for a couple of JBODs. Up until now I''ve been relatively lucky with finding hardware that plays very nicely with ZFS. All my gear currently in production uses LSI SAS controllers (3801e, 9200-16e, 9211-8i) with backplanes powered by LSI SAS expanders (Sun x4250, Sun J4400, etc). But I''m in the market for SAS2 JBODs to support a large number 3.5inch SAS disks (60+ 3TB disks to start). I''m aware of potential issues with SATA drives/interposers and the whole SATA Tunneling Protocol (STP) nonsense, so I''m going to stick to a pure SAS setup. Also, since I''ve had trouble with in the past with daisy-chained SAS JBODs I''ll probably stick with one SAS 4x cable (SFF8088) per JBOD and unless there were a compelling reason for multi-pathing I''d probably stick to a single controller. If possible I''d rather buy 20 packs of enterprise SAS disks with 5yr warranties and have the JBOD come with empty trays, but would also consider buying disks with the JBOD if the price wasn''t too crazy. Does anyone have any positive/negative experiences with any of the following with ZFS: * SuperMicro SC826E16-R500LPB (2U 12 drives, dual 500w PS, single LSI SAS2X28 expander) * SuperMicro SC846BE16-R920B (4U 24 drives, dual 920w PS, single unknown expander) * Dell PowerVault MD 1200 (2U 12 drives, dual 600w PS, dual unknown expanders) * HP StorageWorks D2600 (2U 12 drives, dual 460w PS, single/dual unknown expanders) I''m leaning towards the SuperMicro stuff, but every time I order SuperMicro gear there''s always something missing or wrongly configured so some of the cost savings gets eaten up with my time figuring out where things went wrong and returning/ordering replacements. The Dell/HP gear I''m sure is fine, but buying disks from them gets pricey quick. The last time I looked they charged $150 extra per disk for when the only added value was a proprietary sled a shorter warranty (3yr vs 5yr). I''m open to other JBOD vendors too, was just really just curious what folks were using when they needed more than two dozen 3.5" SAS disks for use with ZFS. Thanks -Peter ------------------------------------------- illumos-discuss Archives: https://www.listbox.com/member/archive/182180/=now RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175751-e450dce1 Modify Your Subscription: https://www.listbox.com/member/?member_id=21175751&id_secret=21175751-e6533b b6 Powered by Listbox: http://www.listbox.com
Sašo Kiselkov
2012-Nov-13 20:42 UTC
[zfs-discuss] [discuss] Hardware Recommendations: SAS2 JBODs
We''ve got a SC847E26-RJBOD1. Takes a bit of getting used to that you have to wire it yourself (plus you need to buy a pair of internal SFF-8087 cables to connect the back and front backplanes - incredible SuperMicro doesn''t provide those out of the box), but other than that, never had a problem with it. Dirt cheap. Just works. No nonsense. Cheers, -- Saso On 11/13/2012 09:08 PM, Peter Tripp wrote:> Hi folks, > > I''m in the market for a couple of JBODs. Up until now I''ve been relatively lucky with finding hardware that plays very nicely with ZFS. All my gear currently in production uses LSI SAS controllers (3801e, 9200-16e, 9211-8i) with backplanes powered by LSI SAS expanders (Sun x4250, Sun J4400, etc). But I''m in the market for SAS2 JBODs to support a large number 3.5inch SAS disks (60+ 3TB disks to start). > > I''m aware of potential issues with SATA drives/interposers and the whole SATA Tunneling Protocol (STP) nonsense, so I''m going to stick to a pure SAS setup. Also, since I''ve had trouble with in the past with daisy-chained SAS JBODs I''ll probably stick with one SAS 4x cable (SFF8088) per JBOD and unless there were a compelling reason for multi-pathing I''d probably stick to a single controller. If possible I''d rather buy 20 packs of enterprise SAS disks with 5yr warranties and have the JBOD come with empty trays, but would also consider buying disks with the JBOD if the price wasn''t too crazy. > > Does anyone have any positive/negative experiences with any of the following with ZFS: > * SuperMicro SC826E16-R500LPB (2U 12 drives, dual 500w PS, single LSI SAS2X28 expander) > * SuperMicro SC846BE16-R920B (4U 24 drives, dual 920w PS, single unknown expander) > * Dell PowerVault MD 1200 (2U 12 drives, dual 600w PS, dual unknown expanders) > * HP StorageWorks D2600 (2U 12 drives, dual 460w PS, single/dual unknown expanders) > > I''m leaning towards the SuperMicro stuff, but every time I order SuperMicro gear there''s always something missing or wrongly configured so some of the cost savings gets eaten up with my time figuring out where things went wrong and returning/ordering replacements. The Dell/HP gear I''m sure is fine, but buying disks from them gets pricey quick. The last time I looked they charged $150 extra per disk for when the only added value was a proprietary sled a shorter warranty (3yr vs 5yr). > > I''m open to other JBOD vendors too, was just really just curious what folks were using when they needed more than two dozen 3.5" SAS disks for use with ZFS. > > Thanks > -Peter > > ------------------------------------------- > illumos-discuss > Archives: https://www.listbox.com/member/archive/182180/=now > RSS Feed: https://www.listbox.com/member/archive/rss/182180/22722377-e9306e56 > Modify Your Subscription: https://www.listbox.com/member/?member_id=22722377&id_secret=22722377-08ac87bf > Powered by Listbox: http://www.listbox.com
Schweiss, Chip
2012-Nov-13 20:42 UTC
[zfs-discuss] [discuss] Hardware Recommendations: SAS2 JBODs
I''ve had perfect success with the SuperMicro SC847E25-RJOB1 It has two back planes and holds 45 3.5 inch drives. It''s built as a JBOD so it has everything that is needed. On Tue, Nov 13, 2012 at 2:08 PM, Peter Tripp <peter at psych.columbia.edu>wrote:> Hi folks, > > I''m in the market for a couple of JBODs. Up until now I''ve been > relatively lucky with finding hardware that plays very nicely with ZFS. > All my gear currently in production uses LSI SAS controllers (3801e, > 9200-16e, 9211-8i) with backplanes powered by LSI SAS expanders (Sun x4250, > Sun J4400, etc). But I''m in the market for SAS2 JBODs to support a large > number 3.5inch SAS disks (60+ 3TB disks to start). > > I''m aware of potential issues with SATA drives/interposers and the whole > SATA Tunneling Protocol (STP) nonsense, so I''m going to stick to a pure SAS > setup. Also, since I''ve had trouble with in the past with daisy-chained > SAS JBODs I''ll probably stick with one SAS 4x cable (SFF8088) per JBOD and > unless there were a compelling reason for multi-pathing I''d probably stick > to a single controller. If possible I''d rather buy 20 packs of enterprise > SAS disks with 5yr warranties and have the JBOD come with empty trays, but > would also consider buying disks with the JBOD if the price wasn''t too > crazy. > > Does anyone have any positive/negative experiences with any of the > following with ZFS: > * SuperMicro SC826E16-R500LPB (2U 12 drives, dual 500w PS, single LSI > SAS2X28 expander) > * SuperMicro SC846BE16-R920B (4U 24 drives, dual 920w PS, single unknown > expander) > * Dell PowerVault MD 1200 (2U 12 drives, dual 600w PS, dual unknown > expanders) > * HP StorageWorks D2600 (2U 12 drives, dual 460w PS, single/dual unknown > expanders) > > I''m leaning towards the SuperMicro stuff, but every time I order > SuperMicro gear there''s always something missing or wrongly configured so > some of the cost savings gets eaten up with my time figuring out where > things went wrong and returning/ordering replacements. The Dell/HP gear > I''m sure is fine, but buying disks from them gets pricey quick. The last > time I looked they charged $150 extra per disk for when the only added > value was a proprietary sled a shorter warranty (3yr vs 5yr). > > I''m open to other JBOD vendors too, was just really just curious what > folks were using when they needed more than two dozen 3.5" SAS disks for > use with ZFS. > > Thanks > -Peter > > ------------------------------------------- > illumos-discuss > Archives: https://www.listbox.com/member/archive/182180/=now > RSS Feed: > https://www.listbox.com/member/archive/rss/182180/21878145-4afe7abf > Modify Your Subscription: > https://www.listbox.com/member/?member_id=21878145&id_secret=21878145-d5e6ec1f > Powered by Listbox: http://www.listbox.com >-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121113/8268a479/attachment-0001.html>
Ray Van Dolson
2012-Nov-13 20:44 UTC
[zfs-discuss] [discuss] Hardware Recommendations: SAS2 JBODs
On Tue, Nov 13, 2012 at 03:08:04PM -0500, Peter Tripp wrote:> Hi folks, > > I''m in the market for a couple of JBODs. Up until now I''ve been > relatively lucky with finding hardware that plays very nicely with > ZFS. All my gear currently in production uses LSI SAS controllers > (3801e, 9200-16e, 9211-8i) with backplanes powered by LSI SAS > expanders (Sun x4250, Sun J4400, etc). But I''m in the market for > SAS2 JBODs to support a large number 3.5inch SAS disks (60+ 3TB disks > to start). > > I''m aware of potential issues with SATA drives/interposers and the > whole SATA Tunneling Protocol (STP) nonsense, so I''m going to stick > to a pure SAS setup. Also, since I''ve had trouble with in the past > with daisy-chained SAS JBODs I''ll probably stick with one SAS 4x > cable (SFF8088) per JBOD and unless there were a compelling reason > for multi-pathing I''d probably stick to a single controller. If > possible I''d rather buy 20 packs of enterprise SAS disks with 5yr > warranties and have the JBOD come with empty trays, but would also > consider buying disks with the JBOD if the price wasn''t too crazy. > > Does anyone have any positive/negative experiences with any of the following with ZFS: > * SuperMicro SC826E16-R500LPB (2U 12 drives, dual 500w PS, single LSI SAS2X28 expander) > * SuperMicro SC846BE16-R920B (4U 24 drives, dual 920w PS, single unknown expander) > * Dell PowerVault MD 1200 (2U 12 drives, dual 600w PS, dual unknown expanders) > * HP StorageWorks D2600 (2U 12 drives, dual 460w PS, single/dual unknown expanders) > > I''m leaning towards the SuperMicro stuff, but every time I order > SuperMicro gear there''s always something missing or wrongly > configured so some of the cost savings gets eaten up with my time > figuring out where things went wrong and returning/ordering > replacements. The Dell/HP gear I''m sure is fine, but buying disks > from them gets pricey quick. The last time I looked they charged $150 > extra per disk for when the only added value was a proprietary sled a > shorter warranty (3yr vs 5yr). > > I''m open to other JBOD vendors too, was just really just curious what > folks were using when they needed more than two dozen 3.5" SAS disks > for use with ZFS. > > Thanks > -PeterWe''ve had good experiences with the Dell MD line. It''s been MD1200 up until now, but are keeping our eyes on their MD3260 (60-bay). You''re right in that their costs are higher for disks and such, but since we are a big Dell shop it simplifies support significantly for us and we have quick turnaround on parts anywhere in the world. If that weren''t a significant issue I''d go SuperMicro or DataON. We used SuperMicro for quite a while with mixed experiences. Best bet was to find a chassis that work and stick with it as long as possible. :) Even if you''re not using Nexenta, their HCL is valuable for finding HW that is likely to work for you. Ray
Peter, Could you please give info or links to clarify "Sata tunneling protocol nonsense" issues? Also have you considered Supermicro''s 45 drive in 4U enclosure - The SC847E26-RJBOD1 ? It''s cheap too at around 2000$ and based on LSI backplanes anyway. SGI has a nice and crazy 81(!) 3.5" disks in 4U JBOD enclosure but they refused to sell it to us without them supplying the disks also. http://www.sgi.com/products/storage/modular/jbod.html Anyone aware of a 60 or more disks in 4U enclosure on top of those mentioned in this message or the data-on? We are trying to build super-high-density-storage racks. Cedric Tineo On 13 nov. 2012, at 21:08, Peter Tripp <peter at psych.columbia.edu> wrote:> Hi folks, > > I''m in the market for a couple of JBODs. Up until now I''ve been relatively lucky with finding hardware that plays very nicely with ZFS. All my gear currently in production uses LSI SAS controllers (3801e, 9200-16e, 9211-8i) with backplanes powered by LSI SAS expanders (Sun x4250, Sun J4400, etc). But I''m in the market for SAS2 JBODs to support a large number 3.5inch SAS disks (60+ 3TB disks to start). > > I''m aware of potential issues with SATA drives/interposers and the whole SATA Tunneling Protocol (STP) nonsense, so I''m going to stick to a pure SAS setup. Also, since I''ve had trouble with in the past with daisy-chained SAS JBODs I''ll probably stick with one SAS 4x cable (SFF8088) per JBOD and unless there were a compelling reason for multi-pathing I''d probably stick to a single controller. If possible I''d rather buy 20 packs of enterprise SAS disks with 5yr warranties and have the JBOD come with empty trays, but would also consider buying disks with the JBOD if the price wasn''t too crazy. > > Does anyone have any positive/negative experiences with any of the following with ZFS: > * SuperMicro SC826E16-R500LPB (2U 12 drives, dual 500w PS, single LSI SAS2X28 expander) > * SuperMicro SC846BE16-R920B (4U 24 drives, dual 920w PS, single unknown expander) > * Dell PowerVault MD 1200 (2U 12 drives, dual 600w PS, dual unknown expanders) > * HP StorageWorks D2600 (2U 12 drives, dual 460w PS, single/dual unknown expanders) > > I''m leaning towards the SuperMicro stuff, but every time I order SuperMicro gear there''s always something missing or wrongly configured so some of the cost savings gets eaten up with my time figuring out where things went wrong and returning/ordering replacements. The Dell/HP gear I''m sure is fine, but buying disks from them gets pricey quick. The last time I looked they charged $150 extra per disk for when the only added value was a proprietary sled a shorter warranty (3yr vs 5yr). > > I''m open to other JBOD vendors too, was just really just curious what folks were using when they needed more than two dozen 3.5" SAS disks for use with ZFS. > > Thanks > -Peter > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121113/d492e971/attachment.html>
I''m quite happy with my HP D2700 and D2600 enclosures. I''m using them with LSI 9205-8e controllers and NexentaStor, but MPxIO definitely works. You will need to find HP drive trays? They''re available on eBay. -- Edmund White On 11/13/12 2:08 PM, "Peter Tripp" <peter at psych.columbia.edu> wrote:>Hi folks, > >I''m in the market for a couple of JBODs. Up until now I''ve been >relatively lucky with finding hardware that plays very nicely with ZFS. >All my gear currently in production uses LSI SAS controllers (3801e, >9200-16e, 9211-8i) with backplanes powered by LSI SAS expanders (Sun >x4250, Sun J4400, etc). But I''m in the market for SAS2 JBODs to support >a large number 3.5inch SAS disks (60+ 3TB disks to start). > >I''m aware of potential issues with SATA drives/interposers and the whole >SATA Tunneling Protocol (STP) nonsense, so I''m going to stick to a pure >SAS setup. Also, since I''ve had trouble with in the past with >daisy-chained SAS JBODs I''ll probably stick with one SAS 4x cable >(SFF8088) per JBOD and unless there were a compelling reason for >multi-pathing I''d probably stick to a single controller. If possible I''d >rather buy 20 packs of enterprise SAS disks with 5yr warranties and have >the JBOD come with empty trays, but would also consider buying disks with >the JBOD if the price wasn''t too crazy. > >Does anyone have any positive/negative experiences with any of the >following with ZFS: > * SuperMicro SC826E16-R500LPB (2U 12 drives, dual 500w PS, single LSI >SAS2X28 expander) > * SuperMicro SC846BE16-R920B (4U 24 drives, dual 920w PS, single unknown >expander) > * Dell PowerVault MD 1200 (2U 12 drives, dual 600w PS, dual unknown >expanders) > * HP StorageWorks D2600 (2U 12 drives, dual 460w PS, single/dual unknown >expanders) > >I''m leaning towards the SuperMicro stuff, but every time I order >SuperMicro gear there''s always something missing or wrongly configured so >some of the cost savings gets eaten up with my time figuring out where >things went wrong and returning/ordering replacements. The Dell/HP gear >I''m sure is fine, but buying disks from them gets pricey quick. The last >time I looked they charged $150 extra per disk for when the only added >value was a proprietary sled a shorter warranty (3yr vs 5yr). > >I''m open to other JBOD vendors too, was just really just curious what >folks were using when they needed more than two dozen 3.5" SAS disks for >use with ZFS. > >Thanks >-Peter >_______________________________________________ >zfs-discuss mailing list >zfs-discuss at opensolaris.org >http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Also consider looking at the HP MDS600 enclosure. http://h10010.www1.hp.com/wwpc/us/en/sm/WF05a/12169-304616-3930445-3930445- 3930445-3936271.html They''re available for <$1000US on eBay, fully ZFS-friendly and hold 70 x 3.5" disks. The only downside is that they were introduced as 3G SAS units. I cannot tell if 6G is possible now (via firmware or otherwise). -- Edmund White On 11/13/12 2:08 PM, "Peter Tripp" <peter at psych.columbia.edu> wrote:>Hi folks, > >I''m in the market for a couple of JBODs. Up until now I''ve been >relatively lucky with finding hardware that plays very nicely with ZFS. >All my gear currently in production uses LSI SAS controllers (3801e, >9200-16e, 9211-8i) with backplanes powered by LSI SAS expanders (Sun >x4250, Sun J4400, etc). But I''m in the market for SAS2 JBODs to support >a large number 3.5inch SAS disks (60+ 3TB disks to start). > >I''m aware of potential issues with SATA drives/interposers and the whole >SATA Tunneling Protocol (STP) nonsense, so I''m going to stick to a pure >SAS setup. Also, since I''ve had trouble with in the past with >daisy-chained SAS JBODs I''ll probably stick with one SAS 4x cable >(SFF8088) per JBOD and unless there were a compelling reason for >multi-pathing I''d probably stick to a single controller. If possible I''d >rather buy 20 packs of enterprise SAS disks with 5yr warranties and have >the JBOD come with empty trays, but would also consider buying disks with >the JBOD if the price wasn''t too crazy. > >Does anyone have any positive/negative experiences with any of the >following with ZFS: > * SuperMicro SC826E16-R500LPB (2U 12 drives, dual 500w PS, single LSI >SAS2X28 expander) > * SuperMicro SC846BE16-R920B (4U 24 drives, dual 920w PS, single unknown >expander) > * Dell PowerVault MD 1200 (2U 12 drives, dual 600w PS, dual unknown >expanders) > * HP StorageWorks D2600 (2U 12 drives, dual 460w PS, single/dual unknown >expanders) > >I''m leaning towards the SuperMicro stuff, but every time I order >SuperMicro gear there''s always something missing or wrongly configured so >some of the cost savings gets eaten up with my time figuring out where >things went wrong and returning/ordering replacements. The Dell/HP gear >I''m sure is fine, but buying disks from them gets pricey quick. The last >time I looked they charged $150 extra per disk for when the only added >value was a proprietary sled a shorter warranty (3yr vs 5yr). > >I''m open to other JBOD vendors too, was just really just curious what >folks were using when they needed more than two dozen 3.5" SAS disks for >use with ZFS. > >Thanks >-Peter >_______________________________________________ >zfs-discuss mailing list >zfs-discuss at opensolaris.org >http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
On Nov 13, 2012, at 12:08 PM, Peter Tripp <peter at psych.columbia.edu> wrote:> Hi folks, > > I''m in the market for a couple of JBODs. Up until now I''ve been relatively lucky with finding hardware that plays very nicely with ZFS. All my gear currently in production uses LSI SAS controllers (3801e, 9200-16e, 9211-8i) with backplanes powered by LSI SAS expanders (Sun x4250, Sun J4400, etc). But I''m in the market for SAS2 JBODs to support a large number 3.5inch SAS disks (60+ 3TB disks to start). > > I''m aware of potential issues with SATA drives/interposers and the whole SATA Tunneling Protocol (STP) nonsense, so I''m going to stick to a pure SAS setup. Also, since I''ve had trouble with in the past with daisy-chained SAS JBODs I''ll probably stick with one SAS 4x cable (SFF8088) per JBOD and unless there were a compelling reason for multi-pathing I''d probably stick to a single controller. If possible I''d rather buy 20 packs of enterprise SAS disks with 5yr warranties and have the JBOD come with empty trays, but would also consider buying disks with the JBOD if the price wasn''t too crazy. > > Does anyone have any positive/negative experiences with any of the following with ZFS: > * SuperMicro SC826E16-R500LPB (2U 12 drives, dual 500w PS, single LSI SAS2X28 expander) > * SuperMicro SC846BE16-R920B (4U 24 drives, dual 920w PS, single unknown expander) > * Dell PowerVault MD 1200 (2U 12 drives, dual 600w PS, dual unknown expanders) > * HP StorageWorks D2600 (2U 12 drives, dual 460w PS, single/dual unknown expanders)I''ve used all of the above and all of the DataOn systems, too (Hi Rocky!) No real complaints, though as others have noted the supermicro gear tends to require more work to get going. -- richard> I''m leaning towards the SuperMicro stuff, but every time I order SuperMicro gear there''s always something missing or wrongly configured so some of the cost savings gets eaten up with my time figuring out where things went wrong and returning/ordering replacements. The Dell/HP gear I''m sure is fine, but buying disks from them gets pricey quick. The last time I looked they charged $150 extra per disk for when the only added value was a proprietary sled a shorter warranty (3yr vs 5yr). > > I''m open to other JBOD vendors too, was just really just curious what folks were using when they needed more than two dozen 3.5" SAS disks for use with ZFS. > > Thanks > -Peter > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- Richard.Elling at RichardElling.com +1-760-896-4422 -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121113/d2a06478/attachment.html>
Hi Cedric, Sorry for the delayed reply, SATA vs SAS is a subject many ZFS sysadmins (including myself until recently) are completely ignorant about. I''m still not super knowledgable about STP, but here goes. Normal SAS devices communicate with your controller via Serial SCSI Protocol (SSP), while SATA can communicate using SATA Tunneling Protocol (STP). Feel free to look into the specifications for each to see real differences, but among other things STP is limited by the command set implemented by the drive in firmware; basically SATA drives have fewer configuration options and among other things may or may not have a SAS WWN other constant GUID. SAS devices implement additional monitoring and failure modes as well as an awareness and methods of communication with multiple initiators. As for what this means in the practical world, when a SATA device is behind a SAS switch (like in a backplane) certain situations must be handled properly by the interposer since the SATA drive itself has no concept of ''disconnect initiator A and connect to initiator B'' or ''this drive is now attached to a new initiator, let''s make sure it has the same GUID as before''. Do you know who wrote/debugged the firmware on your interposer? I don''t have a personal story where an interposter/SAS switch screwed me, but have I have heard multiple folks (Theo @ OmniTI and Garrett @ Nexenta) blame them for really hard to debug problems. I''ll leave you with a quote from Garrett: "At a significant account, I can say that we (meaning Nexenta) have verified that SAS/SATA expanders combined with high loads of ZFS activity have proven conclusively to be highly toxic. So, if you''re designing an enterprise storage solution, please consider using SAS all the way to the disk drives, and just skip those cheaper SATA options. You may think SATA looks like a bargain, but when your array goes offline during ZFS scrub or resilver operations because the expander is choking on cache sync commands, you''ll really wish you had spent the extra cash up front. Really." http://gdamore.blogspot.com/2010/08/why-sas-sata-is-not-such-great-idea.html So wherever possible I''d try and use all SAS. Where I must use a SATA SSD I''ll driectly connect to the SAS controller on a dedicated channel and not behind a SAS switch and interposer. I''m not sure that helps, but at least there''s one less set of firmware between ZFS and my disks. That said, I still have a Sun J4400 JBOD and two Dell MD1000 devices filled with 1TB SATA disks behind SAS interposers, but when they''re retired you can bet the replacement will 100% SAS. Thanks -Pete P.S. Also note, sometimes devices with a SAS interface aren''t what they seem. Supposedly the OCZ Talos SSDs are actually SATA devices with an internal interposer. P.P.S. Hadn''t seen those SGI JBODs, those are silly high density, I can''t imagine those come cheap and I''m sure they are deeper than most things in your rack. Dell/DataOn do 60 disks in 4U, but with eleven of the SGI JBODs in a single rack (44U), that''s a whopping 660 4TB disks (2.4PB raw) per cabinet. Truly silly silly dense. On Nov 13, 2012, at 3:57 PM, Cedric Tineo wrote:> Peter, > > Could you please give info or links to clarify "Sata tunneling protocol nonsense" issues? > > Also have you considered Supermicro''s 45 drive in 4U enclosure - The SC847E26-RJBOD1 ? It''s cheap too at around 2000$ and based on LSI backplanes anyway. > > SGI has a nice and crazy 81(!) 3.5" disks in 4U JBOD enclosure but they refused to sell it to us without them supplying the disks also. http://www.sgi.com/products/storage/modular/jbod.html > > Anyone aware of a 60 or more disks in 4U enclosure on top of those mentioned in this message or the data-on? We are trying to build super-high-density-storage racks. > > Cedric Tineo > > > On 13 nov. 2012, at 21:08, Peter Tripp <peter at psych.columbia.edu> wrote: > >> Hi folks, >> >> I''m in the market for a couple of JBODs. Up until now I''ve been relatively lucky with finding hardware that plays very nicely with ZFS. All my gear currently in production uses LSI SAS controllers (3801e, 9200-16e, 9211-8i) with backplanes powered by LSI SAS expanders (Sun x4250, Sun J4400, etc). But I''m in the market for SAS2 JBODs to support a large number 3.5inch SAS disks (60+ 3TB disks to start). >> >> I''m aware of potential issues with SATA drives/interposers and the whole SATA Tunneling Protocol (STP) nonsense, so I''m going to stick to a pure SAS setup. Also, since I''ve had trouble with in the past with daisy-chained SAS JBODs I''ll probably stick with one SAS 4x cable (SFF8088) per JBOD and unless there were a compelling reason for multi-pathing I''d probably stick to a single controller. If possible I''d rather buy 20 packs of enterprise SAS disks with 5yr warranties and have the JBOD come with empty trays, but would also consider buying disks with the JBOD if the price wasn''t too crazy. >> >> Does anyone have any positive/negative experiences with any of the following with ZFS: >> * SuperMicro SC826E16-R500LPB (2U 12 drives, dual 500w PS, single LSI SAS2X28 expander) >> * SuperMicro SC846BE16-R920B (4U 24 drives, dual 920w PS, single unknown expander) >> * Dell PowerVault MD 1200 (2U 12 drives, dual 600w PS, dual unknown expanders) >> * HP StorageWorks D2600 (2U 12 drives, dual 460w PS, single/dual unknown expanders) >> >> I''m leaning towards the SuperMicro stuff, but every time I order SuperMicro gear there''s always something missing or wrongly configured so some of the cost savings gets eaten up with my time figuring out where things went wrong and returning/ordering replacements. The Dell/HP gear I''m sure is fine, but buying disks from them gets pricey quick. The last time I looked they charged $150 extra per disk for when the only added value was a proprietary sled a shorter warranty (3yr vs 5yr). >> >> I''m open to other JBOD vendors too, was just really just curious what folks were using when they needed more than two dozen 3.5" SAS disks for use with ZFS. >> >> Thanks >> -Peter >> _______________________________________________ >> zfs-discuss mailing list >> zfs-discuss at opensolaris.org >> http://mail.opensolaris.org/mailman/listinfo/zfs-discuss > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20121120/11c21795/attachment.html>