Eugen Leitl
2009-Sep-09 14:37 UTC
[zfs-discuss] alternative hardware configurations for zfs
Inspired by http://www.webhostingtalk.com/showpost.php?p=6334764&postcount=14 I''m considering taking the Supermicro chassis like http://www.supermicro.com/products/chassis/4U/846/SC846E1-R900.cfm populating it with 1 TByte WD Caviar Black WD1001FALS with TLER set to 7 seconds, plus using 2x Intel X25-E and 2x Intel X25-M Gen2 80GB X25-Es as ZILs and X25-Ms as L2ARC cache, taking 3x LSI SAS3081E-R and a well-supported Supermicro board (any suggestions as to which? I understand that the HCL for Opensolaris is not very up-to-date http://www.sun.com/bigadmin/hcl/data/os/systems/views/all_motherboards_all_results.page1.html ) for my next storage system. Do you think above is a sensible choice? -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE
Eric Sproul
2009-Sep-10 17:11 UTC
[zfs-discuss] alternative hardware configurations for zfs
Eugen Leitl wrote:> Inspired by > http://www.webhostingtalk.com/showpost.php?p=6334764&postcount=14 > I''m considering taking the Supermicro chassis like > http://www.supermicro.com/products/chassis/4U/846/SC846E1-R900.cfm > populating it with 1 TByte WD Caviar Black WD1001FALS with TLER > set to 7 seconds,I would not use the Caviar Black drives, regardless of TLER settings. The RE3 or RE4 drives would be a better choice, since they also have better vibration tolerance. This will be a significant factor in a chassis with 20 spinning drives.> Do you think above is a sensible choice?All your other choices seem good. I''ve used a lot of Supermicro gear with good results. The very leading-edge hardware is sometimes not supported, but anything that''s been out for a while should work fine. I presume you''re going for an Intel Xeon solution-- the peripherals on those boards a a bit better supported than the AMD stuff, but even the AMD boards work well. Eric -- Eric Sproul Lead Site Reliability Engineer OmniTI Computer Consulting, Inc. Web Applications & Internet Architectures http://omniti.com
Why do you need 3x LSI SAS3081E-R? The back plane has LSI SAS x36 expander so you only nedd 1x 3081E. If you want multipathing, you need E2 model. Second, I''d say use Seagate ES 2 1TB SAS disk especially if you want multipathing. I believe E2 only supports SAS disks. I have Supermicro 936E1 (LSI SAS X28 expander) as diskshelf and LSI 3080X on head unit, Intel X25-E as ZIL, works like charm. Your setup is very well supported by Solaris. For motherboard, my Supermicro X8SAX and X8ST3 both work well with Solaris. You may want dual proc board that supports more memory. ECC is given on i7 based when using XEON. -- This message posted from opensolaris.org
Eugen Leitl
2009-Sep-11 06:51 UTC
[zfs-discuss] alternative hardware configurations for zfs
On Thu, Sep 10, 2009 at 01:11:49PM -0400, Eric Sproul wrote:> I would not use the Caviar Black drives, regardless of TLER settings. The RE3 > or RE4 drives would be a better choice, since they also have better vibration > tolerance. This will be a significant factor in a chassis with 20 spinning drives.Yes, I''m aware of the issue, and am using 16x RE4 drives in my current box right now (which I unfortunately had to convert to CentOS 5.3 for Oracle/ custom software compatibility reasons). I''ve made very bad experiences with Seagate 7200.11 in RAID in the past. Thanks for your advice against Caviar Black.> > Do you think above is a sensible choice? > > All your other choices seem good. I''ve used a lot of Supermicro gear with good > results. The very leading-edge hardware is sometimes not supported, butI''ve been using http://www.supermicro.com/products/motherboard/QPI/5500/X8DAi.cfm in above box.> anything that''s been out for a while should work fine. I presume you''re going > for an Intel Xeon solution-- the peripherals on those boards a a bit better > supported than the AMD stuff, but even the AMD boards work well.Yes, dual-socket quadcore Xeon. -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE
Markus Kovero
2009-Sep-11 07:00 UTC
[zfs-discuss] alternative hardware configurations for zfs
We''ve been using caviar black 1TB with disk configurations consisting 64 disks or more. They are working just fine. Yours Markus Kovero -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Eugen Leitl Sent: 11. syyskuuta 2009 9:51 To: Eric Sproul; zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] alternative hardware configurations for zfs On Thu, Sep 10, 2009 at 01:11:49PM -0400, Eric Sproul wrote:> I would not use the Caviar Black drives, regardless of TLER settings. The RE3 > or RE4 drives would be a better choice, since they also have better vibration > tolerance. This will be a significant factor in a chassis with 20 spinning drives.Yes, I''m aware of the issue, and am using 16x RE4 drives in my current box right now (which I unfortunately had to convert to CentOS 5.3 for Oracle/ custom software compatibility reasons). I''ve made very bad experiences with Seagate 7200.11 in RAID in the past. Thanks for your advice against Caviar Black.> > Do you think above is a sensible choice? > > All your other choices seem good. I''ve used a lot of Supermicro gear with good > results. The very leading-edge hardware is sometimes not supported, butI''ve been using http://www.supermicro.com/products/motherboard/QPI/5500/X8DAi.cfm in above box.> anything that''s been out for a while should work fine. I presume you''re going > for an Intel Xeon solution-- the peripherals on those boards a a bit better > supported than the AMD stuff, but even the AMD boards work well.Yes, dual-socket quadcore Xeon. -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Tristan Ball
2009-Sep-11 07:03 UTC
[zfs-discuss] alternative hardware configurations for zfs
How long have you had them in production? Were you able to adjust the TLER settings from within solaris? Thanks, Tristan. -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Markus Kovero Sent: Friday, 11 September 2009 5:00 PM To: Eugen Leitl; Eric Sproul; zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] alternative hardware configurations for zfs We''ve been using caviar black 1TB with disk configurations consisting 64 disks or more. They are working just fine. Yours Markus Kovero -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Eugen Leitl Sent: 11. syyskuuta 2009 9:51 To: Eric Sproul; zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] alternative hardware configurations for zfs On Thu, Sep 10, 2009 at 01:11:49PM -0400, Eric Sproul wrote:> I would not use the Caviar Black drives, regardless of TLER settings.The RE3> or RE4 drives would be a better choice, since they also have bettervibration> tolerance. This will be a significant factor in a chassis with 20spinning drives. Yes, I''m aware of the issue, and am using 16x RE4 drives in my current box right now (which I unfortunately had to convert to CentOS 5.3 for Oracle/ custom software compatibility reasons). I''ve made very bad experiences with Seagate 7200.11 in RAID in the past. Thanks for your advice against Caviar Black.> > Do you think above is a sensible choice? > > All your other choices seem good. I''ve used a lot of Supermicro gearwith good> results. The very leading-edge hardware is sometimes not supported,but I''ve been using http://www.supermicro.com/products/motherboard/QPI/5500/X8DAi.cfm in above box.> anything that''s been out for a while should work fine. I presumeyou''re going> for an Intel Xeon solution-- the peripherals on those boards a a bitbetter> supported than the AMD stuff, but even the AMD boards work well.Yes, dual-socket quadcore Xeon. -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________
Markus Kovero
2009-Sep-11 07:10 UTC
[zfs-discuss] alternative hardware configurations for zfs
Couple months, nope. I guess there is this DOS utility provided by WD that allows you change TLER settings having TLER disabled can be problem, faulty disks timeout randomly and zfs doesn''t always want to mark them as failed, sometimes it does though. Yours Markus Kovero -----Original Message----- From: Tristan Ball [mailto:Tristan.Ball at leica-microsystems.com] Sent: 11. syyskuuta 2009 10:04 To: Markus Kovero; zfs-discuss at opensolaris.org Subject: RE: [zfs-discuss] alternative hardware configurations for zfs How long have you had them in production? Were you able to adjust the TLER settings from within solaris? Thanks, Tristan. -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Markus Kovero Sent: Friday, 11 September 2009 5:00 PM To: Eugen Leitl; Eric Sproul; zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] alternative hardware configurations for zfs We''ve been using caviar black 1TB with disk configurations consisting 64 disks or more. They are working just fine. Yours Markus Kovero -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of Eugen Leitl Sent: 11. syyskuuta 2009 9:51 To: Eric Sproul; zfs-discuss at opensolaris.org Subject: Re: [zfs-discuss] alternative hardware configurations for zfs On Thu, Sep 10, 2009 at 01:11:49PM -0400, Eric Sproul wrote:> I would not use the Caviar Black drives, regardless of TLER settings.The RE3> or RE4 drives would be a better choice, since they also have bettervibration> tolerance. This will be a significant factor in a chassis with 20spinning drives. Yes, I''m aware of the issue, and am using 16x RE4 drives in my current box right now (which I unfortunately had to convert to CentOS 5.3 for Oracle/ custom software compatibility reasons). I''ve made very bad experiences with Seagate 7200.11 in RAID in the past. Thanks for your advice against Caviar Black.> > Do you think above is a sensible choice? > > All your other choices seem good. I''ve used a lot of Supermicro gearwith good> results. The very leading-edge hardware is sometimes not supported,but I''ve been using http://www.supermicro.com/products/motherboard/QPI/5500/X8DAi.cfm in above box.> anything that''s been out for a while should work fine. I presumeyou''re going> for an Intel Xeon solution-- the peripherals on those boards a a bitbetter> supported than the AMD stuff, but even the AMD boards work well.Yes, dual-socket quadcore Xeon. -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss ______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________
Eugen Leitl
2009-Sep-11 08:34 UTC
[zfs-discuss] alternative hardware configurations for zfs
On Thu, Sep 10, 2009 at 11:54:16AM -0700, Chris Du wrote:> Why do you need 3x LSI SAS3081E-R? The back plane has LSI SAS x36 expander so you only nedd 1x 3081E. If you want multipathing, you need E2 model.Can you use SATA drives with expanders at all? (I have to stick to enterprise/nearline SATA (100 EUR/TByte vs. 60 EUR/TByte consumer SATA) for cost reasons). Also, won''t you get oversubscription at such large disk populations on a single host adapter? I was thinking 8x SATA on an 8-lane PCI Express (what, about 8*200 MByte/s nominal bandwidth?) was a more conservative setting.> Second, I''d say use Seagate ES 2 1TB SAS disk especially if you want multipathing. I believe E2 only supports SAS disks. > > I have Supermicro 936E1 (LSI SAS X28 expander) as diskshelf andWhat is the advantage of using external disk expanders? They use up more rack height units and add hardware expense and cabling hassle for very little to show for it, IMHO. Supermicro makes 24-drive chassis with redundant power supplies which can take server motherboards with enough PCI Express slots and CPU power to serve them. If you need more storage, a cluster file system (e.g. pNFS aka NFS 4.1 or PVFS2) can be used to build up nodes. Granted, you''ll probably need InfiniBand in order to make optimal use of it within the cluster as even channel-bonded GBit Ethernet will peak at 480 MBytes/s, or so.> LSI 3080X on head unit, Intel X25-E as ZIL, works like charm. > Your setup is very well supported by Solaris.Thank you for the confirmation.> For motherboard, my Supermicro X8SAX and X8ST3 both work well with Solaris. > You may want dual proc board that supports more memory. ECC is given on i7 based when using XEON.Given that this is a hybrid application (application in Solaris containers accessing zfs pool on the same machine) I realize ECC is very important. -- Eugen* Leitl <a href="http://leitl.org">leitl</a> http://leitl.org ______________________________________________________________ ICBM: 48.07100, 11.36820 http://www.ativel.com http://postbiota.org 8B29F6BE: 099D 78BA 2FD3 B014 B08A 7779 75B0 2443 8B29 F6BE
Joseph L. Casale
2009-Sep-11 11:48 UTC
[zfs-discuss] alternative hardware configurations for zfs
>Can you use SATA drives with expanders at all? (I have to stick >to enterprise/nearline SATA (100 EUR/TByte vs. 60 EUR/TByte >consumer SATA) for cost reasons).Yes, the expander has nothing to do with the drive in front of it. I have several SAS expanders with SATA drives on them.>What is the advantage of using external disk expanders? >They use up more rack height units and add hardware expense >and cabling hassle for very little to show for it, IMHO.Because a single port on any HBA/RAID controller far exceeds the speed of any single disc. Your analogy doesn''t scal well past what you can fit into a single server. You also assume every disc enclosure only holds the discs in a similar fashion to the way they would be oriented in the supermicro chassis. Here''s a nice read: http://www.scsita.org/aboutscsi/sas/SAS_Expander.doc jlc
>>Can you use SATA drives with expanders at all? (I have to stick to enterprise/nearline SATA (100 EUR/TByte vs. 60 EUR/TByte consumer SATA) for cost reasons).Yes you can in E1 model. E1 is single path model which supports both SAS and SATA. You need to know what you are buying. The Supermicro case you buy has backplane with SAS expander. In E2 model which supports multipathing, dual-port is required because each path needs access to the disk at the same time, thus SAS is required, SATA is single-port. If you want more bandwidth between your HBA and disks and better redundancy, you need multipathing so E2. I still suggest you go with nearline SAS. SAS is dual-port design, it has much better IOPS, command queue and error recovery. Data transfer speed is same between NS2 SAS and SATA. I know a lot of people have big problem with 7200.11 but this is not the same disk. It does cost even more than enterprise SATA. We have SATA disk shelves in NetApp, I say SATA doesn''t belong to Enterprise. Granted, SATA is only used in dev environment, production uses 15K FC disks which we never have performance issue. The reason I use external disk shelf is I ran out of disk trays in head unit. Adding SAS shelf is the quick, easy and cheap way to expand storage. I won''t touch cluster file system as it gets too complicated and way toooooo expensive. -- This message posted from opensolaris.org
On Fri, Sep 11, 2009 at 12:48 PM, Chris Du <dilidolo at gmail.com> wrote:> >>Can you use SATA drives with expanders at all? (I have to stick to > enterprise/nearline SATA (100 EUR/TByte vs. 60 EUR/TByte consumer SATA) for > cost reasons). > > Yes you can in E1 model. E1 is single path model which supports both SAS > and SATA. You need to know what you are buying. The Supermicro case you buy > has backplane with SAS expander. > > In E2 model which supports multipathing, dual-port is required because each > path needs access to the disk at the same time, thus SAS is required, SATA > is single-port. If you want more bandwidth between your HBA and disks and > better redundancy, you need multipathing so E2. > > I still suggest you go with nearline SAS. SAS is dual-port design, it has > much better IOPS, command queue and error recovery. Data transfer speed is > same between NS2 SAS and SATA. I know a lot of people have big problem with > 7200.11 but this is not the same disk. It does cost even more than > enterprise SATA. > > We have SATA disk shelves in NetApp, I say SATA doesn''t belong to > Enterprise. Granted, SATA is only used in dev environment, production uses > 15K FC disks which we never have performance issue. > > The reason I use external disk shelf is I ran out of disk trays in head > unit. Adding SAS shelf is the quick, easy and cheap way to expand storage. I > won''t touch cluster file system as it gets too complicated and way toooooo > expensive. >Better IOPS? Do you have some numbers to back that claim up? I''ve never heard of anyone getting "much better" IOPS out of a drive by simply changing the interface from SATA to SAS. Or SATA to FATA for that matter. A 7200RPM drive is limited by the 7200RPM''s, not the interface it''s attached to. --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090911/a2cca5f8/attachment.html>
Eric D. Mudama
2009-Sep-11 20:20 UTC
[zfs-discuss] alternative hardware configurations for zfs
On Fri, Sep 11 at 13:14, Tim Cook wrote:> Better IOPS? Do you have some numbers to back that claim up? I''ve never > heard of anyone getting "much better" IOPS out of a drive by simply > changing the interface from SATA to SAS. Or SATA to FATA for that > matter. A 7200RPM drive is limited by the 7200RPM''s, not the interface > it''s attached to.Depends on the model of drive. A number of vendors put relatively larger magnets and stronger actuators in their enterprise designs, where the customers are willing to pay for it. This can significantly decrease track-to-track seek times, which improves IOPS. On top of that, many enterprise drives are using smaller platters and/or higher RPM, both of which also help IOPS at shallow queue depths. At infinitely high queue depth, IOPS basically becomes a function of how quickly the servo system can settle, since seek distances approach zero (both linearly and rotationally) as the number of operations to choose from goes to infinity. -- Eric D. Mudama edmudama at mail.bounceswoosh.org
On Fri, Sep 11, 2009 at 3:20 PM, Eric D. Mudama <edmudama at bounceswoosh.org>wrote:> On Fri, Sep 11 at 13:14, Tim Cook wrote: > >> Better IOPS? Do you have some numbers to back that claim up? I''ve never >> heard of anyone getting "much better" IOPS out of a drive by simply >> changing the interface from SATA to SAS. Or SATA to FATA for that >> matter. A 7200RPM drive is limited by the 7200RPM''s, not the interface >> it''s attached to. >> > > Depends on the model of drive. > > A number of vendors put relatively larger magnets and stronger > actuators in their enterprise designs, where the customers are willing > to pay for it. This can significantly decrease track-to-track seek > times, which improves IOPS. > > On top of that, many enterprise drives are using smaller platters > and/or higher RPM, both of which also help IOPS at shallow queue > depths. > > At infinitely high queue depth, IOPS basically becomes a function of > how quickly the servo system can settle, since seek distances approach > zero (both linearly and rotationally) as the number of operations to > choose from goes to infinity. > >The question wasn''t about consumer vs. enterprise drives. He said the SAS interface improves IOPS. Please don''t change the topic of discussion mid-thread. --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090911/66946014/attachment.html>
You can optimize for better IOPS or for transfer speed. NS2 SATA and SAS share most of the design, but they are still different, cache, interface, firmware are all different. Then by much better, I don''t mean just IOPS, it''s all the 3, better IOPS, command queue and error recovery, etc. -- This message posted from opensolaris.org
Eric D. Mudama
2009-Sep-11 22:32 UTC
[zfs-discuss] alternative hardware configurations for zfs
On Fri, Sep 11 at 16:15, Tim Cook wrote:> The question wasn''t about consumer vs. enterprise drives. He said the SAS > interface improves IOPS. Please don''t change the topic of discussion > mid-thread.Sorry, wasn''t trying to derail, but most people don''t make the distinctions you do. I think SATA actually has slightly lower protocol overhead than SAS, and thus should be capable of slightly more IOPS if not limited elsewhere. --eric -- Eric D. Mudama edmudama at mail.bounceswoosh.org
On Fri, Sep 11, 2009 at 4:46 PM, Chris Du <dilidolo at gmail.com> wrote:> You can optimize for better IOPS or for transfer speed. NS2 SATA and SAS > share most of the design, but they are still different, cache, interface, > firmware are all different. > > Then by much better, I don''t mean just IOPS, it''s all the 3, better IOPS, > command queue and error recovery, etc. >And I''m asking you to provide a factual basis for the interface playing any role in IOPS. I know for a fact it has nothing to do with error recovery or command queue. Cache size and type isn''t changed based on interface, whoever told you that was selling you beachfront property in Phoenix. It also won''t change IOPS because sooner or later you''re going to spinning media. It may help with bursty sequential transfers, it will not do a thing for sustained random I/O. Firmware may be different but the only differences there that I''ve ever seen are to provide the proper intelligence to talk SAS, SATA, or FC respectively. Regardless, I''ve never seen either one provide any significant change in IOPS. I feel fairly confident stating that within the storage industry there''s a pretty well known range of IOPS provided for 7200, 10K, and 15K drives respectively, regardless of interface. You appear to be saying this isn''t the case, so I''d like to know what data you''re using as a reference point. --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090911/272280c8/attachment.html>
Damjan Perenic
2009-Sep-12 15:17 UTC
[zfs-discuss] alternative hardware configurations for zfs
On Sat, Sep 12, 2009 at 7:25 AM, Tim Cook <tim at cook.ms> wrote:> > > On Fri, Sep 11, 2009 at 4:46 PM, Chris Du <dilidolo at gmail.com> wrote: >> >> You can optimize for better IOPS or for transfer speed. NS2 SATA and SAS >> share most of the design, but they are still different, cache, interface, >> firmware are all different. > > And I''m asking you to provide a factual basis for the interface playing any > role in IOPS.? I know for a fact it has nothing to do with error recovery or > command queue. > > Regardless, I''ve never seen either one provide any significant change in > IOPS.? I feel fairly confident stating that within the storage industry > there''s a pretty well known range of IOPS provided for 7200, 10K, and 15K > drives respectively, regardless of interface.? You appear to be saying this > isn''t the case, so I''d like to know what data you''re using as a reference > point.I shopped for 1TB 7200rpm drives recently and I noticed Seagate Barracude ES.2 has 1TB version with SATA and SAS interface. In their datasheet at http://www.seagate.com/www/en-us/products/servers/barracuda_es/ and product overview they claim following: --- Choose SAS for the seamless Tier 2 enterprise experience, with improved data integrity and a 135 percent average performance boost over SATA. SAS also reduces integration complexity and optimizes system performance for rich media, reference data storage and enterprise backup applications. --- With a choice of either SATA or SAS interfaces, the Barracuda ES.2 drive utilizes perpendicular recording technology to deliver the industry?s highest-capacity 4-platter drive. SAS delivers up to a 38 percent IOPS/watt improvement over SATA. --- And in Product overview: --- ? Full internal IOEDC/IOECC* data integrity protection on SAS models ? Dual-ported, multi-initiator SAS provides full-duplex compatibility and a 135 percent average** performance improvement over SATA. *IOEDC/IOECC on SATA (writes only), IOEDC/IOECC on SAS (both reads and writes) **Averaged from random/sequential, read/write activities with write cache off -- I admit I have no clue why SAS version should be/is faster. I just pass on things I found out. But I am interested in opinion if there is any substance in this marketing material. Kind regards, Damjan
On Sat, Sep 12, 2009 at 10:17 AM, Damjan Perenic < damjan.perenic at guest.arnes.si> wrote:> On Sat, Sep 12, 2009 at 7:25 AM, Tim Cook <tim at cook.ms> wrote: > > > > > > On Fri, Sep 11, 2009 at 4:46 PM, Chris Du <dilidolo at gmail.com> wrote: > >> > >> You can optimize for better IOPS or for transfer speed. NS2 SATA and SAS > >> share most of the design, but they are still different, cache, > interface, > >> firmware are all different. > > > > And I''m asking you to provide a factual basis for the interface playing > any > > role in IOPS. I know for a fact it has nothing to do with error recovery > or > > command queue. > > > > Regardless, I''ve never seen either one provide any significant change in > > IOPS. I feel fairly confident stating that within the storage industry > > there''s a pretty well known range of IOPS provided for 7200, 10K, and 15K > > drives respectively, regardless of interface. You appear to be saying > this > > isn''t the case, so I''d like to know what data you''re using as a reference > > point. > > I shopped for 1TB 7200rpm drives recently and I noticed Seagate > Barracude ES.2 has 1TB version with SATA and SAS interface. > > In their datasheet at > http://www.seagate.com/www/en-us/products/servers/barracuda_es/ and > product overview they claim following: > > --- > Choose SAS for the seamless Tier 2 enterprise experience, with > improved data integrity and a 135 percent average performance > boost over SATA. SAS also reduces integration complexity and > optimizes system performance for rich media, reference data > storage and enterprise backup applications. > --- > With a choice of either SATA or SAS > interfaces, the Barracuda ES.2 drive > utilizes perpendicular recording technology > to deliver the industry?s highest-capacity > 4-platter drive. SAS delivers up to a 38 > percent IOPS/watt improvement over > SATA. > --- > > And in Product overview: > --- > ? Full internal IOEDC/IOECC* data integrity protection on SAS models > ? Dual-ported, multi-initiator SAS provides full-duplex compatibility > and a 135 percent average** performance improvement over SATA. > > *IOEDC/IOECC on SATA (writes only), IOEDC/IOECC on SAS (both reads and > writes) > **Averaged from random/sequential, read/write activities with write cache > off > -- > > I admit I have no clue why SAS version should be/is faster. I just > pass on things I found out. But I am interested in opinion if there is > any substance in this marketing material. > > Kind regards, > Damjan >The two *''s leave much room to be desired. Averaged? How about some real numbers with testing methodology. I''m not at all surprised they claim 2x performance from the drive they charge twice as much for. On the flip side, according to storage review, the SATA version trumps the SAS version in pretty much everything but throughput (which is still negligible). http://www.storagereview.com/php/benchmark/suite_v4.php?typeID=10&testbedID=4&osID=6&raidconfigID=1&numDrives=1&devID_0=354&devID_1=362&devCnt=2 --Tim -------------- next part -------------- An HTML attachment was scrubbed... URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20090912/0c663bb2/attachment.html>
Jens Elkner
2009-Sep-14 02:38 UTC
[zfs-discuss] alternative hardware configurations for zfs
On Sat, Sep 12, 2009 at 02:37:35PM -0500, Tim Cook wrote:> On Sat, Sep 12, 2009 at 10:17 AM, Damjan Perenic...> I shopped for 1TB 7200rpm drives recently and I noticed Seagate > Barracude ES.2 has 1TB version with SATA and SAS interface. > > On the flip side, according to storage review, the SATA version trumps the > SAS version in pretty much everything but throughput (which is still > negligible). > [5]http://www.storagereview.com/php/benchmark/suite_v4.php?typeID=10&testbed > ID=4&osID=6&raidconfigID=1&numDrives=1&devID_0=354&devID_1=362&devCnt=2 > --TimJust in case if interested in SATA, perhaps this helps (made on an almost idle system): elkner.sol /pool2 > uname -a SunOS sol 5.11 snv_98 i86pc i386 i86xpv elkner.sol /rpool > prtdiag System Configuration: Intel S5000PAL BIOS Configuration: Intel Corporation S5000.86B.10.00.0091.081520081046 08/15/2008 BMC Configuration: IPMI 2.0 (KCS: Keyboard Controller Style) ==== Processor Sockets =================================== Version Location Tag -------------------------------- -------------------------- Intel(R) Xeon(R) CPU E5440 @ 2.83GHz CPU1 Intel(R) Xeon(R) CPU E5440 @ 2.83GHz CPU2 ... elkner.sol /pool2 > + /usr/X11/bin/scanpci | grep -i sata Intel Corporation 631xESB/632xESB SATA AHCI Controller elkner.sol ~ > iostat -E | \ awk ''/^sd/ { print $1; getline; print; getline; print }'' sd0 Vendor: ATA Product: ST3250310NS Revision: SN05 Serial No: Size: 250.06GB <250059350016 bytes> sd1 Vendor: ATA Product: ST3250310NS Revision: SN04 Serial No: Size: 250.06GB <250059350016 bytes> sd2 Vendor: ATA Product: ST3250310NS Revision: SN04 Serial No: Size: 250.06GB <250059350016 bytes> sd3 Vendor: ATA Product: ST3250310NS Revision: SN05 Serial No: Size: 250.06GB <250059350016 bytes> sd5 Vendor: ATA Product: ST31000340NS Revision: SN06 Serial No: Size: 1000.20GB <1000204886016 bytes> sd6 Vendor: ATA Product: ST31000340NS Revision: SN06 Serial No: Size: 1000.20GB <1000204886016 bytes> elkner.sol ~ > zpool status | grep ONLINE state: ONLINE pool1 ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t2d0 ONLINE 0 0 0 c1t3d0 ONLINE 0 0 0 state: ONLINE pool2 ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t4d0 ONLINE 0 0 0 c1t5d0 ONLINE 0 0 0 state: ONLINE rpool ONLINE 0 0 0 mirror ONLINE 0 0 0 c1t0d0s0 ONLINE 0 0 0 c1t1d0s0 ONLINE 0 0 0 elkner.sol /pool2 > + time sh -c "mkfile 4g xx; sync; echo ST31000340NS" ST31000340NS real 3:55.2 user 0.0 sys 1.9 elkner.sol ~ > iostat -zmnx c1t4d0 c1t5d0 5 | grep -v device 0.0 154.2 0.0 19739.4 3.0 32.0 19.4 207.5 100 100 c1t4d0 0.0 125.8 0.0 16103.9 3.0 32.0 23.8 254.3 100 100 c1t5d0 0.0 133.0 0.0 16366.9 2.4 25.9 17.9 194.4 80 82 c1t4d0 0.0 158.0 0.0 19592.5 2.8 30.3 17.6 191.7 93 96 c1t5d0 0.0 159.4 0.0 20054.8 2.8 30.3 17.7 190.2 94 95 c1t4d0 0.0 140.2 0.0 17597.2 2.8 30.3 20.1 216.4 94 95 c1t5d0 0.0 134.8 0.0 16298.7 2.0 23.0 15.2 170.8 68 76 c1t4d0 0.0 154.4 0.0 18807.5 2.7 29.3 17.3 189.9 89 94 c1t5d0 0.0 188.4 0.0 24115.5 3.0 32.0 15.9 169.8 100 100 c1t4d0 0.0 159.8 0.0 20454.6 3.0 32.0 18.8 200.2 100 100 c1t5d0 0.0 120.0 0.0 14328.3 2.0 22.2 16.4 184.9 66 71 c1t4d0 0.0 143.2 0.0 17169.9 2.6 28.2 18.0 197.1 86 93 c1t5d0 0.0 157.0 0.0 19140.9 2.6 29.3 16.5 186.9 87 96 c1t4d0 0.0 169.2 0.0 20676.9 2.2 24.8 13.2 146.6 75 79 c1t5d0 0.0 156.2 0.0 19993.8 3.0 32.0 19.2 204.8 100 100 c1t4d0 0.0 140.4 0.0 17971.3 3.0 32.0 21.3 227.9 100 100 c1t5d0 0.0 138.8 0.0 16759.6 2.6 29.3 18.4 210.9 86 94 c1t4d0 0.0 146.6 0.0 17809.2 2.7 29.6 18.4 201.7 90 94 c1t5d0 0.0 133.8 0.0 16196.8 2.5 28.0 18.9 209.3 85 90 c1t4d0 0.0 134.0 0.0 16222.4 2.6 28.7 19.5 214.3 87 94 c1t5d0 r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device elkner.sol /pool1 > + time sh -c ''mkfile 4g xx; sync; echo ST3250310NS'' ST3250310NS real 1:33.5 user 0.0 sys 2.0 elkner.sol ~ > iostat -zmnx c1t2d0 c1t3d0 5 | grep -v device 0.2 408.6 1.6 49336.8 25.7 0.8 62.8 1.9 79 79 c1t3d0 0.2 432.6 1.6 53284.4 29.9 0.9 69.0 2.1 89 89 c1t2d0 0.2 456.0 1.6 56280.0 28.6 0.9 62.6 1.9 86 86 c1t3d0 0.8 389.8 17.6 45360.7 25.8 0.8 66.0 2.1 81 80 c1t2d0 0.4 368.6 3.2 42698.0 21.1 0.6 57.3 1.8 65 65 c1t3d0 1.0 432.4 8.0 52615.8 30.2 0.9 69.6 2.1 91 91 c1t2d0 0.0 457.8 0.0 56423.3 28.3 0.8 61.8 1.8 85 84 c1t3d0 1.0 418.4 11.2 50232.5 28.7 0.9 68.4 2.1 89 89 c1t2d0 0.8 406.4 8.0 48089.0 25.0 0.8 61.5 2.0 80 79 c1t3d0 0.6 381.4 4.8 42019.3 23.9 0.8 62.7 2.1 80 80 c1t2d0 0.4 387.0 3.2 42580.9 21.7 0.7 56.0 1.8 70 70 c1t3d0 0.4 392.8 4.8 48067.8 30.6 0.9 77.9 2.3 92 92 c1t2d0 0.4 372.4 3.2 45343.2 27.5 0.8 73.8 2.2 82 83 c1t3d0 0.6 324.4 16.0 38879.6 29.8 0.9 91.6 2.8 90 91 c1t2d0 0.8 329.0 40.0 39828.9 28.6 0.9 86.8 2.6 87 87 c1t3d0 0.0 296.2 0.0 34579.9 26.5 0.9 89.6 3.1 90 90 c1t2d0 0.4 288.2 14.4 33361.7 23.6 0.7 81.8 2.6 73 74 c1t3d0 0.8 341.8 6.4 41018.7 29.1 1.2 85.0 3.6 88 95 c1t2d0 2.4 344.2 20.8 41377.1 29.3 1.7 84.6 4.9 89 96 c1t3d0 3.8 338.4 209.6 38739.9 26.3 0.9 76.8 2.5 85 86 c1t2d0 3.6 340.4 207.6 38893.5 23.4 0.8 68.0 2.2 76 76 c1t3d0 0.2 401.6 1.6 48459.1 27.2 0.8 67.7 2.1 85 85 c1t2d0 0.6 411.8 6.4 49058.0 24.9 0.8 60.3 1.8 77 76 c1t3d0 0.2 443.0 1.6 54939.5 30.5 0.9 68.8 2.1 92 92 c1t2d0 0.4 451.4 3.2 56184.2 29.0 0.9 64.3 1.9 87 87 c1t3d0 1.4 412.6 12.8 49317.5 26.8 0.9 64.8 2.1 85 85 c1t2d0 r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device elkner.sol /rpool > + time sh -c ''mkfile 4g xx; sync; echo ST3250310NS'' ST3250310NS real 55.2 user 0.0 sys 1.8 elkner.sol ~ > iostat -zmnx c1t0d0 c1t1d0 5 | grep -v device 0.0 660.8 0.0 81796.7 11.7 16.3 17.7 24.7 78 83 c1t0d0 0.0 657.0 0.0 81335.9 11.3 18.7 17.2 28.5 83 89 c1t1d0 0.0 607.2 0.0 75039.6 10.9 15.2 18.0 25.0 73 77 c1t0d0 0.0 614.8 0.0 75986.9 10.7 18.2 17.3 29.6 81 86 c1t1d0 0.0 649.0 0.0 81454.7 12.4 15.6 19.1 24.1 80 81 c1t0d0 0.0 660.8 0.0 83016.3 12.7 19.6 19.2 29.7 91 94 c1t1d0 0.0 723.0 0.0 91469.4 13.7 17.9 18.9 24.8 90 92 c1t0d0 0.0 683.8 0.0 86452.1 13.5 18.8 19.7 27.4 90 94 c1t1d0 0.0 462.6 0.0 56758.7 5.9 22.0 12.8 47.5 78 83 c1t0d0 0.0 501.0 0.0 61725.3 6.4 24.5 12.7 48.8 87 91 c1t1d0 0.0 568.4 0.0 71384.5 8.9 20.9 15.6 36.7 85 88 c1t0d0 0.0 559.8 0.0 70334.9 8.1 24.6 14.4 44.0 92 95 c1t1d0 0.0 636.0 0.0 80171.5 14.0 17.4 22.0 27.3 89 91 c1t0d0 0.0 584.6 0.0 73693.8 13.9 18.9 23.8 32.3 94 95 c1t1d0 0.0 604.2 0.0 75505.2 10.0 16.9 16.5 28.0 76 79 c1t0d0 0.0 630.2 0.0 78730.2 11.2 19.3 17.7 30.7 86 90 c1t1d0 0.0 563.6 0.0 69714.6 9.6 17.2 17.0 30.5 75 79 c1t0d0 0.0 595.0 0.0 73632.5 9.6 21.6 16.1 36.3 88 92 c1t1d0 0.0 573.0 0.0 71691.8 12.5 15.6 21.9 27.2 80 82 c1t0d0 0.0 578.4 0.0 72306.2 12.9 18.9 22.3 32.7 90 92 c1t1d0 r/s w/s kr/s kw/s wait actv wsvc_t asvc_t %w %b device Finally: 17.4 vs. 43.8 vs. 74.2 MB/s It seems odd, since rpool (SATA0,SATA1) and pool1 (SATA2,SATA3) have the same disks and capacity of ~60%, pool2 (SATA4,SATA5) has 0%. So very roughly 1st 600, 2nd 400, 3rd 200 iops. My guess is, that there is a driver or BIOS/FW problem. Upgrade to snv_b124 and BIOS/FW to latest release is scheduled for the end of october, than I may know a little bit more ... Regards, jel. PS: Sorry, for the long posting :( -- Otto-von-Guericke University http://www.cs.uni-magdeburg.de/ Department of Computer Science Geb. 29 R 027, Universitaetsplatz 2 39106 Magdeburg, Germany Tel: +49 391 67 12768