eneal at businessgrade.com
2009-Aug-31 14:16 UTC
[zfs-discuss] ZFS read performance scalability
Hi. I''ve been doing some simple read/write tests using filebench on a mirrored pool. Essentially, I''ve been scaling up the number of disks in the pool before each test between 4, 8 and 12. I''ve noticed that for individual disks, ZFS write performance scales very well between 4, 8 and 12 disks. This may be due to the fact that I''m using a SSD as a logging device. But I''m seeing individual disk performance drop by as much as 14MB per disk between 4 and 12 disks. Across the entire pool that means I''ve lost 168MB of raw throughput just by adding two mirror sets. I''m curious to know if there are any dials I can turn to improve this. System details are below: HW: Dual Quad Core 2.33 Xeon 8GB RAM Disks: Seagate Savio 10K 146GB and LSI 1068e HBA latest firmware OS: SCXE snv_121 Thank in advance.. -------------------------------------------------------------------------------- This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. This communication may contain material protected by the attorney-client privilege. If you are not the intended recipient, be advised that any use, dissemination, forwarding, printing or copying is strictly prohibited. If you have received this email in error, please contact the sender and delete all copies.
On Mon, 31 Aug 2009, eneal at businessgrade.com wrote:> Hi. I''ve been doing some simple read/write tests using filebench on a > mirrored pool. Essentially, I''ve been scaling up the number of disks in the > pool before each test between 4, 8 and 12. I''ve noticed that for individual > disks, ZFS write performance scales very well between 4, 8 and 12 disks. This > may be due to the fact that I''m using a SSD as a logging device. But I''m > seeing individual disk performance drop by as much as 14MB per disk between 4 > and 12 disks. Across the entire pool that means I''ve lost 168MB of raw > throughput just by adding two mirror sets. I''m curious to know if there are > any dials I can turn to improve this. System details are below:Sun is currently working on several prefetch bugs (complete loss of prefetch & insufficient prefetch) which have been identified. Perhaps you were not on this list in July when a huge amount of discussion traffic was dominated by the topic "Why is Solaris 10 ZFS performance so terrible?", "http://mail.opensolaris.org/pipermail/zfs-discuss/2009-July/029340.html". It turned out that the subject was over-specific since current OpenSolaris suffers from the same issues as proven by test results run by many people on a wide variety of hardware. Eventually Rich Morris posted a preliminary analysis of the performance problem at "http://mail.opensolaris.org/pipermail/zfs-discuss/2009-July/030169.html" Hopefully Sun will get the prefetch algorithm and timing perfected so that we may enjoy the full benefit of our hardware. Bob -- Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Hi; You may be hitting a bottleneck at your HBA. Try using multiple HBA''s or drive channels Mertol Mertol Ozyoney Storage Practice - Sales Manager Sun Microsystems, TR Istanbul TR Phone +902123352200 Mobile +905339310752 Fax +902123352222 Email mertol.ozyoney at sun.com -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of eneal at businessgrade.com Sent: Monday, August 31, 2009 5:16 PM To: zfs-discuss at opensolaris.org Subject: [zfs-discuss] ZFS read performance scalability Hi. I''ve been doing some simple read/write tests using filebench on a mirrored pool. Essentially, I''ve been scaling up the number of disks in the pool before each test between 4, 8 and 12. I''ve noticed that for individual disks, ZFS write performance scales very well between 4, 8 and 12 disks. This may be due to the fact that I''m using a SSD as a logging device. But I''m seeing individual disk performance drop by as much as 14MB per disk between 4 and 12 disks. Across the entire pool that means I''ve lost 168MB of raw throughput just by adding two mirror sets. I''m curious to know if there are any dials I can turn to improve this. System details are below: HW: Dual Quad Core 2.33 Xeon 8GB RAM Disks: Seagate Savio 10K 146GB and LSI 1068e HBA latest firmware OS: SCXE snv_121 Thank in advance.. ---------------------------------------------------------------------------- ---- This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. This communication may contain material protected by the attorney-client privilege. If you are not the intended recipient, be advised that any use, dissemination, forwarding, printing or copying is strictly prohibited. If you have received this email in error, please contact the sender and delete all copies. _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
There is around a zillion possible reasons for this. In my experience, most folks don''t or can''t create enough load. Make sure you have enough threads creating work. Other than that, the scientific method would suggest creating experiments, making measurements, running regressions, etc. -- richard On Aug 31, 2009, at 7:16 AM, eneal at businessgrade.com wrote:> Hi. I''ve been doing some simple read/write tests using filebench on > a mirrored pool. Essentially, I''ve been scaling up the number of > disks in the pool before each test between 4, 8 and 12. I''ve noticed > that for individual disks, ZFS write performance scales very well > between 4, 8 and 12 disks. This may be due to the fact that I''m > using a SSD as a logging device. But I''m seeing individual disk > performance drop by as much as 14MB per disk between 4 and 12 disks. > Across the entire pool that means I''ve lost 168MB of raw throughput > just by adding two mirror sets. I''m curious to know if there are any > dials I can turn to improve this. System details are below: > > HW: Dual Quad Core 2.33 Xeon 8GB RAM > Disks: Seagate Savio 10K 146GB and LSI 1068e HBA latest firmware > OS: SCXE snv_121 > > Thank in advance.. > > > > > > -------------------------------------------------------------------------------- > > This email and any files transmitted with it are confidential and > are intended solely for the use of the individual or entity to whom > they are addressed. This communication may contain material > protected by the attorney-client privilege. If you are not the > intended recipient, be advised that any use, dissemination, > forwarding, printing or copying is strictly prohibited. If you have > received this email in error, please contact the sender and delete > all copies. > > > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
eneal at businessgrade.com
2009-Aug-31 16:28 UTC
[zfs-discuss] ZFS read performance scalability
Quoting Bob Friesenhahn <bfriesen at simple.dallas.tx.us>:> On Mon, 31 Aug 2009, eneal at businessgrade.com wrote: > >> Hi. I''ve been doing some simple read/write tests using filebench on >> a mirrored pool. Essentially, I''ve been scaling up the number of >> disks in the pool before each test between 4, 8 and 12. I''ve >> noticed that for individual disks, ZFS write performance scales >> very well between 4, 8 and 12 disks. This may be due to the fact >> that I''m using a SSD as a logging device. But I''m seeing individual >> disk performance drop by as much as 14MB per disk between 4 and 12 >> disks. Across the entire pool that means I''ve lost 168MB of raw >> throughput just by adding two mirror sets. I''m curious to know if >> there are any dials I can turn to improve this. System details are >> below: > > Sun is currently working on several prefetch bugs (complete loss of > prefetch & insufficient prefetch) which have been identified. Perhaps > you were not on this list in July when a huge amount of discussion > traffic was dominated by the topic "Why is Solaris 10 ZFS performance > so terrible?", > "http://mail.opensolaris.org/pipermail/zfs-discuss/2009-July/029340.html". It > turned out that the subject was over-specific since current OpenSolaris > suffers from the same issues as proven by test results run by many > people on a wide variety of hardware. > > Eventually Rich Morris posted a preliminary analysis of the performance > problem at > "http://mail.opensolaris.org/pipermail/zfs-discuss/2009-July/030169.html" > > Hopefully Sun will get the prefetch algorithm and timing perfected so > that we may enjoy the full benefit of our hardware. >Thanks Bob. Can you or anyone else comment on how this bug would interact with a zvol that''s being remotely accessed? I can see clearly how this would come into play in on a local ZFS filesystem, but how about a remote system using the zvol through iscsi or FC? -------------------------------------------------------------------------------- This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. This communication may contain material protected by the attorney-client privilege. If you are not the intended recipient, be advised that any use, dissemination, forwarding, printing or copying is strictly prohibited. If you have received this email in error, please contact the sender and delete all copies.
eneal at businessgrade.com
2009-Aug-31 16:31 UTC
[zfs-discuss] ZFS read performance scalability
Quoting Mertol Ozyoney <Mertol.Ozyoney at Sun.COM>:> Hi; > > You may be hitting a bottleneck at your HBA. Try using multiple HBA''s or > drive channels > Mertol > >I''m pretty sure it''s not a HBA issue. As I commented, my per-disk write throughput stayed pretty consistent for 4, 8 and 12 disk pools and varied between 80 and 90MB. The overall rough average was about 85MB per second, per disk. -------------------------------------------------------------------------------- This email and any files transmitted with it are confidential and are intended solely for the use of the individual or entity to whom they are addressed. This communication may contain material protected by the attorney-client privilege. If you are not the intended recipient, be advised that any use, dissemination, forwarding, printing or copying is strictly prohibited. If you have received this email in error, please contact the sender and delete all copies.