John Tracy
2008-Feb-19 20:01 UTC
[zfs-discuss] five megabytes per second with Microsoft iSCSI initiator (2.06)
Hello All- I''ve been creating iSCSI targets on the following two boxes: - Sun Ultra 40 M2 with eight 10K SATA disks - Sun x2200 M2, with two 15K RPM SAS drives Both were running build 82 I''m creating a zfs volume, and sharing it with "zfs set shareiscsi=on poolname/volume". I can access the iSCSI volume without any problems, but IO is terribly slow, as in five megabytes per second sustained transfers. I''ve tried creating an iSCSI target stored on a UFS filesystem, and get the same slow IO. I''ve tried every level of RAID available in ZFS with the same results. The client machines are Windows 2003 Enterprise Edition SP2, running Microsoft iSCSI initiator 2.06, and Windows XP SP2, running MS iSCSI initiator 2.06. I''ve tried moving some of the client machines to the same physical switch as the target servers, and get the same results. I''ve tried another switch, and get the same results. I''ve even physically isolated the computers from my network, and get the same results. I''m not sure where to go from here and what to try next. The network is all gigabit. I normally have the Solaris boxes in a 802.3ad LAG group, tying two physical NICs together which should give me a max of 2gb/s of bandwidth (250 megabytes per second). Of course, I''ve tried no LAG connections with the same results. In short, I''ve tried every combination of everything I know to try, except using a different iSCSI client/server software stack (well, I did try the 2.05 version of MS''s iSCSI initiator client--same result). Here is what I''m seeing with performance logs on the Windows side- On any of the boxes, I see the queue length for the "hard disk" (iSCSI target) go from under 1 to 600+, and then back to under 1 about every four or five seconds. On the Solaris side, I''m running "iostat -xtc 1" which shows me lots of IO activity on the hard drives associated with my ZFS pool, and then about three or four seconds of pause, and then lots of activity again for a second or two, and then a lull again, and the cycle repeats as long as I''m doing active sustained IO against the iSCSI target. The output of prstat doesn''t show any heavy processor/memory usage on the Solaris box. I''m not sure what other monitors to run on either side to get a better picture. Any recommendations on how to proceed? Does anybody else use the Solaris iSCSI target software to export iSCSI targets to initiators running the MS iSCSI initiator? Thank you- John This message posted from opensolaris.org
Marion Hakanson
2008-Feb-19 20:19 UTC
[zfs-discuss] five megabytes per second with Microsoft iSCSI initiator (2.06)
john.tracy at covenant.edu said:> I''m creating a zfs volume, and sharing it with "zfs set shareiscsi=on > poolname/volume". I can access the iSCSI volume without any problems, but IO > is terribly slow, as in five megabytes per second sustained transfers. > > I''ve tried creating an iSCSI target stored on a UFS filesystem, and get the > same slow IO. I''ve tried every level of RAID available in ZFS with the same > results.Apologies if you''ve already done so, but try testing your network (without iSCSI and storage). You can use "ttcp" from blastwave.org on the Solaris side, and PCATTCP on the Windows side. That should tell you if your TCP/IP stacks and network hardware are in good condition. Regards, Marion
Bob Friesenhahn
2008-Feb-19 20:44 UTC
[zfs-discuss] five megabytes per second with Microsoft iSCSI initiator (2.06)
It would be useful if people here who have used iSCSI on top of ZFS could share their performance experiences. It is very easy to waste a lot of time trying to realize unrealistic expectations. Hopefully iSCSI on top of ZFS normally manages to transfer much more than 5MB/second! Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Erast Benson
2008-Feb-19 20:50 UTC
[zfs-discuss] five megabytes per second with Microsoft iSCSI initiator (2.06)
http://blogs.sun.com/constantin/entry/x4500_solaris_zfs_iscsi_perfect On Tue, 2008-02-19 at 14:44 -0600, Bob Friesenhahn wrote:> It would be useful if people here who have used iSCSI on top of ZFS > could share their performance experiences. It is very easy to waste a > lot of time trying to realize unrealistic expectations. Hopefully > iSCSI on top of ZFS normally manages to transfer much more than > 5MB/second! > > Bob > =====================================> Bob Friesenhahn > bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss >
Chris Linton-Ford
2008-Feb-20 10:44 UTC
[zfs-discuss] five megabytes per second with Microsoft iSCSI initiator (2.06)
Over a pretty quiet Gigabit link, 3x 10k SCSI disks, all on the same controller, in a RAIDZ pool exported using shareiscsi: # time dd if=/dev/urandom of=/var/log/maillogs/test bs=1024 count=100000 100000+0 records in 100000+0 records out real 0m6.958s user 0m0.065s sys 0m6.893s (second try) real 0m6.858s user 0m0.065s sys 0m6.783s (third) real 0m6.881s user 0m0.065s sys 0m6.815s The only problem I''ve had is with directories that have a lot of file creation and deletion - sometimes listing these directories can be incredibly slow, even with only a couple of files in them, but I think that might be a bug in the (rather old) version of zfs I''m using. HTH, Chris On Tue, 2008-02-19 at 14:44 -0600, Bob Friesenhahn wrote:> It would be useful if people here who have used iSCSI on top of ZFS > could share their performance experiences. It is very easy to waste a > lot of time trying to realize unrealistic expectations. Hopefully > iSCSI on top of ZFS normally manages to transfer much more than > 5MB/second! > > Bob > =====================================> Bob Friesenhahn > bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ > GraphicsMagick Maintainer, http://www.GraphicsMagick.org/ > > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Mertol Ozyoney
2008-Feb-20 14:42 UTC
[zfs-discuss] five megabytes per second with Microsoft iSCSI initiator (2.06)
Hi John; This is a know bug of MS. You need to do a few tricks to optimize the performance on windows clients. I have attached a file where you will find a easy method to optimize windows clients. Let me know the results. PS: While there are some bugs in the Solaris iSCSI target system (Mostly about failover and recovery processes) it performs quite well. The problem is as always MS itself. :) Mertol Ozyoney Storage Practice - Sales Manager Sun Microsystems, TR Istanbul TR Phone +902123352200 Mobile +905339310752 Fax +902123352222 Email mertol.ozyoney at Sun.COM -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of John Tracy Sent: 19 ?ubat 2008 Sal? 22:02 To: zfs-discuss at opensolaris.org Subject: [zfs-discuss] five megabytes per second with Microsoft iSCSI initiator (2.06) Hello All- I''ve been creating iSCSI targets on the following two boxes: - Sun Ultra 40 M2 with eight 10K SATA disks - Sun x2200 M2, with two 15K RPM SAS drives Both were running build 82 I''m creating a zfs volume, and sharing it with "zfs set shareiscsi=on poolname/volume". I can access the iSCSI volume without any problems, but IO is terribly slow, as in five megabytes per second sustained transfers. I''ve tried creating an iSCSI target stored on a UFS filesystem, and get the same slow IO. I''ve tried every level of RAID available in ZFS with the same results. The client machines are Windows 2003 Enterprise Edition SP2, running Microsoft iSCSI initiator 2.06, and Windows XP SP2, running MS iSCSI initiator 2.06. I''ve tried moving some of the client machines to the same physical switch as the target servers, and get the same results. I''ve tried another switch, and get the same results. I''ve even physically isolated the computers from my network, and get the same results. I''m not sure where to go from here and what to try next. The network is all gigabit. I normally have the Solaris boxes in a 802.3ad LAG group, tying two physical NICs together which should give me a max of 2gb/s of bandwidth (250 megabytes per second). Of course, I''ve tried no LAG connections with the same results. In short, I''ve tried every combination of everything I know to try, except using a different iSCSI client/server software stack (well, I did try the 2.05 version of MS''s iSCSI initiator client--same result). Here is what I''m seeing with performance logs on the Windows side- On any of the boxes, I see the queue length for the "hard disk" (iSCSI target) go from under 1 to 600+, and then back to under 1 about every four or five seconds. On the Solaris side, I''m running "iostat -xtc 1" which shows me lots of IO activity on the hard drives associated with my ZFS pool, and then about three or four seconds of pause, and then lots of activity again for a second or two, and then a lull again, and the cycle repeats as long as I''m doing active sustained IO against the iSCSI target. The output of prstat doesn''t show any heavy processor/memory usage on the Solaris box. I''m not sure what other monitors to run on either side to get a better picture. Any recommendations on how to proceed? Does anybody else use the Solaris iSCSI target software to export iSCSI targets to initiators running the MS iSCSI initiator? Thank you- John This message posted from opensolaris.org _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss -------------- next part -------------- An embedded message was scrubbed... From: <Saved by Windows Internet Explorer 7> Subject: X4500 + Solaris ZFS + iSCSI = Perfect Video Editing Storage - Constantin''s Blooog Date: Wed, 02 Jan 2008 20:46:54 +0200 Size: 225632 URL: <http://mail.opensolaris.org/pipermail/zfs-discuss/attachments/20080220/eb71ee88/attachment.mht>
Mertol Ozyoney
2008-Feb-20 14:44 UTC
[zfs-discuss] five megabytes per second with Microsoft iSCSI initiator (2.06)
Please also check http://www.microsoft.com/downloads/details.aspx?familyid=12CB3C1A-15D6-4585- B385-BEFD1319F825&displaylang=en best regards Mertol Ozyoney Storage Practice - Sales Manager Sun Microsystems, TR Istanbul TR Phone +902123352200 Mobile +905339310752 Fax +902123352222 Email mertol.ozyoney at Sun.COM -----Original Message----- From: zfs-discuss-bounces at opensolaris.org [mailto:zfs-discuss-bounces at opensolaris.org] On Behalf Of John Tracy Sent: 19 ?ubat 2008 Sal? 22:02 To: zfs-discuss at opensolaris.org Subject: [zfs-discuss] five megabytes per second with Microsoft iSCSI initiator (2.06) Hello All- I''ve been creating iSCSI targets on the following two boxes: - Sun Ultra 40 M2 with eight 10K SATA disks - Sun x2200 M2, with two 15K RPM SAS drives Both were running build 82 I''m creating a zfs volume, and sharing it with "zfs set shareiscsi=on poolname/volume". I can access the iSCSI volume without any problems, but IO is terribly slow, as in five megabytes per second sustained transfers. I''ve tried creating an iSCSI target stored on a UFS filesystem, and get the same slow IO. I''ve tried every level of RAID available in ZFS with the same results. The client machines are Windows 2003 Enterprise Edition SP2, running Microsoft iSCSI initiator 2.06, and Windows XP SP2, running MS iSCSI initiator 2.06. I''ve tried moving some of the client machines to the same physical switch as the target servers, and get the same results. I''ve tried another switch, and get the same results. I''ve even physically isolated the computers from my network, and get the same results. I''m not sure where to go from here and what to try next. The network is all gigabit. I normally have the Solaris boxes in a 802.3ad LAG group, tying two physical NICs together which should give me a max of 2gb/s of bandwidth (250 megabytes per second). Of course, I''ve tried no LAG connections with the same results. In short, I''ve tried every combination of everything I know to try, except using a different iSCSI client/server software stack (well, I did try the 2.05 version of MS''s iSCSI initiator client--same result). Here is what I''m seeing with performance logs on the Windows side- On any of the boxes, I see the queue length for the "hard disk" (iSCSI target) go from under 1 to 600+, and then back to under 1 about every four or five seconds. On the Solaris side, I''m running "iostat -xtc 1" which shows me lots of IO activity on the hard drives associated with my ZFS pool, and then about three or four seconds of pause, and then lots of activity again for a second or two, and then a lull again, and the cycle repeats as long as I''m doing active sustained IO against the iSCSI target. The output of prstat doesn''t show any heavy processor/memory usage on the Solaris box. I''m not sure what other monitors to run on either side to get a better picture. Any recommendations on how to proceed? Does anybody else use the Solaris iSCSI target software to export iSCSI targets to initiators running the MS iSCSI initiator? Thank you- John This message posted from opensolaris.org _______________________________________________ zfs-discuss mailing list zfs-discuss at opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Wade.Stuart at fallon.com
2008-Feb-20 16:31 UTC
[zfs-discuss] five megabytes per second with Microsoft iSCSI initiator (2.06)
zfs-discuss-bounces at opensolaris.org wrote on 02/20/2008 08:42:55 AM:> Hi John; > > This is a know bug of MS. You need to do a few tricks to optimize the > performance on windows clients. > > I have attached a file where you will find a easy method to optimizewindows> clients. > > Let me know the results. > > PS: While there are some bugs in the Solaris iSCSI target system (Mostly > about failover and recovery processes) it performs quite well. Theproblem> is as always MS itself. :) >Whatever you did to create this email -- please don''t. Links work fine. Thanks! Wade Stuart
I''m now getting about 90 megabytes of sustained IO from the Windows client to an iSCSI target. I''m using HD Tach (http://www.simplisoftware.com) to measure IO. Real world operations now are also averaging about 90 megabytes/second on read and write speeds (large hunks of continuous IO, of course copying lots of smaller files is different). Just for perspective, the target server is a Sun Ultra 40 M2, with seven 750GB sata drives in a double parity raid-z pool, linked with 2 GB nics in a LAG group. The client is a Dell Poweredge 2850 (2x3.2ghz Xeons, 4 GB RAM, dual GB NICS linked together in a LAG group). Interestingly enough, I have every link between the target and initiator in a LAG group, with two 1GB interfaces grouped together. Apparently, the load balancing mechanisms of one or multiple of the switches/NICS along the way means that an IP stream from a single source will only get transferred over one member of the LAG group. I''ve read that this behavior can be expected depending on how the LAG is setup, whether it divides hashes up the data on a per packet or per source/destination basis/or other options. If anybody has any pointers in this arena, that I may be able to bump the IO speed up more (at least as far as the network subsystem is concerned), I''d appreciate it. Thanks to all who replied to this thread. This is a great community. -John This message posted from opensolaris.org
On Thu, 2008-02-21 at 11:06 -0800, John Tracy wrote:> I''ve read that this behavior can be expected depending on how the LAG > is setup, whether it divides hashes up the data on a per packet or per > source/destination basis/or other options.(this is a generic answer, not specific to zfs exported via iSCSI). round-robin by packet often results in packet reordering which is often toxic to tcp performance; tcp can misinterpret significant amounts of reordering as a sign of packet loss due to congestion and slow down in the name of congestion avoidance. round-robin by "flow" (I''m oversimplifying, but in this context "flow" generally means "same addresses *and* ports") works fairly well if you have that option and have enough different TCP connections; each connection will go over a single path but with enough connections and enough paths it will even out. - Bill