Recently I''ve installed SXCE nv86 for the first time in hopes of getting rid of my linux file server and using Solaris and ZFS for my new file server. After setting up a simple ZFS mirror of 2 disks, I enabled smb and set about moving over all of my data from my old storage server. What I noticed was the dismal performance while writing. I have tried to find information regarding performance and possible expectations, but I''ve yet to come across anything with any real substance that can help me out. I''m sure there is some guide on tuning for CIFS, but I''ve not been able to locate it. The write speeds for NFS described in this post http://opensolaris.org/jive/thread.jspa?threadID=55764&tstart=0 made me want to look into NFS. However, after disabling sharing, turning off smb, enabling NFS, and sharing the pool again I see the same if not worse performance on write speeds (ms windows SFU may be partially to blame, so I''ve gone back to learning how to fix smb instead of learning and tweaking NFS). What I''m doing is mounting the smb share with WinXP and pulling data from the ZFS mirror pool at 2.3MiB/s across the network. Writing to the same share from the WinXP host I get a fairly consistent 342KiB/s speed. Copying data locally from an IDE drive to the zpool mirror (2 SATAII drives) I get much faster performance. As I do with copying data from one zpool mirror (1 SATA1 drive and 1 SATAII drive) to another zpool mirror (2 SATAII drives) on the same host. I''m not sure on performance numbers but it takes *substantially* less time to transfer. The research I''ve done thus far indicates that I''ve got to use a file that''s double the size of my ram to ensure that caching doesn''t skew the results. So these tests are all done with an 8GB file. I would imagine that write speeds and read speeds across the network should be much closer. At this point, I''m assuming that I''m doing something wrong here. Anyone want to let me know what I''m missing? rick This message posted from opensolaris.org
On Thu, 2008-04-24 at 09:46 -0700, Rick wrote:> Recently I''ve installed SXCE nv86 for the first time in hopes of getting rid of my linux file server and using Solaris and ZFS for my new file server. After setting up a simple ZFS mirror of 2 disks, I enabled smb and set about moving over all of my data from my old storage server. What I noticed was the dismal performance while writing. I have tried to find information regarding performance and possible expectations, but I''ve yet to come across anything with any real substance that can help me out. I''m sure there is some guide on tuning for CIFS, but I''ve not been able to locate it. The write speeds for NFS described in this post http://opensolaris.org/jive/thread.jspa?threadID=55764&tstart=0 made me want to look into NFS. However, after disabling sharing, turning off smb, enabling NFS, and sharing the pool again I see the same if not worse performance on write speeds (ms windows SFU may be partially to blame, so I''ve gone back to learning how to fix smb instead of learning> and tweaking NFS). > > What I''m doing is mounting the smb share with WinXP and pulling data from the ZFS mirror pool at 2.3MiB/s across the network. Writing to the same share from the WinXP host I get a fairly consistent 342KiB/s speed. > > Copying data locally from an IDE drive to the zpool mirror (2 SATAII drives) I get much faster performance. As I do with copying data from one zpool mirror (1 SATA1 drive and 1 SATAII drive) to another zpool mirror (2 SATAII drives) on the same host. I''m not sure on performance numbers but it takes *substantially* less time to transfer. > > The research I''ve done thus far indicates that I''ve got to use a file that''s double the size of my ram to ensure that caching doesn''t skew the results. So these tests are all done with an 8GB file. > > I would imagine that write speeds and read speeds across the network should be much closer. At this point, I''m assuming that I''m doing something wrong here. Anyone want to let me know what I''m missing? > > rick >What performance are you getting with transfers over other protocols, e.g. SCP with Putty? Have you tried using a different network card? If you are getting bad one-way performance with file copies your network stack may be a culprit - check that your NIC is operating in full-duplex mode (methods to check this are card-dependent, but may well involve ndd(1M)). Chris
-snip-> > What I''m doing is mounting the smb share with WinXP > and pulling data from the ZFS mirror pool at 2.3MiB/s > across the network. Writing to the same share from > the WinXP host I get a fairly consistent 342KiB/s > speed.-snip-> What performance are you getting with transfers over > other protocols, > e.g. SCP with Putty? Have you tried using a different > network card? If > you are getting bad one-way performance with file > copies your network > stack may be a culprit - check that your NIC is > operating in full-duplex > mode (methods to check this are card-dependent, but > may well involve > ndd(1M)). > > ChrisChris, I didn''t cross post the following, it was sent only to the cifs discuss list: nge. Onboard nvidia nforce 570 Ultra MCP onboard gigabit nic for the Asus M2N-E board. nge: [ID 801725 kern.info] NOTICE: nge0: Using FIXED interrupt type mac: [ID 469746 kern.info] NOTICE: nge0 registered mac: [ID 435574 kern.info] NOTICE: nge0 link up, 100 Mbps, full duplex This is actually the first time I checked the nic. Looks like ndd reports it as being in 100Mbps. Not gigabit. I''m not sure what that''s all about. Although I did just find out that dladm replaces ndd. ... -end repost- As for the other protocols, I have stats for those: FTP from WinXP to Solaris ZFS Read: 11MiB/s Write: 4.7MiB/s, spiking to 6.0MiB/s SFTP from WinXP to Solaris ZFS Read: 500KiB/s Write: 1.9MiB/s (odd, no fluctuation in speed, and I''m sure this is a windows thing as I had the same dismal performance when attempting SFTP to a linux box) FTP from Linux to Solaris Read: 2.6MiB/s Write: 4.4MiB/s SFTP from Linux to Solaris Read: 2.6MiB/s Write: 4.6MiB/s Those just from single file transfers. It''s not like I did 10 tests. Jose Borrego noted the following in the cifs discuss list: I should put back a fix related to performance this week CR 6686647. I don''t know if it''s related to what you are experiencing but the performance boost is quite significant. Thanks for the help. rick This message posted from opensolaris.org
> I didn''t cross post the following, it was sent only to the cifs discuss list: > nge. Onboard nvidia nforce 570 Ultra MCP onboard gigabit nic for the Asus M2N-E board. > > nge: [ID 801725 kern.info] NOTICE: nge0: Using FIXED interrupt type > mac: [ID 469746 kern.info] NOTICE: nge0 registered > mac: [ID 435574 kern.info] NOTICE: nge0 link up, 100 Mbps, full duplex > > This is actually the first time I checked the nic. Looks like ndd reports it as being in 100Mbps. Not gigabit. I''m not sure what that''s all about. Although I did just find out that dladm replaces ndd. ... >If your switch is gigabit as well there may have been some problems with speed negotiation. I actually had SXDE running on the same motherboard a while ago and had some problems with the stock nge driver - I eventually installed an older Intel 100Mb NIC and got much better bulk transfer performance from it. This may have been addressed by now though.> -end repost- > > As for the other protocols, I have stats for those: > FTP from WinXP to Solaris ZFS > Read: 11MiB/s > Write: 4.7MiB/s, spiking to 6.0MiB/s > > SFTP from WinXP to Solaris ZFS > Read: 500KiB/s > Write: 1.9MiB/s (odd, no fluctuation in speed, and I''m sure this is a windows thing as I had the same dismal performance when attempting SFTP to a linux box) > > FTP from Linux to Solaris > Read: 2.6MiB/s > Write: 4.4MiB/s > > SFTP from Linux to Solaris > Read: 2.6MiB/s > Write: 4.6MiB/s > > Those just from single file transfers. It''s not like I did 10 tests.> Jose Borrego noted the following in the cifs discuss list: > I should put back a fix related to performance this week CR 6686647. I > don''t know if it''s related to what you are experiencing but the > performance boost is quite significant.It wouldn''t explain the SFTP performance from XP to Solaris though. Unless the processing overhead for encrypted protocols is too much for your Windows box, I think your problem lies elsewhere than in the type of filesystem. Chris
Rick, I have the same motherboard on my backup machine and got 48MBytes/sec sustained on a 650GB transfer (but that was using iSCSI), so I suggest two things: 1. Make sure you are using the latest stable -- i.e. not beta, BIOS update. You can use a USB thumbdrive to install it, and can save the old one on there too, in case you want to return to it. 2. I assume you have at least Category 5e ethernet cables between all boxes linked to your gigabit switch. If not, that could be the cause as Cat. 5 might not be sufficient. I use Cat. 6 because I wanted to be sure if I got low speeds that it wasn''t the cables letting me down. You might also be able to do a cable test as part of the system''s POST, set within the BIOS (I know my other M2N-SLI Deluxe can do this anyway). Hope it helps. Simon This message posted from opensolaris.org
> Rick, I have the same motherboard on my backup > machine and got 48MBytes/sec sustained on a 650GB > transfer (but that was using iSCSI), so I suggest two > things: > > 1. Make sure you are using the latest stable -- i.e. > not beta, BIOS update. You can use a USB thumbdrive > to install it, and can save the old one on there too, > in case you want to return to it. > > 2. I assume you have at least Category 5e ethernet > cables between all boxes linked to your gigabit > switch. If not, that could be the cause as Cat. 5 > might not be sufficient. I use Cat. 6 because I > wanted to be sure if I got low speeds that it wasn''t > the cables letting me down. You might also be able to > do a cable test as part of the system''s POST, set > within the BIOS (I know my other M2N-SLI Deluxe can > do this anyway). > > Hope it helps. > > SimonSimon, Hey Simon. I have not updated to the latest BIOS. I''m 1 or 2 revisions behind. I will check into updating, however I think that''s not my issue. I do have a Cat6 cable that I''m plugging in to a 10/100/1000 switch. It''s a SOHO switch which does not have a configuration interface. I do know the port works as 1000 because my last mobo was connected to the same cable and port at 1000. For whatever reason, I cannot set my nge interface to 1000 though. It runs fine, but only connects at 100 full duplex. On the same switch I also have a PS3 which connects at 1000. The typical speed from the Solaris ZFS shared data to the PS3 is 700KiB/s. That''s streaming data though. The actual "copy" speed is 9.3MiB/s. Although, to be honest, I''m not sure of the protocol it uses to copy data. MediaTomb is the DLNA application that feeds my PS3. Anyway, all other devices use Cat5e and connect to the same switch (at least, all other devices that are involved in this little fiasco.) The 2MiB/s speed isn''t so bad, or at least uncommon in my office. Copying from another windows box (laptop) to the winxp box, that was used in the tests quoted before, results in 2.2MiB/s - 3.4MiB/s while using MS SMB. I''m less concerned with speeds around that mark. The only speed that really concerns me is the 342KiB/s write speed. It''s not a great situation when your backup solution only accepts 342KiB/s on writes. :) And for those getting annoyed, I''m aware that this is probably not a direct fault of ZFS. However, I''m not sure how exactly the code between CIFS and ZFS interfaces, so I included "zfs discuss" on my initial post. Because the read/write speeds of other protocols is faster on the same ZFS mirror, I have to assume that there is definitely an issue with CIFS or my configuration of CIFS. Still, I have not found any information at all on tweaking CIFS in Solaris. I can''t even find a config file for it. :( rick This message posted from opensolaris.org
Hey, hi Rick! The obvious thing that is wrong is the network being recognised as 100Mbps and not 1000. Hopefully, the read/write speeds will fix themselves once the network problem is fixed. As it''s the same cable you had working previously at 1000Mbps on your other computer and the same switch, I suppose, then it all points to problems with the network on the solaris box. The first thing to try is replace the cable with another one if you have another one around, although I suppose you would already have tried that. Then, see if there''s anything that could possibly be setup wrongly in the BIOS, but I don''t recall there being anything to change, although I could be wrong. Then you could try to see if you can cause your networking on Solaris to be re-setup somehow. You could try: # svcadm disable /network/physical:nwam # svcadm enable /network/physical:nwam But I suppose this won''t change anything. Also I would investigate the possibility of getting the latest BIOS, unless you know of a good reason not to. This might be a bug in the BIOS. Lastly, could there be a bug in the build 86 regarding the nge driver? That''s all I can think of, good luck! :) Simon This message posted from opensolaris.org
Simon,> The obvious thing that is wrong is the network being > recognised as 100Mbps and not 1000. Hopefully, the > read/write speeds will fix themselves once the > network problem is fixed.You are absolutely correct. My troubleshooting skills for things of this nature are pretty bad. I should have started there.> As it''s the same cable you had working previously at > 1000Mbps on your other computer and the same switch, > I suppose, then it all points to problems with the > network on the solaris box.There was a very small problem. My old mobo has 2 nics. I cabled up the wrong cable. I would very much like to pretend something, anything, otherwise. But the truth of it is that I was plugged in with the wrong cable.> But I suppose this won''t change anything. To see the > speed negotiated, try: > > # dladm show-dev > LINK STATE SPEED DUPLEX > nge0 up 1000Mb full > ## dladm show-dev LINK STATE SPEED DUPLEX nge0 up 1000Mb full I think I''m back in business now. I need to validate write/read speeds again.> Also I would investigate the possibility of getting > the latest BIOS, unless you know of a good reason not > to. This might be a bug in the BIOS.I''m paranoid about flashing the BIOS. I just don''t want to introduce a new issue. However, I should really do just that. I''ll have to look into how to boot off a thumb drive to do it. It appears that the 1401 BIOS was just released on 2008-04-24.> Lastly, could there be a bug in the build 86 > regarding the nge driver?So far, looks like something I did. :(> That''s all I can think of, good luck! :)Thanks! I''ll check throughput numbers and post back here with the results. rick This message posted from opensolaris.org
On Tue, 29 Apr 2008, Simon Breden wrote:> The obvious thing that is wrong is the network being recognised as > 100Mbps and not 1000. Hopefully, the read/write speeds will fix > themselves once the network problem is fixed. > > As it''s the same cable you had working previously at 1000Mbps on > your other computer and the same switch, I suppose, then it all > points to problems with the network on the solaris box. > > The first thing to try is replace the cable with another one if you > have another one around, although I suppose you would already have > tried that.100Mbit is actually more challenging for the wiring than 1Gbit except that gigabit ethernet uses all the wires in the cable. A single bad connection in the cable will defeat gigabit ethernet. Check all the pins in the connectors (bent pin?) and try a different port if possible. Replacing the cable is certainly the first thing to try. The cheapy "SOHO" switch is the next most likely suspect. Device drivers are least suspect unless they have been configured to force the hardware to particular settings. Bob =====================================Bob Friesenhahn bfriesen at simple.dallas.tx.us, http://www.simplesystems.org/users/bfriesen/ GraphicsMagick Maintainer, http://www.GraphicsMagick.org/
Rick, Glad it worked ;-) Now if I were you, I would not upgrade the BIOS unless you really want/need to. I look forward to seeing your revised speed test data for reads and writes with the gigabit network speed working correctly. I think it should make a little difference -- I''m guessing you''ll get 40 - 45 MBytes/sec sustained assuming disks are fast enough at the other end, AND you have a Cat 5e/6 cable at the other end too. Simon This message posted from opensolaris.org
Hi Rick, I have the same problem as you. (sorry for my english) I have installed the same OS on a gigabyte mother board. I wanted to make a NAS with the nice ZFS. First I tried the new smb kernel implementation: file navigation (on windows) and streaming were too slow. File transfer was acceptable. I thought that was because of the young kernel implementation. So I removed every configuration related to it and I tried samba. Samba brought significant boost in file navigation. (I was happy) but streaming were even worst and file transfer were weird: when I transfer 2 files the speed were around 10MB/s and for one file the speed is around 100Kb/s ... I clearly, the wires are able to transfer at full speed, clearly ZFS is able to undertake it (and much more I think) but something weird is going on. At that point, I was going to reinstall UBUNTU and forget about solaris and ZFS but your story gave me hope. I''m up to again to find out what''s going on. This message posted from opensolaris.org
FYI - If you''re doing anything with CIFS and performance, you''ll want this fix: 6686647 smbsrv scalability impacted by memory management issues Which was putback into build 89 of nevada. - Eric On Thu, Apr 24, 2008 at 09:46:04AM -0700, Rick wrote:> Recently I''ve installed SXCE nv86 for the first time in hopes of getting rid of my linux file server and using Solaris and ZFS for my new file server. After setting up a simple ZFS mirror of 2 disks, I enabled smb and set about moving over all of my data from my old storage server. What I noticed was the dismal performance while writing. I have tried to find information regarding performance and possible expectations, but I''ve yet to come across anything with any real substance that can help me out. I''m sure there is some guide on tuning for CIFS, but I''ve not been able to locate it. The write speeds for NFS described in this post http://opensolaris.org/jive/thread.jspa?threadID=55764&tstart=0 made me want to look into NFS. However, after disabling sharing, turning off smb, enabling NFS, and sharing the pool again I see the same if not worse performance on write speeds (ms windows SFU may be partially to blame, so I''ve gone back to learning how to fix smb instead of learning > and tweaking NFS). > > What I''m doing is mounting the smb share with WinXP and pulling data from the ZFS mirror pool at 2.3MiB/s across the network. Writing to the same share from the WinXP host I get a fairly consistent 342KiB/s speed. > > Copying data locally from an IDE drive to the zpool mirror (2 SATAII drives) I get much faster performance. As I do with copying data from one zpool mirror (1 SATA1 drive and 1 SATAII drive) to another zpool mirror (2 SATAII drives) on the same host. I''m not sure on performance numbers but it takes *substantially* less time to transfer. > > The research I''ve done thus far indicates that I''ve got to use a file that''s double the size of my ram to ensure that caching doesn''t skew the results. So these tests are all done with an 8GB file. > > I would imagine that write speeds and read speeds across the network should be much closer. At this point, I''m assuming that I''m doing something wrong here. Anyone want to let me know what I''m missing? > > rick > > > This message posted from opensolaris.org > _______________________________________________ > zfs-discuss mailing list > zfs-discuss at opensolaris.org > http://mail.opensolaris.org/mailman/listinfo/zfs-discuss-- Eric Schrock, Fishworks http://blogs.sun.com/eschrock
> If you''re doing anything with CIFS and performance, > you''ll want this > fix: > > 6686647 smbsrv scalability impacted by memory > management issues > > Which was putback into build 89 of nevada. > > - EricThank you Eric. This is the second time someone has mentioned this to me. I imagine it''s a significant change. When build 89 is released, I''ll be sure to upgrade. As for the speed issue, the bottom line is that I discovered a faulty NIC on the WinXP box. Through gathering performance results, I found that the WinXP box performance on read was less than a WinXP vm! After swapping out cables and using different ports on the switch I found no change. However, after swapping NICs on the mobo I gained a huge increase in write speed. I''m not sure why only CIFS had issue, probably because it''s one of the chattiest protocols ever, but that''s just a random jab and has no basis in fact. :) On to the results: Stats are taken with System Monitor (v2.18.2) on Solaris. I''ve verified the stats are similar, if not identical, on the WinXP virtual machine and on the linux box through conky. All boxes are 100fdx. The PS3 is 1000fdx. They are all plugged into the same swtich (Netgear G5605 v2, 5 port 10/100/1000). File transfers are with a 4GB file for SMB and FTP. The HTTP transfer is with a 1.6GB file. WinXP(1) = virtual machine on the linux box WinXP(2) = physical machine WinXP(1): FTP Not tested HTTP Read: 500KiB/s - 2.7MiB/s SMB Read: 1.1MiB/s - 3.4MiB/s Write: 2.2MiB/s - 3.4MiB/s WinXP(2): FTP Read: 4.2MiB/s - 9.9MiB/s Write: 1.9MiB/s - 6.0MiB/s HTTP Read: 2.4MiB/s - 6.5MiB/s SMB Read: 7.6MiB/s - 8.3MiB/s Write: 7.9MiB/s - 8.9MiB/s Linux (100fdx) FTP Read: 1.8MiB/s - 3.7MiB/s Write: 3.6MiB/s - 3.7MiB/s HTTP Read: 2.5MiB/s - 4MiB/s PS3 (1000fdx): DLNA Read: 9.4MiB/s The end result is that I have 63.75 Mbps - 74.66 Mbps read/write via CIFS. Not too bad considering it''s really just a 100Mbit network link. Sorry to spam. I thought for sure that the high read/write speeds for FTP and HTTP showed that the issue was with CIFS. I''m still kind of baffled as to why CIFS was so terrible for the write speed only when, in the end, the issue was the NIC. Thanks to those that helped. rick This message posted from opensolaris.org
Hi Rick, So just to verify, you never managed to get more than 10 MBytes/sec across the link due to the network only giving you a 100 Mbps connection? Simon This message posted from opensolaris.org
> So just to verify, you never managed to get more than > 10 MBytes/sec across the link due to the network only > giving you a 100 Mbps connection?Hi Simon, I''ll try to clear this up. Sorry for the confusion. The server the Solaris M2N-E is replacing had 2 NICs. When I removed the physical box, I left the cat5 cables laying on the desk. When I plugged up the Solaris box, I must have plugged in the cable that ended in a 10/100 switch. This caused Solaris to boot up and negotiate a 100Mbit full duplex link. After reading your post today, I looked at the cable and realized that I''d plugged in the wrong cable. After a move and a reboot, the box came up in 1000Mbit full duplex. Solaris side fixed. The initial problem that started this whole thread was that I wasn''t able to get above 342KiB/s (2.80 Mbps) when writing to the ZFS CIFS share from WinXP. I was able to achieve better speeds with FTP and HTTP. After I fixed the cable snafu on the Solaris box, I attempted to re-create the read/write speed tests to see if the slow write issue was resolved. It was not. After some more troubleshooting, the poor write speed performance turned out to be because of an issue with the second NIC on the WinXP motherboard. Once I switched the cable to the primary NIC on that mobo, I was able to achieve around 8.9KiB/s (74.66 Mbps) write speeds from the WinXP box to the Solaris ZFS CIFS share. WinXP to Solaris ZFS, slow write speed fixed. I hope that answers your question. I''m not sure though. I don''t think I can get much higher than the speed I have now. With overhead, and other network traffic, I believe that 75-85Mpbs is the most I can really hope to achieve where almost all the devices talk at 100Mbit. I have re-read your question a few times. I just tested and found that with 2 WinXP boxes and the PS3 pulling data from the Solaris ZFS share, I get 20.3MiB/s or 170.29 Mbps total output from the Solaris box. I''m sure that if another one of my networked devices (PS3 doesn''t count) could talk at 1000Mbits/s I''d get much greater speeds. As it stands now, the linux box is a laptop (5400rpm drive) and the WinXP box has a SATA 1.5Gbit/s drive. As I said before, they both connect at 100Mbit. Those are probably the limiting factors in cresting 10MiB/s plus. Although, even on an empty network I believe that 11.5 or so is the most you''d be able to get on a 100Mbit link. Of course, I''ve not talked network in a bit, so maybe I have my terms mixed. I''ve been trying to review external references just to make sure I''m speaking in the correct terms. Feel free to let me know if I have something wrong. Thanks! rick This message posted from opensolaris.org
Hi Rick, OK, thanks for clarifying. As, it seems there''s different devices with (1) mixed speed NICs and (2) mixed category cabling being used in your setup, I will simplify things by saying that if you want to get much faster speeds then I think you''ll need to ensure you (1) use at least Cat. 5e cables between all devices talking on your LAN, and (2) ensure you use Gigabit NICs, and (3) confirm negotiated NIC speed before performing speed tests. Then, assuming your disks are reasonably fast, you should, using CIFS sharing, be able to get around 40+ MBytes/sec sustained throughput using one NIC on each box, with your Gigabit switch. Hope that helps. Simon This message posted from opensolaris.org
Hello eschrock, I''m a newbe on solaris, would you tell me how I can get/install build 89 of nevada? Fabrice. This message posted from opensolaris.org
dh wrote:> Hello eschrock, > > I''m a newbe on solaris, would you tell me how I can get/install build 89 of nevada? > > Fabrice.Hi Fabrice, I think a good place to start is http://www.opensolaris.org/os/newbies/ - I don''t know whether they give you access to build 89 yet, but you can certainly get some practice so that wen you get b89, you know what to do. (and no, eschrock is not a pseudonym of mine ;-) HTH Michael -- Michael Schuster http://blogs.sun.com/recursion Recursion, n.: see ''Recursion''