Hi, I am trying to optimize my gigabit network. I have two Intel 1000 MT Gigabit Server Adapters, which support Jumbo Frames -- as well as a Switch that supports Jumbo Frames. However, I am observing some strange behavior in my file transfers from Windows XP to Linux and I am wondering if it has anything to do with the way the Samba variables are set on my Linux box? The "strange behavior" is that when I set both NICs to use Jumbo Frames [MTU=9014 on the Windows side (includes IP headers) , 9000 on the Linux side (doesn't include the headers], I am getting about half the throughput that I get when I set both NICs to use the standard MTU of 1514/1500. I see the same behavior even if I take the switch out of the system and connect the Windows XP and Linux machines directly to each other (crossover cable not required for computer-to-computer connection with these NICs -- and by the way all of my cables are CAT6). On the Linux side, I am using Samba 3.0.0 on Mandrake Linux 9.2 with all of Mandrake's current updates -- kernel = 2.4.22-21enterprisemdk. The Linux machine is a P4-3.06 Ghz with 1 GB of RAM -- running in hyperthreading mode. I am wondering if any of the Samba socket options settings like tcp_nodelay, so_sndbuf=8192 or so_rcvbuf=8192 are affecting my throughput -- particularly when I am using Jumbo Frames? And are there any other Samba settings that might be interacting in a negative way with my TCP/IP and NIC driver settings that are causing me to get lower throughput with Jumbo Frames instead of higher throughput (which is what I am told I should be getting). Any guidance would be appreciated. I have purchased "The Official Samba 3 HOW-TO and Reference Guide" but it really isn't very helpful when it comes to understanding how to tune these options and how various socket options settings interact with other network settings and hardware. Andy Liebman Resolute Films 119 Braintree Street, Suite 410 Boston, MA 02134 Tel: 617-782-0479 Cell: 617-308-0488 Fax: 617-782-1071
Hi, Sorry if this is a duplicate message. I forgot whether or not this list accepts html posts so I am sending it again as a plain text message. I am trying to optimize my gigabit network. I have two Intel 1000 MT Gigabit Server Adapters, which support Jumbo Frames -- as well as a Switch that supports Jumbo Frames. However, I am observing some strange behavior in my file transfers from Windows XP to Linux and I am wondering if it has anything to do with the way the Samba variables are set on my Linux box? The "strange behavior" is that when I set both NICs to use Jumbo Frames [MTU=9014 on the Windows side (includes IP headers) , 9000 on the Linux side (doesn't include the headers], I am getting about half the throughput that I get when I set both NICs to use the standard MTU of 1514/1500. I see the same behavior even if I take the switch out of the system and connect the Windows XP and Linux machines directly to each other (crossover cable not required for computer-to-computer connection with these NICs -- and by the way all of my cables are CAT6). On the Linux side, I am using Samba 3.0.0 on Mandrake Linux 9.2 with all of Mandrake's current updates -- kernel = 2.4.22-21enterprisemdk. The Linux machine is a P4-3.06 Ghz with 1 GB of RAM -- running in hyperthreading mode. I am wondering if any of the Samba socket options settings like tcp_nodelay, so_sndbuf=8192 or so_rcvbuf=8192 are affecting my throughput -- particularly when I am using Jumbo Frames? And are there any other Samba settings that might be interacting in a negative way with my TCP/IP and NIC driver settings that are causing me to get lower throughput with Jumbo Frames instead of higher throughput (which is what I am told I should be getting). Any guidance would be appreciated. I have purchased "The Official Samba 3 HOW-TO and Reference Guide" but it really isn't very helpful when it comes to understanding how to tune these options and how various socket options settings interact with other network settings and hardware. Andy Liebman Resolute Films 119 Braintree Street, Suite 410 Boston, MA 02134 Tel: 617-782-0479 Cell: 617-308-0488 Fax: 617-782-1071
Perhaps your TCP window is too small You should try the following global settings: read size = 65535 max xmit = 65535 socket options = TCP_NODELAY SO_SNDBUF=65535 SO_RCVBUF=65535 Rgds Per AndyLiebman@aol.com wrote:> > Hi, > > I am trying to optimize my gigabit network. I have two Intel 1000 MT Gigabit > Server Adapters, which support Jumbo Frames -- as well as a Switch that > supports Jumbo Frames. However, I am observing some strange behavior in my file > transfers from Windows XP to Linux and I am wondering if it has anything to do > with the way the Samba variables are set on my Linux box? > > The "strange behavior" is that when I set both NICs to use Jumbo Frames > [MTU=9014 on the Windows side (includes IP headers) , 9000 on the Linux side > (doesn't include the headers], I am getting about half the throughput that I get > when I set both NICs to use the standard MTU of 1514/1500. I see the same > behavior even if I take the switch out of the system and connect the Windows XP and > Linux machines directly to each other (crossover cable not required for > computer-to-computer connection with these NICs -- and by the way all of my cables > are CAT6). > > On the Linux side, I am using Samba 3.0.0 on Mandrake Linux 9.2 with all of > Mandrake's current updates -- kernel = 2.4.22-21enterprisemdk. The Linux > machine is a P4-3.06 Ghz with 1 GB of RAM -- running in hyperthreading mode. > > I am wondering if any of the Samba socket options settings like tcp_nodelay, > so_sndbuf=8192 or so_rcvbuf=8192 are affecting my throughput -- particularly > when I am using Jumbo Frames? And are there any other Samba settings that might > be interacting in a negative way with my TCP/IP and NIC driver settings that > are causing me to get lower throughput with Jumbo Frames instead of higher > throughput (which is what I am told I should be getting). > > Any guidance would be appreciated. I have purchased "The Official Samba 3 > HOW-TO and Reference Guide" but it really isn't very helpful when it comes to > understanding how to tune these options and how various socket options settings > interact with other network settings and hardware. > > Andy Liebman > Resolute Films > 119 Braintree Street, Suite 410 > Boston, MA 02134 > > Tel: 617-782-0479 > Cell: 617-308-0488 > Fax: 617-782-1071 > -- > To unsubscribe from this list go to the following URL and read the > instructions: http://lists.samba.org/mailman/listinfo/samba
Thanks for the reply. Do you know (and if so, caan you tell me) what the relationship is between these Samba settings and Linux settings such as net.core.rmem_default (or _max), net.core.wmem_default (or _max), net.ipv4.tcp_rmem and net.ipv4.tcp_wmem. Do the Samba options override the Linux socket options, or do they act as another layer of limits and buffers? Perhaps your TCP window is too small You should try the following global settings: read size = 65535 max xmit = 65535 socket options = TCP_NODELAY SO_SNDBUF=65535 SO_RCVBUF=65535 Rgds Per AndyLiebman@aol.com wrote:> > Hi, > > I am trying to optimize my gigabit network. I have two Intel 1000 MT Gigabit > Server Adapters, which support Jumbo Frames -- as well as a Switch that > supports Jumbo Frames. However, I am observing some strange behavior in myfile> transfers from Windows XP to Linux and I am wondering if it has anything todo> with the way the Samba variables are set on my Linux box? > > The "strange behavior" is that when I set both NICs to use Jumbo Frames > [MTU=9014 on the Windows side (includes IP headers) , 9000 on the Linux side > (doesn't include the headers], I am getting about half the throughput thatI get> when I set both NICs to use the standard MTU of 1514/1500. I see the same > behavior even if I take the switch out of the system and connect theWindows XP and> Linux machines directly to each other (crossover cable not required for > computer-to-computer connection with these NICs -- and by the way all of mycables> are CAT6). > > On the Linux side, I am using Samba 3.0.0 on Mandrake Linux 9.2 with all of > Mandrake's current updates -- kernel = 2.4.22-21enterprisemdk. The Linux > machine is a P4-3.06 Ghz with 1 GB of RAM -- running in hyperthreading mode. > > I am wondering if any of the Samba socket options settings like tcp_nodelay, > so_sndbuf=8192 or so_rcvbuf=8192 are affecting my throughput -- particularly > when I am using Jumbo Frames? And are there any other Samba settings thatmight> be interacting in a negative way with my TCP/IP and NIC driver settings that > are causing me to get lower throughput with Jumbo Frames instead of higher > throughput (which is what I am told I should be getting). > > Any guidance would be appreciated. I have purchased "The Official Samba 3 > HOW-TO and Reference Guide" but it really isn't very helpful when it comesto> understanding how to tune these options and how various socket optionssettings> interact with other network settings and hardware. > > Andy Liebman > Resolute Films > 119 Braintree Street, Suite 410 > Boston, MA 02134 > > Tel: 617-782-0479 > Cell: 617-308-0488 > Fax: 617-782-1071 > -- > To unsubscribe from this list go to the following URL and read the > instructions: http://lists.samba.org/mailman/listinfo/samba
So, here's the answer to the puzzle... I did as "Per" suggested -- set the so_sndbuf and so_rcvbuf to 65535 each, as well as read size and max xmit to the same. And low and behold, the throughput between my Windows XP box and my Linux Box's Firewire RAID 5 arrays went from around 20 MB/sec to around 30 MB/sec. And when I raised all those values by another factor of 4 (up to 262140 for each one) my throughput increased to around 36-37 MB/sec. Raising those values further didn't seem to make any difference. And, and, and, ... at the 262140 level, now I was able to set the MTUs to 4088 on the Windows side and 4074 on the Linux side (4K Jumbo Frames). While throughput essentially remained the same as what I was getting with a standard 1514/1500 MTU, I was happy that it didn't DROP by 50 percent, as it did before when I was trying to use 4K and 9K Jumbo Frames with the buffers set to 8092. And although throughput didn't go UP with the 4K frames, the CPU usage in each machine was pretty much cut in half. That made my Windows XP video editing applications happy. Incidentally, I still get a serious DROP in throughput, about 50 percent, if I try to use 9K Jumbo Frames -- even with those big buffers (I even tried up to 1 MB buffers). Perhaps that could be due to the fact that both my Windows and Linux boxes only have 32-bit PCI slots. I should add a few other notes. First, I am using Intel Gigabit Server NICs because they seem to have good Linux drivers -- and the Intel Tech Support folks were totally blown away by the fact that a few variables in a Samba configuration file could affect network throughput so dramatically. Alas, Samba is a mystery to many folks in this world -- myself included. Second, after hours of playing with TCP/IP settings on both the Windows and Linux sides -- TcpWindowSize in Windows, and a bunch of wmem, rmem, and mem values in Linux -- I came to the conclusion that none of my changes gave me any improvement over the default configurations of both operating systems (at least the way Linux is configured in Mandrake 9.2). Before I started tweaking, I had tried all sorts of Window sizes using the program "Iperf" on both ends -- and found pretty much that I got to the maximum throughput (a pure bandwidth of about 780 Mbits) at about 64K TCPWindows. But both OS's must do that by default, so tweaking was a waste of time in my case, where I have a fast LAN with almost zero latency. Finally, I have to say that I am impressed with the Intel Gigabit Server Ethernet adapters. The Intel e1000 drivers (not the eepro1000 driver that came with my Mandrake 9.2 distribution) give you fantastic flexibility in both Linux and Windows. For instance, the driver has a number of methods to let you choose whether the NIC interrupts the CPU each time it receives or transmits a packet, or whether the NIC stores up lots of packets and interrupts less frequently. Playing around with that and other settings in the driver allowed me to trade a 15 percent reduction in maximum throughput (and a slight increase in latency) for a 75 percent reduction in CPU usage! Which let my Windows video editing application flawlessly playback uncompressed video through some hardware that's also vying for attention on the PCI bus. Without the ability to do this kind of tweaking, my video editing application was getting interrupted too often to work properly. Surely there are other real time applications that can benefit from such control. I certainly had a happier experience with the Intel NIC than with either Linksys or SysKonnect NICs -- although I hear the SysKonnect NICs are a little faster than most. Hope all this information is useful to somebody out there. As a last word, of course my switch supports Jumbo Frames. I'm trying out a fairly new SMC switch, the SMC8508T. It's fast, cheap and out of control... No just kidding. It's fast and cheap (around $ 140 US) and as far as I can see it's the only switch any where near that price that supports Jumbo Frames. Says so right on the box. It's an unmanaged switch. But I know it's not the switch that's caused my original problems with Jumbo Frames, because I had the same problems when I took the switch out of the system and connected the computers directly. And that's still the case with the 9K Jumbo Frames. Finally, I put "use sendfile=yes" in my samba configuration file. But what does it do??? I suppose I should at least try taking it out and seeing what it does. Thanks for your suggestions. It's because of people like you that Linux keeps getting better, and Linux users keep getting better results. Andy In a message dated 12/17/2003 10:10:10 AM Eastern Standard Time, greg@max-t.com writes: I believe samba just does setsockopt or ioctl on the sockets. Do you get any errors on the interfaces in jumbo? Does your switch support jumbo? Setting "use sendfile=yes" will help alot on read speeds from samba. On the windows side check the settings. I think the e1000 has some adaptive spacing setting that kills throughput. Also some things to check on the linux side. e1000 module options like rxIntDelay, etc. You will not get much more performance out of jumbo unless your CPUs are maxed but you should not get less. What kind of numbers are you seeing? hope this helps. Greg On Wednesday 17 December 2003 08:23 am, AndyLiebman@aol.com wrote:> Thanks for the reply. Do you know (and if so, caan you tell me) what the > relationship is between these Samba settings and Linux settings such as > net.core.rmem_default (or _max), net.core.wmem_default (or _max), > net.ipv4.tcp_rmem and net.ipv4.tcp_wmem. Do the Samba options override the > Linux socket options, or do they act as another layer of limits and > buffers? > > Perhaps your TCP window is too small > You should try the following global settings: > > read size = 65535 > max xmit = 65535 > socket options = TCP_NODELAY SO_SNDBUF=65535 SO_RCVBUF=65535 > > > Rgds Per > > AndyLiebman@aol.com wrote: > > Hi, > > > > I am trying to optimize my gigabit network. I have two Intel 1000 MT > > Gigabit Server Adapters, which support Jumbo Frames -- as well as a > > Switch that supports Jumbo Frames. However, I am observing some strange > > behavior in my > > file > > > transfers from Windows XP to Linux and I am wondering if it has anything > > to > > do > > > with the way the Samba variables are set on my Linux box? > > > > The "strange behavior" is that when I set both NICs to use Jumbo Frames > > [MTU=9014 on the Windows side (includes IP headers) , 9000 on the Linux > > side (doesn't include the headers], I am getting about half the > > throughput that > > I get > > > when I set both NICs to use the standard MTU of 1514/1500. I see the same > > behavior even if I take the switch out of the system and connect the > > Windows XP and > > > Linux machines directly to each other (crossover cable not required for > > computer-to-computer connection with these NICs -- and by the way all of > > my > > cables > > > are CAT6). > > > > On the Linux side, I am using Samba 3.0.0 on Mandrake Linux 9.2 with all > > of Mandrake's current updates -- kernel = 2.4.22-21enterprisemdk. The > > Linux machine is a P4-3.06 Ghz with 1 GB of RAM -- running in > > hyperthreading mode. > > > > I am wondering if any of the Samba socket options settings like > > tcp_nodelay, so_sndbuf=8192 or so_rcvbuf=8192 are affecting my throughput > > -- particularly when I am using Jumbo Frames? And are there any other > > Samba settings that > > might > > > be interacting in a negative way with my TCP/IP and NIC driver settings > > that are causing me to get lower throughput with Jumbo Frames instead of > > higher throughput (which is what I am told I should be getting). > > > > Any guidance would be appreciated. I have purchased "The Official Samba 3 > > HOW-TO and Reference Guide" but it really isn't very helpful when it > > comes > > to > > > understanding how to tune these options and how various socket options > > settings > > > interact with other network settings and hardware. > > > > Andy Liebman > > Resolute Films > > 119 Braintree Street, Suite 410 > > Boston, MA 02134 > > > > Tel: 617-782-0479 > > Cell: 617-308-0488 > > Fax: 617-782-1071 > > -- > > To unsubscribe from this list go to the following URL and read the > > instructions: http://lists.samba.org/mailman/listinfo/samba