After upgrading to 7.2 (amd64) some customers complained of very poor bandwidth. Upon investigation all the effected customers were ATT DSL clients located all over the USA, not in a single city, nor were other ISPs effected. The server is a Supermicro with dual (quad core) processors with a single Intel fxp network card on a 100mbit connection. Kernel is GENERIC for both 7.1_release and 7.2_release. Normally a client can max out their download connection, but for ATT DSL customers the transfer rate would be about 5-10KB/s even though the server and client where both idle. Repeated tests were done, from multiple clients in different geographical locations. The problem manifested itself regardless of whether ftp, http, smtp, pop, or scp was used, and regardless of the OS of the client. Believing it to be a routing issue we changed the route and even changed the local router the server is connected to so that a different NIC port would be used to talk to ATT DSL customers, but no change in performance. Turns out it is somehow related to differences in FreeBSD 7.1 and 7.2. If I boot the same server with 7.1, all clients work as you would expect. But, if 7.2 is used all clients with the exception of ATT DSL clients would work normally, ATT customers would be limited to 5-10KB/s. I have no reason to believe there is anything wrong with the ATT DSL network, it just happen to be effected by whatever causes the problem. Any theories? A special thanks to cybercon.com tech support for being so helpful. If you need a data center, they have good tech support.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi David, David Samms wrote:> After upgrading to 7.2 (amd64) some customers complained of very poor > bandwidth. Upon investigation all the effected customers were ATT DSL > clients located all over the USA, not in a single city, nor were other > ISPs effected. The server is a Supermicro with dual (quad core) > processors with a single Intel fxp network card on a 100mbit connection.Could you please try if this would help: sysctl net.inet.tcp.tso=0 Cheers, - -- Xin LI <delphij@delphij.net> http://www.delphij.net/ FreeBSD - The Power to Serve! -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.11 (FreeBSD) iEYEARECAAYFAkoJ3vAACgkQi+vbBBjt66CKqQCgwPkg1IZnI61Q1+PWfr5sOvVm n5IAnAzbI5HQXQqyPg+DmzHvCNhzhelI =oHGO -----END PGP SIGNATURE-----
Xin LI wrote:> Hi David, > > David Samms wrote: >> After upgrading to 7.2 (amd64) some customers complained of very poor >> bandwidth. Upon investigation all the effected customers were ATT DSL >> clients located all over the USA, not in a single city, nor were other >> ISPs effected. The server is a Supermicro with dual (quad core) >> processors with a single Intel fxp network card on a 100mbit connection. > > Could you please try if this would help: > > sysctl net.inet.tcp.tso=0 > > Cheers, > - -- > Xin LI <delphij@delphij.net> http://www.delphij.net/ > FreeBSD - The Power to Serve!Xin LI, Thank you for your help. Setting sysctl net.inet.tcp.tso=0 resolved the issue completely. What does sysctl net.inet.tcp.tso=0 do? Where can I read more about the option? I captured tcpdumps of a single file transfer to 7.1, 7.2 and 7.2 with sysctl net.inet.tcp.tso=0, but they are to large to attach to this list. Let me know if you are interested in viewing the dump files. Thanks again for your assistance!
On Tue, May 12, 2009 at 05:31:01PM -0400, David Samms wrote:> > Setting sysctl net.inet.tcp.tso=0 resolved the issue completely. What > does sysctl net.inet.tcp.tso=0 do?# sysctl -d net.inet.tcp.tso net.inet.tcp.tso: Enable TCP Segmentation Offload I had a similar problem with a different NIC. This option controls whether we offload segmenting to the NIC. My NIC seemed to be limited by the number of interrupts which could be delivered. You can also do this on a card-by-card basis using "ifconfig <interface> -tso". -- Rick C. Petty
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, David, David Samms wrote:> Xin LI wrote: >> Hi David, >> >> David Samms wrote: >>> After upgrading to 7.2 (amd64) some customers complained of very poor >>> bandwidth. Upon investigation all the effected customers were ATT DSL >>> clients located all over the USA, not in a single city, nor were other >>> ISPs effected. The server is a Supermicro with dual (quad core) >>> processors with a single Intel fxp network card on a 100mbit connection. >> >> Could you please try if this would help: >> >> sysctl net.inet.tcp.tso=0 >> >> Cheers, >> - -- >> Xin LI <delphij@delphij.net> http://www.delphij.net/ >> FreeBSD - The Power to Serve! > > Xin LI, > > Thank you for your help. > > Setting sysctl net.inet.tcp.tso=0 resolved the issue completely. What > does sysctl net.inet.tcp.tso=0 do? Where can I read more about the > option? I captured tcpdumps of a single file transfer to 7.1, 7.2 and > 7.2 with sysctl net.inet.tcp.tso=0, but they are to large to attach to > this list. Let me know if you are interested in viewing the dump files. > > Thanks again for your assistance!Thanks for the offer but I think this is a known problem so perhaps the dump files are no longer necessary. The problem was caused by the reciever side (usually PPPoE clients, e.g. DSL users) which proposes a smaller MSS than the interface MTU, the previous implementation sets the packet length to interface MTU instead of the negotiated one, which would cause problem. Setting net.inet.tcp.tso=0 would turn off TCP Segment Offloading completely. The previous release of FreeBSD does not include this feature. I think yongari@ has committed a fix as revision 191867 (RELENG_7) and 190982 (HEAD): Index: if_fxp.c ==================================================================- --- if_fxp.c (revision 190981) +++ if_fxp.c (revision 190982) @@ -1485,7 +1485,8 @@ * checksum in the first frame driver should compute it. */ ip->ip_sum = 0; - - ip->ip_len = htons(ifp->if_mtu); + ip->ip_len = htons(m->m_pkthdr.tso_segsz + (ip->ip_hl << 2) + + (tcp->th_off << 2)); tcp->th_sum = in_pseudo(ip->ip_src.s_addr, ip->ip_dst.s_addr, htons(IPPROTO_TCP + (tcp->th_off << 2) + m->m_pkthdr.tso_segsz)); To re@: Perhaps we should issue an errata for this, at least document it in errata (I can do this)? Cheers, - -- Xin LI <delphij@delphij.net> http://www.delphij.net/ FreeBSD - The Power to Serve! -----BEGIN PGP SIGNATURE----- Version: GnuPG v2.0.11 (FreeBSD) iEYEARECAAYFAkoJ89AACgkQi+vbBBjt66B85ACeNJjEuVXitnceaC6GRG+9zWtB OaUAoLqikyZXMEngwkLEtHboaDiQp8QI =mcFR -----END PGP SIGNATURE-----
On 13/5/09 8:41 AM, Xin LI wrote:> -----BEGIN PGP SIGNED MESSAGE----- > Hash: SHA1 > > Hi David, > > David Samms wrote: >> After upgrading to 7.2 (amd64) some customers complained of very poor >> bandwidth. Upon investigation all the effected customers were ATT DSL >> clients located all over the USA, not in a single city, nor were other >> ISPs effected. The server is a Supermicro with dual (quad core) >> processors with a single Intel fxp network card on a 100mbit connection. > > Could you please try if this would help: > > sysctl net.inet.tcp.tso=0 > > Cheers, > - -- > Xin LI<delphij@delphij.net> http://www.delphij.net/Thank you! This hint has saved me a lot of troubleshooting. I was having the same issue as David with 3 servers recently upgraded to 7.2. Clients (MS Windows) were complaining that they were having intermittent connectivity issues talking to these servers (https, imaps). They too have fxp network interface cards, no issues with other servers upgraded to 7.2 with em cards. Thanks again. Regards, Nigel.
On Wed, May 13, 2009 at 10:52:34AM +1200, Nigel Wohlers wrote:> On 13/5/09 8:41 AM, Xin LI wrote: > >-----BEGIN PGP SIGNED MESSAGE----- > >Hash: SHA1 > > > >Hi David, > > > >David Samms wrote: > >>After upgrading to 7.2 (amd64) some customers complained of very poor > >>bandwidth. Upon investigation all the effected customers were ATT DSL > >>clients located all over the USA, not in a single city, nor were other > >>ISPs effected. The server is a Supermicro with dual (quad core) > >>processors with a single Intel fxp network card on a 100mbit connection. > > > >Could you please try if this would help: > > > > sysctl net.inet.tcp.tso=0 > > > >Cheers, > >- -- > >Xin LI<delphij@delphij.net> http://www.delphij.net/ > > > Thank you! This hint has saved me a lot of troubleshooting. > > I was having the same issue as David with 3 servers recently upgraded to > 7.2. Clients (MS Windows) were complaining that they were having > intermittent connectivity issues talking to these servers (https, imaps). > > They too have fxp network interface cards, no issues with other servers > upgraded to 7.2 with em cards. >Instead of disabling TSO in network stack, just disable TSO in fxp(4) as a workaround. Fix already is in RELENG_7(r191867) so you can extract the patch and apply it by hand if you want. For instance, #cd /tmp #fetch -o fxp.tso.patch "http://svn.freebsd.org/viewvc/base/head/sys/dev/fxp/if_fxp.c?r1=190982&r2=188176&view=patch" #cd /usr/src/sys/dev/fxp #patch -p4 < /tmp/fxp.tso.patch And rebuild kernel.