Dear listers, We recently found our new FreeBSD server (located in some foreign region) has poor network performance. After doing some tcpdump and iperf testing, we found that out-of-order TCP packets are not inserted into queue. This is an 100Mbps line, and TSO is disabled. % uname -a FreeBSD bsd 7.1-RC2 FreeBSD 7.1-RC2 #2: Wed Dec 31 03:12:39 CST 2008 root@bsd:/usr/obj/usr/src/sys/KERNEL amd64 % iperf -c 10.1.1.250 ------------------------------------------------------------ Client connecting to office, TCP port 5001 TCP window size: 3.07 MByte (default) ------------------------------------------------------------ [ 4] local 10.1.1.210 port 61488 connected with 10.1.1.250 port 5001 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.2 sec 5.74 MBytes 4.74 Mbits/sec 03:47:21.146397 IP 10.1.1.210.54919 > 10.1.1.250.5001: . 159305:160753(1448) ack 1 win 1040 <nop,nop,timestamp 555928950 3216612488> 03:47:21.146409 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 160753 win 12568 <nop,nop,timestamp 3216612488 555928950> 03:47:21.146473 IP 10.1.1.210.54919 > 10.1.1.250.5001: . 160753:162201(1448) ack 1 win 1040 <nop,nop,timestamp 555928950 3216612488> 03:47:21.146485 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 162201 win 12568 <nop,nop,timestamp 3216612489 555928950> 03:47:21.146972 IP 10.1.1.210.54919 > 10.1.1.250.5001: . 163649:165097(1448) ack 1 win 1040 <nop,nop,timestamp 555928950 3216612488> 03:47:21.146983 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 162201 win 12573 <nop,nop,timestamp 3216612489 555928950> 03:47:21.146985 IP 10.1.1.210.54919 > 10.1.1.250.5001: . 162201:163649(1448) ack 1 win 1040 <nop,nop,timestamp 555928950 3216612488> 03:47:21.146996 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 163649 win 12568 <nop,nop,timestamp 3216612489 555928950> 03:47:21.146998 IP 10.1.1.210.54919 > 10.1.1.250.5001: . 165097:166545(1448) ack 1 win 1040 <nop,nop,timestamp 555928950 3216612488> 03:47:21.147006 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 163649 win 12573 <nop,nop,timestamp 3216612489 555928950> 03:47:21.147009 IP 10.1.1.210.54919 > 10.1.1.250.5001: . 166545:167993(1448) ack 1 win 1040 <nop,nop,timestamp 555928950 3216612488> 03:47:21.147017 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 163649 win 12573 <nop,nop,timestamp 3216612489 555928950> 03:47:21.147019 IP 10.1.1.210.54919 > 10.1.1.250.5001: . 167993:169441(1448) ack 1 win 1040 <nop,nop,timestamp 555928950 3216612488> * You can see "ack 163649" repeating, but the packet is transmitted before 163649:165097. % cat /etc/sysctl.conf # $FreeBSD: src/etc/sysctl.conf,v 1.8 2003/03/13 18:43:50 mux Exp $ # # This file is read when going to multi-user and its contents piped thru # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details. # # Uncomment this to prevent users from seeing information about processes that # are being run under another UID. #security.bsd.see_other_uids=0 debug.bootverbose=1 kern.ipc.somaxconn=8192 kern.maxfiles=65536 kern.maxfilesperproc=32768 kern.maxprocperuid=65536 net.inet.ip.fastforwarding=1 net.inet.tcp.delayed_ack=0 vm.pmap.shpgperproc=2000 kern.ipc.maxsockbuf=8388608 net.inet.tcp.sendspace=3217968 net.inet.tcp.recvspace=3217968 Is our configuration wrong? Or it is an known bug? I have searched stable & net list, but found no similar discussion. Thank you all in advance!
Lin Jui-Nan Eric wrote, On 30.12.2008 21:06:> Dear listers, > > We recently found our new FreeBSD server (located in some foreign > region) has poor network performance. After doing some tcpdump and > iperf testing, we found that out-of-order TCP packets are not inserted > into queue. > > This is an 100Mbps line, and TSO is disabled. > > % uname -a > FreeBSD bsd 7.1-RC2 FreeBSD 7.1-RC2 #2: Wed Dec 31 03:12:39 CST 2008 > root@bsd:/usr/obj/usr/src/sys/KERNEL amd64 > > % iperf -c 10.1.1.250 > ------------------------------------------------------------ > Client connecting to office, TCP port 5001 > TCP window size: 3.07 MByte (default) > ------------------------------------------------------------ > [ 4] local 10.1.1.210 port 61488 connected with 10.1.1.250 port 5001 > [ ID] Interval Transfer Bandwidth > [ 4] 0.0-10.2 sec 5.74 MBytes 4.74 Mbits/sec > > 03:47:21.146397 IP 10.1.1.210.54919 > 10.1.1.250.5001: . > 159305:160753(1448) ack 1 win 1040 <nop,nop,timestamp 555928950 > 3216612488> > 03:47:21.146409 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 160753 > win 12568 <nop,nop,timestamp 3216612488 555928950> > 03:47:21.146473 IP 10.1.1.210.54919 > 10.1.1.250.5001: . > 160753:162201(1448) ack 1 win 1040 <nop,nop,timestamp 555928950 > 3216612488> > 03:47:21.146485 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 162201 > win 12568 <nop,nop,timestamp 3216612489 555928950> > 03:47:21.146972 IP 10.1.1.210.54919 > 10.1.1.250.5001: . > 163649:165097(1448) ack 1 win 1040 <nop,nop,timestamp 555928950 > 3216612488> > 03:47:21.146983 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 162201 > win 12573 <nop,nop,timestamp 3216612489 555928950> > 03:47:21.146985 IP 10.1.1.210.54919 > 10.1.1.250.5001: . > 162201:163649(1448) ack 1 win 1040 <nop,nop,timestamp 555928950 > 3216612488> > 03:47:21.146996 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 163649 > win 12568 <nop,nop,timestamp 3216612489 555928950> > 03:47:21.146998 IP 10.1.1.210.54919 > 10.1.1.250.5001: . > 165097:166545(1448) ack 1 win 1040 <nop,nop,timestamp 555928950 > 3216612488> > 03:47:21.147006 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 163649 > win 12573 <nop,nop,timestamp 3216612489 555928950> > 03:47:21.147009 IP 10.1.1.210.54919 > 10.1.1.250.5001: . > 166545:167993(1448) ack 1 win 1040 <nop,nop,timestamp 555928950 > 3216612488> > 03:47:21.147017 IP 10.1.1.250.5001 > 10.1.1.210.54919: . ack 163649 > win 12573 <nop,nop,timestamp 3216612489 555928950> > 03:47:21.147019 IP 10.1.1.210.54919 > 10.1.1.250.5001: . > 167993:169441(1448) ack 1 win 1040 <nop,nop,timestamp 555928950 > 3216612488> > > * You can see "ack 163649" repeating, but the packet is transmitted > before 163649:165097. > > % cat /etc/sysctl.conf > # $FreeBSD: src/etc/sysctl.conf,v 1.8 2003/03/13 18:43:50 mux Exp $ > # > # This file is read when going to multi-user and its contents piped thru > # ``sysctl'' to adjust kernel values. ``man 5 sysctl.conf'' for details. > # > > # Uncomment this to prevent users from seeing information about processes that > # are being run under another UID. > #security.bsd.see_other_uids=0 > debug.bootverbose=1 > kern.ipc.somaxconn=8192 > kern.maxfiles=65536 > kern.maxfilesperproc=32768 > kern.maxprocperuid=65536 > net.inet.ip.fastforwarding=1 > net.inet.tcp.delayed_ack=0 > vm.pmap.shpgperproc=2000 > kern.ipc.maxsockbuf=8388608 > net.inet.tcp.sendspace=3217968 > net.inet.tcp.recvspace=3217968 > > Is our configuration wrong? Or it is an known bug? I have searched > stable & net list, but found no similar discussion. > Thank you all in advance!Do you have net.inet.tcp.sack.enable=1? -- VH
Dear All listers, After running "netstat -s -p tcp", we found that lots of packets are discarded due to memory problems. We googled for it, and found that sysctl oid "net.inet.tcp.reass.maxsegments" became 0, therefore packets never reassembled. Then we checked our /boot/loader.conf and /etc/sysctl.conf, and found that setting kern.ipc.nmbclusters="0" makes net.inet.tcp.reass.maxsegments=0. After setting net.inet.tcp.reass.maxsegments="1600" in /boot/loader.conf, the network works perfectly now. Thank you all for the help!
On Fri, 2 Jan 2009, Lin Jui-Nan Eric wrote:> After running "netstat -s -p tcp", we found that lots of packets are > discarded due to memory problems. We googled for it, and found that sysctl > oid "net.inet.tcp.reass.maxsegments" became 0, therefore packets never > reassembled. > > Then we checked our /boot/loader.conf and /etc/sysctl.conf, and found that > setting kern.ipc.nmbclusters="0" makes net.inet.tcp.reass.maxsegments=0. > After setting net.inet.tcp.reass.maxsegments="1600" in /boot/loader.conf, > the network works perfectly now.Was it set to 0 through a configuration error, or did the system auto-tune improperly? Robert N M Watson Computer Laboratory University of Cambridge> > Thank you all for the help! > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >
Hi Robert, I thought that the system auto-tune improperly in this case. On Mon, Jan 5, 2009 at 9:13 PM, Robert Watson <rwatson@freebsd.org> wrote:> On Fri, 2 Jan 2009, Lin Jui-Nan Eric wrote: > >> After running "netstat -s -p tcp", we found that lots of packets are >> discarded due to memory problems. We googled for it, and found that sysctl >> oid "net.inet.tcp.reass.maxsegments" became 0, therefore packets never >> reassembled. >> >> Then we checked our /boot/loader.conf and /etc/sysctl.conf, and found that >> setting kern.ipc.nmbclusters="0" makes net.inet.tcp.reass.maxsegments=0. >> After setting net.inet.tcp.reass.maxsegments="1600" in /boot/loader.conf, >> the network works perfectly now. > > Was it set to 0 through a configuration error, or did the system auto-tune > improperly? > > Robert N M Watson > Computer Laboratory > University of Cambridge > >> >> Thank you all for the help! >> _______________________________________________ >> freebsd-stable@freebsd.org mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >> >
On Tue, 6 Jan 2009, Lin Jui-Nan Eric wrote:> I thought that the system auto-tune improperly in this case.Hmm. Do you have a custom setting for kern.ipc.nmbclusters in loader.conf or sysctl.conf? What does kern.ipc.nmbclusters configure itself to on your system? Also, could you send me the output of uname -a on the system? Thanks, Robert N M Watson Computer Laboratory University of Cambridge> On Mon, Jan 5, 2009 at 9:13 PM, Robert Watson <rwatson@freebsd.org> wrote: >> On Fri, 2 Jan 2009, Lin Jui-Nan Eric wrote: >> >>> After running "netstat -s -p tcp", we found that lots of packets are >>> discarded due to memory problems. We googled for it, and found that sysctl >>> oid "net.inet.tcp.reass.maxsegments" became 0, therefore packets never >>> reassembled. >>> >>> Then we checked our /boot/loader.conf and /etc/sysctl.conf, and found that >>> setting kern.ipc.nmbclusters="0" makes net.inet.tcp.reass.maxsegments=0. >>> After setting net.inet.tcp.reass.maxsegments="1600" in /boot/loader.conf, >>> the network works perfectly now. >> >> Was it set to 0 through a configuration error, or did the system auto-tune >> improperly? >> >> Robert N M Watson >> Computer Laboratory >> University of Cambridge >> >>> >>> Thank you all for the help! >>> _______________________________________________ >>> freebsd-stable@freebsd.org mailing list >>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >>> To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >>> >> >
Oops, we surely have kern.ipc.nmbclusters="0" in loader.conf, but I think that should not modify net.inet.tcp.reass.maxsegments to "0" since we wish unlimited nmbclusters but not zero TCP reassembly segments. On Tue, Jan 6, 2009 at 4:16 PM, Robert Watson <rwatson@freebsd.org> wrote:> > On Tue, 6 Jan 2009, Lin Jui-Nan Eric wrote: > >> I thought that the system auto-tune improperly in this case. > > Hmm. Do you have a custom setting for kern.ipc.nmbclusters in loader.conf > or sysctl.conf? What does kern.ipc.nmbclusters configure itself to on your > system? Also, could you send me the output of uname -a on the system? > > Thanks, > > Robert N M Watson > Computer Laboratory > University of Cambridge > >> On Mon, Jan 5, 2009 at 9:13 PM, Robert Watson <rwatson@freebsd.org> wrote: >>> >>> On Fri, 2 Jan 2009, Lin Jui-Nan Eric wrote: >>> >>>> After running "netstat -s -p tcp", we found that lots of packets are >>>> discarded due to memory problems. We googled for it, and found that >>>> sysctl >>>> oid "net.inet.tcp.reass.maxsegments" became 0, therefore packets never >>>> reassembled. >>>> >>>> Then we checked our /boot/loader.conf and /etc/sysctl.conf, and found >>>> that >>>> setting kern.ipc.nmbclusters="0" makes net.inet.tcp.reass.maxsegments=0. >>>> After setting net.inet.tcp.reass.maxsegments="1600" in >>>> /boot/loader.conf, >>>> the network works perfectly now. >>> >>> Was it set to 0 through a configuration error, or did the system >>> auto-tune >>> improperly? >>> >>> Robert N M Watson >>> Computer Laboratory >>> University of Cambridge >>> >>>> >>>> Thank you all for the help! >>>> _______________________________________________ >>>> freebsd-stable@freebsd.org mailing list >>>> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >>>> To unsubscribe, send any mail to >>>> "freebsd-stable-unsubscribe@freebsd.org" >>>> >>> >> >