bugzilla-daemon@bugzilla.netfilter.org
2006-Sep-15 05:57 UTC
[Bug 511] New: Premature ip_conntrack timer expiry on 3+ window size advertisements
https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=511 Summary: Premature ip_conntrack timer expiry on 3+ window size advertisements Product: netfilter/iptables Version: linux-2.6.x Platform: All OS/Version: All Status: NEW Severity: minor Priority: P2 Component: ip_conntrack AssignedTo: laforge@netfilter.org ReportedBy: georgeh@anstat.com.au The linux firewall loses track of valid TCP connections, due to premature expiry of the ip_conntrack timer. This can result in one end of the TCP connection being kept open indefinitely, waiting for more data. This data is now being dropped by the firewall, due to the ip_conntrack timer having expired. Example: A tomcat server which is supplying data to a front-end apache server typically has a limit to the number of "threads" it allows. These threads are related to the number of open TCP/IP connections. TCP/IP connections affected by this problem are kept open, and accumulate over time. Eventually the tomcat server reaches the limit of allowed threads, and stops serving requests from the apache server. This problem is caused by insufficient criteria used for "dead-peer" detection. The current algorithm is: if 3 of the "same" packets are seen in one direction, with no traffic in the other direction, the receiver is treated as a "dead-peer" and the ip_conntrack timer is reset to 5 minutes. The criteria for being the "same" is based on the TCP sequence number and acknowledgement number, but does not include the TCP window size advertisement. Sample occurance : Extract from /proc/net/ip_conntrack 08:46:51.811 431999 ESTABLISHED src=192.168.81.245 dst=192.168.49.30 sport=48610 dport=8009 08:47:01.532 431998 ESTABLISHED src=192.168.81.245 dst=192.168.49.30 sport=48610 dport=8009 08:47:01.920 000299 ESTABLISHED src=192.168.81.245 dst=192.168.49.30 sport=48610 dport=8009 08:47:02.925 000298 ESTABLISHED src=192.168.81.245 dst=192.168.49.30 sport=48610 dport=8009 08:47:03.919 000297 ESTABLISHED src=192.168.81.245 dst=192.168.49.30 sport=48610 dport=8009 ... 08:47:56.922 431999 ESTABLISHED src=192.168.81.245 dst=192.168.49.30 sport=48610 dport=8009 Extract from tcpdump 08:46:59.997669 IP tomcat-server.8009 > apache-server.48610: . 347295:348743(1448) ack 889 win 5896 <nop,nop,timestamp 3345910565 508393589> 08:46:59.997681 IP tomcat-server.8009 > apache-server.48610: . 348743:350191(1448) ack 889 win 5896 <nop,nop,timestamp 3345910565 508393589> 08:46:59.997693 IP tomcat-server.8009 > apache-server.48610: . 350191:351639(1448) ack 889 win 5896 <nop,nop,timestamp 3345910565 508393589> 08:46:59.997702 IP tomcat-server.8009 > apache-server.48610: P 351639:352129(490) ack 889 win 5896 <nop,nop,timestamp 3345910565 508393589> 08:47:00.035237 IP apache-server.48610 > tomcat-server.8009: . ack 352129 win 2406 <nop,nop,timestamp 508393593 3345910565> 08:47:00.162163 IP apache-server.48610 > tomcat-server.8009: . ack 352129 win 10136 <nop,nop,timestamp 508393605 3345910565> 08:47:00.515958 IP apache-server.48610 > tomcat-server.8009: . ack 352129 win 24616 <nop,nop,timestamp 508393641 3345910565> 08:47:01.915395 IP apache-server.48610 > tomcat-server.8009: . ack 352129 win 56472 <nop,nop,timestamp 508393780 3345910565> 08:47:56.910459 IP apache-server.48610 > tomcat-server.8009: P 889:1636(747) ack 352129 win 63712 <nop,nop,timestamp 508399279 3345910565> 08:47:56.910484 IP tomcat-server.8009 > apache-server.48610: . ack 1636 win 5896 <nop,nop,timestamp 3345967478 508399279> 08:47:57.464866 IP tomcat-server.8009 > apache-server.48610: P 352129:352216(87) ack 1636 win 5896 <nop,nop,timestamp 3345968032 508399279> This problem occurs frequently in real-life, but it is difficult to reproduce "at will". I suspect another symptom of this problem are dropped "FIN" packets on a firewall. Once the ip_conntrack timer has expired, and the client or server try to close the connection (without more data being transfered from the client), the FIN packet it classified as "INVALID", and dropped. To show any "lost" tcp connections, use: netstat -na | gawk '/ESTAB/ { local=$4; gsub(".*:","",local) ; remote=$5 ; gsub(".*:","",remote); result=system("grep -q port=" local ".\\*port=" remote " /proc/net/ip_conntrack"); print local"/"remote " = " ( result ? "not found" : "found" );}' Note that the busier the apache and tomcat servers get, the less likely they are to suffer from this problem A poosible patch would as follows, but this has not been tested due to the problem being difficult to reproduce under controlled conditions (any suggestions are welcome). --- /usr/src/linux-2.6.17.13/include/linux/netfilter/nf_conntrack_tcp.h.orig 2006-09-09 13:23:25.000000000 +1000 +++ /usr/src/linux-2.6.17.13/include/linux/netfilter/nf_conntrack_tcp.h 2006-09-15 11:18:28.000000000 +1000 @@ -49,6 +49,7 @@ u_int32_t last_seq; /* Last sequence number seen in dir */ u_int32_t last_ack; /* Last sequence number seen in opposite dir */ u_int32_t last_end; /* Last seq + len */ + u_int16_t last_win; /* Last window advertisement seen in dir */ }; #endif /* __KERNEL__ */ --- /usr/src/linux-2.6.17.13/net/ipv4/netfilter/ip_conntrack_proto_tcp.c.orig 2006-09-09 13:23:25.000000000 +1000 +++ /usr/src/linux-2.6.17.13/net/ipv4/netfilter/ip_conntrack_proto_tcp.c 2006-09-15 11:16:07.000000000 +1000 @@ -732,12 +732,14 @@ if (state->last_dir == dir && state->last_seq == seq && state->last_ack == ack + && state->last_win == win && state->last_end == end) state->retrans++; else { state->last_dir = dir; state->last_seq = seq; state->last_ack = ack; + state->last_win = win; state->last_end = end; state->retrans = 0; } -- Configure bugmail: https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email ------- You are receiving this mail because: ------- You are the assignee for the bug, or are watching the assignee.
Maybe Matching Threads
- [Bug 511] Premature ip_conntrack timer expiry on 3+ window size advertisements
- Problem : Debian xen package removal & installation
- Changing colour of continuous time-series in ggplot2
- Icecast not listening ports?
- BRIstuff/QuadBRI problem: Ring requested on unconfigured channel 255/255 span 5