<florian.feldhaus@tu-dortmund.de>
2011-Feb-22 12:40 UTC
Problem with duplicate ACK packages
Hi! We have a very strange Shorewall Problem at our site. We''re using a 1to1 NAT setup for an HPC cluster (250 nodes). It is running fine, but sometimes established connections are stalling. We debugged several connections with tcpdump and found out that on the WAN interface of our gateway there are packages with duplicate ACKs which we can''t find on the internal interface of the gateway. The duplicate ACKs are probably send due to delayed or lost packages. As the duplicate ACKs are not delivered to the nodes behind the firewall, the connection stalls after a while. Unfortunately, we can''t find the reason, why shorewall / iptables is blocking the packages with duplicate ACKs. Can anyone give us a hint how to solve/further debug this problem? /sbin/shorewall version 4.4.15.1 ip addr show ip route show shorewall show -> see attachments Cheers, Florian Feldhaus ------------------------------------------------------------------------------ Index, Search & Analyze Logs and other IT data in Real-Time with Splunk Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. Free Software Download: http://p.sf.net/sfu/splunk-dev2dev
On 2/22/11 4:40 AM, florian.feldhaus@tu-dortmund.de wrote:> We have a very strange Shorewall Problem at our site. We''re using a 1to1 > NAT setup for an HPC cluster (250 nodes). It is running fine, but > sometimes established connections are stalling. We debugged several > connections with tcpdump and found out that on the WAN interface of our > gateway there are packages with duplicate ACKs which we can''t find on the > internal interface of the gateway. The duplicate ACKs are probably send > due to delayed or lost packages. As the duplicate ACKs are not delivered > to the nodes behind the firewall, the connection stalls after a while. > Unfortunately, we can''t find the reason, why shorewall / iptables is > blocking the packages with duplicate ACKs. Can anyone give us a hint how > to solve/further debug this problem?This has nothing to do with Shorewall configuration. Please try inserting this into /etc/shorewall/init and see if it helps: echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Index, Search & Analyze Logs and other IT data in Real-Time with Splunk Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. Free Software Download: http://p.sf.net/sfu/splunk-dev2dev
<florian.feldhaus@tu-dortmund.de>
2011-Feb-22 16:39 UTC
Re: Problem with duplicate ACK packages
Hi Tom, thank you very much for that hint. It seems to solve our problem. Unfortunately I can''t find much information on that parameter other than the fact that it tells netfilter to be more liberal with packages out of the current TCP window. I will search for more information on the parameter but would be glad for any link to additional information - especially why setting this parameter becomes necessary for our configuration. Cheers, Florian Am 22.02.11 16:40 schrieb "Tom Eastep" unter <teastep@shorewall.net>:>On 2/22/11 4:40 AM, florian.feldhaus@tu-dortmund.de wrote: > >> We have a very strange Shorewall Problem at our site. We''re using a 1to1 >> NAT setup for an HPC cluster (250 nodes). It is running fine, but >> sometimes established connections are stalling. We debugged several >> connections with tcpdump and found out that on the WAN interface of our >> gateway there are packages with duplicate ACKs which we can''t find on >>the >> internal interface of the gateway. The duplicate ACKs are probably send >> due to delayed or lost packages. As the duplicate ACKs are not delivered >> to the nodes behind the firewall, the connection stalls after a while. >> Unfortunately, we can''t find the reason, why shorewall / iptables is >> blocking the packages with duplicate ACKs. Can anyone give us a hint how >> to solve/further debug this problem? > >This has nothing to do with Shorewall configuration. Please try >inserting this into /etc/shorewall/init and see if it helps: > > echo 1 > /proc/sys/net/netfilter/nf_conntrack_tcp_be_liberal > >-Tom >-- >Tom Eastep \ When I die, I want to go like my Grandfather who >Shoreline, \ died peacefully in his sleep. Not screaming like >Washington, USA \ all of the passengers in his car >http://shorewall.net \________________________________________________ > >-------------------------------------------------------------------------- >---- >Index, Search & Analyze Logs and other IT data in Real-Time with Splunk >Collect, index and harness all the fast moving IT data generated by your >applications, servers and devices whether physical, virtual or in the >cloud. >Deliver compliance at lower cost and gain new business insights. >Free Software Download: >http://p.sf.net/sfu/splunk-dev2dev________________________________________ >_______ >Shorewall-users mailing list >Shorewall-users@lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/shorewall-users------------------------------------------------------------------------------ Index, Search & Analyze Logs and other IT data in Real-Time with Splunk Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. Free Software Download: http://p.sf.net/sfu/splunk-dev2dev
On 2/22/11 8:39 AM, florian.feldhaus@tu-dortmund.de wrote:> thank you very much for that hint. It seems to solve our problem. > Unfortunately I can''t find much information on that parameter other than > the fact that it tells netfilter to be more liberal with packages out of > the current TCP window. > > I will search for more information on the parameter but would be glad for > any link to additional information - especially why setting this parameter > becomes necessary for our configuration.Hi Florian, I have no quick reference to additional information beyond what a Google search turns up. But experience has shown that where TCP stalls are occurring, setting that parameter usually corrects the problem. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Free Software Download: Index, Search & Analyze Logs and other IT data in Real-Time with Splunk. Collect, index and harness all the fast moving IT data generated by your applications, servers and devices whether physical, virtual or in the cloud. Deliver compliance at lower cost and gain new business insights. http://p.sf.net/sfu/splunk-dev2dev