netfilter buglog - Feb 2006 - [Bug 443] 2.6 kernel failing in NAT with significant outbound traffic

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443





------- Additional Comments From nothingel@hotmail.com  2006-02-08 05:35 MET
-------
I also, the situation described in bug ID 322 seemed related and I tried the
patch from Phil Oester but it did not make a difference.

-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443





------- Additional Comments From nothingel@hotmail.com  2006-02-08 05:41 MET
-------
Created an attachment (id=206)
 -->
(https://bugzilla.netfilter.org/bugzilla/attachment.cgi?id=206&action=view)
nicely formatted tcpdump captures

the tcpdump output in the report is hard-to-read.  I'm attaching this file
in
hopes it will be much easier.

-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443





------- Additional Comments From nothingel@hotmail.com  2006-02-09 00:44 MET
-------
just tested the following:

Fedora Core 2 (stock kernel, 2.6.5-1.358) does NOT exhibit the problem
described.  It works perfectly!

Fedora Core 2 (upgraded to kernel 2.6.10-1.771 via the updates) DOES EXHIBIT the
problem described above.  So...some code has changed it seems.

--
I've attempted to draw a diagram of the network layout.  It's important
to note
that the problem is ONLY observed when data is being tranferred TO
"internetserver1" (as the arrows indicate).  Data flowing FROM
internetserver1
does not have a problem.



+---------+
| client1 |
+---------+

   -->

       +-------------------------+
       |          linux1         |
       |  (the problem machine   |
       | with SNAT or MASQUERADE)|
       +-------------------------+

          -->

               +-----------------------+
               | at least 2 routers    |
               | plus at least one     |
               | Cisco PIX.  NO ICMP is|
               | allowed so the exact  |
               | number is unknown.    |
               |                       |
               | I have no knowledge or|
               | control at this level |
               +-----------------------+

                          --->

                               +-------------+
                               |  Internet   |
                               +-------------+

                                         ---->

                                              +------------------+
                                              |  internetserver1 |
                                              | under my control |
                                              +------------------+





-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443


laforge@netfilter.org changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
           Severity|blocker                     |normal




------- Additional Comments From laforge@netfilter.org  2006-02-09 15:48 MET
-------
have you tried to disable window scaling via sysctl?

This really looks like it isn't a linux/netfilter/iptables problem, but
rather a
problem of some [broken?] box sitting in between that tries to do nasty things
to packets.  The rewritten MSS is one indication for that.  Also, the ICMP block
will break PMTU, maybe that's why somebody tries to workaround by decreasing
the
MSS.

What happens if you set the MSS to 1000 (or even change the MTU of the outbound
interface to 1000 ?

-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You reported the bug, or are watching the reporter.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443





------- Additional Comments From nothingel@hotmail.com  2006-02-09 23:56 MET
-------

I don't doubt that a mis-configured box is sitting in the middle.  But so
far I
have not found a way to work around it (or them).  I do know that older kernels
DO work...so, from my point of view, something has changed.

I did try setting the outgoing interface MTU of "linux1" to 1000. 
I've also
played around with other values less than 1500.  In all cases, the initial login
and directory listing works but the actual transfer hangs.  It actually makes
the situation "worse" than leaving the MTU at 1500.

I've also tried commands like:

iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu

and also:

iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --set-mss [various
values]

I also tried:

echo 0 > /proc/sys/net/ipv4/tcp_window_scaling



-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443





------- Additional Comments From nothingel@hotmail.com  2006-02-15 22:15 MET
-------
Just wanted to clarify that the problem is ONLY visible on recent kernels AND
when  "linux1" acts as a NAT router.

When configured as a normal router (i.e. iptables flushed and only
/proc/sys/net/ipv4/ip_forward=1 enabled), the problem does not occur.

Any further thoughts on this?  Is there anything else I can provide that might
shed some light on this issue?  Plan B is that I'll revert to an older
distro
with an older kernel....but I would hate to do that because upgrades would
become very limited.



-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443





------- Additional Comments From nothingel@hotmail.com  2006-02-20 22:48 MET
-------
I did some more digging and kernel-2.6.8 works as expected but 2.6.9 breaks.

After reading through the Changelog for 2.6.9
(http://www.kernel.org/pub/linux/kernel/v2.6/ChangeLog-2.6.9), I noticed the
option "ip_conntrack_tcp_be_liberal".

"echo 1 > /proc/sys/net/ipv4/netfilter/ip_conntrack_tcp_be_liberal"
causes the
problem to be resolved.

When set to 0 (the default), the problem is manifested.

Can anyone shed more light on this "liberal" option?  What is it's
intended use?
 Apparently it does more harm than good for me.



-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443





------- Additional Comments From kaber@trash.net  2006-02-21 02:41 MET -------
TCP window tracking was introduced in 2.6.9, the "be liberal" option
basically
disables it for all but RST packets. But this is good to know, for some reason
TCP window tracking apparently thinks your packets are invalid. You can log
the reason by doing "echo 255
>/proc/sys/net/ipv4/netfilter/ipt_LOG" after
loading the ipt_LOG module.

-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443





------- Additional Comments From kaber@trash.net  2006-02-21 02:41 MET -------
TCP window tracking was introduced in 2.6.9, the "be liberal" option
basically
disables it for all but RST packets. But this is good to know, for some reason
TCP window tracking apparently thinks your packets are invalid. You can log
the reason by doing "echo 255
>/proc/sys/net/ipv4/netfilter/ipt_LOG" after
loading the ipt_LOG module.

-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You reported the bug, or are watching the reporter.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443





------- Additional Comments From nothingel@hotmail.com  2006-02-21 04:15 MET
-------
Hmm, I don't seem to have a "/proc/sys/net/ipv4/netfilter/ipt_LOG"
but I did
find a "/proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid" and so
I set it
to 255.

Upon failing (with the "liberal" option set to 0), I see the following
in the log:

kernel: ip_ct_tcp: ACK is over the upper bound (ACKed data not seen yet).

-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443


netfilter@linuxace.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |netfilter@linuxace.com




------- Additional Comments From netfilter@linuxace.com  2006-03-05 18:19 MET
-------
There were many problems with tcp window tracking on its initial merge.  Any
kernel between 2.6.9 and 2.6.11 (or perhaps 2.6.12) has issues.  So please focus
your tests only on recent kernels.

Does the liberal setting have any impact on your results in recent kernels?


-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443


netfilter@linuxace.com changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
                 CC|                            |netfilter@linuxace.com




------- Additional Comments From netfilter@linuxace.com  2006-03-05 18:19 MET
-------
There were many problems with tcp window tracking on its initial merge.  Any
kernel between 2.6.9 and 2.6.11 (or perhaps 2.6.12) has issues.  So please focus
your tests only on recent kernels.

Does the liberal setting have any impact on your results in recent kernels?


-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You reported the bug, or are watching the reporter.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443





------- Additional Comments From nothingel@hotmail.com  2006-03-05 20:45 MET
-------
I began using older kernels in an effort to discover WHICH version had the
"problem" and that's when I discovered the "liberal"
setting.  My "real" system
is Fedora Core 4 running 2.6.15.  Without the "liberal" option (set to
"0"), the
problem I've been describing _does_ exist.

With the liberal option enabled (set to "1"), 2.6.15 becomes as
reliable as
2.6.8 (and prior).

At this point, I'm content to continue using the "liberal" option
indefinitely...but I am also willing to help diagnose this further if anyone is
interested.


-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443





------- Additional Comments From nothingel@hotmail.com  2006-03-05 20:45 MET
-------
I began using older kernels in an effort to discover WHICH version had the
"problem" and that's when I discovered the "liberal"
setting.  My "real" system
is Fedora Core 4 running 2.6.15.  Without the "liberal" option (set to
"0"), the
problem I've been describing _does_ exist.

With the liberal option enabled (set to "1"), 2.6.15 becomes as
reliable as
2.6.8 (and prior).

At this point, I'm content to continue using the "liberal" option
indefinitely...but I am also willing to help diagnose this further if anyone is
interested.


-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are on the CC list for the bug, or are watching someone who is.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443





------- Additional Comments From netfilter@linuxace.com  2006-03-05 20:55 MET
-------
Please post output (as a text attachment) from 'tcpdump -Snn' on the
linux
gateway during a problematic transfer, and append any log entries received via
setting '/proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid' to 255.

-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

https://bugzilla.netfilter.org/bugzilla/show_bug.cgi?id=443





------- Additional Comments From netfilter@linuxace.com  2006-03-05 20:55 MET
-------
Please post output (as a text attachment) from 'tcpdump -Snn' on the
linux
gateway during a problematic transfer, and append any log entries received via
setting '/proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid' to 255.

-- 
Configure bugmail:
https://bugzilla.netfilter.org/bugzilla/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You reported the bug, or are watching the reporter.