netfilter buglog - Nov 2009 - [Bug 620] New: Connection reset by peer

http://bugzilla.netfilter.org/show_bug.cgi?id=620

           Summary: Connection reset by peer
           Product: netfilter/iptables
           Version: unspecified
          Platform: i386
        OS/Version: All
            Status: NEW
          Severity: blocker
          Priority: P1
         Component: unknown
        AssignedTo: laforge at netfilter.org
        ReportedBy: vshiray at gmail.com


There are many such complaints can be found on Google with next search line:
  "Read error at byte" "Connection reset by peer"
There are no visible answer on it.

We have such issues periodically for long time.

My tests show that this error is happened only when NAT
rules in the linux kernel applies to TCP connection between
end points.

There are tcpdump files exist for different tests.

The error was succesful reproduced many times for next kernels:
  2.6.31.5  (original, i686)
  2.6.26.8  (original, i686)
  2.6.18-164.el5 (CentOS 5, i686)

How must we provide addidional information about tests
(network topology details, dump files, ...) ?


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #1 from kaber at trash.net  2009-11-13 13:25 -------
A packet dump (binary with -s0 please) would be a good start.


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #2 from vshiray at gmail.com  2009-11-15 14:11 -------
Created an attachment (id=307)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=307&action=view)
Test  N1, dump from H1


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #3 from vshiray at gmail.com  2009-11-15 14:13 -------
Created an attachment (id=308)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=308&action=view)
Test  N1, dump from R0


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #4 from vshiray at gmail.com  2009-11-15 14:14 -------
Created an attachment (id=309)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=309&action=view)
Test N1, dump from R1


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #5 from vshiray at gmail.com  2009-11-15 14:16 -------
Created an attachment (id=310)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=310&action=view)
Network topology


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #6 from vshiray at gmail.com  2009-11-15 14:19 -------
Test N1

Test coonection from H1 to 134.76.12.3
NAT rule was applied on R1

There are dumps exist:

R0:  tcpdump -n -i eth2 -w t1-r0.pcap -s 0 host 134.76.12.3
R1:  tcpdump -n -i eth1 -w t1-r1.pcap -s 0 host 134.76.12.3
H1:  tcpdump -n -i eth0 -w t1-h1.pcap -s 0 host 134.76.12.3

PS. See details in "Network topology".


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #7 from vshiray at gmail.com  2009-11-15 14:35 -------
Created an attachment (id=311)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=311&action=view)
Test N2, dump from H1


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #8 from vshiray at gmail.com  2009-11-15 14:35 -------
Created an attachment (id=312)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=312&action=view)
Test N2, dump from R0


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #9 from vshiray at gmail.com  2009-11-15 14:36 -------
Created an attachment (id=313)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=313&action=view)
Test N2, dump from R1


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #10 from vshiray at gmail.com  2009-11-15 14:38 -------
Test N2 (it's the same as N1)

Test coonection from H1 to 134.76.12.3
NAT rule was applied on R1

There are dumps exist:

R0:  tcpdump -n -i eth2 -w t2-r0.pcap -s 0 host 134.76.12.3
R1:  tcpdump -n -i eth1 -w t2-r1.pcap -s 0 host 134.76.12.3
H1:  tcpdump -n -i eth0 -w t2-h1.pcap -s 0 host 134.76.12.3


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #11 from vshiray at gmail.com  2009-11-15 14:42 -------
Created an attachment (id=314)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=314&action=view)
Test N3, dump from H0


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #12 from vshiray at gmail.com  2009-11-15 14:42 -------
Created an attachment (id=315)
 --> (http://bugzilla.netfilter.org/attachment.cgi?id=315&action=view)
Test N3, dump from R0


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #13 from vshiray at gmail.com  2009-11-15 14:45 -------
Test N3

Test connection from H0 to 134.76.12.3
NAT rule was applied on H0 itself.

There are dumps exist:

R0:  tcpdump -n -i eth2 -w t3-r0.pcap -s 0 host 134.76.12.3
H0:  tcpdump -n -i eth0 -w t3-h0.pcap -s 0 host 134.76.12.3


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #14 from vshiray at gmail.com  2009-11-15 14:53 -------
I produced also many test connection from H0 to 134.76.12.3
but without NAT was being applied.

All of them finished succesfully.

I have a dumps from the good connection also.
But them are bit large (5.5Mb).


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #15 from kaber at trash.net  2009-11-24 14:20 -------
I couldn't spot anything in these dumps so far. Does adding this rule:

iptables -t mangle -A PREROUTING -m state --state INVALID -j DROP

on the box doing NAT make any difference?


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #16 from vshiray at gmail.com  2009-11-24 14:47 -------
As I see from quick tests adding this rule solve the problem.

Without this rule I got two errors during several minuites.
After I add it the same tests was finished OK.

Must I do other tests ?


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620


kaber at trash.net changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |INVALID




------- Comment #17 from kaber at trash.net  2009-11-24 14:52 -------
No, it sounds like we found the cause.

When packets from a NATed connection are recognized as invalid (which might
happen for multiple reasons, f.i. retransmissions for which an ACK already
passed through the firewall) they are not associated with the conntrack,
meaning they'll have no NAT applied and are delivered to the unNATed
destination, which doesn't know about the connection and resets it.


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #18 from vshiray at gmail.com  2009-11-24 16:11 -------

As I see from dumps the first RST packets was sent by NAT router itself.
So it assumes that connection must be reseted despite the NAT connection
exists already. 

Workaround is working but do you think that it's a normal behaviour?


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #19 from kaber at trash.net  2009-11-24 16:14 -------
Yes, thats expected. When the packet is not associated to the existing
connection, its not NATed and delivered locally, therefore the NAT machine
generates an RST.


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #20 from vshiray at gmail.com  2009-11-24 16:29 -------
In such case this rule must be default for most installs which use NAT.
But I never found any such recomendation before.

How can we make it known to many other peoples which have the same issue?


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.

http://bugzilla.netfilter.org/show_bug.cgi?id=620





------- Comment #21 from kaber at trash.net  2009-11-24 16:36 -------
(In reply to comment #20)
> In such case this rule must be default for most installs which use NAT.
> But I never found any such recomendation before.
Well, its mentioned in quite a few guides.
> How can we make it known to many other peoples which have the same issue?
You could send a patch to clarify this in the manpage. But please take this to
netfilter-devel at vger.kernel.org.


-- 
Configure bugmail: http://bugzilla.netfilter.org/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug, or are watching the assignee.