Dr Josef Karthauser
2016-Apr-07 16:08 UTC
IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3
I?m scratching my head with an IPFW / NAT configuration; could someone please throw me a bone? I?ve got a jail, and I?m NATing using IPFW to connect it to the outside world. In particular I?m forwarding port 8080 from the host?s public address to the jail?s private address. When I pull an HTTP connection from port publicip:8080 I get the first packet of the TCP stream twice, and then the HTTP connection fails. That ought not to happen :(. The firewall rule is very simple nat 1 config if vlan10 reset redirect_port tcp 10.17.0.16:8080 8080 // NAT for jails - forward to portal on 8080 nat 1 ip from any to any via vlan10 in nat 1 ip from any to any via vlan10 out add allow ip from any to any If I tcpdump on the host: # tcpdump -i vlan10 port 8080 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on vlan10, link-type EN10MB (Ethernet), capture size 65535 bytes 17:02:02.478760 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [S], seq 3088565770, win 65535, options [mss 1460,nop,wscale 5,nop,nop,TS val 672977930 ecr 0,sackOK,eol], length 0 17:02:02.478797 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [S.], seq 425576427, ack 3088565771, win 65535, options [mss 1460,nop,wscale 6,sackOK,TS val 1035319863 ecr 672977930], length 0 17:02:02.480137 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 1, win 4117, options [nop,nop,TS val 672977931 ecr 1035319863], length 0 17:02:02.480393 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [P.], seq 1:86, ack 1, win 4117, options [nop,nop,TS val 672977931 ecr 1035319863], length 85 17:02:02.714225 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [P.], seq 1:86, ack 1, win 4117, options [nop,nop,TS val 672978161 ecr 1035319863], length 85 17:02:02.975220 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [P.], seq 1:86, ack 1, win 4117, options [nop,nop,TS val 672978421 ecr 1035319863], length 85 17:02:02.975239 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 1:1449, ack 86, win 1040, options [nop,nop,TS val 1035320360 ecr 672977931], length 1448 17:02:03.079324 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 1449, win 4096, options [nop,nop,TS val 672978522 ecr 1035320360], length 0 17:02:03.079336 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 1449:4345, ack 86, win 1040, options [nop,nop,TS val 1035320464 ecr 672978522], length 2896 17:02:03.080931 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 4345, win 4050, options [nop,nop,TS val 672978523 ecr 1035320464], length 0 17:02:03.578732 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 4345:5793, ack 86, win 1040, options [nop,nop,TS val 1035320963 ecr 672978523], length 1448 17:02:03.725858 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 5793, win 4096, options [nop,nop,TS val 672979158 ecr 1035320963], length 0 17:02:03.725888 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 5793:8689, ack 86, win 1040, options [nop,nop,TS val 1035321110 ecr 672979158], length 2896 17:02:03.727352 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 8689, win 4050, options [nop,nop,TS val 672979159 ecr 1035321110], length 0 17:02:04.260416 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 8689:10137, ack 86, win 1040, options [nop,nop,TS val 1035321645 ecr 672979159], length 1448 17:02:04.340844 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [.], ack 10137, win 4096, options [nop,nop,TS val 672979770 ecr 1035321645], length 0 17:02:04.340855 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 10137:13033, ack 86, win 1040, options [nop,nop,TS val 1035321725 ecr 672979770], length 2896 17:02:04.342775 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [F.], seq 86, ack 11585, win 4096, options [nop,nop,TS val 672979771 ecr 1035321725], length 0 17:02:04.342803 IP X.X.X.216.8080 > X.X.X.211.63289: Flags [.], seq 13033:15929, ack 87, win 1040, options [nop,nop,TS val 1035321727 ecr 672979771], length 2896 17:02:04.343154 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [R], seq 3088565856, win 0, length 0 17:02:04.344440 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [R], seq 3088565857, win 0, length 0 17:02:04.344740 IP X.X.X.211.63289 > X.X.X.216.8080: Flags [R], seq 3088565857, win 0, length 0 And the client doing the http request gets: phoenix:~ joe$ curl -v http://X.X.X.216:8080/ * Trying 31.210.26.216... * Connected to X.X.X.216 port 8080 (#0)> GET / HTTP/1.1 > Host: x.x.com:8080 > User-Agent: curl/7.43.0 > Accept: */* >< HTTP/1.1 200 OK < Server: Apache-Coyote/1.1 < Content-Type: text/html;charset=ISO-8859-1 < Transfer-Encoding: chunked < Date: Thu, 07 Apr 2016 16:02:02 GMT < <!DOCTYPE html> <html lang="en"> <head> <title>Apache Tomcat/7.0.68</title> <link href="favicon.ico" rel="icon" type="image/x-icon" /> <link href="favicon.ico" rel="shortcut icon" type="image/x-icon" /> <link href="tomcat.css" rel="stylesheet" type="text/css" /> </head> <body> <div id="wrapper"> <div id="navigation" class="curved container"> <span id="nav-home"><a href="http://tomcat.apache.org/">Home</a></span> <span id="nav-hosts"><a href="/docs/">Documentation</a></span> <span id="nav-config"><a href="/docs/config/">Configuration</a></span> <span id="nav-examples"><a href="/examples/">Examples</a></span> <span id="nav-wiki"><a href="http://wiki.apache.org/tomcat/FrontPage">Wiki</a></span> [CUT] <div class="col20"> <div class="container"> <h4>Other Documentation</h4> <ul> <li><a href="http://tomcat.apache.org/connectors-doc/">Tomcat Connectors</a></li> <li><a href="http://tomcat.apache.org/connectors-doc/">mod_jk Documentation</a></li> HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Content-Type: text/html;charset=ISO-8859-1 Transfer-Encoding: chunked Date: Thu, 07 Apr 2016 16:02:02 GMT 2000 <!DOCTYPE html> <html lang="en"> <head> <title>Apache Tomcat/7.0.68</title> <link href="favicon.ico" rel="icon" type="image/x-icon" /> <link href="favicon.ico" rel="shortcut icon" type="image/x-icon" /> <link href="tomcat.css" rel="stylesheet" type="text/css" /> </head> <body> <div id="wrapper"> <div id="navigation" class="curved container"> [CUT] </div> </div> <div id="actions"> <div class="button"> <a class="container shadow" href="/manager/status"><span>Server Status</span></a> * Malformed encoding found in chunked-encoding * Closing connection 0 curl: (56) Malformed encoding found in chunked-encoding phoenix:~ joe$ Looks like the first packet is being retransmitted, which means that the nat is probably misconfigured and the TCP connection is broken in some strange way. Does anyone have a clue as to where to look? The ipfw rules are simple enough - what have I missed? Thanks, Joe p.s. I also have one_pass disabled: # sysctl net.inet.ip.fw.one_pass net.inet.ip.fw.one_pass: 0 ? Dr Josef Karthauser Chief Technical Officer (01225) 300371 / (07703) 596893 www.truespeed.com <http://www.truespeed.com/> / theTRUESPEED <http://www.facebook.com/theTRUESPEED> @theTRUESPEED <https://twitter.com/thetruespeed> This email contains TrueSpeed information, which may be privileged or confidential. It's meant only for the individual(s) or entity named above. If you're not the intended recipient, note that disclosing, copying, distributing or using this information is prohibited. If you've received this email in error, please let me know immediately on the email address above. Thank you. We monitor our email system, and may record your emails.
Dr Josef Karthauser
2016-Apr-07 23:11 UTC
IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3
> On 7 Apr 2016, at 17:08, Dr Josef Karthauser <joe at truespeed.com> wrote: > > Looks like the first packet is being retransmitted, which means that the nat is probably misconfigured and the TCP connection is broken in some strange way. > > Does anyone have a clue as to where to look? The ipfw rules are simple enough - what have I missed?Ok, the packet definitely isn?t being retransmitted. I?ve done a tcpdump/pcap capture and taken a look and I get a packet that I?ve included below. It?s got a 'HTTP/1.1 200 OK? inserted mid-flow right in the middle of an HTTP response. Looking at this I?d be inclined to think it?s a bug in the webserver/tomcat, however, what?s strange is that if I ?curl' the jailed web server directly from the host machine on the private IP address (bypassing the NAT), the HTTP response received is perfectly fine. It?s only when I do an HTTP request to the public IP address and go through the NAT that I experience the problem. How could this happen? Is it a buggy packet reassembly in the kernel perhaps? Joe p.s here?s the strange packet with an HTTP response injected in the middle of a HTML stream: 23:01:07.204016 IP (tos 0x0, ttl 64, id 4190, offset 0, flags [DF], proto TCP (6), length 1500) 31.210.26.216.8080 > infiniverse.karthauser.co.uk.62475: Flags [.], cksum 0xda1c (incorrect -> 0x7ff7), seq 8689:10137, ack 86, win 1040, options [nop,nop,TS val 124159447 ecr 1737359970], length 1448 .........g.)............. .f..g..b <h4>Other Documentation</h4> <ul> <li><a href="http://tomcat.apache.org/connectors-doc/">Tomcat Connectors</a></li> <li><a href="http://tomcat.apache.org/connectors-doc/">mod_jk Documentation</a></li> HTTP/1.1 200 OK Server: Apache-Coyote/1.1 Content-Type: text/html;charset=ISO-8859-1 Transfer-Encoding: chunked Date: Thu, 07 Apr 2016 23:01:05 GMT 2000 <!DOCTYPE html> <html lang="en"> <head> <title>Apache Tomcat/7.0.68</title> <link href="favicon.ico" rel="icon" type="image/x-icon" /> <link href="favicon.ico" rel="shortcut icon" type="image/x-icon" /> <link href="tomcat.css" rel="stylesheet" type="text/css" /> </head> <body> <div id="wrapper"> <div id="navigation" class="curved container"> <span id="nav-home"><a href="http://tomcat.apache.org/">Home</a></span> <span id="nav-hosts"><a href="/docs/">Documentation</a></span> <span id="nav-config"><a href="/docs/config/">Configuration</a></span> <span id="nav-examples"><a href="/examples/">Examples</a></span> <span id="nav-wiki"><a href="http://wiki.apache.org/tomcat/FrontPage">Wiki</a></span> <span id="nav-lists"><a href="http://tomcat.apache.org/lists.html">Mailing Lists</a></span> <s
Ian Smith
2016-Apr-08 05:51 UTC
IPFW with NAT : Problems with duplicate packets on FreeBSD 10.3-RC3
On Thu, 7 Apr 2016 17:08:38 +0100, Dr Josef Karthauser wrote: [ AppleMail msgs fail to quote properly in pine, so a partial quote: ] > Looks like the first packet is being retransmitted, which means that > the nat is probably misconfigured and the TCP connection is broken in > some strange way. > Does anyone have a clue as to where to look? The ipfw rules are > simple enough - what have I missed? Do you have TSO enabled on that NIC? If so, see ipfw(8) BUGS, third last para. If not, no idea .. cheers, Ian