Jesse Patterson
2005-Apr-22 14:28 UTC
I have a problem similar to FAQ 2 scenario, but reply packets don''t seem to be recognized.
Hello,
I am running Shorewall 2.0.2f, on SuSE 9.2 distro, kernel
2.6.8-24.11-default
My ip addr show output is as follows:
1: lo: <LOOPBACK,UP> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:00:f4:c3:9d:ac brd ff:ff:ff:ff:ff:ff
inet 192.168.0.100/16 brd 192.168.255.255 scope global eth0
inet6 fe80::200:f4ff:fec3:9dac/64 scope link
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0f:b5:42:7f:aa brd ff:ff:ff:ff:ff:ff
inet 65.40.217.150/24 brd 65.40.217.255 scope global eth1
inet6 fe80::20f:b5ff:fe42:7faa/64 scope link
valid_lft forever preferred_lft forever
4: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0f:b5:45:4a:1e brd ff:ff:ff:ff:ff:ff
5: eth3: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:30:bd:b5:24:5f brd ff:ff:ff:ff:ff:ff
inet 71.16.82.82/28 brd 71.16.82.95 scope global eth3
inet 71.16.82.83/28 brd 71.16.82.95 scope global secondary eth3:V83
inet 71.16.82.84/28 brd 71.16.82.95 scope global secondary eth3:V84
inet 71.16.82.85/28 brd 71.16.82.95 scope global secondary eth3:V85
inet 71.16.82.86/28 brd 71.16.82.95 scope global secondary eth3:V86
inet 71.16.82.87/28 brd 71.16.82.95 scope global secondary eth3:V87
inet 71.16.82.88/28 brd 71.16.82.95 scope global secondary eth3:V88
inet 71.16.82.89/28 brd 71.16.82.95 scope global secondary eth3:V89
inet 71.16.82.90/28 brd 71.16.82.95 scope global secondary eth3:V90
inet 71.16.82.91/28 brd 71.16.82.95 scope global secondary eth3:V91
inet 71.16.82.92/28 brd 71.16.82.95 scope global secondary eth3:V92
inet 71.16.82.93/28 brd 71.16.82.95 scope global secondary eth3:V93
inet 71.16.82.94/28 brd 71.16.82.95 scope global secondary eth3:V94
inet6 fe80::230:bdff:feb5:245f/64 scope link
valid_lft forever preferred_lft forever
6: sit0: <NOARP> mtu 1480 qdisc noqueue
link/sit 0.0.0.0 brd 0.0.0.0
My ip route show output is as follows:
71.16.82.80/28 dev eth3 scope link src 71.16.82.93
65.40.217.0/24 dev eth1 scope link src 65.40.217.150
169.254.0.0/16 dev eth0 scope link
192.168.0.0/16 dev eth0 proto kernel scope link src 192.168.0.100
127.0.0.0/8 dev lo scope link
default via 65.40.217.129 dev eth1
======================================================
The problem I am experiencing is that I have a web server on my internal
network and external clients are being DNATed to that server with no issues.
Internal clients trying to get to this same server via the external IP are
unable to do so.
The FAQ entry " <http://www.shorewall.net/FAQ.htm#faq2> (FAQ 2) I
port
forward www requests to www.mydomain.com (IP 130.151.100.69) to system
192.168.1.5 in my local network. External clients can browse
http://www.mydomain.com but internal clients can''t." seemed to be
exactly
our situation and I followed the instructions as described.
This is being tested with a text-based browser "links", and it was
executing
an https query, with the normal "443" dst port.
Still having what appeared to be a failure, I started TCPDUMPs to see what
traffic was going where.
I could see the traffic from the internal client machine destined for the
external IP, in this case from 192.168.0.150 to 65.40.217.150.
I saw the traffic arrive on the Shorewall machine (192.168.0.100 internal
interface) and saw it rerouted to the internal web server. so now the
traffic looks like 192.168.0.100 to 192.168.0.99.
I saw the traffic arrive on the web server, and I saw the response go back
to the Shorewall machine. 192.168.0.99 to 192.168.0.100.
I then saw what looked to be the packet get re-addressed so that it appeared
as 65.40.217.150 to 192.168.0.150 -- this to me looks like it is doing
exactly what it is supposed to be doing...
I see this packet arrive back on the 192.168.0.150 machine. To me, this
looks like a successful round-trip. BUT, the response packet does not seem
to be acknowledged as such. The browser just keeps trying to establish a
connection.
Now from an external client, I see the traffic coming in on 65.40.217.150,
it gets routed to 192.168.0.99, the response goes back to 192.168.0.100,
then out the public IP of 65.40.217.150 and back to my external client with
no issues -- this also was using the text-based "links" browser, so
that I
could verify it wasn''t just a browser issue.
Attached is a short TCPDUMP from the internal client (192.168.0.150). The
dump would appear to show the packets coming back just fine -- I just
don''t
understand what could be different in the return packet that would somehow
result in it seemingly be unrecognized as a proper response.
TCPDUMP results:
dale:~ # tcpdump -eni eth0 port 443 -vvvv -xx
tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96
bytes
09:07:03.606045 00:11:43:fd:4d:ca > 00:00:f4:c3:9d:ac, ethertype IPv4
(0x0800), length 74: IP (tos 0x0, ttl 64, id 46520, offset 0, flags [DF],
length: 60) 192.168.0.150.35238 > 65.40.217.150.443: S [tcp sum ok]
353484362:353484362(0) win 5840 <mss 1460,sackOK,timestamp 3624012786
0,nop,wscale 2>
0x0000: 0000 f4c3 9dac 0011 43fd 4dca 0800 4500 ........C.M...E.
0x0010: 003c b5b8 4000 4006 a906 c0a8 0096 4128 .<..@.@.......A(
0x0020: d996 89a6 01bb 1511 be4a 0000 0000 a002 .........J......
0x0030: 16d0 1286 0000 0204 05b4 0402 080a d802 ................
0x0040: 0bf2 0000 0000 0103 0302 ..........
09:07:03.607781 00:0f:b5:42:7f:aa > 00:02:3b:02:78:06, ethertype IPv4
(0x0800), length 78: IP (tos 0x0, ttl 127, id 59957, offset 0, flags [DF],
length: 64) 65.40.217.150.443 > 192.168.0.150.35238: S [tcp sum ok]
2760737647:2760737647(0) ack 353484363 win 17520 <mss 1460,nop,wscale
0,nop,nop,timestamp 0 0,nop,nop,sackOK>
0x0000: 0002 3b02 7806 000f b542 7faa 0800 4500 ..;.x....B....E.
0x0010: 0040 ea35 4000 7f06 3585 4128 d996 c0a8 .@.5@...5.A(....
0x0020: 0096 01bb 89a6 a48d 836f 1511 be4b b012 .........o...K..
0x0030: 4470 8ec8 0000 0204 05b4 0103 0300 0101 Dp..............
0x0040: 080a 0000 0000 0000 0000 0101 0402 ..............
09:07:06.601794 00:0f:b5:42:7f:aa > 00:02:3b:02:78:06, ethertype IPv4
(0x0800), length 78: IP (tos 0x0, ttl 127, id 59960, offset 0, flags [DF],
length: 64) 65.40.217.150.443 > 192.168.0.150.35238: S [tcp sum ok]
2760737647:2760737647(0) ack 353484363 win 17520 <mss 1460,nop,wscale
0,nop,nop,timestamp 0 0,nop,nop,sackOK>
0x0000: 0002 3b02 7806 000f b542 7faa 0800 4500 ..;.x....B....E.
0x0010: 0040 ea38 4000 7f06 3582 4128 d996 c0a8 .@.8@...5.A(....
0x0020: 0096 01bb 89a6 a48d 836f 1511 be4b b012 .........o...K..
0x0030: 4470 8ec8 0000 0204 05b4 0103 0300 0101 Dp..............
0x0040: 080a 0000 0000 0000 0000 0101 0402 ..............
09:07:06.604662 00:11:43:fd:4d:ca > 00:00:f4:c3:9d:ac, ethertype IPv4
(0x0800), length 74: IP (tos 0x0, ttl 64, id 46521, offset 0, flags [DF],
length: 60) 192.168.0.150.35238 > 65.40.217.150.443: S [tcp sum ok]
353484362:353484362(0) win 5840 <mss 1460,sackOK,timestamp 3624015786
0,nop,wscale 2>
0x0000: 0000 f4c3 9dac 0011 43fd 4dca 0800 4500 ........C.M...E.
0x0010: 003c b5b9 4000 4006 a905 c0a8 0096 4128 .<..@.@.......A(
0x0020: d996 89a6 01bb 1511 be4a 0000 0000 a002 .........J......
0x0030: 16d0 06ce 0000 0204 05b4 0402 080a d802 ................
0x0040: 17aa 0000 0000 0103 0302 ..........
09:07:12.604614 00:11:43:fd:4d:ca > 00:00:f4:c3:9d:ac, ethertype IPv4
(0x0800), length 74: IP (tos 0x0, ttl 64, id 46522, offset 0, flags [DF],
length: 60) 192.168.0.150.35238 > 65.40.217.150.443: S [tcp sum ok]
353484362:353484362(0) win 5840 <mss 1460,sackOK,timestamp 3624021786
0,nop,wscale 2>
0x0000: 0000 f4c3 9dac 0011 43fd 4dca 0800 4500 ........C.M...E.
0x0010: 003c b5ba 4000 4006 a904 c0a8 0096 4128 .<..@.@.......A(
0x0020: d996 89a6 01bb 1511 be4a 0000 0000 a002 .........J......
0x0030: 16d0 ef5d 0000 0204 05b4 0402 080a d802 ...]............
0x0040: 2f1a 0000 0000 0103 0302 /.........
09:07:12.609806 00:0f:b5:42:7f:aa > 00:02:3b:02:78:06, ethertype IPv4
(0x0800), length 78: IP (tos 0x0, ttl 127, id 60103, offset 0, flags [DF],
length: 64) 65.40.217.150.443 > 192.168.0.150.35238: S [tcp sum ok]
2760737647:2760737647(0) ack 353484363 win 17520 <mss 1460,nop,wscale
0,nop,nop,timestamp 0 0,nop,nop,sackOK>
0x0000: 0002 3b02 7806 000f b542 7faa 0800 4500 ..;.x....B....E.
0x0010: 0040 eac7 4000 7f06 34f3 4128 d996 c0a8 .@..@...4.A(....
0x0020: 0096 01bb 89a6 a48d 836f 1511 be4b b012 .........o...K..
0x0030: 4470 8ec8 0000 0204 05b4 0103 0300 0101 Dp..............
0x0040: 080a 0000 0000 0000 0000 0101 0402 ..............
6 packets captured
6 packets received by filter
0 packets dropped by kernel
dale:~ #
===================================================
I''ve also tried this on -other- internal client machines with the same
result - this isn''t a behavior unique to the one box. It is almost as
if
the packet is going in and out the same interface on the Shorewall machine
(eth0), something gets mangled. If the packet goes in and out different
interfaces (as with external originated connections), there is no issue.
I hope I''ve made this help request clear enough and didn''t
commit any
serious faux pas in the process of asking. My apologies in advance if I
have. I _did_ try to resolve this first via the documentation, guides, and
FAQ.
Oh, to avoid the obvious question: Yes, I can connect directly from
192.168.0.150 to the internal web server at 192.168.0.99 without a problem.
It is only when I try to connect to it by browsing to the external IP that
this fails.
Thanks in advance for any advice you are able to provide.
-Jesse
Tom Eastep
2005-Apr-22 15:02 UTC
Re: I have a problem similar to FAQ 2 scenario, but reply packets don''t seem to be recognized.
Jesse Patterson wrote:> > dale:~ # tcpdump -eni eth0 port 443 -vvvv -xx > tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 > bytes > 09:07:03.606045 00:11:43:fd:4d:ca > 00:00:f4:c3:9d:ac, ethertype IPv4 > (0x0800), length 74: IP (tos 0x0, ttl 64, id 46520, offset 0, flags [DF], > length: 60) 192.168.0.150.35238 > 65.40.217.150.443: S [tcp sum ok] > 353484362:353484362(0) win 5840 <mss 1460,sackOK,timestamp 3624012786 > 0,nop,wscale 2> > 0x0000: 0000 f4c3 9dac 0011 43fd 4dca 0800 4500 ........C.M...E. > 0x0010: 003c b5b8 4000 4006 a906 c0a8 0096 4128 .<..@.@.......A( > 0x0020: d996 89a6 01bb 1511 be4a 0000 0000 a002 .........J...... > 0x0030: 16d0 1286 0000 0204 05b4 0402 080a d802 ................ > 0x0040: 0bf2 0000 0000 0103 0302 .......... > 09:07:03.607781 00:0f:b5:42:7f:aa > 00:02:3b:02:78:06, ethertype IPv4 > (0x0800), length 78: IP (tos 0x0, ttl 127, id 59957, offset 0, flags [DF], > length: 64) 65.40.217.150.443 > 192.168.0.150.35238: S [tcp sum ok] > 2760737647:2760737647(0) ack 353484363 win 17520 <mss 1460,nop,wscale > 0,nop,nop,timestamp 0 0,nop,nop,sackOK>The MAC addresses in the above sequence are very puzzling. The SYN goes from 00:11:43:fd:4d:ca > 00:00:f4:c3:9d:ac while the SYN,ACK goes from 00:0f:b5:42:7f:aa > 00:02:3b:02:78:06. ??? Contrast with a working example (using port 22 rather than port 443 since I don''t raise foxes in my henhouse): 07:59:57.160527 00:02:e3:08:48:4c > 00:a0:c9:15:39:78, ethertype IPv4 (0x0800), length 74: IP 192.168.1.5.1726 > 206.124.146.179.22: S 2510303663:2510303663(0) win 5840 <mss 1460,sackOK,timestamp 1295516337 0,nop,wscale 2> 07:59:57.161197 00:a0:c9:15:39:78 > 00:02:e3:08:48:4c, ethertype IPv4 (0x0800), length 74: IP 206.124.146.179.22 > 192.168.1.5.1726: S 2508207209:2508207209(0) ack 2510303664 win 5792 <mss 1460,sackOK,timestamp 1295516338 1295516337,nop,wscale 2> 07:59:57.161218 00:02:e3:08:48:4c > 00:a0:c9:15:39:78, ethertype IPv4 (0x0800), length 66: IP 192.168.1.5.1726 > 206.124.146.179.22: . ack 1 win 1460 <nop,nop,timestamp 1295516338 1295516338> 0 -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
Tom Eastep
2005-Apr-22 16:35 UTC
Re: I have a problem similar to FAQ 2 scenario, but reply packets don''t seem to be recognized.
Tom Eastep wrote:> Jesse Patterson wrote: > >> >>dale:~ # tcpdump -eni eth0 port 443 -vvvv -xx >>tcpdump: listening on eth0, link-type EN10MB (Ethernet), capture size 96 >>bytes >>09:07:03.606045 00:11:43:fd:4d:ca > 00:00:f4:c3:9d:ac, ethertype IPv4 >>(0x0800), length 74: IP (tos 0x0, ttl 64, id 46520, offset 0, flags [DF], >>length: 60) 192.168.0.150.35238 > 65.40.217.150.443: S [tcp sum ok] >>353484362:353484362(0) win 5840 <mss 1460,sackOK,timestamp 3624012786 >>0,nop,wscale 2> >> 0x0000: 0000 f4c3 9dac 0011 43fd 4dca 0800 4500 ........C.M...E. >> 0x0010: 003c b5b8 4000 4006 a906 c0a8 0096 4128 .<..@.@.......A( >> 0x0020: d996 89a6 01bb 1511 be4a 0000 0000 a002 .........J...... >> 0x0030: 16d0 1286 0000 0204 05b4 0402 080a d802 ................ >> 0x0040: 0bf2 0000 0000 0103 0302 .......... >>09:07:03.607781 00:0f:b5:42:7f:aa > 00:02:3b:02:78:06, ethertype IPv4 >>(0x0800), length 78: IP (tos 0x0, ttl 127, id 59957, offset 0, flags [DF], >>length: 64) 65.40.217.150.443 > 192.168.0.150.35238: S [tcp sum ok] >>2760737647:2760737647(0) ack 353484363 win 17520 <mss 1460,nop,wscale >>0,nop,nop,timestamp 0 0,nop,nop,sackOK> > > The MAC addresses in the above sequence are very puzzling. The SYN goes > from 00:11:43:fd:4d:ca > 00:00:f4:c3:9d:ac while the SYN,ACK goes from > 00:0f:b5:42:7f:aa > 00:02:3b:02:78:06. >In fact, the reply seems to be coming from eth1 and being sent to an oddball ethernet address (00:02:3b:02:78:06) which doesn''t appear to be the MAC of 192.168.0.150 (which the above trace shows as 00:11:43:fd:4d:ca). 3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:0f:b5:42:7f:aa brd ff:ff:ff:ff:ff:ff ----------------- inet 65.40.217.150/24 brd 65.40.217.255 scope global eth1 inet6 fe80::20f:b5ff:fe42:7faa/64 scope link valid_lft forever preferred_lft forever What does "arp -na" on the firewall box show? -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
Jesse Patterson
2005-Apr-22 17:06 UTC
RE: I have a problem similar to FAQ 2 scenario, but reply packets don''t seem to be recognized.
Your pointing out the oddball MAC address was helpful indeed -- That MAC address is actually the address of our eth1 uplink gateway: Arp -na on the Shorewall box gives this: chief2:~ # arp -na ? (192.168.0.7) at 00:11:11:CE:89:DA [ether] on eth0 ? (192.168.0.5) at 00:0D:56:0C:69:D3 [ether] on eth0 ? (192.168.0.178) at 00:C0:9F:30:2A:E4 [ether] on eth0 ? (192.168.0.4) at 00:0F:1F:4C:C2:1E [ether] on eth0 ? (192.168.0.3) at 00:07:E9:7B:5B:9F [ether] on eth0 ? (192.168.0.99) at 00:06:5B:14:62:C4 [ether] on eth0 ? (192.168.0.10) at 00:11:11:EA:76:E5 [ether] on eth0 ? (192.168.0.21) at 00:11:43:FD:7D:F2 [ether] on eth0 ? (192.168.0.120) at 00:11:43:FD:4D:1F [ether] on eth0 ? (192.168.0.133) at 00:11:43:FD:4D:1F [ether] on eth0 ? (192.168.0.131) at 00:11:43:FD:4D:1F [ether] on eth0 ? (65.40.217.129) at 00:02:3B:02:78:06 [ether] on eth1 ? (192.168.0.125) at 00:11:43:FD:4D:1F [ether] on eth0 ? (192.168.0.90) at 00:40:CA:59:08:E2 [ether] on eth0 ? (192.168.0.126) at 00:11:43:FD:4D:1F [ether] on eth0 ? (192.168.0.200) at 00:C0:9F:1D:70:78 [ether] on eth0 ? (192.168.0.127) at 00:11:43:FD:4D:1F [ether] on eth0 ? (71.16.82.81) at 00:A0:C8:0F:53:70 [ether] on eth3 ? (192.168.0.61) at 00:C0:9F:30:2A:E4 [ether] on eth0 ? (192.168.0.136) at 00:11:43:FD:4D:1F [ether] on eth0 ? (192.168.0.137) at 00:11:43:FD:4D:1F [ether] on eth0 Where you can see the odd address is 65.40.217.129 - our uplink gateway to sprint. I''m looking into the routing tables again to see if there is something going on that is pushing the packets the wrong way. I very much appreciate your help and sharp eye on spotting that inconsistency, Tom. -Jesse -----Original Message----- From: shorewall-users-bounces@lists.shorewall.net [mailto:shorewall-users-bounces@lists.shorewall.net] On Behalf Of Tom Eastep Sent: Friday, April 22, 2005 12:36 PM To: Mailing List for Shorewall Users Subject: Re: [Shorewall-users] I have a problem similar to FAQ 2 scenario,but reply packets don''t seem to be recognized. Tom Eastep wrote:> The MAC addresses in the above sequence are very puzzling. The SYN > goes from 00:11:43:fd:4d:ca > 00:00:f4:c3:9d:ac while the SYN,ACK goes > from 00:0f:b5:42:7f:aa > 00:02:3b:02:78:06.In fact, the reply seems to be coming from eth1 and being sent to an oddball ethernet address (00:02:3b:02:78:06) which doesn''t appear to be the MAC of 192.168.0.150 (which the above trace shows as 00:11:43:fd:4d:ca). 3: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:0f:b5:42:7f:aa brd ff:ff:ff:ff:ff:ff ----------------- inet 65.40.217.150/24 brd 65.40.217.255 scope global eth1 inet6 fe80::20f:b5ff:fe42:7faa/64 scope link valid_lft forever preferred_lft forever What does "arp -na" on the firewall box show?
Tom Eastep
2005-Apr-23 15:03 UTC
Re: I have a problem similar to FAQ 2 scenario, but reply packets don''t seem to be recognized.
Jesse Patterson wrote:> > I''m looking into the routing tables again to see if there is something going > on that is pushing the packets the wrong way. >You also need to understand why packets being sent on eth1 are visible to tcpdump on eth0! Are these two interfaces connected to the same hub/switch? You probably don''t want that... -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key