Dear list. I have a working Multiple ISP configuration running on a debian etch box with shorewall version 3.2.6-2 (I''ll upgrade soon, I promise!) My two internet uplinks (eth4 and ppp0) belongs to the same "net" zone. Everything is working fine but I have a problem with natting. Behind the firewall I have some services I want to be accessible from outside, eg the SMTP server, which is listening on port 25/tcp on an internal server. Interfaces are like this: net eth4 detect dhcp,blacklist,tcpflags net ppp0 detect dhcp,blacklist,tcpflags lan1 eth1 detect arp_filter lan2 eth2 detect arp_filter road tun+ The problematic rule is this: DNAT net lan1:<internal IP of my mail server> tcp 25 If I try to nmap the port on the first public IP (which is routed to ppp0) from an external server I get # nmap -p 25 <public IP #1> Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2012-04-11 09:40 CEST Interesting ports on <public IP #1> : PORT STATE SERVICE 25/tcp filtered smtp Nmap finished: 1 IP address (1 host up) scanned in 6.887 seconds On the firewall I can see (with tcpdump) packets coming thru ppp0 # tcpdump -i ppp0 dst port 25 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on ppp0, link-type LINUX_SLL (Linux cooked), capture size 96 bytes 09:40:42.913240 IP <sourceIP>.62519 > <public IP #1>.smtp: S 2099963655:2099963655(0) win 3072 <mss 1452> 09:40:43.016305 IP <sourceIP> .62520 > <public IP #1> .smtp: S 2100029190:2100029190(0) win 1024 <mss 1452> If I try the same with the other public IP (which is routed to eth4) I get # nmap -p 25 <public IP #2> Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2012-04-11 09:48 CEST Interesting ports on <public IP #2> : PORT STATE SERVICE 25/tcp open smtp Nmap finished: 1 IP address (1 host up) scanned in 6.873 seconds I really don''t understand where is the fault. It doesn''t seem to be a routing problem so I''m asking your support. Please ask if you need additional elements to diagnose. Thanks. Alessandro ------------------------------------------------------------------------------ Better than sec? Nothing is better than sec when it comes to monitoring Big Data applications. Try Boundary one-second resolution app monitoring today. Free. http://p.sf.net/sfu/Boundary-dev2dev
On 04/11/2012 12:57 AM, Alessandro Faglia wrote:> My two internet uplinks (eth4 and ppp0) belongs to the same "net" zone. > Everything is working fine but I have a problem with natting. > > The problematic rule is this: > DNAT net lan1:<internal IP of my mail server> tcp 25 >Have you followed the DNAT troubleshooting procedure described in Shorewall FAQs 1a and 1b? -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Better than sec? Nothing is better than sec when it comes to monitoring Big Data applications. Try Boundary one-second resolution app monitoring today. Free. http://p.sf.net/sfu/Boundary-dev2dev
Alessandro Faglia
2012-Apr-11 15:17 UTC
Re: Problem with nat on a multiple isp configuration
On Wed, Apr 11, 2012 at 4:38 PM, Tom Eastep <teastep@shorewall.net> wrote:> On 04/11/2012 12:57 AM, Alessandro Faglia wrote: > > > My two internet uplinks (eth4 and ppp0) belongs to the same "net" zone. > > Everything is working fine but I have a problem with natting. > > > > The problematic rule is this: > > DNAT net lan1:<internal IP of my mail server> tcp 25 > > > > Have you followed the DNAT troubleshooting procedure described in > Shorewall FAQs 1a and 1b? >I did :-( I created a LOG rule to track 25/tcp packets and in the syslog I see Apr 11 17:06:56 <sw-box> kernel: Shorewall:net2lan1:LOG:IN=ppp0 OUT=eth1 SRC=<src-ip> DST=192.168.1.9 LEN=44 TOS=0x00 PREC=0x00 TTL=42 I D=36272 PROTO=TCP SPT=47814 DPT=25 WINDOW=2048 RES=0x00 SYN URGP=0 Apr 11 17:06:56 <sw-box> kernel: Shorewall:net2lan1:LOG:IN=ppp0 OUT=eth1 SRC=<src-ip> DST=192.168.1.9 LEN=44 TOS=0x00 PREC=0x00 TTL=39 I D=51095 PROTO=TCP SPT=47815 DPT=25 WINDOW=3072 RES=0x00 SYN URGP=0 Here <sw-box> is the hostname of the box where shorewall is running (local IP is 192.168.1.1) and <src-ip> is the public IP of the other box I''m running nmap to test. In the target box the gateway is poiting to the shorewall box: # route -n Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 bond0 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 bond0 Maybe the bond interface on the target server is involved in the issue? but in this case it won''t work even when scanning the other IP, at least I think so... I don''t have a clue... Alessandro ------------------------------------------------------------------------------ Better than sec? Nothing is better than sec when it comes to monitoring Big Data applications. Try Boundary one-second resolution app monitoring today. Free. http://p.sf.net/sfu/Boundary-dev2dev
On 04/11/2012 08:17 AM, Alessandro Faglia wrote:> On Wed, Apr 11, 2012 at 4:38 PM, Tom Eastep <teastep@shorewall.net > <mailto:teastep@shorewall.net>> wrote: > > On 04/11/2012 12:57 AM, Alessandro Faglia wrote: > > > My two internet uplinks (eth4 and ppp0) belongs to the same "net" > zone. > > Everything is working fine but I have a problem with natting. > > > > The problematic rule is this: > > DNAT net lan1:<internal IP of my mail server> tcp 25 > > > > Have you followed the DNAT troubleshooting procedure described in > Shorewall FAQs 1a and 1b? > > > I did :-( > > I created a LOG rule to track 25/tcp packets and in the syslog I see > Apr 11 17:06:56 <sw-box> kernel: Shorewall:net2lan1:LOG:IN=ppp0 OUT=eth1 > SRC=<src-ip> DST=192.168.1.9 LEN=44 TOS=0x00 PREC=0x00 TTL=42 I > D=36272 PROTO=TCP SPT=47814 DPT=25 WINDOW=2048 RES=0x00 SYN URGP=0 > Apr 11 17:06:56 <sw-box> kernel: Shorewall:net2lan1:LOG:IN=ppp0 > OUT=eth1 SRC=<src-ip> DST=192.168.1.9 LEN=44 TOS=0x00 PREC=0x00 TTL=39 I > D=51095 PROTO=TCP SPT=47815 DPT=25 WINDOW=3072 RES=0x00 SYN URGP=0 > > Here <sw-box> is the hostname of the box where shorewall is running > (local IP is 192.168.1.1) and <src-ip> is the public IP of the other box > I''m running nmap to test. > > In the target box the gateway is poiting to the shorewall box: > # route -n > Kernel IP routing table > Destination Gateway Genmask Flags Metric Ref Use > Iface > 192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 > bond0 > 0.0.0.0 192.168.1.1 0.0.0.0 UG 0 0 0 > bond0 > > Maybe the bond interface on the target server is involved in the issue? > but in this case it won''t work even when scanning the other IP, at least > I think so... > > > I don''t have a clue...Have you looked at eth1 with tcpdump when doing this test? If you use the -e option (e.g., tcpdump -nei eth1 port 25 and host <nmap-host-ip>) you can see if the mail server is responding and with what destination MAC. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Better than sec? Nothing is better than sec when it comes to monitoring Big Data applications. Try Boundary one-second resolution app monitoring today. Free. http://p.sf.net/sfu/Boundary-dev2dev
On 04/11/2012 08:17 AM, Alessandro Faglia wrote:> I don''t have a clue...One thing I will point out is that we don''t recommend running a multiple ISP configuration on any version of Shorewall earlier than 4.2.0 and then only when using the Perl-based rules compiler (the only compiler included in Shorewall 4.4.0 and later). There were many problems with the old code and those problems were going to be very expensive to fix. That''s the reason that I made the decision to stop developing the shell-based compiler and write the Perl-based one. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Better than sec? Nothing is better than sec when it comes to monitoring Big Data applications. Try Boundary one-second resolution app monitoring today. Free. http://p.sf.net/sfu/Boundary-dev2dev
Alessandro Faglia
2012-Apr-12 06:20 UTC
Re: Problem with nat on a multiple isp configuration
On Wed, Apr 11, 2012 at 5:35 PM, Tom Eastep <teastep@shorewall.net> wrote:> > Have you looked at eth1 with tcpdump when doing this test? If you use > the -e option (e.g., tcpdump -nei eth1 port 25 and host <nmap-host-ip>) > you can see if the mail server is responding and with what destination MAC. >On the shorewall box: # ip l 1: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: eth0: <BROADCAST,MULTICAST> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:18:8b:e7:34:90 brd ff:ff:ff:ff:ff:ff 3: eth3: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 100 link/ether 00:04:23:e0:8a:50 brd ff:ff:ff:ff:ff:ff 4: eth4: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 100 link/ether 00:04:23:e0:8a:51 brd ff:ff:ff:ff:ff:ff 5: eth1: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether *00:04:23:e0:8c:20* brd ff:ff:ff:ff:ff:ff 6: eth2: <BROADCAST,MULTICAST,UP,10000> mtu 1500 qdisc pfifo_fast qlen 1000 link/ether 00:04:23:e0:8c:21 brd ff:ff:ff:ff:ff:ff 7: sit0: <NOARP> mtu 1480 qdisc noop link/sit 0.0.0.0 brd 0.0.0.0 44: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,10000> mtu 1500 qdisc pfifo_fast qlen 100 link/[65534] 45: ppp0: <POINTOPOINT,MULTICAST,NOARP,UP,10000> mtu 1492 qdisc pfifo_fast qlen 3 link/ppp On the mail box: # ip l 1: eth0: <BROADCAST,MULTICAST,SLAVE,UP,10000> mtu 1500 qdisc pfifo_fast master bond0 qlen 1000 link/ether *00:15:c5:60:86:b3* brd ff:ff:ff:ff:ff:ff 2: eth1: <BROADCAST,MULTICAST,SLAVE,UP,10000> mtu 1500 qdisc pfifo_fast master bond0 qlen 1000 link/ether *00:10:18:1c:57:b7* brd ff:ff:ff:ff:ff:ff 3: eth2: <BROADCAST,MULTICAST> mtu 1500 qdisc noop qlen 1000 link/ether 00:15:c5:60:86:b4 brd ff:ff:ff:ff:ff:ff 4: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 5: bond0: <BROADCAST,MULTICAST,MASTER,UP,10000> mtu 1500 qdisc noqueue link/ether 00:10:18:1c:57:b7 brd ff:ff:ff:ff:ff:ff When I run the test from the nmap-host-ip I see on shorewall box # tcpdump -nei eth1 port 25 and host <nmap-host-ip> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes 08:01:11.411377 *00:04:23:e0:8c:20* > *00:15:c5:60:86:b3*, ethertype IPv4 (0x0800), length 58: <nmap-host-ip>.53627 > 192.168.1.9.25: S 641556343:641556343(0) win 2048 <mss 1452> 08:01:11.411468 *00:10:18:1c:57:b7* > *00:04:23:e0:8c:20*, ethertype IPv4 (0x0800), length 60: 192.168.1.9.25 > <nmap-host-ip>.53627: S 2281575337:2281575337(0) ack 641556344 win 5840 <mss 1460> 08:01:11.512664 *00:04:23:e0:8c:20* > *00:15:c5:60:86:b3*, ethertype IPv4 (0x0800), length 58: <nmap-host-ip>.53628 > 192.168.1.9.25: S 641490806:641490806(0) win 3072 <mss 1452> 08:01:11.512782 *00:10:18:1c:57:b7* > *00:04:23:e0:8c:20*, ethertype IPv4 (0x0800), length 60: 192.168.1.9.25 > <nmap-host-ip>.53628: S 2381413036:2381413036(0) ack 641490807 win 5840 <mss 1460> 08:01:15.723446 *00:10:18:1c:57:b7* > *00:04:23:e0:8c:20*, ethertype IPv4 (0x0800), length 60: 192.168.1.9.25 > <nmap-host-ip>.53627: S 2281575337:2281575337(0) ack 641556344 win 5840 <mss 1460> 08:01:15.923454 *00:10:18:1c:57:b7* > *00:04:23:e0:8c:20*, ethertype IPv4 (0x0800), length 60: 192.168.1.9.25 > <nmap-host-ip> .53628: S 2381413036:2381413036(0) ack 641490807 win 5840 <mss 1460> [...] For what I understand I shouldn''t have any output from tcpdump, or is it normal? Do you see routing issues? About the Shorewall version earlier than 4.2.0, I already planned an upgrade for the future, but I was wondering if this problem comes from a configuration mistake or what else. It''s always a challenge to fix such kind of issues. Thanks Tom for your precious advises and support, and for making Shorewall a valuable tool. Alessandro ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2
Alessandro Faglia
2012-Apr-12 13:14 UTC
Re: Problem with nat on a multiple isp configuration
On Thu, Apr 12, 2012 at 8:20 AM, Alessandro Faglia < alessandro.faglia@serioplast.com> wrote:> On Wed, Apr 11, 2012 at 5:35 PM, Tom Eastep <teastep@shorewall.net> wrote: > >> >> Have you looked at eth1 with tcpdump when doing this test? If you use >> the -e option (e.g., tcpdump -nei eth1 port 25 and host <nmap-host-ip>) >> you can see if the mail server is responding and with what destination >> MAC. >> > > [...] > >Moreover: this is the providers file on the shorewall box <isp1> 1 1 main eth4 192.168.0.1 track,balance,optional eth1,eth2,tun* <isp2> 2 2 main ppp0 detect track,balance,optional eth1,eth2 eth4 is connected to a Cisco router which is in turn connected to an optical fiber uplink (the public IP is bound to the WAN NIC of this device). The LAN IP of this box is 192.168.0.1 and eth4 is 192.168.0.2. The cable from the PPP antenna is connected to eth3 and the pppd daemon is managing the connection. The tun* you see is used by OpenVPN which is running on the same server. If I understand correctly the track option should "magically" route back packets via the incoming NIC (in this case ppp0), is it correct? I''m just trying to better understand how the system is dealing such packets. Thanks. Alessandro ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2
On 04/11/2012 11:20 PM, Alessandro Faglia wrote:> > For what I understand I shouldn''t have any output from tcpdump, or is it > normal? Do you see routing issues? >That looks okay. Now try running tcpdump on eth4 while you are testing; do you see response packets being sent out of eth4 rather than ppp0? -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2
Alessandro Faglia
2012-Apr-12 14:10 UTC
Re: Problem with nat on a multiple isp configuration
On Thu, Apr 12, 2012 at 3:19 PM, Tom Eastep <teastep@shorewall.net> wrote:> On 04/11/2012 11:20 PM, Alessandro Faglia wrote: > > > > > For what I understand I shouldn''t have any output from tcpdump, or is it > > normal? Do you see routing issues? > > > > That looks okay. Now try running tcpdump on eth4 while you are testing; > do you see response packets being sent out of eth4 rather than ppp0? >Yes I do: # tcpdump -nei eth4 port 25 and host <nmap-host-ip> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth4, link-type EN10MB (Ethernet), capture size 96 bytes 16:05:53.308093 00:04:23:e0:8a:51 > 00:25:9c:ca:6d:46, ethertype IPv4 (0x0800), length 58: <wan-ip>.25 > <nmap-host-ip> .36640: S 283332995:283332995(0) ack 2424569839 win 5840 <mss 1460> 16:05:53.406159 00:04:23:e0:8a:51 > 00:25:9c:ca:6d:46, ethertype IPv4 (0x0800), length 58: <wan-ip> .25 > <nmap-host-ip> .36641: S 382851284:382851284(0) ack 2424504304 win 5840 <mss 1460> 16:05:57.032048 00:04:23:e0:8a:51 > 00:25:9c:ca:6d:46, ethertype IPv4 (0x0800), length 58: <wan-ip> .25 > <nmap-host-ip> .36640: S 283332995:283332995(0) ack 2424569839 win 5840 <mss 1460> 16:05:57.831952 00:04:23:e0:8a:51 > 00:25:9c:ca:6d:46, ethertype IPv4 (0x0800), length 58: <wan-ip> .25 > <nmap-host-ip> .36641: S 382851284:382851284(0) ack 2424504304 win 5840 <mss 1460> In this case <wan-ip> is the public IP (#1 in my previous examples) I''m running nmap against from the test host: # nmap -p 25 <wan-ip> Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2012-04-12 16:05 CEST Interesting ports on <wan-ip> : PORT STATE SERVICE 25/tcp filtered smtp Nmap finished: 1 IP address (1 host up) scanned in 6.890 seconds So I have packets flowing back thru eth4 that shouldn''t be there, am I correct? Is it a setup problem? Thanks. Alessandro ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2
On 04/12/2012 07:10 AM, Alessandro Faglia wrote:> On Thu, Apr 12, 2012 at 3:19 PM, Tom Eastep <teastep@shorewall.net > <mailto:teastep@shorewall.net>> wrote: > > On 04/11/2012 11:20 PM, Alessandro Faglia wrote: > > > > > For what I understand I shouldn''t have any output from tcpdump, or > is it > > normal? Do you see routing issues? > > > > That looks okay. Now try running tcpdump on eth4 while you are testing; > do you see response packets being sent out of eth4 rather than ppp0? > > > Yes I do: > > # tcpdump -nei eth4 port 25 and host <nmap-host-ip> > tcpdump: verbose output suppressed, use -v or -vv for full protocol decode > listening on eth4, link-type EN10MB (Ethernet), capture size 96 bytes > 16:05:53.308093 00:04:23:e0:8a:51 > 00:25:9c:ca:6d:46, ethertype IPv4 > (0x0800), length 58: <wan-ip>.25 > <nmap-host-ip> .36640: S > 283332995:283332995(0) ack 2424569839 win 5840 <mss 1460> > 16:05:53.406159 00:04:23:e0:8a:51 > 00:25:9c:ca:6d:46, ethertype IPv4 > (0x0800), length 58: <wan-ip> .25 > <nmap-host-ip> .36641: S > 382851284:382851284(0) ack 2424504304 win 5840 <mss 1460> > 16:05:57.032048 00:04:23:e0:8a:51 > 00:25:9c:ca:6d:46, ethertype IPv4 > (0x0800), length 58: <wan-ip> .25 > <nmap-host-ip> .36640: S > 283332995:283332995(0) ack 2424569839 win 5840 <mss 1460> > 16:05:57.831952 00:04:23:e0:8a:51 > 00:25:9c:ca:6d:46, ethertype IPv4 > (0x0800), length 58: <wan-ip> .25 > <nmap-host-ip> .36641: S > 382851284:382851284(0) ack 2424504304 win 5840 <mss 1460> > > In this case <wan-ip>is the public IP (#1 in my previous examples) I''m > running nmap against from the test host: > # nmap -p 25 <wan-ip> > > Starting Nmap 4.11 ( http://www.insecure.org/nmap/ ) at 2012-04-12 16:05 > CEST > Interesting ports on <wan-ip> : > PORT STATE SERVICE > 25/tcp filtered smtp > > Nmap finished: 1 IP address (1 host up) scanned in 6.890 seconds > > So I have packets flowing back thru eth4 that shouldn''t be there, am I > correct? Is it a setup problem?Most likely it is a bug in the ancient version of Shorewall you are running. You can try: - shorewall stop - /etc/init.d/networking restart - shorewall start and see if that fixes it. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2
Alessandro Faglia
2012-Apr-12 14:52 UTC
Re: Problem with nat on a multiple isp configuration
On Thu, Apr 12, 2012 at 4:15 PM, Tom Eastep <teastep@shorewall.net> wrote:> > Most likely it is a bug in the ancient version of Shorewall you are > running. You can try: > > - shorewall stop > - /etc/init.d/networking restart > - shorewall start > > and see if that fixes it. >OK I''ll try as soon as I can and I''ll let the list to know. Thanks for your patience. Alessandro ------------------------------------------------------------------------------ For Developers, A Lot Can Happen In A Second. Boundary is the first to Know...and Tell You. Monitor Your Applications in Ultra-Fine Resolution. Try it FREE! http://p.sf.net/sfu/Boundary-d2dvs2