Trent O''Callaghan
2010-Feb-04 05:40 UTC
SNAT/ARP issue - shorewall-shell_4.0.15-1_all.deb
MASQ/SNAT and ARP are interacting in a way that is causing outbound connectivity issues in periods of low traffic (when ARP entries timeout). Tcpdump of ARP packets shows who-has packets with the SNAT IP address when I need them to have the Firewall''s Interface IP address as their source. I have modified MASQ to SNAT to the Firewall''s Interface IP address for the Peering network (via 198.32.212.73), but outbound traffic is normally to more distant networks and my default route is to the Paid Internet (via 121.200.226.210). I have seen some have scripted ARP watchers that could assist but I believe this is something Shorewall can cope with, but I am lacking in the knowledge. root@per-r1:/etc/shorewall# ifconfig -a eth0 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 inet addr:121.200.226.210 Bcast:121.200.226.211 Mask:255.255.255.252 eth0:1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 inet addr:198.32.212.73 Bcast:198.32.212.255 Mask:255.255.255.0 eth0:2 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 inet addr:180.233.131.7 Bcast:180.233.131.255 Mask:255.255.255.0 eth1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:91 inet addr:10.240.0.1 Bcast:10.240.0.255 Mask:255.255.255.0 root@per-r1:/etc/shorewall# ip route show table main | grep -v zebra 121.200.226.208/30 dev eth0 proto kernel scope link src 121.200.226.210 198.32.212.0/24 dev eth0 proto kernel scope link src 198.32.212.73 180.233.131.0/24 dev eth0 proto kernel scope link src 180.233.131.7 10.240.1.0/24 dev eth1 proto kernel scope link src 10.240.1.1 default via 121.200.226.209 dev eth0 metric 100 # # Shorewall version 4 - Masq file # eth0:!198.32.212.0/24 eth1:!10.240.1.7 180.233.131.7 eth0:198.32.212.0/24 10.240.1.0/24 198.32.212.73 root@per-r1:/etc/shorewall# cat shorewall.conf | grep -v \# | grep -v ^$ STARTUP_ENABLED=Yes VERBOSITY=1 SHOREWALL_COMPILERLOGFILE=/var/log/messages LOGFORMAT="Shorewall:%s:%s:" LOGTAGONLY=No LOGRATE=12/minute LOGBURST=3 LOGALLNEWBLACKLIST_LOGLEVELMACLIST_LOG_LEVEL=info TCP_FLAGS_LOG_LEVEL=info RFC1918_LOG_LEVEL=info SMURF_LOG_LEVEL=info LOG_MARTIANS=No IPTABLESPATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin SHOREWALL_SHELL=/bin/sh SUBSYSLOCK="" MODULESDIRCONFIG_PATH=/etc/shorewall:/usr/share/shorewall RESTOREFILEIPSECFILE=zones LOCKFILEDROP_DEFAULT="Drop" REJECT_DEFAULT="Reject" ACCEPT_DEFAULT="none" QUEUE_DEFAULT="none" NFQUEUE_DEFAULT="none" RSH_COMMAND=''ssh ${root}@${system} ${command}'' RCP_COMMAND=''scp ${files} ${root}@${system}:${destination}'' IP_FORWARDING=On ADD_IP_ALIASES=Yes ADD_SNAT_ALIASES=No RETAIN_ALIASES=No TC_ENABLED=No TC_EXPERT=No CLEAR_TC=No MARK_IN_FORWARD_CHAIN=No CLAMPMSS=No ROUTE_FILTER=Yes DETECT_DNAT_IPADDRS=No MUTEX_TIMEOUT=60 ADMINISABSENTMINDED=Yes BLACKLISTNEWONLY=Yes DELAYBLACKLISTLOAD=No MODULE_SUFFIXDISABLE_IPV6=Yes BRIDGING=No DYNAMIC_ZONES=No PKTTYPE=Yes RFC1918_STRICT=No MACLIST_TABLE=filter MACLIST_TTLSAVE_IPSETS=No MAPOLDACTIONS=No FASTACCEPT=Yes IMPLICIT_CONTINUE=Yes HIGH_ROUTE_MARKS=No USE_ACTIONS=Yes OPTIMIZE=0 EXPORTPARAMS=Yes EXPAND_POLICIES=Yes KEEP_RT_TABLES=No DELETE_THEN_ADD=Yes MULTICAST=No DONT_LOADBLACKLIST_DISPOSITION=DROP MACLIST_DISPOSITION=REJECT TCP_FLAGS_DISPOSITION=DROP Only masq, hosts, rules, zones, policy, interfaces, params, and shorewall.conf have non default settings. root@per-r1:/etc/shorewall# ip neighbor | grep 198.32.212 198.32.212.12 dev eth0 lladdr 00:1f:12:8b:28:29 DELAY 198.32.212.2 dev eth0 lladdr 00:24:14:2b:98:00 REACHABLE 198.32.212.37 dev eth0 lladdr 00:1b:0d:ef:7e:80 REACHABLE 198.32.212.17 dev eth0 lladdr 00:02:17:ee:ee:01 REACHABLE 198.32.212.64 dev eth0 lladdr 00:90:69:df:60:7e REACHABLE 198.32.212.15 dev eth0 lladdr 00:1a:2f:c6:9e:a0 STALE 198.32.212.22 dev eth0 lladdr 00:0d:65:ec:20:01 DELAY 198.32.212.240 dev eth0 lladdr 00:24:dc:12:ad:80 REACHABLE 198.32.212.253 dev eth0 lladdr 00:24:dc:0b:eb:80 REACHABLE 198.32.212.46 dev eth0 lladdr 00:1b:2b:4f:44:0a DELAY 198.32.212.5 dev eth0 lladdr 00:0b:60:a5:78:1b REACHABLE 198.32.212.44 dev eth0 lladdr 00:02:fc:fa:9c:19 DELAY 198.32.212.33 dev eth0 lladdr 00:13:5f:1c:8a:80 DELAY 198.32.212.52 dev eth0 lladdr 00:d0:ff:e2:58:38 STALE 198.32.212.57 dev eth0 lladdr 00:13:1a:27:a2:1b STALE 198.32.212.16 dev eth0 lladdr 00:01:63:39:ec:1b STALE 198.32.212.40 dev eth0 lladdr 00:19:e7:71:d7:a8 STALE 198.32.212.7 dev eth0 lladdr 00:0f:8f:e2:a1:01 STALE 198.32.212.72 dev eth0 lladdr 00:0f:35:a2:00:19 STALE TCPDUMP of ARP traffic: root@per-r1:/etc/shorewall# tcpdump -nei eth0 arp tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes 12:06:38.102489 00:15:17:cc:dd:90 > 00:1b:2b:4f:44:0a, ethertype ARP (0x0806), length 42: arp who-has 198.32.212.46 tell 198.32.212.73 : correct tell IP with ucast dest mac 12:06:38.103212 00:1b:2b:4f:44:0a > 00:15:17:cc:dd:90, ethertype ARP (0x0806), length 60: arp reply 198.32.212.46 is-at 00:1b:2b:4f:44:0a 12:06:38.132488 00:15:17:cc:dd:90 > 00:21:05:46:63:db, ethertype ARP (0x0806), length 42: arp who-has 121.200.226.209 tell 121.200.226.210 12:06:38.134412 00:21:05:46:63:db > 00:15:17:cc:dd:90, ethertype ARP (0x0806), length 60: arp reply 121.200.226.209 is-at 00:21:05:46:63:db 12:06:39.123738 00:15:17:cc:dd:90 > 00:1f:12:8b:28:29, ethertype ARP (0x0806), length 42: arp who-has 198.32.212.12 tell 198.32.212.73 : correct tell IP with ucast dest mac 12:06:39.125163 00:1f:12:8b:28:29 > 00:15:17:cc:dd:90, ethertype ARP (0x0806), length 60: arp reply 198.32.212.12 is-at 00:1f:12:8b:28:29 12:06:40.871240 00:15:17:cc:dd:90 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 198.32.212.41 tell 180.233.131.7 ! wrong tell IP with bcast dest mac "ff:ff:ff:ff:ff:ff" 12:06:40.984990 00:15:17:cc:dd:90 > 00:0d:65:ec:20:01, ethertype ARP (0x0806), length 42: arp who-has 198.32.212.22 tell 198.32.212.73 : correct tell IP with ucast dest mac 12:06:40.985862 00:0d:65:ec:20:01 > 00:15:17:cc:dd:90, ethertype ARP (0x0806), length 60: arp reply 198.32.212.22 is-at 00:0d:65:ec:20:01 12:06:41.871240 00:15:17:cc:dd:90 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 198.32.212.41 tell 180.233.131.7 ! wrong tell IP with bcast dest mac "ff:ff:ff:ff:ff:ff" 12:06:42.871240 00:15:17:cc:dd:90 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 198.32.212.41 tell 180.233.131.7 ! wrong tell IP with bcast dest mac "ff:ff:ff:ff:ff:ff" 12:06:43.701238 00:15:17:cc:dd:90 > 00:02:fc:fa:9c:19, ethertype ARP (0x0806), length 42: arp who-has 198.32.212.44 tell 198.32.212.73 12:06:43.702478 00:02:fc:fa:9c:19 > 00:15:17:cc:dd:90, ethertype ARP (0x0806), length 60: arp reply 198.32.212.44 is-at 00:02:fc:fa:9c:19 12:06:44.018817 00:17:0e:87:6a:c1 > 00:15:17:cc:dd:90, ethertype ARP (0x0806), length 60: arp who-has 198.32.212.73 (00:15:17:cc:dd:90) tell 198.32.212.206 12:06:44.018832 00:15:17:cc:dd:90 > 00:17:0e:87:6a:c1, ethertype ARP (0x0806), length 42: arp reply 198.32.212.73 is-at 00:15:17:cc:dd:90 ^C 15 packets captured 18 packets received by filter 0 packets dropped by kernel Since capturing this tcpdump I have improved performance with: printf "net.ipv4.neigh.eth0.gc_stale_time = 3600" >> /etc/sysctl.conf sudo /sbin/sysctl -p /etc/sysctl.conf now seeing: root@per-r1:/etc/shorewall# ip neighbor | grep 198.32.212 | wc 65 390 3578 Was 22 132 1247 While performance has improved the gc_stale_time change could have undesirable side effects so I would still like to find out if Shorewall can improve the SNAT / ARP issue at the root of this? Kind regards, Trent O''Callaghan www.nearmap.com ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Trent O''Callaghan wrote:> > While performance has improved the gc_stale_time change could have > undesirable side effects so I would still like to find out if Shorewall can > improve the SNAT / ARP issue at the root of this? >Trent, At cannot. To mangle ARP packets, you need to use the arptables utility which Shorewall does not currently provide support for. I''m actually a bit surprised that the Linux IP stack is behaving this way but then I''ve never seen a configuration quite like yours either. Sorry that I can''t be of more help, -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Tom Eastep wrote:> Trent O''Callaghan wrote: >> While performance has improved the gc_stale_time change could have >> undesirable side effects so I would still like to find out if Shorewall can >> improve the SNAT / ARP issue at the root of this? >> > Trent, > > At cannot. To mangle ARP packets, you need to use the arptables utilityHmm -- I shouldn''t answer emails before having my morning coffee. That should have been "It cannot.". -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Trent O''Callaghan wrote:> MASQ/SNAT and ARP are interacting in a way that is causing outbound > connectivity issues in periods of low traffic (when ARP entries timeout). > Tcpdump of ARP packets shows who-has packets with the SNAT IP address when I > need them to have the Firewall''s Interface IP address as their source. > > I have modified MASQ to SNAT to the Firewall''s Interface IP address for the > Peering network (via 198.32.212.73), but outbound traffic is normally to > more distant networks and my default route is to the Paid Internet (via > 121.200.226.210). > > I have seen some have scripted ARP watchers that could assist but I believe > this is something Shorewall can cope with, but I am lacking in the > knowledge. > > root@per-r1:/etc/shorewall# ifconfig -a > eth0 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 > inet addr:121.200.226.210 Bcast:121.200.226.211 > Mask:255.255.255.252 > eth0:1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 > inet addr:198.32.212.73 Bcast:198.32.212.255 Mask:255.255.255.0 > eth0:2 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 > inet addr:180.233.131.7 Bcast:180.233.131.255 Mask:255.255.255.0 > eth1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:91 > inet addr:10.240.0.1 Bcast:10.240.0.255 Mask:255.255.255.0 > > root@per-r1:/etc/shorewall# ip route show table main | grep -v zebra > 121.200.226.208/30 dev eth0 proto kernel scope link src 121.200.226.210 > 198.32.212.0/24 dev eth0 proto kernel scope link src 198.32.212.73 > 180.233.131.0/24 dev eth0 proto kernel scope link src 180.233.131.7 > 10.240.1.0/24 dev eth1 proto kernel scope link src 10.240.1.1 > default via 121.200.226.209 dev eth0 metric 100 > > # > # Shorewall version 4 - Masq file > # > eth0:!198.32.212.0/24 eth1:!10.240.1.7 180.233.131.7Ah! I took one more look at your report and I seriously doubt that the above rule does what you expect. Rewrite it as: eth0:!198.32.212.0/24 10.240.0.0/24!10.240.1.7 -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Tom Eastep wrote:> Trent O''Callaghan wrote: >> MASQ/SNAT and ARP are interacting in a way that is causing outbound >> connectivity issues in periods of low traffic (when ARP entries timeout). >> Tcpdump of ARP packets shows who-has packets with the SNAT IP address when I >> need them to have the Firewall''s Interface IP address as their source. >> >> I have modified MASQ to SNAT to the Firewall''s Interface IP address for the >> Peering network (via 198.32.212.73), but outbound traffic is normally to >> more distant networks and my default route is to the Paid Internet (via >> 121.200.226.210). >> >> I have seen some have scripted ARP watchers that could assist but I believe >> this is something Shorewall can cope with, but I am lacking in the >> knowledge. >> >> root@per-r1:/etc/shorewall# ifconfig -a >> eth0 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >> inet addr:121.200.226.210 Bcast:121.200.226.211 >> Mask:255.255.255.252 >> eth0:1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >> inet addr:198.32.212.73 Bcast:198.32.212.255 Mask:255.255.255.0 >> eth0:2 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >> inet addr:180.233.131.7 Bcast:180.233.131.255 Mask:255.255.255.0 >> eth1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:91 >> inet addr:10.240.0.1 Bcast:10.240.0.255 Mask:255.255.255.0 >> >> root@per-r1:/etc/shorewall# ip route show table main | grep -v zebra >> 121.200.226.208/30 dev eth0 proto kernel scope link src 121.200.226.210 >> 198.32.212.0/24 dev eth0 proto kernel scope link src 198.32.212.73 >> 180.233.131.0/24 dev eth0 proto kernel scope link src 180.233.131.7 >> 10.240.1.0/24 dev eth1 proto kernel scope link src 10.240.1.1 >> default via 121.200.226.209 dev eth0 metric 100 >> >> # >> # Shorewall version 4 - Masq file >> # >> eth0:!198.32.212.0/24 eth1:!10.240.1.7 180.233.131.7 > > Ah! I took one more look at your report and I seriously doubt that the > above rule does what you expect. Rewrite it as: > > eth0:!198.32.212.0/24 10.240.0.0/24!10.240.1.7In fact, the current version of Shorewall (4.4.6) rejects that type of rule: gateway:/etc/shorewall# shorewall check Compiling... WARNING: Using an interface as the masq SOURCE requires the interface to be up and configured when Shorewall starts/restarts : /etc/shorewall/masq (line 7) ERROR: SOURCE interface may not be specified with a source IP address in the POSTROUTING chain : /etc/shorewall/masq (line 7) gateway:/etc/shorewall# Are you still using Shorewall-shell? If so, I highly recommend migrating to Shorewall-perl at the first opportunity. It does a much better job of validating the configuration at compile-time (and it does it much faster as well). -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Tom Eastep wrote:> Tom Eastep wrote: >> Trent O''Callaghan wrote: >>> MASQ/SNAT and ARP are interacting in a way that is causing outbound >>> connectivity issues in periods of low traffic (when ARP entries timeout). >>> Tcpdump of ARP packets shows who-has packets with the SNAT IP address when I >>> need them to have the Firewall''s Interface IP address as their source. >>> >>> I have modified MASQ to SNAT to the Firewall''s Interface IP address for the >>> Peering network (via 198.32.212.73), but outbound traffic is normally to >>> more distant networks and my default route is to the Paid Internet (via >>> 121.200.226.210). >>> >>> I have seen some have scripted ARP watchers that could assist but I believe >>> this is something Shorewall can cope with, but I am lacking in the >>> knowledge. >>> >>> root@per-r1:/etc/shorewall# ifconfig -a >>> eth0 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:121.200.226.210 Bcast:121.200.226.211 >>> Mask:255.255.255.252 >>> eth0:1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:198.32.212.73 Bcast:198.32.212.255 Mask:255.255.255.0 >>> eth0:2 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:180.233.131.7 Bcast:180.233.131.255 Mask:255.255.255.0 >>> eth1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:91 >>> inet addr:10.240.0.1 Bcast:10.240.0.255 Mask:255.255.255.0 >>> >>> root@per-r1:/etc/shorewall# ip route show table main | grep -v zebra >>> 121.200.226.208/30 dev eth0 proto kernel scope link src 121.200.226.210 >>> 198.32.212.0/24 dev eth0 proto kernel scope link src 198.32.212.73 >>> 180.233.131.0/24 dev eth0 proto kernel scope link src 180.233.131.7 >>> 10.240.1.0/24 dev eth1 proto kernel scope link src 10.240.1.1 >>> default via 121.200.226.209 dev eth0 metric 100 >>> >>> # >>> # Shorewall version 4 - Masq file >>> # >>> eth0:!198.32.212.0/24 eth1:!10.240.1.7 180.233.131.7 >> Ah! I took one more look at your report and I seriously doubt that the >> above rule does what you expect. Rewrite it as: >> >> eth0:!198.32.212.0/24 10.240.0.0/24!10.240.1.7 > > In fact, the current version of Shorewall (4.4.6) rejects that type of rule: > > gateway:/etc/shorewall# shorewall check > Compiling... > WARNING: Using an interface as the masq SOURCE requires the interface > to be up and configured when Shorewall starts/restarts : > /etc/shorewall/masq (line 7) > ERROR: SOURCE interface may not be specified with a source IP address > in the POSTROUTING chain : /etc/shorewall/masq (line 7) > gateway:/etc/shorewall# > > Are you still using Shorewall-shell?Duh -- just looked at the Subject again. I suggest that you look at http://www.shorewall.net/LennyToSqueeze.html. Also, note that the Debian Shorewall maintainer has Shorewall 4.4 packages available for Lenny; a link to his site can be found on the Shorewall download page. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Trent O''Callaghan
2010-Feb-05 00:42 UTC
Re: SNAT/ARP issue - shorewall-shell_4.0.15-1_all.deb
Thanks Tom, I will upgrade and retest. Then I will post an update on how I go. Kind regards, Trent O''Callaghan Network Manager www.nearmap.com From: Tom Eastep [mailto:teastep@shorewall.net] Sent: Thursday, 4 February 2010 11:28 PM To: Shorewall Users Cc: trent.ocallaghan@nearmap.com Subject: Re: [Shorewall-users] SNAT/ARP issue - shorewall-shell_4.0.15-1_all.deb Tom Eastep wrote:> Tom Eastep wrote: >> Trent O''Callaghan wrote: >>> MASQ/SNAT and ARP are interacting in a way that is causing outbound >>> connectivity issues in periods of low traffic (when ARP entriestimeout).>>> Tcpdump of ARP packets shows who-has packets with the SNAT IP addresswhen I>>> need them to have the Firewall''s Interface IP address as their source. >>> >>> I have modified MASQ to SNAT to the Firewall''s Interface IP address forthe>>> Peering network (via 198.32.212.73), but outbound traffic is normally to >>> more distant networks and my default route is to the Paid Internet (via >>> 121.200.226.210). >>> >>> I have seen some have scripted ARP watchers that could assist but Ibelieve>>> this is something Shorewall can cope with, but I am lacking in the >>> knowledge. >>> >>> root@per-r1:/etc/shorewall# ifconfig -a >>> eth0 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:121.200.226.210 Bcast:121.200.226.211 >>> Mask:255.255.255.252 >>> eth0:1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:198.32.212.73 Bcast:198.32.212.255Mask:255.255.255.0>>> eth0:2 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:180.233.131.7 Bcast:180.233.131.255Mask:255.255.255.0>>> eth1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:91 >>> inet addr:10.240.0.1 Bcast:10.240.0.255 Mask:255.255.255.0 >>> >>> root@per-r1:/etc/shorewall# ip route show table main | grep -v zebra >>> 121.200.226.208/30 dev eth0 proto kernel scope link src121.200.226.210>>> 198.32.212.0/24 dev eth0 proto kernel scope link src 198.32.212.73 >>> 180.233.131.0/24 dev eth0 proto kernel scope link src 180.233.131.7 >>> 10.240.1.0/24 dev eth1 proto kernel scope link src 10.240.1.1 >>> default via 121.200.226.209 dev eth0 metric 100 >>> >>> # >>> # Shorewall version 4 - Masq file >>> # >>> eth0:!198.32.212.0/24 eth1:!10.240.1.7 180.233.131.7 >> Ah! I took one more look at your report and I seriously doubt that the >> above rule does what you expect. Rewrite it as: >> >> eth0:!198.32.212.0/24 10.240.0.0/24!10.240.1.7 > > In fact, the current version of Shorewall (4.4.6) rejects that type ofrule:> > gateway:/etc/shorewall# shorewall check > Compiling... > WARNING: Using an interface as the masq SOURCE requires the interface > to be up and configured when Shorewall starts/restarts : > /etc/shorewall/masq (line 7) > ERROR: SOURCE interface may not be specified with a source IP address > in the POSTROUTING chain : /etc/shorewall/masq (line 7) > gateway:/etc/shorewall# > > Are you still using Shorewall-shell?Duh -- just looked at the Subject again. I suggest that you look at http://www.shorewall.net/LennyToSqueeze.html. Also, note that the Debian Shorewall maintainer has Shorewall 4.4 packages available for Lenny; a link to his site can be found on the Shorewall download page. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Trent O''Callaghan
2010-Feb-09 04:03 UTC
Re: SNAT/ARP issue - shorewall-shell_4.0.15-1_all.deb
My Issue has been resolved by: echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce root@# cat /proc/sys/net/ipv4/conf/eth0/arp_announce 2 Documentation is in the 2.6 kernel docs (linux/Documentation/networking/ip-sysctl.txt) (here from the 2.6.17 kernel). arp_announce - INTEGER Define different restriction levels for announcing the local source IP address from IP packets in ARP requests sent on interface: 0 - (default) Use any local address, configured on any interface 1 - Try to avoid local addresses that are not in the target''s subnet for this interface. This mode is useful when target hosts reachable via this interface require the source IP address in ARP requests to be part of their logical network configured on the receiving interface. When we generate the request we will check all our subnets that include the target IP and will preserve the source address if it is from such subnet. If there is no such subnet we select source address according to the rules for level 2. 2 - Always use the best local address for this target. In this mode we ignore the source address in the IP packet and try to select local address that we prefer for talks with the target host. Such local address is selected by looking for primary IP addresses on all our subnets on the outgoing interface that include the target IP address. If no suitable local address is found we select the first local address we have on the outgoing interface or on all other interfaces, with the hope we will receive reply for our request and even sometimes no matter the source IP address we announce. The max value from conf/{all,interface}/arp_announce is used. Increasing the restriction level gives more chance for receiving answer from the resolved target while decreasing the level announces more valid sender''s information. Tested successfully with shorewall-perl 4.4.6-1 on Debian and shorewall-shell_4.0.15-1_all.deb 11:57:31.184988 00:15:17:cc:dd:90 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: arp who-has 198.32.212.41 tell 198.32.212.73 11:57:31.185397 00:0c:db:34:83:00 > 00:15:17:cc:dd:90, ethertype ARP (0x0806), length 60: arp reply 198.32.212.41 is-at 00:0c:db:34:83:00 Many Thanks, Trent O''Callaghan Network Manager www.nearmap.com From: Tom Eastep [mailto:teastep@shorewall.net] Sent: Thursday, 4 February 2010 11:28 PM To: Shorewall Users Cc: trent.ocallaghan@nearmap.com Subject: Re: [Shorewall-users] SNAT/ARP issue - shorewall-shell_4.0.15-1_all.deb Tom Eastep wrote:> Tom Eastep wrote: >> Trent O''Callaghan wrote: >>> MASQ/SNAT and ARP are interacting in a way that is causing outbound >>> connectivity issues in periods of low traffic (when ARP entriestimeout).>>> Tcpdump of ARP packets shows who-has packets with the SNAT IP addresswhen I>>> need them to have the Firewall''s Interface IP address as their source. >>> >>> I have modified MASQ to SNAT to the Firewall''s Interface IP address forthe>>> Peering network (via 198.32.212.73), but outbound traffic is normally to >>> more distant networks and my default route is to the Paid Internet (via >>> 121.200.226.210). >>> >>> I have seen some have scripted ARP watchers that could assist but Ibelieve>>> this is something Shorewall can cope with, but I am lacking in the >>> knowledge. >>> >>> root@per-r1:/etc/shorewall# ifconfig -a >>> eth0 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:121.200.226.210 Bcast:121.200.226.211 >>> Mask:255.255.255.252 >>> eth0:1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:198.32.212.73 Bcast:198.32.212.255Mask:255.255.255.0>>> eth0:2 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:180.233.131.7 Bcast:180.233.131.255Mask:255.255.255.0>>> eth1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:91 >>> inet addr:10.240.0.1 Bcast:10.240.0.255 Mask:255.255.255.0 >>> >>> root@per-r1:/etc/shorewall# ip route show table main | grep -v zebra >>> 121.200.226.208/30 dev eth0 proto kernel scope link src121.200.226.210>>> 198.32.212.0/24 dev eth0 proto kernel scope link src 198.32.212.73 >>> 180.233.131.0/24 dev eth0 proto kernel scope link src 180.233.131.7 >>> 10.240.1.0/24 dev eth1 proto kernel scope link src 10.240.1.1 >>> default via 121.200.226.209 dev eth0 metric 100 >>> >>> # >>> # Shorewall version 4 - Masq file >>> # >>> eth0:!198.32.212.0/24 eth1:!10.240.1.7 180.233.131.7 >> Ah! I took one more look at your report and I seriously doubt that the >> above rule does what you expect. Rewrite it as: >> >> eth0:!198.32.212.0/24 10.240.0.0/24!10.240.1.7 > > In fact, the current version of Shorewall (4.4.6) rejects that type ofrule:> > gateway:/etc/shorewall# shorewall check > Compiling... > WARNING: Using an interface as the masq SOURCE requires the interface > to be up and configured when Shorewall starts/restarts : > /etc/shorewall/masq (line 7) > ERROR: SOURCE interface may not be specified with a source IP address > in the POSTROUTING chain : /etc/shorewall/masq (line 7) > gateway:/etc/shorewall# > > Are you still using Shorewall-shell?Duh -- just looked at the Subject again. I suggest that you look at http://www.shorewall.net/LennyToSqueeze.html. Also, note that the Debian Shorewall maintainer has Shorewall 4.4 packages available for Lenny; a link to his site can be found on the Shorewall download page. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com