Trent O''Callaghan
2010-Feb-04 05:40 UTC
SNAT/ARP issue - shorewall-shell_4.0.15-1_all.deb
MASQ/SNAT and ARP are interacting in a way that is causing outbound
connectivity issues in periods of low traffic (when ARP entries timeout).
Tcpdump of ARP packets shows who-has packets with the SNAT IP address when I
need them to have the Firewall''s Interface IP address as their source.
I have modified MASQ to SNAT to the Firewall''s Interface IP address for
the
Peering network (via 198.32.212.73), but outbound traffic is normally to
more distant networks and my default route is to the Paid Internet (via
121.200.226.210).
I have seen some have scripted ARP watchers that could assist but I believe
this is something Shorewall can cope with, but I am lacking in the
knowledge.
root@per-r1:/etc/shorewall# ifconfig -a
eth0 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90
inet addr:121.200.226.210 Bcast:121.200.226.211
Mask:255.255.255.252
eth0:1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90
inet addr:198.32.212.73 Bcast:198.32.212.255 Mask:255.255.255.0
eth0:2 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90
inet addr:180.233.131.7 Bcast:180.233.131.255 Mask:255.255.255.0
eth1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:91
inet addr:10.240.0.1 Bcast:10.240.0.255 Mask:255.255.255.0
root@per-r1:/etc/shorewall# ip route show table main | grep -v zebra
121.200.226.208/30 dev eth0 proto kernel scope link src 121.200.226.210
198.32.212.0/24 dev eth0 proto kernel scope link src 198.32.212.73
180.233.131.0/24 dev eth0 proto kernel scope link src 180.233.131.7
10.240.1.0/24 dev eth1 proto kernel scope link src 10.240.1.1
default via 121.200.226.209 dev eth0 metric 100
#
# Shorewall version 4 - Masq file
#
eth0:!198.32.212.0/24 eth1:!10.240.1.7 180.233.131.7
eth0:198.32.212.0/24 10.240.1.0/24 198.32.212.73
root@per-r1:/etc/shorewall# cat shorewall.conf | grep -v \# | grep -v ^$
STARTUP_ENABLED=Yes
VERBOSITY=1
SHOREWALL_COMPILERLOGFILE=/var/log/messages
LOGFORMAT="Shorewall:%s:%s:"
LOGTAGONLY=No
LOGRATE=12/minute
LOGBURST=3
LOGALLNEWBLACKLIST_LOGLEVELMACLIST_LOG_LEVEL=info
TCP_FLAGS_LOG_LEVEL=info
RFC1918_LOG_LEVEL=info
SMURF_LOG_LEVEL=info
LOG_MARTIANS=No
IPTABLESPATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin
SHOREWALL_SHELL=/bin/sh
SUBSYSLOCK=""
MODULESDIRCONFIG_PATH=/etc/shorewall:/usr/share/shorewall
RESTOREFILEIPSECFILE=zones
LOCKFILEDROP_DEFAULT="Drop"
REJECT_DEFAULT="Reject"
ACCEPT_DEFAULT="none"
QUEUE_DEFAULT="none"
NFQUEUE_DEFAULT="none"
RSH_COMMAND=''ssh ${root}@${system} ${command}''
RCP_COMMAND=''scp ${files} ${root}@${system}:${destination}''
IP_FORWARDING=On
ADD_IP_ALIASES=Yes
ADD_SNAT_ALIASES=No
RETAIN_ALIASES=No
TC_ENABLED=No
TC_EXPERT=No
CLEAR_TC=No
MARK_IN_FORWARD_CHAIN=No
CLAMPMSS=No
ROUTE_FILTER=Yes
DETECT_DNAT_IPADDRS=No
MUTEX_TIMEOUT=60
ADMINISABSENTMINDED=Yes
BLACKLISTNEWONLY=Yes
DELAYBLACKLISTLOAD=No
MODULE_SUFFIXDISABLE_IPV6=Yes
BRIDGING=No
DYNAMIC_ZONES=No
PKTTYPE=Yes
RFC1918_STRICT=No
MACLIST_TABLE=filter
MACLIST_TTLSAVE_IPSETS=No
MAPOLDACTIONS=No
FASTACCEPT=Yes
IMPLICIT_CONTINUE=Yes
HIGH_ROUTE_MARKS=No
USE_ACTIONS=Yes
OPTIMIZE=0
EXPORTPARAMS=Yes
EXPAND_POLICIES=Yes
KEEP_RT_TABLES=No
DELETE_THEN_ADD=Yes
MULTICAST=No
DONT_LOADBLACKLIST_DISPOSITION=DROP
MACLIST_DISPOSITION=REJECT
TCP_FLAGS_DISPOSITION=DROP
Only masq, hosts, rules, zones, policy, interfaces, params, and
shorewall.conf have non default settings.
root@per-r1:/etc/shorewall# ip neighbor | grep 198.32.212
198.32.212.12 dev eth0 lladdr 00:1f:12:8b:28:29 DELAY
198.32.212.2 dev eth0 lladdr 00:24:14:2b:98:00 REACHABLE
198.32.212.37 dev eth0 lladdr 00:1b:0d:ef:7e:80 REACHABLE
198.32.212.17 dev eth0 lladdr 00:02:17:ee:ee:01 REACHABLE
198.32.212.64 dev eth0 lladdr 00:90:69:df:60:7e REACHABLE
198.32.212.15 dev eth0 lladdr 00:1a:2f:c6:9e:a0 STALE
198.32.212.22 dev eth0 lladdr 00:0d:65:ec:20:01 DELAY
198.32.212.240 dev eth0 lladdr 00:24:dc:12:ad:80 REACHABLE
198.32.212.253 dev eth0 lladdr 00:24:dc:0b:eb:80 REACHABLE
198.32.212.46 dev eth0 lladdr 00:1b:2b:4f:44:0a DELAY
198.32.212.5 dev eth0 lladdr 00:0b:60:a5:78:1b REACHABLE
198.32.212.44 dev eth0 lladdr 00:02:fc:fa:9c:19 DELAY
198.32.212.33 dev eth0 lladdr 00:13:5f:1c:8a:80 DELAY
198.32.212.52 dev eth0 lladdr 00:d0:ff:e2:58:38 STALE
198.32.212.57 dev eth0 lladdr 00:13:1a:27:a2:1b STALE
198.32.212.16 dev eth0 lladdr 00:01:63:39:ec:1b STALE
198.32.212.40 dev eth0 lladdr 00:19:e7:71:d7:a8 STALE
198.32.212.7 dev eth0 lladdr 00:0f:8f:e2:a1:01 STALE
198.32.212.72 dev eth0 lladdr 00:0f:35:a2:00:19 STALE
TCPDUMP of ARP traffic:
root@per-r1:/etc/shorewall# tcpdump -nei eth0 arp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 96 bytes
12:06:38.102489 00:15:17:cc:dd:90 > 00:1b:2b:4f:44:0a, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.46 tell 198.32.212.73
: correct tell IP with ucast dest mac
12:06:38.103212 00:1b:2b:4f:44:0a > 00:15:17:cc:dd:90, ethertype ARP
(0x0806), length 60: arp reply 198.32.212.46 is-at 00:1b:2b:4f:44:0a
12:06:38.132488 00:15:17:cc:dd:90 > 00:21:05:46:63:db, ethertype ARP
(0x0806), length 42: arp who-has 121.200.226.209 tell 121.200.226.210
12:06:38.134412 00:21:05:46:63:db > 00:15:17:cc:dd:90, ethertype ARP
(0x0806), length 60: arp reply 121.200.226.209 is-at 00:21:05:46:63:db
12:06:39.123738 00:15:17:cc:dd:90 > 00:1f:12:8b:28:29, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.12 tell 198.32.212.73
: correct tell IP with ucast dest mac
12:06:39.125163 00:1f:12:8b:28:29 > 00:15:17:cc:dd:90, ethertype ARP
(0x0806), length 60: arp reply 198.32.212.12 is-at 00:1f:12:8b:28:29
12:06:40.871240 00:15:17:cc:dd:90 > ff:ff:ff:ff:ff:ff, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.41 tell 180.233.131.7
! wrong tell IP with bcast dest mac "ff:ff:ff:ff:ff:ff"
12:06:40.984990 00:15:17:cc:dd:90 > 00:0d:65:ec:20:01, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.22 tell 198.32.212.73
: correct tell IP with ucast dest mac
12:06:40.985862 00:0d:65:ec:20:01 > 00:15:17:cc:dd:90, ethertype ARP
(0x0806), length 60: arp reply 198.32.212.22 is-at 00:0d:65:ec:20:01
12:06:41.871240 00:15:17:cc:dd:90 > ff:ff:ff:ff:ff:ff, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.41 tell 180.233.131.7
! wrong tell IP with bcast dest mac "ff:ff:ff:ff:ff:ff"
12:06:42.871240 00:15:17:cc:dd:90 > ff:ff:ff:ff:ff:ff, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.41 tell 180.233.131.7
! wrong tell IP with bcast dest mac "ff:ff:ff:ff:ff:ff"
12:06:43.701238 00:15:17:cc:dd:90 > 00:02:fc:fa:9c:19, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.44 tell 198.32.212.73
12:06:43.702478 00:02:fc:fa:9c:19 > 00:15:17:cc:dd:90, ethertype ARP
(0x0806), length 60: arp reply 198.32.212.44 is-at 00:02:fc:fa:9c:19
12:06:44.018817 00:17:0e:87:6a:c1 > 00:15:17:cc:dd:90, ethertype ARP
(0x0806), length 60: arp who-has 198.32.212.73 (00:15:17:cc:dd:90) tell
198.32.212.206
12:06:44.018832 00:15:17:cc:dd:90 > 00:17:0e:87:6a:c1, ethertype ARP
(0x0806), length 42: arp reply 198.32.212.73 is-at 00:15:17:cc:dd:90
^C
15 packets captured
18 packets received by filter
0 packets dropped by kernel
Since capturing this tcpdump I have improved performance with:
printf "net.ipv4.neigh.eth0.gc_stale_time = 3600" >>
/etc/sysctl.conf
sudo /sbin/sysctl -p /etc/sysctl.conf
now seeing:
root@per-r1:/etc/shorewall# ip neighbor | grep 198.32.212 | wc
65 390 3578
Was 22 132 1247
While performance has improved the gc_stale_time change could have
undesirable side effects so I would still like to find out if Shorewall can
improve the SNAT / ARP issue at the root of this?
Kind regards,
Trent O''Callaghan
www.nearmap.com
------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com
Trent O''Callaghan wrote:> > While performance has improved the gc_stale_time change could have > undesirable side effects so I would still like to find out if Shorewall can > improve the SNAT / ARP issue at the root of this? >Trent, At cannot. To mangle ARP packets, you need to use the arptables utility which Shorewall does not currently provide support for. I''m actually a bit surprised that the Linux IP stack is behaving this way but then I''ve never seen a configuration quite like yours either. Sorry that I can''t be of more help, -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Tom Eastep wrote:> Trent O''Callaghan wrote: >> While performance has improved the gc_stale_time change could have >> undesirable side effects so I would still like to find out if Shorewall can >> improve the SNAT / ARP issue at the root of this? >> > Trent, > > At cannot. To mangle ARP packets, you need to use the arptables utilityHmm -- I shouldn''t answer emails before having my morning coffee. That should have been "It cannot.". -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Trent O''Callaghan wrote:> MASQ/SNAT and ARP are interacting in a way that is causing outbound > connectivity issues in periods of low traffic (when ARP entries timeout). > Tcpdump of ARP packets shows who-has packets with the SNAT IP address when I > need them to have the Firewall''s Interface IP address as their source. > > I have modified MASQ to SNAT to the Firewall''s Interface IP address for the > Peering network (via 198.32.212.73), but outbound traffic is normally to > more distant networks and my default route is to the Paid Internet (via > 121.200.226.210). > > I have seen some have scripted ARP watchers that could assist but I believe > this is something Shorewall can cope with, but I am lacking in the > knowledge. > > root@per-r1:/etc/shorewall# ifconfig -a > eth0 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 > inet addr:121.200.226.210 Bcast:121.200.226.211 > Mask:255.255.255.252 > eth0:1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 > inet addr:198.32.212.73 Bcast:198.32.212.255 Mask:255.255.255.0 > eth0:2 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 > inet addr:180.233.131.7 Bcast:180.233.131.255 Mask:255.255.255.0 > eth1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:91 > inet addr:10.240.0.1 Bcast:10.240.0.255 Mask:255.255.255.0 > > root@per-r1:/etc/shorewall# ip route show table main | grep -v zebra > 121.200.226.208/30 dev eth0 proto kernel scope link src 121.200.226.210 > 198.32.212.0/24 dev eth0 proto kernel scope link src 198.32.212.73 > 180.233.131.0/24 dev eth0 proto kernel scope link src 180.233.131.7 > 10.240.1.0/24 dev eth1 proto kernel scope link src 10.240.1.1 > default via 121.200.226.209 dev eth0 metric 100 > > # > # Shorewall version 4 - Masq file > # > eth0:!198.32.212.0/24 eth1:!10.240.1.7 180.233.131.7Ah! I took one more look at your report and I seriously doubt that the above rule does what you expect. Rewrite it as: eth0:!198.32.212.0/24 10.240.0.0/24!10.240.1.7 -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Tom Eastep wrote:> Trent O''Callaghan wrote: >> MASQ/SNAT and ARP are interacting in a way that is causing outbound >> connectivity issues in periods of low traffic (when ARP entries timeout). >> Tcpdump of ARP packets shows who-has packets with the SNAT IP address when I >> need them to have the Firewall''s Interface IP address as their source. >> >> I have modified MASQ to SNAT to the Firewall''s Interface IP address for the >> Peering network (via 198.32.212.73), but outbound traffic is normally to >> more distant networks and my default route is to the Paid Internet (via >> 121.200.226.210). >> >> I have seen some have scripted ARP watchers that could assist but I believe >> this is something Shorewall can cope with, but I am lacking in the >> knowledge. >> >> root@per-r1:/etc/shorewall# ifconfig -a >> eth0 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >> inet addr:121.200.226.210 Bcast:121.200.226.211 >> Mask:255.255.255.252 >> eth0:1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >> inet addr:198.32.212.73 Bcast:198.32.212.255 Mask:255.255.255.0 >> eth0:2 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >> inet addr:180.233.131.7 Bcast:180.233.131.255 Mask:255.255.255.0 >> eth1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:91 >> inet addr:10.240.0.1 Bcast:10.240.0.255 Mask:255.255.255.0 >> >> root@per-r1:/etc/shorewall# ip route show table main | grep -v zebra >> 121.200.226.208/30 dev eth0 proto kernel scope link src 121.200.226.210 >> 198.32.212.0/24 dev eth0 proto kernel scope link src 198.32.212.73 >> 180.233.131.0/24 dev eth0 proto kernel scope link src 180.233.131.7 >> 10.240.1.0/24 dev eth1 proto kernel scope link src 10.240.1.1 >> default via 121.200.226.209 dev eth0 metric 100 >> >> # >> # Shorewall version 4 - Masq file >> # >> eth0:!198.32.212.0/24 eth1:!10.240.1.7 180.233.131.7 > > Ah! I took one more look at your report and I seriously doubt that the > above rule does what you expect. Rewrite it as: > > eth0:!198.32.212.0/24 10.240.0.0/24!10.240.1.7In fact, the current version of Shorewall (4.4.6) rejects that type of rule: gateway:/etc/shorewall# shorewall check Compiling... WARNING: Using an interface as the masq SOURCE requires the interface to be up and configured when Shorewall starts/restarts : /etc/shorewall/masq (line 7) ERROR: SOURCE interface may not be specified with a source IP address in the POSTROUTING chain : /etc/shorewall/masq (line 7) gateway:/etc/shorewall# Are you still using Shorewall-shell? If so, I highly recommend migrating to Shorewall-perl at the first opportunity. It does a much better job of validating the configuration at compile-time (and it does it much faster as well). -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Tom Eastep wrote:> Tom Eastep wrote: >> Trent O''Callaghan wrote: >>> MASQ/SNAT and ARP are interacting in a way that is causing outbound >>> connectivity issues in periods of low traffic (when ARP entries timeout). >>> Tcpdump of ARP packets shows who-has packets with the SNAT IP address when I >>> need them to have the Firewall''s Interface IP address as their source. >>> >>> I have modified MASQ to SNAT to the Firewall''s Interface IP address for the >>> Peering network (via 198.32.212.73), but outbound traffic is normally to >>> more distant networks and my default route is to the Paid Internet (via >>> 121.200.226.210). >>> >>> I have seen some have scripted ARP watchers that could assist but I believe >>> this is something Shorewall can cope with, but I am lacking in the >>> knowledge. >>> >>> root@per-r1:/etc/shorewall# ifconfig -a >>> eth0 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:121.200.226.210 Bcast:121.200.226.211 >>> Mask:255.255.255.252 >>> eth0:1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:198.32.212.73 Bcast:198.32.212.255 Mask:255.255.255.0 >>> eth0:2 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:180.233.131.7 Bcast:180.233.131.255 Mask:255.255.255.0 >>> eth1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:91 >>> inet addr:10.240.0.1 Bcast:10.240.0.255 Mask:255.255.255.0 >>> >>> root@per-r1:/etc/shorewall# ip route show table main | grep -v zebra >>> 121.200.226.208/30 dev eth0 proto kernel scope link src 121.200.226.210 >>> 198.32.212.0/24 dev eth0 proto kernel scope link src 198.32.212.73 >>> 180.233.131.0/24 dev eth0 proto kernel scope link src 180.233.131.7 >>> 10.240.1.0/24 dev eth1 proto kernel scope link src 10.240.1.1 >>> default via 121.200.226.209 dev eth0 metric 100 >>> >>> # >>> # Shorewall version 4 - Masq file >>> # >>> eth0:!198.32.212.0/24 eth1:!10.240.1.7 180.233.131.7 >> Ah! I took one more look at your report and I seriously doubt that the >> above rule does what you expect. Rewrite it as: >> >> eth0:!198.32.212.0/24 10.240.0.0/24!10.240.1.7 > > In fact, the current version of Shorewall (4.4.6) rejects that type of rule: > > gateway:/etc/shorewall# shorewall check > Compiling... > WARNING: Using an interface as the masq SOURCE requires the interface > to be up and configured when Shorewall starts/restarts : > /etc/shorewall/masq (line 7) > ERROR: SOURCE interface may not be specified with a source IP address > in the POSTROUTING chain : /etc/shorewall/masq (line 7) > gateway:/etc/shorewall# > > Are you still using Shorewall-shell?Duh -- just looked at the Subject again. I suggest that you look at http://www.shorewall.net/LennyToSqueeze.html. Also, note that the Debian Shorewall maintainer has Shorewall 4.4 packages available for Lenny; a link to his site can be found on the Shorewall download page. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Trent O''Callaghan
2010-Feb-05 00:42 UTC
Re: SNAT/ARP issue - shorewall-shell_4.0.15-1_all.deb
Thanks Tom, I will upgrade and retest. Then I will post an update on how I go. Kind regards, Trent O''Callaghan Network Manager www.nearmap.com From: Tom Eastep [mailto:teastep@shorewall.net] Sent: Thursday, 4 February 2010 11:28 PM To: Shorewall Users Cc: trent.ocallaghan@nearmap.com Subject: Re: [Shorewall-users] SNAT/ARP issue - shorewall-shell_4.0.15-1_all.deb Tom Eastep wrote:> Tom Eastep wrote: >> Trent O''Callaghan wrote: >>> MASQ/SNAT and ARP are interacting in a way that is causing outbound >>> connectivity issues in periods of low traffic (when ARP entriestimeout).>>> Tcpdump of ARP packets shows who-has packets with the SNAT IP addresswhen I>>> need them to have the Firewall''s Interface IP address as their source. >>> >>> I have modified MASQ to SNAT to the Firewall''s Interface IP address forthe>>> Peering network (via 198.32.212.73), but outbound traffic is normally to >>> more distant networks and my default route is to the Paid Internet (via >>> 121.200.226.210). >>> >>> I have seen some have scripted ARP watchers that could assist but Ibelieve>>> this is something Shorewall can cope with, but I am lacking in the >>> knowledge. >>> >>> root@per-r1:/etc/shorewall# ifconfig -a >>> eth0 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:121.200.226.210 Bcast:121.200.226.211 >>> Mask:255.255.255.252 >>> eth0:1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:198.32.212.73 Bcast:198.32.212.255Mask:255.255.255.0>>> eth0:2 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90 >>> inet addr:180.233.131.7 Bcast:180.233.131.255Mask:255.255.255.0>>> eth1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:91 >>> inet addr:10.240.0.1 Bcast:10.240.0.255 Mask:255.255.255.0 >>> >>> root@per-r1:/etc/shorewall# ip route show table main | grep -v zebra >>> 121.200.226.208/30 dev eth0 proto kernel scope link src121.200.226.210>>> 198.32.212.0/24 dev eth0 proto kernel scope link src 198.32.212.73 >>> 180.233.131.0/24 dev eth0 proto kernel scope link src 180.233.131.7 >>> 10.240.1.0/24 dev eth1 proto kernel scope link src 10.240.1.1 >>> default via 121.200.226.209 dev eth0 metric 100 >>> >>> # >>> # Shorewall version 4 - Masq file >>> # >>> eth0:!198.32.212.0/24 eth1:!10.240.1.7 180.233.131.7 >> Ah! I took one more look at your report and I seriously doubt that the >> above rule does what you expect. Rewrite it as: >> >> eth0:!198.32.212.0/24 10.240.0.0/24!10.240.1.7 > > In fact, the current version of Shorewall (4.4.6) rejects that type ofrule:> > gateway:/etc/shorewall# shorewall check > Compiling... > WARNING: Using an interface as the masq SOURCE requires the interface > to be up and configured when Shorewall starts/restarts : > /etc/shorewall/masq (line 7) > ERROR: SOURCE interface may not be specified with a source IP address > in the POSTROUTING chain : /etc/shorewall/masq (line 7) > gateway:/etc/shorewall# > > Are you still using Shorewall-shell?Duh -- just looked at the Subject again. I suggest that you look at http://www.shorewall.net/LennyToSqueeze.html. Also, note that the Debian Shorewall maintainer has Shorewall 4.4 packages available for Lenny; a link to his site can be found on the Shorewall download page. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ The Planet: dedicated and managed hosting, cloud storage, colocation Stay online with enterprise data centers and the best network in the business Choose flexible plans and management services without long-term contracts Personal 24x7 support from experience hosting pros just a phone call away. http://p.sf.net/sfu/theplanet-com
Trent O''Callaghan
2010-Feb-09 04:03 UTC
Re: SNAT/ARP issue - shorewall-shell_4.0.15-1_all.deb
My Issue has been resolved by:
echo 2 > /proc/sys/net/ipv4/conf/eth0/arp_announce
root@# cat /proc/sys/net/ipv4/conf/eth0/arp_announce
2
Documentation is in the 2.6 kernel docs
(linux/Documentation/networking/ip-sysctl.txt) (here from the 2.6.17
kernel).
arp_announce - INTEGER
Define different restriction levels for announcing the local
source IP address from IP packets in ARP requests sent on
interface:
0 - (default) Use any local address, configured on any interface
1 - Try to avoid local addresses that are not in the target''s
subnet for this interface. This mode is useful when target
hosts reachable via this interface require the source IP
address in ARP requests to be part of their logical network
configured on the receiving interface. When we generate the
request we will check all our subnets that include the
target IP and will preserve the source address if it is from
such subnet. If there is no such subnet we select source
address according to the rules for level 2.
2 - Always use the best local address for this target.
In this mode we ignore the source address in the IP packet
and try to select local address that we prefer for talks with
the target host. Such local address is selected by looking
for primary IP addresses on all our subnets on the outgoing
interface that include the target IP address. If no suitable
local address is found we select the first local address
we have on the outgoing interface or on all other interfaces,
with the hope we will receive reply for our request and
even sometimes no matter the source IP address we announce.
The max value from conf/{all,interface}/arp_announce is used.
Increasing the restriction level gives more chance for
receiving answer from the resolved target while decreasing
the level announces more valid sender''s information.
Tested successfully with shorewall-perl 4.4.6-1 on Debian and
shorewall-shell_4.0.15-1_all.deb
11:57:31.184988 00:15:17:cc:dd:90 > ff:ff:ff:ff:ff:ff, ethertype ARP
(0x0806), length 42: arp who-has 198.32.212.41 tell 198.32.212.73
11:57:31.185397 00:0c:db:34:83:00 > 00:15:17:cc:dd:90, ethertype ARP
(0x0806), length 60: arp reply 198.32.212.41 is-at 00:0c:db:34:83:00
Many Thanks,
Trent O''Callaghan
Network Manager
www.nearmap.com
From: Tom Eastep [mailto:teastep@shorewall.net]
Sent: Thursday, 4 February 2010 11:28 PM
To: Shorewall Users
Cc: trent.ocallaghan@nearmap.com
Subject: Re: [Shorewall-users] SNAT/ARP issue -
shorewall-shell_4.0.15-1_all.deb
Tom Eastep wrote:> Tom Eastep wrote:
>> Trent O''Callaghan wrote:
>>> MASQ/SNAT and ARP are interacting in a way that is causing outbound
>>> connectivity issues in periods of low traffic (when ARP entries
timeout).>>> Tcpdump of ARP packets shows who-has packets with the SNAT IP
address
when I>>> need them to have the Firewall''s Interface IP address as
their source.
>>>
>>> I have modified MASQ to SNAT to the Firewall''s Interface
IP address for
the>>> Peering network (via 198.32.212.73), but outbound traffic is
normally to
>>> more distant networks and my default route is to the Paid Internet
(via
>>> 121.200.226.210).
>>>
>>> I have seen some have scripted ARP watchers that could assist but I
believe>>> this is something Shorewall can cope with, but I am lacking in the
>>> knowledge.
>>>
>>> root@per-r1:/etc/shorewall# ifconfig -a
>>> eth0 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90
>>> inet addr:121.200.226.210 Bcast:121.200.226.211
>>> Mask:255.255.255.252
>>> eth0:1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90
>>> inet addr:198.32.212.73 Bcast:198.32.212.255
Mask:255.255.255.0>>> eth0:2 Link encap:Ethernet HWaddr 00:15:17:cc:dd:90
>>> inet addr:180.233.131.7 Bcast:180.233.131.255
Mask:255.255.255.0>>> eth1 Link encap:Ethernet HWaddr 00:15:17:cc:dd:91
>>> inet addr:10.240.0.1 Bcast:10.240.0.255
Mask:255.255.255.0
>>>
>>> root@per-r1:/etc/shorewall# ip route show table main | grep -v
zebra
>>> 121.200.226.208/30 dev eth0 proto kernel scope link src
121.200.226.210>>> 198.32.212.0/24 dev eth0 proto kernel scope link src
198.32.212.73
>>> 180.233.131.0/24 dev eth0 proto kernel scope link src
180.233.131.7
>>> 10.240.1.0/24 dev eth1 proto kernel scope link src 10.240.1.1
>>> default via 121.200.226.209 dev eth0 metric 100
>>>
>>> #
>>> # Shorewall version 4 - Masq file
>>> #
>>> eth0:!198.32.212.0/24 eth1:!10.240.1.7 180.233.131.7
>> Ah! I took one more look at your report and I seriously doubt that the
>> above rule does what you expect. Rewrite it as:
>>
>> eth0:!198.32.212.0/24 10.240.0.0/24!10.240.1.7
>
> In fact, the current version of Shorewall (4.4.6) rejects that type of
rule:>
> gateway:/etc/shorewall# shorewall check
> Compiling...
> WARNING: Using an interface as the masq SOURCE requires the interface
> to be up and configured when Shorewall starts/restarts :
> /etc/shorewall/masq (line 7)
> ERROR: SOURCE interface may not be specified with a source IP address
> in the POSTROUTING chain : /etc/shorewall/masq (line 7)
> gateway:/etc/shorewall#
>
> Are you still using Shorewall-shell?
Duh -- just looked at the Subject again. I suggest that you look at
http://www.shorewall.net/LennyToSqueeze.html. Also, note that the Debian
Shorewall maintainer has Shorewall 4.4 packages available for Lenny; a
link to his site can be found on the Shorewall download page.
-Tom
--
Tom Eastep \ When I die, I want to go like my Grandfather who
Shoreline, \ died peacefully in his sleep. Not screaming like
Washington, USA \ all of the passengers in his car
http://shorewall.net \________________________________________________
------------------------------------------------------------------------------
The Planet: dedicated and managed hosting, cloud storage, colocation
Stay online with enterprise data centers and the best network in the business
Choose flexible plans and management services without long-term contracts
Personal 24x7 support from experience hosting pros just a phone call away.
http://p.sf.net/sfu/theplanet-com