Recently our network switches were replaced. Since that time, Shorewall will not reply to who-has arp packets for any system defined in proxyarp or nat file. It actually worked for a few days, then suddenly the switch sends out who-has requests and no system replies. Any suggestions on how to debug this? Thanks, Chop ------------------------------------------------------------------------------ Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with <2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1
On 04/29/2013 09:52 PM, chop wow wrote:> Recently our network switches were replaced. > > Since that time, Shorewall will not reply to who-has arp packets for any > system defined in proxyarp or nat file. It actually worked for a few > days, then suddenly the switch sends out who-has requests and no system > replies. > > > Any suggestions on how to debug this? > > Thanks,Just to be clear, Shorewall itself does not process packets. Shorewall is a tool that configures your kernel based on the input it receives. You can see the proxy ARP entries made in the kernel''s IPv4 neighbor table using ''ARP -na''. Do those look correct? -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with <2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1
Thanks for the reply Tom, * The arp table looks fine with my RF1918 IPs, however I do not see any entries for the external NAT''d IPs - is this expected? ... ? (10.95.100.49) at 90:e6:ba:ed:4b:39 [ether] on eth1 (should I see the NAT''d IP on eth0?) * While pinging a server on the failed route, I see the following on the firewall eth0 (tcpdump -nei eth0 host <MY EXT NAT''d IP>): ... 09:25:15.285583 fa:c0:01:7b:4a:91 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 60: Request who-has <MY EXT NAT''d IP> tell <SWITCH IP>, length 46 * The failure also occurs from a system I have proxyarp directly behind firewall eth3: ... ? (PROXYIP) at 00:1b:21:77:b2:fc [ether] on eth3 ? (PROXYIP) at <from_interface> PERM PUB on eth0 Network doesn''t completely fail on the NAT''d systems, only a few select routes. However when it does fail, the flurry of who-has packets ensues. Also, all other masqueraded systems successfully access all routes. Thanks again! Chop On Tue, Apr 30, 2013 at 7:10 AM, Tom Eastep <teastep@shorewall.net> wrote:> On 04/29/2013 09:52 PM, chop wow wrote: > > Recently our network switches were replaced. > > > > Since that time, Shorewall will not reply to who-has arp packets for any > > system defined in proxyarp or nat file. It actually worked for a few > > days, then suddenly the switch sends out who-has requests and no system > > replies. > > > > > > Any suggestions on how to debug this? > > > > Thanks, > > Just to be clear, Shorewall itself does not process packets. Shorewall > is a tool that configures your kernel based on the input it receives. > > You can see the proxy ARP entries made in the kernel''s IPv4 neighbor > table using ''ARP -na''. Do those look correct? > > -Tom > -- > Tom Eastep \ When I die, I want to go like my Grandfather who > Shoreline, \ died peacefully in his sleep. Not screaming like > Washington, USA \ all of the passengers in his car > http://shorewall.net \________________________________________________ > > > > ------------------------------------------------------------------------------ > Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET > Get 100% visibility into your production application - at no cost. > Code-level diagnostics for performance bottlenecks with <2% overhead > Download for free and get started troubleshooting in minutes. > http://p.sf.net/sfu/appdyn_d2d_ap1 > _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users > >------------------------------------------------------------------------------ Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with <2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1
On 4/30/13 9:55 AM, "chop wow" <chop.encore@gmail.com> wrote:> Thanks for the reply Tom, > > > > * The arp table looks fine with my RF1918 IPs, however I do not see any > entries for the external NAT''d IPs - is this expected? > ... > ? (10.95.100.49) at 90:e6:ba:ed:4b:39 [ether] on eth1 > (should I see the NAT''d IP on eth0?) > > > * While pinging a server on the failed route, I see the following on the > firewall eth0 (tcpdump -nei eth0 host <MY EXT NAT''d IP>): > ... > 09:25:15.285583 fa:c0:01:7b:4a:91 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), > length 60: Request who-has <MY EXT NAT''d IP> tell <SWITCH IP>, length 46 > > > * The failure also occurs from a system I have proxyarp directly behind > firewall eth3: > ... > ? (PROXYIP) at 00:1b:21:77:b2:fc [ether] on eth3 > ? (PROXYIP) at <from_interface> PERM PUB on eth0You are asking us questions that assume that we know all about your configuration; you have told us almost nothing and what you have told us you have obscured (PROXYIP). The PERM PUB entry is what is added by Shorewall. You should see one of those for each entry in your proxyarp file.That causes who-has requests for PROXYIP arriving on eth0 to be replied. What does your /etc/shorewall/interfaces file entry for eth0 look like? -Tom You do not need a parachute to skydive. You only need a parachute to skydive twice. ------------------------------------------------------------------------------ Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with <2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1
On 04/30/2013 12:20 PM, Tom Eastep wrote:> > The PERM PUB entry is what is added by Shorewall. You should see one of > those for each entry in your proxyarp file.That causes who-has requests > for PROXYIP arriving on eth0 to be replied. What does your > /etc/shorewall/interfaces file entry for eth0 look like? >Also, please forward the output from: for f in /proc/sys/net/ipv4/conf/eth0/*; do echo $f=$(cat $f); done Thanks, -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Introducing AppDynamics Lite, a free troubleshooting tool for Java/.NET Get 100% visibility into your production application - at no cost. Code-level diagnostics for performance bottlenecks with <2% overhead Download for free and get started troubleshooting in minutes. http://p.sf.net/sfu/appdyn_d2d_ap1
Tom et al. First and foremost, I want to apologize for submitting such a obscure and obvious lacking email request for support. I should have followed http://www.shorewall.net/support.htm before sending out initial email. To my embarrassment I didn''t even provide the Shorewall version level. I do realize - I am lucky I even received a response. My plan was to correct the lack of details and follow-up with a proper communication. However, I have since found the issue. Ultimately the problem/solution was clear and somewhat trivial. I had NAT''d to an external IP. This IP was then ''acquired'' by my NOC to use for their new switch in an existing VRRP group (we had an additional switch installed for dedicated wireless network). The problem was further complicated as the forward DNS record was updated - but the reverse was not. This explains the routing issues/failures I experienced where the routing failure only occurred to very specific destinations. This may also explain why the routing would work all weekend and start failing again midday on Monday - perhaps when the my user would start to use the NAT''d external IP. Once the routing failed, all NAT''d and proxy arp systems routing would fail. In conclusion - thanks again for a wonderful tool and help. Any future communications, I''ll insure to provide adequate information/detail. ------------------------------------------------------------------------------ AlienVault Unified Security Management (USM) platform delivers complete security visibility with the essential security capabilities. Easily and efficiently configure, manage, and operate all of your security controls from a single console and one unified framework. Download a free trial. http://p.sf.net/sfu/alienvault_d2d
On 5/13/13 2:10 PM, "chop wow" <chop.encore@gmail.com> wrote:> Tom et al. > > First and foremost, I want to apologize for submitting such a obscure and > obvious lacking email request for support. > I should have followed http://www.shorewall.net/support.htm before sending out > initial email. To my embarrassment I didn''t even provide the Shorewall > version level. I do realize - I am lucky I even received a response. My > plan was to correct the lack of details and follow-up with a proper > communication. However, I have since found the issue. > > Ultimately the problem/solution was clear and somewhat trivial. I had NAT''d > to an external IP. This IP was then ''acquired'' by my NOC to use for their new > switch in an existing VRRP group (we had an additional switch installed for > dedicated wireless network). The problem was further complicated as the > forward DNS record was updated - but the reverse was not. This explains the > routing issues/failures I experienced where the routing failure only occurred > to very specific destinations. This may also explain why the routing would > work all weekend and start failing again midday on Monday - perhaps when the > my user would start to use the NAT''d external IP. Once the routing failed, > all NAT''d and proxy arp systems routing would fail. > > In conclusion - thanks again for a wonderful tool and help. Any future > communications, I''ll insure to provide adequate information/detail.No problem glad to hear that you got it sorted out. -Tom You do not need a parachute to skydive. You only need a parachute to skydive twice. ------------------------------------------------------------------------------ AlienVault Unified Security Management (USM) platform delivers complete security visibility with the essential security capabilities. Easily and efficiently configure, manage, and operate all of your security controls from a single console and one unified framework. Download a free trial. http://p.sf.net/sfu/alienvault_d2d