Here is a puzzle. I have a network with several servers. It''s a mess. It''s a /24 and pieces and servers are all over the place inside this /24 block, on both sides of the firewall. For example, the router at 1.2.3.1 is outside the firewall and many of the servers at 1.2.3.nnn/24 are behind the firewall. (Obviously, 1.2.3.nnn is a fudged network.) eth0 points outward to the Internet. eth1 points inward to the serers. Both eth0 and eth1 have IP Address 1.2.3.2. I setup proxy ARP like this: echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp echo 1 > /proc/sys/net/ipv4/conf/eth1/proxy_arp And I set up appropriate routes to the systems on both sides of the firewall. This all works - all the systems route the way they are supposed to route. Here is the problem. Behind the firewall is a Coyote Point Equalizer at 1.2.3.10, with a high-volume website behind it spread across several servers. Every time I put this proxy ARP firewall in place, that nasty Coyote Point box dies and this breaks the high volume website behind it and makes lots of people mad. I''ve never seen a Coyote Point Equalizer but I have a hunch it might not get along well with a proxy ARP device in its same network. Here are my questions: Proxy ARP really means proxy ARP - that firewall answers ARP requests for anything and everything it sees, for any network. This also has consequences for new devices that try to be polite when they set IP Addresses for themselves by ARPing to see if anyone else answers at that address. Is there a way to limit proxy ARP to a list of IP Addresses? Or - should I forget proxy ARP and look at bridging instead? Can I do bridging and still access the bridged interfaces remotely? Thanks - Greg Scott GregScott@InfraSupportEtc.com
On 5/30/2007 6:46 PM, Greg Scott wrote:> Here is the problem. Behind the firewall is a Coyote Point Equalizer > at 1.2.3.10, with a high-volume website behind it spread across > several servers. Every time I put this proxy ARP firewall in place, > that nasty Coyote Point box dies and this breaks the high volume > website behind it and makes lots of people mad. I''ve never seen a > Coyote Point Equalizer but I have a hunch it might not get along well > with a proxy ARP device in its same network.Hrm...> Proxy ARP really means proxy ARP - that firewall answers ARP requests > for anything and everything it sees, for any network. This also has > consequences for new devices that try to be polite when they set IP > Addresses for themselves by ARPing to see if anyone else answers at > that address. Is there a way to limit proxy ARP to a list of IP > Addresses?This will be Proxy ARP implementation specific. I have no idea whether or not Linux can be configured to behave as you are asking or not.> Or - should I forget proxy ARP and look at bridging instead?Having just (briefly) brushed up on Proxy ARPing, I can see how it would be a problem for a load balancer. Most load balancers work on a couple of different levels, either IP <-> MAC spoofing, or NATing. The former method is probably what is happening and thus having a problem with your Proxy ARP router / firewall. Consider if you will a host out side of the Proxy ARP router / firewall trying to connect to an IP address that is both behind the Proxy ARP router / firewall AND the load balancer. If the load balancer changes the MAC address that the IP address belongs to, the Proxy ARP router / firewall will inevitably end up pointing to the wrong internal MAC. How will the load balancer handle the traffic when it does not start flowing to the alternative MAC like it wants? I can not say. But, I do see a very big potential for a conflict. In said conflict, I can not say any thing to how any of the equipment will fail. Thus, you could end up in the scenario you are in now. I can''t say for sure as to whether or not you should forget about proxy ARP or not, but I can say for sure that bridging will do what you are wanting to do very well. Bridging will pass the ARP requests in directly to the load balancer like it is expecting so that it can control things the way that it wants to. This means that when the load balancer alters the IP <-> MAC mapping, the upstream device on the other side of the bridge will see the changed MAC address. I think I would go the bridging route.> Can I do bridging and still access the bridged interfaces remotely?Most definitely! Put your IP address on the bridge interface. I.e. eth0 and eth1 are bridged together by br0. ifconfig br0 1.2.3.2 netmask 255.255.255.0 You will be able to access 1.2.3.2 from either side of the bridge. That is presuming that you do not use EBTables / IPTables to filter the bridged traffic. In other words, so long as you are not doing any layer 2 filtering yes.
Greg Scott wrote:> > Here is a puzzle. > > I have a network with several servers. It''s a mess. It''s a /24 and > pieces and servers are all over the place inside this /24 block, on both > sides of the firewall. For example, the router at 1.2.3.1 is outside > the firewall and many of the servers at 1.2.3.nnn/24 are behind the > firewall. (Obviously, 1.2.3.nnn is a fudged network.) > > eth0 points outward to the Internet. > eth1 points inward to the serers. > > Both eth0 and eth1 have IP Address 1.2.3.2. I setup proxy ARP like > this: > > echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp > echo 1 > /proc/sys/net/ipv4/conf/eth1/proxy_arp > > And I set up appropriate routes to the systems on both sides of the > firewall. > > This all works - all the systems route the way they are supposed to > route. > > Here is the problem. Behind the firewall is a Coyote Point Equalizer at > 1.2.3.10, with a high-volume website behind it spread across several > servers. Every time I put this proxy ARP firewall in place, that nasty > Coyote Point box dies and this breaks the high volume website behind it > and makes lots of people mad. I''ve never seen a Coyote Point Equalizer > but I have a hunch it might not get along well with a proxy ARP device > in its same network. > > Here are my questions: > > Proxy ARP really means proxy ARP - that firewall answers ARP requests > for anything and everything it sees, for any network. This also has > consequences for new devices that try to be polite when they set IP > Addresses for themselves by ARPing to see if anyone else answers at that > address. Is there a way to limit proxy ARP to a list of IP Addresses? > > Or - should I forget proxy ARP and look at bridging instead? Can I do > bridging and still access the bridged interfaces remotely? > > Thanks > > - Greg Scott > GregScott@InfraSupportEtc.comSee http://yesican.chsoft.biz/lartc/proxy-arp.conf and http://yesican.chsoft.biz/lartc/proxy-arp.sh to see if that helps. The LAN interface (eth0) uses the /proc-/proxy_arp setting while the WAN (eth1) interface uses the script. FWIW, those are my working setups. One computer has a WAN connection (eth1) and all other servers inside connect to its eth0. The above script and config file are on that computer. Note that both eth1 and eth0 have the same IP (66.209.101.198) and netmask. This machine has a third interface (eth2) to the LAN, but that is not material here. If the ISP changes things, which they have done a couple of times, I have to ask them to flush their ARP cache manually because their retention is HUGE (~70 minutes), but except for that, I''ve never had any problems with this setup. I had no success at all trying to use /proc on eth1. -- gypsy
I was thinking about what Grant said in his earlier reply about bridging, and the more I think about it, the more I think Grant is right. I was watching tcpdump and some firewall accounting rules when we tried this the other day and I saw all kinds of traffic that had absolutely nothing to do with my network - and I was blocking it! And I kept wondering, why was I blocking traffic that had a destination IP address nowhere near me that I should not care about? Well, this is a co-location site, so an Ethernet connects this network to the Internet, not a point to point serial or any kind of WAN. No doubt lots of other folks had their networks in racks in the same room on the same extended Ethernet as my stuff. But still, why did my box care about any of this traffic at all? Why would I specifically block it - why didn''t the NIC in my box just ignore it, the way most normal systems do in an Ethernet for traffic it doesn''t care about? Well, duh! It''s proxy ARP. Every time anyone, anywhere on this Ethernet, sends out an ARP request and I see it, I answer - yup, here is my MAC Address and it belongs to the IP Address you just asked about. I don''t care what IP Address, I answer ARP requests for ALL IP Addresses! I was essentialy ARP spoofing the whole world! Well, at least the whole world on that Ethernet that morning. Holy moley - based on that analysis, proxy ARP should be outlawed from any co-location site, at least anything directly exposed to all the public traffic. For that few minutes, I''ll bet I messed up systems belonging to who-knows-how-many customers! Fortunately, it was in the middle of the night in my corner of the world. For Gypsy - it''s not my own ARP cache I was messing with, it was everyone else''s ARP caches. Anyway, lesson learned. Maybe this writeup will help somebody else out there. Definitely do bridging. - Greg Scott -----Original Message----- From: gypsy [mailto:gypsy@iswest.com] Sent: Thursday, May 31, 2007 1:50 AM To: lartc@mailman.ds9a.nl Cc: Greg Scott Subject: Re: [LARTC] Proxy ARP with a Coyote Point equalizer Greg Scott wrote:> > Here is a puzzle. > > I have a network with several servers. It''s a mess. It''s a /24 and > pieces and servers are all over the place inside this /24 block, on > both sides of the firewall. For example, the router at 1.2.3.1 is > outside the firewall and many of the servers at 1.2.3.nnn/24 are > behind the firewall. (Obviously, 1.2.3.nnn is a fudged network.) > > eth0 points outward to the Internet. > eth1 points inward to the serers. > > Both eth0 and eth1 have IP Address 1.2.3.2. I setup proxy ARP like > this: > > echo 1 > /proc/sys/net/ipv4/conf/eth0/proxy_arp > echo 1 > /proc/sys/net/ipv4/conf/eth1/proxy_arp > > And I set up appropriate routes to the systems on both sides of the > firewall. > > This all works - all the systems route the way they are supposed to > route. > > Here is the problem. Behind the firewall is a Coyote Point Equalizer > at 1.2.3.10, with a high-volume website behind it spread across > several servers. Every time I put this proxy ARP firewall in place, > that nasty Coyote Point box dies and this breaks the high volume > website behind it and makes lots of people mad. I''ve never seen a > Coyote Point Equalizer but I have a hunch it might not get along well > with a proxy ARP device in its same network. > > Here are my questions: > > Proxy ARP really means proxy ARP - that firewall answers ARP requests > for anything and everything it sees, for any network. This also has > consequences for new devices that try to be polite when they set IP > Addresses for themselves by ARPing to see if anyone else answers at > that address. Is there a way to limit proxy ARP to a list of IPAddresses?> > Or - should I forget proxy ARP and look at bridging instead? Can I do> bridging and still access the bridged interfaces remotely? > > Thanks > > - Greg Scott > GregScott@InfraSupportEtc.comSee http://yesican.chsoft.biz/lartc/proxy-arp.conf and http://yesican.chsoft.biz/lartc/proxy-arp.sh to see if that helps. The LAN interface (eth0) uses the /proc-/proxy_arp setting while the WAN (eth1) interface uses the script. FWIW, those are my working setups. One computer has a WAN connection (eth1) and all other servers inside connect to its eth0. The above script and config file are on that computer. Note that both eth1 and eth0 have the same IP (66.209.101.198) and netmask. This machine has a third interface (eth2) to the LAN, but that is not material here. If the ISP changes things, which they have done a couple of times, I have to ask them to flush their ARP cache manually because their retention is HUGE (~70 minutes), but except for that, I''ve never had any problems with this setup. I had no success at all trying to use /proc on eth1. -- gypsy
On 05/31/07 13:41, Greg Scott wrote:> I was thinking about what Grant said in his earlier reply about > bridging, and the more I think about it, the more I think Grant is > right. I was watching tcpdump and some firewall accounting rules > when we tried this the other day and I saw all kinds of traffic that > had absolutely nothing to do with my network - and I was blocking it! > And I kept wondering, why was I blocking traffic that had a > destination IP address nowhere near me that I should not care about?That''s not good.> Well, this is a co-location site, so an Ethernet connects this > network to the Internet, not a point to point serial or any kind of > WAN. No doubt lots of other folks had their networks in racks in the > same room on the same extended Ethernet as my stuff.Probably. However I have to ask, why are so many different networks sharing one broadcast domain? Or could it be that traffic was trying to be routed to the target subnet, but your system responded to the ARP for the IP of the target router??? I wonder... Either way, this is not good. This is also why I have seriously considered statically setting some IP to MAC address mappings in some more stringent environments. If the upstream router knows the MAC address of my router, it will not need to ARP for it and thus I do not care if someone else claims to use my IP or not because the router will know which MAC address to use.> But still, why did my box care about any of this traffic at all? Why > would I specifically block it - why didn''t the NIC in my box just > ignore it, the way most normal systems do in an Ethernet for traffic > it doesn''t care about? Well, duh! It''s proxy ARP. Every time > anyone, anywhere on this Ethernet, sends out an ARP request and I see > it, I answer - yup, here is my MAC Address and it belongs to the IP > Address you just asked about. I don''t care what IP Address, I answer > ARP requests for ALL IP Addresses! I was essentially ARP spoofing the > whole world! Well, at least the whole world on that Ethernet that > morning.Oh NO! That was not a good thing. I wonder how many people you effected while doing that.> Holy moley - based on that analysis, proxy ARP should be outlawed > from any co-location site, at least anything directly exposed to all > the public traffic. For that few minutes, I''ll bet I messed up > systems belonging to who-knows-how-many customers! Fortunately, it > was in the middle of the night in my corner of the world.Just because it is the middle of the night for you does not mean that it was not 9:30 in the morning for someone else who potentially has their box co-located there. I think I would be tempted to contact your co-location provider and let them know what happened. Consider if you were there administrator trying to figure out why a bunch of clients traffic did not route correctly for a short time and then repaired its self with out any explanation at all. I know that I would want to know if such a thing happened in one of my data centers. At the very least I would want to know that it was not a bug in the router but rather something that is a valid explanation for what happened. As far as outlawing Proxy ARP, I don''t know. I do know that I see no reason to use Proxy ARP when bridging is sow much more powerful and safer. Just think, you an do any and all IPTables (layer 3 and above) firewalling ON a layer 2 device. You can be REALLY devious with this. If course there is also EBTables and ARPTables too.> For Gypsy - it''s not my own ARP cache I was messing with, it was > everyone else''s ARP caches.Gulp!> Anyway, lesson learned. Maybe this writeup will help somebody else > out there. Definitely do bridging.Here! HERE! Grant. . . .