LARTC - Dec 2010 - default route with two nexthops and MASQUERADE problem

Dear all, 

I''ve the following problem with routing + NAT: 
If I''ve two ISP and I''m using two nexthop in default route
with MASQUERADE on both ISP links, I see routing cache regenerated, but
sometimes packets sent to a new link (after cache regeneration) uses wrong
source address for masquerading.

Here is the config. 

I''ve two links to outside via two different providers: eth1 and eth2 
eth0 is the LAN 

# ip a (part of output, since we have 3 more interfaces disabled) 
2: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
1000
link/ether 00:1a:92:9e:66:e8 brd ff:ff:ff:ff:ff:ff 
inet 192.168.1.254/24 brd 192.168.1.255 scope global eth1 
valid_lft forever preferred_lft forever 
3: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
1000
link/ether d8:5d:4c:80:6b:2b brd ff:ff:ff:ff:ff:ff 
inet 192.168.2.254/24 brd 192.168.2.255 scope global eth2 
valid_lft forever preferred_lft forever 
6: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast qlen
1000
link/ether 00:1a:92:9e:76:82 brd ff:ff:ff:ff:ff:ff 
inet 192.168.5.1/24 brd 192.168.5.255 scope global eth0 
valid_lft forever preferred_lft forever 

# ip r (main table) 
192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.1 
192.168.2.0/24 dev eth2 proto kernel scope link src 192.168.2.254 
192.168.1.0/24 dev eth1 proto kernel scope link src 192.168.1.254 
default 
nexthop via 192.168.1.1 dev eth1 weight 1 
nexthop via 192.168.2.1 dev eth2 weight 1 

# ip r s t eth1 
default via 192.168.1.1 dev eth1 

# ip r s t eth2 
default via 192.168.2.1 dev eth2 

# ip ru 
0: from all lookup local 
32450: from 192.168.2.254 lookup eth2 
32717: from 192.168.5.124 lookup eth1 
32766: from all lookup main 
32767: from all lookup default 

Q1: if I do pings from two PC in LAN: 5.137 and 5.147, to the same IP how can
they go via different links (ping 195.60.x.x is run on both computers)?

# ip r g 195.60.x.x from 192.168.5.137 iif eth0 
195.60.169.6 from 192.168.5.137 via 192.168.1.1 dev eth1 src 192.168.5.1 
cache <src-direct> mtu 1500 advmss 1460 hoplimit 128 iif eth0 

# ip r g 195.60.x.x from 192.168.5.147 iif eth0 
195.60.169.6 from 192.168.5.147 via 192.168.2.1 dev eth2 src 192.168.5.1 
cache <src-direct> mtu 1500 advmss 1460 hoplimit 128 iif eth0 

The routing in my case should be the same for all users. it shoul send packets
to the same destination via the same link always (even if the source IP is
different). isn''t it?

Q2: Sometimes I see in tcpdump on external ifaces that the routing cache was
regenerated. This can be forced by "ip r f t cache". This sometimes
results in change of the link for my pings. But one of two machines suddenly
looses connection. After the tcpdump it is because the routing has decided to
use another link, but the MASQUERADE was not updated at that time:

# tcpdump -i eth1 
IP 192.168.2.254 > 195.60.x.x: ICMP echo request, id 10677, seq 242, length
64 (request from .5.147 with wrong source address due to MASQUERADE not updated
according to the routing cache purge - hence, no reply, since the source address
of the MASQUERADEd packet is wrong)
IP 192.168.1.254 > 195.60.x.x: ICMP echo request, id 37387, seq 244, length
64 (request from .5.137)
IP 195.60.x.x > 192.168.1.254: ICMP echo reply, id 37387, seq 244, length 64 

Here is my MASQUERADE setting 
# iptables -L -t nat 
Chain POSTROUTING (policy ACCEPT 752K packets, 48M bytes) 
pkts bytes target prot opt in out source destination 
2840K 256M MASQUERADE all -- any eth1 192.168.5.0/24 anywhere 
2491K 229M MASQUERADE all -- any eth2 192.168.5.0/24 anywhere 


I understand that I can use conntrack to mark packets, but it is a little bit
more complicated. I would prefer to use destination IP as the key for routing.
What is wrong in this scenario? why routing cache purges does not notify
NAT-engine about changes in routing?

PoltoS 


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc