Hi, I have a linux box which balances load between two interfaces ( say WAN1 and WAN2). I have masquerading on for any request coming from LAN to the outside world. The setup is in such a way that WAN1 drops packets with source ip belonging to WAN2''s network and viceversa. For some strange reason, I find that packet coming out from the WAN interface has source address of WAN2 and thereby getting dropped. When I check the route cache , I find that for the same source and destination, I have two route cache entries 192.168.52.66 192.168.26.73 192.168.19.76 0 0 0 eth1 192.168.52.66 192.168.26.73 192.168.20.25 i 0 0 23 eth2 Here 192.168.19.76 is the WAN1 gateway and 192.168.20.25 is WAN2 gateway, as we see the packets are going out through WAN2 , but the masquerading has happened to the WAN1 IP address. It seems to me that the root of the problem is the creation of the two cache entries. Any idea why this happens, and how it can be avoided. Thanks and Regards, Vinod C
Hi Erik,
Ip route get returns
192.168.26.73 via 192.168.19.76 dev eth1 src 192.168.19.29
cache mtu 1500 advmss 1460
Here too it gives me WAN1s IP address which is the same as the
masqueraded one, but the packets are seen going out through WAN2.
Thanks and Regards,
Vinod C
Erik Slagter wrote:
>On Tue, 2006-05-30 at 20:02 +0530, Vinod Chandran wrote:
>
>
>>It seems to me that the root of the problem is the creation of the two
>>cache entries. Any idea why this happens, and how it can be avoided.
>>
>>
>
>What does "ip route get" say? I think this tool will be the key to
the
>solution, look at the "src address".
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>LARTC mailing list
>LARTC@mailman.ds9a.nl
>http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>
>
On Tue, 2006-05-30 at 20:02 +0530, Vinod Chandran wrote:> It seems to me that the root of the problem is the creation of the two > cache entries. Any idea why this happens, and how it can be avoided.What does "ip route get" say? I think this tool will be the key to the solution, look at the "src address". _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>AFAICR using MASQUERADE with multipath is not recommended (but maybe thisis outdated, plz correct me if I''m wrong) I have had problems using MASQUERADE with multipath on a 2.4.31 kernel box, with two outbound default routes I got messages about "rustys brain broke" and things like ICMP etc suffered packet loss. Not recommended, SNAT works perfectly :) Andy Registered Office: J.O. Sims Ltd, Pudding Lane, Pinchbeck, Spalding, Lincs. PE11 3TJ Company reg No: 2084187 Vat reg No: GB 437 4621 47 Tel: +44 (0) 1775 842100 Fax: +44 (0) 1775 842101 Web: www.josims.com Email:enquiries@josims.com The information contained in this e-mail is confidential and is intended for the addressee only. The contents of this e-mail must not be disclosed or copied without the sender''s consent. If you are not the intended recipient of the message, please notify the sender immediately, and delete the message. The statements and opinions expressed in this message are those of the author and do not necessarily reflect those of the company. No commitment may be inferred from the contents unless explicitly stated. The company does not take any responsibility for the personal views of the author. This message has been scanned for viruses before sending, but the company does not accept any responsibility for infection and recommends that you scan any attachments.JOSEDV001TAG
On Tuesday 30 May 2006 11:32, Vinod Chandran wrote:> Hi, > > I have a linux box which balances load between two interfaces ( say WAN1 > and WAN2). I have masquerading on for any request coming from LAN to the > outside world.If WAN ips are static, you can use iptables -t nat ... -o WAN1 -j SNAT --to x.x.x.x iptables -t nat ... -o WAN2 -j SNAT --to x.x.x.x AFAICR using MASQUERADE with multipath is not recommended (but maybe this is outdated, plz correct me if I''m wrong)> The setup is in such a way that WAN1 drops packets with source ip > belonging to WAN2''s network and viceversa. > For some strange reason, I find that packet coming out from the WAN > interface has source address of WAN2 and thereby getting dropped. > > When I check the route cache , I find that for the same source and > destination, I have two route cache entriesTOS is also computed to make a route cache entry, so it is possible and natural to 2 entries with same src and dst. Be aware that some ssh clients change the packet TOS after negotiation has been done, this could cause you get routed over a different link in the middle of your session, making impossible to do a simple ssh.> 192.168.52.66 192.168.26.73 192.168.19.76 0 0 0 > eth1 192.168.52.66 192.168.26.73 192.168.20.25 i 0 0 > 23 eth2 > > Here 192.168.19.76 is the WAN1 gateway and 192.168.20.25 is WAN2 > gateway, as we see the packets are going out through WAN2 , but the > masquerading has happened to the WAN1 IP address. > > It seems to me that the root of the problem is the creation of the two > cache entries. Any idea why this happens, and how it can be avoided.i think the root of the problem is that you are using MASQUERADE and not SNAT. And somehow MASQUERADE has not an accurate method to guess the right ip address to do the NAT. Besides that, you need to solve the problems that multipath will arise, like TOS situation described above or route cache expiration, that could made long term conns to be routed over a new iface. The solutions i know are CONNMARK(kernel>=2.6.12) and julian''s patches[1]. Personally i prefer CONNMARK. [1] http://www.ssi.bg/~ja/ -- Luciano
My apologies for sending email disclaimer crap to the list, I forgot the magic string to prevent it from being appended :(. Oops. Andy JOSEDV001TAG -----Original Message----- From: Andrew Lyon [mailto:andrew.lyon@josims.com] Sent: 30 May 2006 17:13 To: ''lartc@mailman.ds9a.nl'' Subject: RE: [LARTC] Problems with Routing and Masquerading>AFAICR using MASQUERADE with multipath is not recommended (but maybe >thisis outdated, plz correct me if I''m wrong) I have had problems using MASQUERADE with multipath on a 2.4.31 kernel box, with two outbound default routes I got messages about "rustys brain broke" and things like ICMP etc suffered packet loss. Not recommended, SNAT works perfectly :) Andy Registered Office: J.O. Sims Ltd, Pudding Lane, Pinchbeck, Spalding, Lincs. PE11 3TJ Company reg No: 2084187 Vat reg No: GB 437 4621 47 Tel: +44 (0) 1775 842100 Fax: +44 (0) 1775 842101 Web: www.josims.com Email:enquiries@josims.com The information contained in this e-mail is confidential and is intended for the addressee only. The contents of this e-mail must not be disclosed or copied without the sender''s consent. If you are not the intended recipient of the message, please notify the sender immediately, and delete the message. The statements and opinions expressed in this message are those of the author and do not necessarily reflect those of the company. No commitment may be inferred from the contents unless explicitly stated. The company does not take any responsibility for the personal views of the author. This message has been scanned for viruses before sending, but the company does not accept any responsibility for infection and recommends that you scan any attachments.JOSEDV001TAG _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Luciano Ruete wrote: <snip>> Besides that, you need to solve the problems that multipath will arise, like > TOS situation described above or route cache expiration, that could made > long > term conns to be routed over a new iface. The solutions i know are > CONNMARK(kernel>=2.6.12) and julian''s patches[1]. > Personally i prefer CONNMARK.Could you elaborate a little more on the CONNMARK method? Thanks.
On Tuesday 30 May 2006 13:23, Jason Boxman wrote:> Luciano Ruete wrote: > <snip> > > > Besides that, you need to solve the problems that multipath will arise, > > like TOS situation described above or route cache expiration, that could > > made long > > term conns to be routed over a new iface. The solutions i know are > > CONNMARK(kernel>=2.6.12) and julian''s patches[1]. > > Personally i prefer CONNMARK. > > Could you elaborate a little more on the CONNMARK method?#by-pass rules if it is already MARKed iptables -t mangle -A POSTROUTING -m mark --mark ! 0 -j ACCEPT #1st packets(from a connection) will arrive here iptables -t mangle -A POSTROUTING -o eth1 -j MARK --set-mark 0x1 iptables -t mangle -A POSTROUTING -o eth2 -j MARK --set-mark 0x2 iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark #restore mark before ROUTING decision iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark #route commands ip ro add default nexthop via x.x.x.x dev eth1 weight 1 nexthop via y.y.y.y dev eth2 ip route add default table provider1 via x.x.x.x dev eth1 ip route add default table provider2 via y.y.y.y dev eth2 # and most important ip rule add fwmark 0x1 table provider1 ip rule add fwmark 0x2 table provider2 Some notes: -The example uses 2 ifaces, but is scalable to any (i have it working with 5) -FORWARD could be used instead of POSRTOUTING, it depends on your needs -If you have a large network, think in change the default conntrack table size and hash-size ip_conntrack hashsize=xxx echo xxx > /proc/sys/net/ipv4/ip_conntrack_max #lnstat is your friend, will help to find the magic numbers lnstat -f ip_conntrack -i 1 -c 1 -Full discussion about this solution on this thread[1][2] (sorry spanish only) -Credits to diego woitasen who point me out to this kind of solution [1] http://www.lugmen.org.ar/pipermail/lug-list/2006-April/041078.html [2] http://www.lugmen.org.ar/pipermail/lug-list/2006-May/041084.html -- Luciano
>>>>> "Jason" == Jason Boxman <jasonb@edseek.com> writes:Jason> Luciano Ruete wrote: <snip> >> Besides that, you need to solve the problems that multipath >> will arise, like TOS situation described above or route cache >> expiration, that could made long term conns to be routed over a >> new iface. The solutions i know are CONNMARK(kernel>=2.6.12) >> and julian''s patches[1]. Personally i prefer CONNMARK. Jason> Could you elaborate a little more on the CONNMARK method? I second that motion -- not too clear on the interaction between SNAT, multiple interfaces, multiple default routes and CONNMARK </aol mode="metoo">. If someone could take out the time to make a complete example with (say) 2 outgoing interfaces, I promise a small GPL script in exchange which would automate the whole process. Actually the script''s already made, but it doesn''t use CONNMARK and suffers from the problems Jason describes and as documented in: http://mailman.ds9a.nl/pipermail/lartc/2006q1/018220.html Regards, -- Raju -- Raj Mathur raju@kandalaya.org http://kandalaya.org/ GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F It is the mind that moves
Hi, Thanks Jason for the solution. With CONNMARK, I was able to route the packets properly. Yeah, the problem was seen only for SSH sessions, I didnot see the problem with the Telnet and Ping sessions. TOS could be the answer to that. The only change I had to do as far the CONNMARK solution was that in the PREROUTING chain, I had to add the rule with "-i eth0" where eth0 is my LAN, otherwise the return packets were not reaching the box in LAN. Thanks and Regards, Vinod C Raj Mathur wrote:>>>>>>"Jason" == Jason Boxman <jasonb@edseek.com> writes: >>>>>> >>>>>> > > Jason> Luciano Ruete wrote: <snip> > >> Besides that, you need to solve the problems that multipath > >> will arise, like TOS situation described above or route cache > >> expiration, that could made long term conns to be routed over a > >> new iface. The solutions i know are CONNMARK(kernel>=2.6.12) > >> and julian''s patches[1]. Personally i prefer CONNMARK. > > Jason> Could you elaborate a little more on the CONNMARK method? > >I second that motion -- not too clear on the interaction between SNAT, >multiple interfaces, multiple default routes and CONNMARK </aol >mode="metoo">. If someone could take out the time to make a complete >example with (say) 2 outgoing interfaces, I promise a small GPL script >in exchange which would automate the whole process. > >Actually the script''s already made, but it doesn''t use CONNMARK and >suffers from the problems Jason describes and as documented in: > > http://mailman.ds9a.nl/pipermail/lartc/2006q1/018220.html > >Regards, > >-- Raju > >
Vinod Chandran wrote:> Hi, > > Thanks Jason for the solution. With CONNMARK, I was able to route the > packets properly.Cool, but I don''t think that was me.
Oh yeah, my bad! Thanks Luciano for the solution. Regards, Vinod C Jason Boxman wrote:>Vinod Chandran wrote: > > >>Hi, >> >>Thanks Jason for the solution. With CONNMARK, I was able to route the >>packets properly. >> >> > >Cool, but I don''t think that was me. > > > >_______________________________________________ >LARTC mailing list >LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc > > >