Hi,
I have a client connected to the ''net through 3 ISP''s. Have
set up a
Linux box to do routing and load sharing for the 3 connections. A
fourth interface is connected to the LAN with private IP addresses.
Am using iptables to SNAT traffic to the appropriate IP depending on
the interface the packet gets routed onto. The setup looks something
like this:
Interface IP Gateway Table Network
--------- -- ------- ----- -------
intA ipA gwA tableA netA
intB ipB gwB tableB netB
intC ipC gwC tableC netC
[intD is the LAN interface]
intD ipD (private) no gateway global netD
This works fine most of the time, except that once in a while (every
5-10 minutes or so) packets going out on (e.g.) intB suddenly start
getting NAT''ed to source address ipA (i.e. the address of another
interface). Obviously this plays hell with the existing connections
on that link!
The ip commands I''m using are:
/sbin/ip route add netA dev intA src ipA table tableA
/sbin/ip route add netA dev intA src ipA
/sbin/ip route add default via gwA table tableA
/sbin/ip route add netB dev intB src ipB table tableB
/sbin/ip route add netB dev intB src ipB
/sbin/ip route add default via gwB table tableB
/sbin/ip route add netC dev intC src ipC table tableC
/sbin/ip route add netC dev intC src ipC
/sbin/ip route add default via gwC table tableC
/sbin/ip route add default scope global nexthop via gwB dev intB weight 1
nexthop via gwC dev intC weight 2 nexthop via gwA dev intA weight 2
/sbin/ip rule add from ipA table tableA
/sbin/ip rule add from ipB table tableB
/sbin/ip rule add from ipC table tableC
The iptables commands are:
/sbin/iptables -P FORWARD DROP
# Enable full flow on the LAN
/sbin/iptables -I FORWARD -s netD -i intD -j ACCEPT
/sbin/iptables -I FORWARD -d netD -o intD -j ACCEPT
# Allow all packets to go out
/sbin/iptables -I OUTPUT -o intA -j ACCEPT
/sbin/iptables -I OUTPUT -o intB -j ACCEPT
/sbin/iptables -I OUTPUT -o intC -j ACCEPT
/sbin/iptables -I OUTPUT -o intD -j ACCEPT
/sbin/iptables -I INPUT -i intD -j ACCEPT
/sbin/iptables -I INPUT -i lo -j ACCEPT
/sbin/iptables -P INPUT DROP
/sbin/iptables -A INPUT -i ! intD -m state --state RELATED,ESTABLISHED -j ACCEPT
# Hmmm, why is this one there?
/sbin/iptables -A INPUT -i intD -m state --state RELATED,ESTABLISHED -j ACCEPT
# NAT depending on outbound interface
/sbin/iptables -t nat -A POSTROUTING -s netD -o intA -j SNAT --to-source ipA
/sbin/iptables -t nat -A POSTROUTING -s netD -o intB -j SNAT --to-source ipB
/sbin/iptables -t nat -A POSTROUTING -s netD -o intC -j SNAT --to-source ipC
Any idea why connections that are flowing perfectly would suddenly
decide to start getting NAT''ed to the wrong source? Or some place on
the ''net I can start looking?
Regards,
-- Raju
--
Raj Mathur raju@kandalaya.org http://kandalaya.org/
GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F
It is the mind that moves
That''s because the route to a host X has been changed. There''s a routing decision to host X every so or so minutes. Ways to handle that? Boy, that can be tough. I''ve read of prople who use a different LAN IP for every public connection. So If you have 3 public connections, you should use three different IPs on the lan. Then, when you DNAT packets on the way in (from the internet), you mark those packets with a different fwmark for each internet connection. On POSTROUTING do a SNAT according to this fwmark to a different IP each (remember you have three inner IPs to choose from). Then when packets come back from your servers, they will be sent to three different IPs and you can tell packets that should go out one way or the other. Now, if you don''t care about incoming (to your lan) connections but outgoing.... I guess you are very much out of luck.... unless you use some policy routing for stateful connections (I think that''s the name of connections that DO care if you change the IP the connection is going through) so that they use a single interface all the time. On 2/23/06, Raj Mathur <raju@linux-delhi.org> wrote:> Hi, > > I have a client connected to the ''net through 3 ISP''s. Have set up a > Linux box to do routing and load sharing for the 3 connections. A > fourth interface is connected to the LAN with private IP addresses. > Am using iptables to SNAT traffic to the appropriate IP depending on > the interface the packet gets routed onto. The setup looks something > like this: > > Interface IP Gateway Table Network > --------- -- ------- ----- ------- > intA ipA gwA tableA netA > intB ipB gwB tableB netB > intC ipC gwC tableC netC > [intD is the LAN interface] > intD ipD (private) no gateway global netD > > This works fine most of the time, except that once in a while (every > 5-10 minutes or so) packets going out on (e.g.) intB suddenly start > getting NAT''ed to source address ipA (i.e. the address of another > interface). Obviously this plays hell with the existing connections > on that link! > > The ip commands I''m using are: > > /sbin/ip route add netA dev intA src ipA table tableA > /sbin/ip route add netA dev intA src ipA > /sbin/ip route add default via gwA table tableA > /sbin/ip route add netB dev intB src ipB table tableB > /sbin/ip route add netB dev intB src ipB > /sbin/ip route add default via gwB table tableB > /sbin/ip route add netC dev intC src ipC table tableC > /sbin/ip route add netC dev intC src ipC > /sbin/ip route add default via gwC table tableC > /sbin/ip route add default scope global nexthop via gwB dev intB weight 1 nexthop via gwC dev intC weight 2 nexthop via gwA dev intA weight 2 > /sbin/ip rule add from ipA table tableA > /sbin/ip rule add from ipB table tableB > /sbin/ip rule add from ipC table tableC > > The iptables commands are: > > /sbin/iptables -P FORWARD DROP > # Enable full flow on the LAN > /sbin/iptables -I FORWARD -s netD -i intD -j ACCEPT > /sbin/iptables -I FORWARD -d netD -o intD -j ACCEPT > # Allow all packets to go out > /sbin/iptables -I OUTPUT -o intA -j ACCEPT > /sbin/iptables -I OUTPUT -o intB -j ACCEPT > /sbin/iptables -I OUTPUT -o intC -j ACCEPT > /sbin/iptables -I OUTPUT -o intD -j ACCEPT > /sbin/iptables -I INPUT -i intD -j ACCEPT > /sbin/iptables -I INPUT -i lo -j ACCEPT > /sbin/iptables -P INPUT DROP > /sbin/iptables -A INPUT -i ! intD -m state --state RELATED,ESTABLISHED -j ACCEPT > # Hmmm, why is this one there? > /sbin/iptables -A INPUT -i intD -m state --state RELATED,ESTABLISHED -j ACCEPT > # NAT depending on outbound interface > /sbin/iptables -t nat -A POSTROUTING -s netD -o intA -j SNAT --to-source ipA > /sbin/iptables -t nat -A POSTROUTING -s netD -o intB -j SNAT --to-source ipB > /sbin/iptables -t nat -A POSTROUTING -s netD -o intC -j SNAT --to-source ipC > > Any idea why connections that are flowing perfectly would suddenly > decide to start getting NAT''ed to the wrong source? Or some place on > the ''net I can start looking? > > Regards, > > -- Raju > -- > Raj Mathur raju@kandalaya.org http://kandalaya.org/ > GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F > It is the mind that moves > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc >
Raj, I use something just like you make and for this problem I have patched my kernel with diff-routes patch (there is a link in LARTC) and I have make a little script that makes just a ping to an outside address every 5 min (cron job) and flush the route cache after this ping, like this: ip route flush cache So all cache will be lost (off course that conections that still exists will remain with their track to the destination). Att, Nataniel Klug ----- Original Message ----- From: "Raj Mathur" <raju@linux-delhi.org> To: <lartc@mailman.ds9a.nl> Sent: Thursday, February 23, 2006 4:14 PM Subject: [LARTC] Balancing multiple connections and NAT> Hi, > > I have a client connected to the ''net through 3 ISP''s. Have set up a > Linux box to do routing and load sharing for the 3 connections. A > fourth interface is connected to the LAN with private IP addresses. > Am using iptables to SNAT traffic to the appropriate IP depending on > the interface the packet gets routed onto. The setup looks something > like this: > > Interface IP Gateway Table Network > --------- -- ------- ----- ------- > intA ipA gwA tableA netA > intB ipB gwB tableB netB > intC ipC gwC tableC netC > [intD is the LAN interface] > intD ipD (private) no gateway global netD > > This works fine most of the time, except that once in a while (every > 5-10 minutes or so) packets going out on (e.g.) intB suddenly start > getting NAT''ed to source address ipA (i.e. the address of another > interface). Obviously this plays hell with the existing connections > on that link! > > The ip commands I''m using are: > > /sbin/ip route add netA dev intA src ipA table tableA > /sbin/ip route add netA dev intA src ipA > /sbin/ip route add default via gwA table tableA > /sbin/ip route add netB dev intB src ipB table tableB > /sbin/ip route add netB dev intB src ipB > /sbin/ip route add default via gwB table tableB > /sbin/ip route add netC dev intC src ipC table tableC > /sbin/ip route add netC dev intC src ipC > /sbin/ip route add default via gwC table tableC > /sbin/ip route add default scope global nexthop via gwB dev intB weight 1nexthop via gwC dev intC weight 2 nexthop via gwA dev intA weight 2> /sbin/ip rule add from ipA table tableA > /sbin/ip rule add from ipB table tableB > /sbin/ip rule add from ipC table tableC > > The iptables commands are: > > /sbin/iptables -P FORWARD DROP > # Enable full flow on the LAN > /sbin/iptables -I FORWARD -s netD -i intD -j ACCEPT > /sbin/iptables -I FORWARD -d netD -o intD -j ACCEPT > # Allow all packets to go out > /sbin/iptables -I OUTPUT -o intA -j ACCEPT > /sbin/iptables -I OUTPUT -o intB -j ACCEPT > /sbin/iptables -I OUTPUT -o intC -j ACCEPT > /sbin/iptables -I OUTPUT -o intD -j ACCEPT > /sbin/iptables -I INPUT -i intD -j ACCEPT > /sbin/iptables -I INPUT -i lo -j ACCEPT > /sbin/iptables -P INPUT DROP > /sbin/iptables -A INPUT -i ! intD -m state --state RELATED,ESTABLISHED -jACCEPT> # Hmmm, why is this one there? > /sbin/iptables -A INPUT -i intD -m state --state RELATED,ESTABLISHED -jACCEPT> # NAT depending on outbound interface > /sbin/iptables -t nat -A POSTROUTING -s netD -o intA -j SNAT --to-sourceipA> /sbin/iptables -t nat -A POSTROUTING -s netD -o intB -j SNAT --to-sourceipB> /sbin/iptables -t nat -A POSTROUTING -s netD -o intC -j SNAT --to-sourceipC> > Any idea why connections that are flowing perfectly would suddenly > decide to start getting NAT''ed to the wrong source? Or some place on > the ''net I can start looking? > > Regards, > > -- Raju > -- > Raj Mathur raju@kandalaya.org http://kandalaya.org/ > GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F > It is the mind that moves > _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
Am Donnerstag, 23. Februar 2006 20:14 schrieb Raj Mathur:> Hi, > > I have a client connected to the ''net through 3 ISP''s. Have set up a > Linux box to do routing and load sharing for the 3 connections. A > fourth interface is connected to the LAN with private IP addresses. > Am using iptables to SNAT traffic to the appropriate IP depending on > the interface the packet gets routed onto. The setup looks something > like this: > > Interface IP Gateway Table Network > --------- -- ------- ----- ------- > intA ipA gwA tableA netA > intB ipB gwB tableB netB > intC ipC gwC tableC netC > [intD is the LAN interface] > intD ipD (private) no gateway global netD > > This works fine most of the time, except that once in a while (every > 5-10 minutes or so) packets going out on (e.g.) intB suddenly start > getting NAT''ed to source address ipA (i.e. the address of another > interface). Obviously this plays hell with the existing connections > on that link!you need a patch for NAT processing with multiple gateways. this will then save the routing information for each connection inside NAT structures, so that each packet of an established connection will be get routed over the same gateway. you can find the patches here: http://www.ssi.bg/~ja/#routes please read the guides (nano howto or dgd-usage) carefully. -- Markus Schulz
On Fr, 2006-02-24 at 00:44 +0530, Raj Mathur wrote:> I have a client connected to the ''net through 3 ISP''s. Have set up a > Linux box to do routing and load sharing for the 3 connections. A > fourth interface is connected to the LAN with private IP addresses. > Am using iptables to SNAT traffic to the appropriate IP depending on > the interface the packet gets routed onto.I use exactly the same setup with a customer''s conenction, the only difference: I use MASQUERADE instead of SNAT. I did not see anything like the problem you describe. Maybe because MAQUERADE works stateful, SNAT not? If you do not have a special reason for using SNAT, I think you should try MASQUERADE. If your problem persits, please tell me, as I have to look at my customer''s setup very closely then, to catch this before anyone complains.
>>>>> "Sebastian" == Sebastian Bork <sebi@sebi.org> writes:Sebastian> On Fr, 2006-02-24 at 00:44 +0530, Raj Mathur wrote: >> I have a client connected to the ''net through 3 ISP''s. Have >> set up a Linux box to do routing and load sharing for the 3 >> connections. A fourth interface is connected to the LAN with >> private IP addresses. Am using iptables to SNAT traffic to the >> appropriate IP depending on the interface the packet gets >> routed onto. Sebastian> I use exactly the same setup with a customer''s Sebastian> conenction, the only difference: I use MASQUERADE Sebastian> instead of SNAT. I did not see anything like the Sebastian> problem you describe. Maybe because MAQUERADE works Sebastian> stateful, SNAT not? If you do not have a special reason Sebastian> for using SNAT, I think you should try MASQUERADE. If Sebastian> your problem persits, please tell me, as I have to look Sebastian> at my customer''s setup very closely then, to catch this Sebastian> before anyone complains. Well, both MASQUERADE and SNAT are stateful (MASQUERADE is just a special case of SNAT as far as I remember); however it''s worth a shot if it''s working for you. It''s pretty easy to trap the wrong source IP errors -- going back to my example, just run: tcpdump -i intA -q -t -n ! host ipA tcpdump -i intB -q -t -n ! host ipB tcpdump -i intC -q -t -n ! host ipC Any IP packets that get displayed will be those with wrong source IPs. You may need to start some large FTP uploads or similar and watch for a while -- the problem manifests itself for me when the client is uploading 10+ MB files to his public FTP server. Of course, it may be present in other places also, but outgoing FTP comprises the bulk of his traffic so it''s most patent there. Digressing a bit, from the responses I''ve got from this list, it seems that a kernel patch is required to make the whole load sharing + iptables NAT work properly. I''m a bit disappointed that this isn''t part of the mainstream kernel -- any chances of it being rolled in upstream? Regards, -- Raju -- Raj Mathur raju@kandalaya.org http://kandalaya.org/ GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F It is the mind that moves
On Sa, 2006-02-25 at 00:23 +0530, Raj Mathur wrote:> >>>>> "Sebastian" == Sebastian Bork <sebi@sebi.org> writes: > Sebastian> I use exactly the same setup with a customer''s > Sebastian> conenction, the only difference: I use MASQUERADE > Sebastian> instead of SNAT. I did not see anything like the > Sebastian> problem you describe. Maybe because MAQUERADE works > Sebastian> stateful, SNAT not? If you do not have a special reason > Sebastian> for using SNAT, I think you should try MASQUERADE. If > Sebastian> your problem persits, please tell me, as I have to look > Sebastian> at my customer''s setup very closely then, to catch this > Sebastian> before anyone complains. > > Well, both MASQUERADE and SNAT are stateful (MASQUERADE is just a > special case of SNAT as far as I remember); however it''s worth a shot > if it''s working for you. > > It''s pretty easy to trap the wrong source IP errors -- going back to > my example, just run:Done. It happens here, too. But now it gets really strange: the data (I tried scp) goes out on IF1 with IF2''s source address. The ACK packets come in on IF2. The connection works anyway ... *That''s* what I''d call really cool load-balancing.
On Fr, 2006-02-24 at 22:22 +0100, Sebastian Bork wrote:> Done. It happens here, too. But now it gets really strange: the data (I > tried scp) goes out on IF1 with IF2''s source address. The ACK packets > come in on IF2. The connection works anyway ... *That''s* what I''d call > really cool load-balancing.It obviously only worked with my setup because two of the three routes end on the same router of the upstream provider. As soon as the other provider is used for a route, everything breaks down. I''ll try those patches tomorrow.
Em Qui 23 Fev 2006 20:41, Markus Schulz escreveu:> you need a patch for NAT processing with multiple gateways. this will > then save the routing information for each connection inside NAT > structures, so that each packet of an established connection will be > get routed over the same gateway. you can find the patches here: > http://www.ssi.bg/~ja/#routes > please read the guides (nano howto or dgd-usage) carefully.Any idea why these patches are not yet integrated into the upstream kernel?
On 3/5/06, Andreas Hasenack <ahasenack@terra.com.br> wrote:> Em Qui 23 Fev 2006 20:41, Markus Schulz escreveu: > > you need a patch for NAT processing with multiple gateways. this will > > then save the routing information for each connection inside NAT > > structures, so that each packet of an established connection will be > > get routed over the same gateway. you can find the patches here: > > http://www.ssi.bg/~ja/#routes > > please read the guides (nano howto or dgd-usage) carefully. > > Any idea why these patches are not yet integrated into the upstream kernel? >Possibly disagreement on the features included, or any number of other reasons. Personally I had serious problems with the patches in that they didn''t fully function. Multi-path routing was only choosing one path, and if too many routes were requested a kernel panic would occur. - Jody