Anthony Papineau
2004-Apr-14 19:47 UTC
Multiple ISP''s & DNAT - how are the Answer packets routed
Hi there! We have a Firewall with 2 interfaces connected to the Internet. It is set up according to the Mutliple interfaces information in Shorewall FAQ 32 and it mostly works. I''m not going to go through the entire FAQ here as I don''t believe it is necessary to find the answer to this question. We have a number of hosts behind the firewall and use DNAT to access them. This works fine from the Primary ISP network interface which is also the default route - however traffic coming into one of the DNAT''d hosts from the Secondary ISP network interface is properly handled INTO the network, but the answer packets are routed back out through the Primary ISP network interface. The packets do eventually get to where they are supposed to go, so It''s not a Problem exactly... but I''d like to know what is happening. I''m using the kernel packet traveling diagram at http://www.docum.org/stef.coene/qos/kptd/ as a guideline for understanding ''what happens when'' - it is pretty clear how a packet gets DNAT''d on the way in etc, but I''m now wondering about the answer packets and how they go out. Please correct me if I''m mistaken about any of this - but here is my (probably limited) understanding of how things work - I''m only going to deal with the specific scenario where the packet comes in on the Secondary interface. (This may be over simplified but please don''t worry about too much about details & semantics) IF1 = Interface for ISP1''s network IF2 = Interface for ISP2''s network IF3 = Interface for Internal network IF1_IP = IP address of IF1 IF2_IP = IP address of IF2 SRV_IP = ip address of server on internal network. CLIENT_IP = ip address of someone trying to connect to us Packet comes in with SRC=CLIENT_IP and DST=IF2_IP. The packet is DNAT''d and the DST is modified to be SRV_IP. The packet is now routed and filtered (or filtered and routed) and eventually passed out through the Internal Interface IF3 This much seems to be working OK. The trouble is with the answering packets: Answer packet comes in from IF3 with SRC=SRV_IP and DST=CLIENT_IP. Now.. the ip rules choose the routing table by looking at the source address.. if the source address if IF1 it uses IF1''s gateway as the default router and the same for IF2. (See Shorewall FAQ 32 again) So it all comes down to what happens first: A) If the Answer packets are Un-DNAT-ed before the ip rules are checked then the SRC address will have been re-written to the IP of the interface that the Original packet came in on -- a.k.a. the original DST address. So when the packet is routed, it should go back out via IF2. or B) If the ip rules are checked before the packet is un-DNAT-ed then the SRC address is an internal/private IP address and the Default gateway will be chosen and THEN the SRC address is re-written - giving me Answer packets going out of IF1 with a SRC address of IF2. Unfortunately B) appears to be the case. If I sniff the traffic on the Primary interface, when I make a connection to a DNAT''d port via the Secondary interface, I see response packets coming out of the Primary interface (IF1) with a SRC address of the Secondary interface (IF2_IP). So ultimately I have 2 questions. 1) How are DNAT Answer packets passed through the kernel & in what order do things happen - specifically - are they un-DNAT-ed before or after the IP rules determine the route table to use. 2) If the way it works is pretty much what i''ve described in "B" Above, is there anything I can do to ''fix'' it or should we be looking at doing something other than DNAT? Thanks for your help A <><><><><><><><><> Anthony R. Papineau
Tom Eastep
2004-Apr-14 20:04 UTC
Re: Multiple ISP''s & DNAT - how are the Answer packets routed
Anthony Papineau wrote:> Hi there! > > We have a Firewall with 2 interfaces connected to the Internet. It is > set up according to the Mutliple interfaces information in Shorewall FAQ > 32 and it mostly works. > I''m not going to go through the entire FAQ here as I don''t believe it is > necessary to find the answer to this question. >I call your attention to this text in FAQ 32: "The following information regarding setting up routing for this configuration is reproduced from the LARTC HOWTO and has not been verified by the author. If you have questions or problems with the instructions given below, please post to the LARTC mailing list." Possibly someone else on the list can help you but as I approach retirement, I don''t feel that I can afford more than one internet connection at either of our homes. Consequently I have no experience whatsoever with what is described in the excerpt from the LARTC HowTo reproduced in FAQ 32. Be that as it may, it was my understanding that if the routing was set up as described in that text, the *route cache* was used to correctly route responses out through the proper ISP. Again, I can''t verify that one way or another. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net
Jerry Vonau
2004-Apr-14 22:45 UTC
Re: Multiple ISP''s & DNAT - how are the Answer packets routed
I''m doing the 2 isp thing also...
How is you masq file setup?
I have:
eth0 10.2.0.0/24 $EXTIP1
ppp0 10.2.0.0/24 $EXTIP2
The other thing that comes to mind is the output of ip route ls
<snip>
default
nexthop via zz.zzz.140.1 dev eth0 weight 1
nexthop via yyy.yyy.28.28 dev ppp0 weight 1
This is obtained using " ip route add default scope global /
nexthop via $P1 dev $IF1 weight 1 nexthop via $P2 dev $IF2 weight 1 "
when I setup the load balancing.
Email me off list if you like, this maybe OT for shorewall.
You may have to add an "ip rule" for the client to the isp''s
table.
Jerry Vonau
----- Original Message -----
From: "Anthony Papineau" <anthony@exegenix.com>
To: <shorewall-users@lists.shorewall.net>
Sent: Wednesday, April 14, 2004 14:47
Subject: [Shorewall-users] Multiple ISP''s & DNAT - how are the
Answer
packets routed
> Hi there!
>
> We have a Firewall with 2 interfaces connected to the Internet.
It> is set up according to the Mutliple interfaces information in
Shorewall> FAQ 32 and it mostly works.
> I''m not going to go through the entire FAQ here as I
don''t believe
it> is necessary to find the answer to this question.
>
> We have a number of hosts behind the firewall and use DNAT to access
> them. This works fine from the Primary ISP network interface which
is> also the default route - however traffic coming into one of the
DNAT''d> hosts from the Secondary ISP network interface is properly handled
INTO> the network, but the answer packets are routed back out through the
> Primary ISP network interface.
>
> The packets do eventually get to where they are supposed to go, so
It''s> not a Problem exactly... but I''d like to know what is happening.
>
> I''m using the kernel packet traveling diagram at
> http://www.docum.org/stef.coene/qos/kptd/ as a guideline for
> understanding ''what happens when'' - it is pretty clear
how a packet
> gets DNAT''d on the way in etc, but I''m now wondering
about the
answer> packets and how they go out.
>
> Please correct me if I''m mistaken about any of this - but here is
my
> (probably limited) understanding of how things work - I''m only
going
to> deal with the specific scenario where the packet comes in on the
> Secondary interface. (This may be over simplified but please don''t
> worry about too much about details & semantics)
>
> IF1 = Interface for ISP1''s network
> IF2 = Interface for ISP2''s network
> IF3 = Interface for Internal network
> IF1_IP = IP address of IF1
> IF2_IP = IP address of IF2
> SRV_IP = ip address of server on internal network.
> CLIENT_IP = ip address of someone trying to connect to us
>
> Packet comes in with SRC=CLIENT_IP and DST=IF2_IP. The packet is
> DNAT''d and the DST is modified to be SRV_IP.
> The packet is now routed and filtered (or filtered and routed) and
> eventually passed out through the Internal Interface IF3
>
> This much seems to be working OK. The trouble is with the answering
> packets:
>
> Answer packet comes in from IF3 with SRC=SRV_IP and DST=CLIENT_IP.
>
> Now.. the ip rules choose the routing table by looking at the
source> address.. if the source address if IF1 it uses IF1''s gateway as
the
> default router and the same for IF2. (See Shorewall FAQ 32 again)
So> it all comes down to what happens first:
>
> A) If the Answer packets are Un-DNAT-ed before the ip rules are
checked> then the SRC address will have been re-written to the IP of the
> interface that the Original packet came in on -- a.k.a. the original
> DST address. So when the packet is routed, it should go back out
via> IF2.
>
> or
>
> B) If the ip rules are checked before the packet is un-DNAT-ed then
the> SRC address is an internal/private IP address and the Default
gateway> will be chosen and THEN the SRC address is re-written - giving me
> Answer packets going out of IF1 with a SRC address of IF2.
>
> Unfortunately B) appears to be the case. If I sniff the traffic on
> the Primary interface, when I make a connection to a DNAT''d port
via
> the Secondary interface, I see response packets coming out of the
> Primary interface (IF1) with a SRC address of the Secondary
interface> (IF2_IP).
>
> So ultimately I have 2 questions.
>
> 1) How are DNAT Answer packets passed through the kernel & in what
> order do things happen - specifically - are they un-DNAT-ed before
or> after the IP rules determine the route table to use.
>
> 2) If the way it works is pretty much what i''ve described in
"B"
Above,> is there anything I can do to ''fix'' it or should we be
looking at
doing> something other than DNAT?
>
> Thanks for your help
> A
>
> <><><><><><><><><>
> Anthony R. Papineau
>
> _______________________________________________
> Shorewall-users mailing list
> Post: Shorewall-users@lists.shorewall.net
> Subscribe/Unsubscribe:
https://lists.shorewall.net/mailman/listinfo/shorewall-users> Support: http://www.shorewall.net/support.htm
> FAQ: http://www.shorewall.net/FAQ.htm
Anthony Papineau
2004-Apr-15 17:11 UTC
Re: Multiple ISP''s & DNAT - how are the Answer packets routed
Hi Tom! I had seen that part of the FAQ but was hoping someone on the list might have information. I did manage to find the information I was looking for - of course only AFTER I asked for help did I stumble across it myself. FYI here is what is going on: When setup as described in the LARTC howto, the 2 ISP setup will work when using load balancing and for masqueraded traffic - basically for OUTBOUND traffic. It does NOT handle DNAT traffic (inbound) for almost exactly the reasons I described --- the routing that chooses which outbound interface to use happens BEFORE the source address of the outgoing packet is re-written. A good explanation of the problem and what you can do about it can be found here: http://linux-ip.net/html/adv-multi-internet.html#adv-multi-internet- outbound Thanks for your help! Anthony> I call your attention to this text in FAQ 32: > > "The following information regarding setting up routing for this > configuration is reproduced from the LARTC HOWTO and has not been > verified by the author. If you have questions or problems with the > instructions given below, please post to the LARTC mailing list." > > Possibly someone else on the list can help you but as I approach > retirement, I don''t feel that I can afford more than one internet > connection at either of our homes. Consequently I have no experience > whatsoever with what is described in the excerpt from the LARTC HowTo > reproduced in FAQ 32. > > Be that as it may, it was my understanding that if the routing was set > up as described in that text, the *route cache* was used to correctly > route responses out through the proper ISP. Again, I can''t verify that > one way or another. > > -Tom > -- > Tom Eastep \ Nothing is foolproof to a sufficiently talented fool > Shoreline, \ http://shorewall.net > Washington USA \ teastep@shorewall.net ><><><><><><><><><> Anthony R. Papineau