Brian J. Murrell
2004-Nov-25 15:59 UTC
simple dual Internet connection setup not sending return packets on correct interface
I have a very simple setup exactly as described in the HOWTO section "
4.2. Routing for multiple uplinks/providers".
One is cable (eth1: dhcp) and the other is PPPoE (ppp0).
I used the following commands to configure the routing once all of my
interfaces are up and i have configured SNATing for them:
ip route add 66.11.173.0/24 dev ppp0 src 66.11.173.224 table 11
ip route add default via 66.11.190.1 table 11
ip route add 24.235.240.0/22 dev eth1 src 24.235.240.15 table 12
ip route add default via 24.235.240.1 table 12
ip route add 66.11.173.0/24 dev ppp0 src 66.11.173.224
ip route add 24.235.240.0/22 dev eth1 src 24.235.240.15
ip rule add from 66.11.173.224 table 11
ip rule add from 24.235.240.15 table 12
My iptables "nat" setup looks like this:
Chain POSTROUTING (policy ACCEPT 364 packets, 26735 bytes)
pkts bytes target prot opt in out source destination
258 19801 eth1_masq all -- * eth1 0.0.0.0/0 0.0.0.0/0
0 0 ppp0_masq all -- * ppp0 0.0.0.0/0 0.0.0.0/0
Chain eth1_masq (1 references)
pkts bytes target prot opt in out source destination
252 19021 SNAT all -- * * 10.75.22.0/24 0.0.0.0/0
to:24.235.240.15
0 0 SNAT all -- * * 192.168.66.0/24 0.0.0.0/0
to:24.235.240.15
Chain ppp0_masq (1 references)
pkts bytes target prot opt in out source destination
0 0 SNAT all -- * * 10.75.22.0/24 0.0.0.0/0
to:66.11.173.224
0 0 SNAT all -- * * 192.168.66.0/24 0.0.0.0/0
to:66.11.173.224
When an IP packet enters my machine via interface ppp0 everything is
fine. However when a responding packet (i.e. a SYN-ACK in response to a
SYN) is sent from my machine, it is sent with the correct source address
(66.11.173.224) but on the eth1 interface. It is worth noting at this
point that eth1 is the default gateway interface.
Is this a problem in that iproute2 selects the default route before
SNATting is done to change the source address of the packet, which does
not happen of course until POSTROUTING? Surely I am not the first
person who has this need. How does one solve it?
Current routing tables:
# ip route list
66.11.190.1 dev ppp0 proto kernel scope link src 66.11.173.224
66.11.173.0/24 dev ppp0 scope link src 66.11.173.224
10.75.22.0/24 dev eth0 proto kernel scope link src 10.75.22.254
24.235.240.0/22 dev eth1 proto kernel scope link src 24.235.240.15
default via 24.235.240.1 dev eth1
# ip route list table 11
66.11.173.0/24 dev ppp0 scope link src 66.11.173.224
default via 66.11.190.1 dev ppp0
# ip route list table 12
24.235.240.0/22 dev eth1 scope link src 24.235.240.15
default via 24.235.240.1 dev eth1
and current ip rules:
# ip rule list
0: from all lookup local
32764: from 24.235.240.15 lookup 12
32765: from 66.11.173.224 lookup 11
32766: from all lookup main
32767: from all lookup default
Any ideas what I am doing wrong?
b.
gypsy
2004-Nov-26 05:40 UTC
Re: simple dual Internet connection setup not sending return packetson correct interface
"Brian J. Murrell" wrote:> > I have a very simple setup exactly as described in the HOWTO section " > 4.2. Routing for multiple uplinks/providers". > > My iptables "nat" setup looks like this: > > Chain POSTROUTING (policy ACCEPT 364 packets, 26735 bytes) > pkts bytes target prot opt in out source destination > 258 19801 eth1_masq all -- * eth1 0.0.0.0/0 0.0.0.0/0 > 0 0 ppp0_masq all -- * ppp0 0.0.0.0/0 0.0.0.0/0> Any ideas what I am doing wrong? > > b.Guessing from the lack of any mention of KeepState in your iptables setup, my guess is that you ignored the advice to vist Julian Anastasov''s web site. Start with this: http://www.geocities.com/mctiew/ffw/dual.htm You should also google LARTC "Finally: A working case of two adsl load balance". Read Ron Senykoff''s post "load balance a file download across two connections - success!". gypsy _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Brian J. Murrell
2004-Nov-26 13:19 UTC
Re: simple dual Internet connection setup not sending return packetson correct interface
On Thu, 2004-11-25 at 21:40 -0800, gypsy wrote:> > Guessing from the lack of any mention of KeepStateKeepState? If you are referring to: 52459 2774K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED rules, I have those sprinkled throughout my ruleset where necessary. The iptables "snippet" I included in my previous message was just that. Just the relevant portion that does the NATting.> in your iptables > setup,Like I said, the RELATED,ESTABLISHED state rules are in there. My full set of iptables rules is >400. I did not see see a need to post that fully here.> my guess is that you ignored the advice to vist Julian > Anastasov''s web site.No I didn''t ignore it. But what that site is promoting is some kind of floppy disk based router distribution or something.> > Start with this: > http://www.geocities.com/mctiew/ffw/dual.htmI am not looking to replace/rebuild my whole firewall. I simply want to add a second link to my existing one and have the packets use the correct interface -- to travel back out the interface from which they came. I don''t want to do load balancing or failover or anything fancy. I want two interfaces where I use one for all outgoing traffic and the only time the alternate is used is to send response packets to connections that come _in_ that interface or for routes that are specifically directed through that interface via a routing table entry.> You should also google LARTC "Finally: A working case of two adsl load > balance". Read Ron Senykoff''s post "load balance a file download across > two connections - success!".Interesting. Followed a few links too. Looks like a lot of bells and whistles I am not really looking for (load balancing and failover, etc.) but there is some hint of indication that there is a patch needed to make sure NAT uses the right physical interface. Maybe I will go bug the netfilter guys to see if this is the case. Thanx, b.
Brian J. Murrell
2004-Nov-26 14:39 UTC
Re: simple dual Internet connection setup not sending return packets on correct interface
To followup on my own posting, with more information... On Thu, 2004-11-25 at 10:59 -0500, Brian J. Murrell wrote:> I have a very simple setup exactly as described in the HOWTO section " > 4.2. Routing for multiple uplinks/providers". > > One is cable (eth1: dhcp) and the other is PPPoE (ppp0).These are both on the same physical interface, eth1. IOW, the PPPoE packets are sent to the PPPoE "modem" on eth1. eth1 is also plugged into the cable provider''s "modem" as such: +---------- Cable Modem +--------+ | | | +--+--+ | GW eth1 ------| HUB | | | +--+--+ +--------+ | +---------- PPPoE Modem This set up works, physically. I can tcpdump on eth1 and see both regular ethernet traffic going to an from my cable provider, as well as PPPoE encapsulated traffic coming in through my PPPoE connection: 09:29:58.109041 00:08:e2:33:f8:54 > 00:a0:24:2a:1f:72, ethertype IPv4 (0x0800), length 130: IP 66.96.26.190.922 > 24.235.240.15.52814: P 49:113(64) ack 48 win 28800 <nop,nop,timestamp 59750486 1599607031> 09:29:58.109344 00:a0:24:2a:1f:72 > 00:08:e2:33:f8:54, ethertype IPv4 (0x0800), length 66: IP 24.235.240.15.52814 > 66.96.26.190.922: . ack 113 win 32740 <nop,nop,timestamp 1599607172 59750486> 09:29:58.117164 00:90:1a:40:43:d7 > 00:a0:24:2a:1f:72, ethertype PPPoE S (0x8864), length 82: PPPoE [ses 0x1473] PPP-IP (0x0021), length 62: IP 66.96.26.190.52797 > 66.11.173.224.25: S 3517919246:3517919246(0) win 5840 <mss 1400,sackOK,timestamp 59750486 0,nop,wscale 0> 09:29:58.118789 00:a0:24:2a:1f:72 > 00:08:e2:33:f8:54, ethertype IPv4 (0x0800), length 74: IP 66.11.173.224.25 > 66.96.26.190.52797: S 3862223559:3862223559(0) ack 3517919247 win 5792 <mss 1460,sackOK,timestamp 2207063156 59750486,nop,wscale 0> As you can see, packets 1 and 2 are an established TCP session over the cable connection and packet 3 is an incoming PPPoE encapsulated packet coming in on the PPPoE connection and interestingly enough, packet 4 is an erroneously transmitted packet demonstrating exactly my problem. It is the response to packet 3. As you can see it has all of the correct IP and TCP headers, it is just sent physically via eth1 and not ppp0. Heh. Indeed if my cable provider were not filtering packets from me that don''t have my assigned source address, this would all work. Just a reminder of my iptables SNAT rules for context of my point below...> My iptables "nat" setup looks like this: > > Chain POSTROUTING (policy ACCEPT 364 packets, 26735 bytes) > pkts bytes target prot opt in out source destination > 258 19801 eth1_masq all -- * eth1 0.0.0.0/0 0.0.0.0/0 > 0 0 ppp0_masq all -- * ppp0 0.0.0.0/0 0.0.0.0/0 > > Chain eth1_masq (1 references) > pkts bytes target prot opt in out source destination > 252 19021 SNAT all -- * * 10.75.22.0/24 0.0.0.0/0 to:24.235.240.15 > 0 0 SNAT all -- * * 192.168.66.0/24 0.0.0.0/0 to:24.235.240.15 > > Chain ppp0_masq (1 references) > pkts bytes target prot opt in out source destination > 0 0 SNAT all -- * * 10.75.22.0/24 0.0.0.0/0 to:66.11.173.224 > 0 0 SNAT all -- * * 192.168.66.0/24 0.0.0.0/0 to:66.11.173.224> Is this a problem in that iproute2 selects the default route before > SNATting is done to change the source address of the packet, which does > not happen of course until POSTROUTING?The answer to this is of course no. Because the source address of the erroneously-sent-via-eth1 packet (#4 in the above trace) has been correctly re-written (NATted) to 66.11.173.224, then according to the SNAT rules above, the packet is being sent through the correct interface (ppp0). What still remains inconsistent however is that according to the rules above and currently on my gateway, the "ppp0_masq" rules show 0 hits. How can the source address be correctly re-written to 66.11.173.224 and the rule that does the re-writing show 0 hits? There is definitely something fishy going on here. b.
Brian J. Murrell
2004-Nov-26 15:21 UTC
Re: simple dual Internet connection setup not sending return packets on correct interface
On Fri, 2004-11-26 at 09:39 -0500, Brian J. Murrell wrote:> To followup on my own posting, with more information...And yet more...> On Thu, 2004-11-25 at 10:59 -0500, Brian J. Murrell wrote: > > I have a very simple setup exactly as described in the HOWTO section " > > 4.2. Routing for multiple uplinks/providers". > > > > One is cable (eth1: dhcp) and the other is PPPoE (ppp0). > > These are both on the same physical interface, eth1. IOW, the PPPoE > packets are sent to the PPPoE "modem" on eth1. eth1 is also plugged > into the cable provider''s "modem" as such: > > +---------- Cable Modem > +--------+ | > | | +--+--+ > | GW eth1 ------| HUB | > | | +--+--+ > +--------+ | > +---------- PPPoE ModemWhich is irrelevant. I have just put a third NIC in the machine to put the PPPoE and Cable connections on different NICs and still the same problem. Packets have PPPoE''s source address, but are sent physically on Cable connected NIC. b.
diab
2004-Nov-26 16:17 UTC
Re[2]: simple dual Internet connection setup not sending return packets on correct interface
BJM> Which is irrelevant. I have just put a third NIC in the machine to put
BJM> the PPPoE and Cable connections on different NICs and still the same
BJM> problem. Packets have PPPoE''s source address, but are sent
physically
BJM> on Cable connected NIC.
iirc, to have two working internet connections on one (nat''ing)
computer you basically need two things (in my example its eth0 and
eth1)
1) SNAT to the right source address, like
iptables -A POSTROUTING -j nat -t SNAT [-s from.where or -d to.where]\
--to-source source.addr.of.eth0
iptables -A POSTROUTING -j nat -t SNAT [-s from.where or -d to.where]\
--to-source source.addr.of.eth1
2) two routing tables, like
ip route add default via eth0.gateway.ip.address dev eth0 table 1
ip route add default via eth1.gateway.ip.address dev eth1 table 2
maybe you dont even need the "via xx" thing, the dev xxx is enough.
then you can classify packets to use the connection you want using
ip rule add WHATEVER lookup N (whatever could be "to x.x.x.x" or
"from
x.x.x.x", same as in the SNAT example, N could be 1 or 2)
if you want the router to respond to packets correcty (ie. to answer
ping on both interfaces) you need to
ip rule add iif eth0 lookup 1
ip rule add iif eth1 lookup 2
so packets coming from eth0/eth1 are routed using the correct routing
table.
i wrote all this from scratch so accept my apologies for any errors i
might have done but in my understanding this is how it should work in
general.
-
diab
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
please, what do those 2 following lines do ? mainly that "drop flowid
:1"
?
tc qdisc add dev $WAN handle ffff: ingress
tc filter add dev $WAN parent ffff: protocol ip prio 50 u32 match ip src
0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Brian J. Murrell
2004-Nov-26 16:44 UTC
Re: simple dual Internet connection setup not sending return packets on correct interface
On Fri, 2004-11-26 at 17:17 +0100, diab wrote:> iirc, to have two working internet connections on one (nat''ing) > computer you basically need two things (in my example its eth0 and > eth1) > > 1) SNAT to the right source address, like > iptables -A POSTROUTING -j nat -t SNAT [-s from.where or -d to.where]\ > --to-source source.addr.of.eth0Surely you mean -t nat -j SNAT?> iptables -A POSTROUTING -j nat -t SNAT [-s from.where or -d to.where]\ > --to-source source.addr.of.eth1Ditto on the transposition of -j and -t. But these two iptables rules conflict with each other. If -s "from.where" is my internal lan and the same in both rules, they are both trying to do the SNATting of the same packets. In my two rules, I added a -o <iface> (where <iface> is the interface matching the source.addr.of.<iface>).> > 2) two routing tables, like > ip route add default via eth0.gateway.ip.address dev eth0 table 1got it: ip route add 0/0 via 66.11.190.1 dev ppp0 table 1> ip route add default via eth1.gateway.ip.address dev eth1 table 2got it: ip route add 0/0 via 24.235.240.1 dev eth1 table 2> maybe you dont even need the "via xx" thing, the dev xxx is enough. > > then you can classify packets to use the connection you want using > ip rule add WHATEVER lookup N (whatever could be "to x.x.x.x" or "from > x.x.x.x", same as in the SNAT example, N could be 1 or 2) > > if you want the router to respond to packets correcty (ie. to answer > ping on both interfaces) you need to > ip rule add iif eth0 lookup 1 > ip rule add iif eth1 lookup 2I have: ip rule add from 66.11.173.224 lookup 1 ip rule add from 24.235.240.15 lookup 2 what is "iif" in your above examples? I don''t see an "iif" syntax when I do "ip rule help". I get: Usage: ip rule [ list | add | del ] SELECTOR ACTION SELECTOR := [ from PREFIX ] [ to PREFIX ] [ tos TOS ] [ fwmark FWMARK ] [ dev STRING ] [ pref NUMBER ] ACTION := [ table TABLE_ID ] [ nat ADDRESS ] [ prohibit | reject | unreachable ] [ realms [SRCREALM/]DSTREALM ] TABLE_ID := [ local | main | default | NUMBER ] Thanx much for your input! b.
On Fri, Nov 26, 2004 at 02:41:26PM -0200, James Lista wrote:> please, what do those 2 following lines do ? mainly that "drop flowid :1" > ? > > tc qdisc add dev $WAN handle ffff: ingresstells the kernel that it should be ready to do ingress filtering on device $WAN (well, actually it is called policing not filtering, because ingress'' functionality is limited)> tc filter add dev $WAN parent ffff: protocol ip prio 50 u32 match ip src > 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1the aforementioned ingress policy should match all ip packets, rate limit them to ${DOWNLINK} kbit/s with a 10k burst, and overlimits should be dropped. The flowid :1 is in my guess necessary in order for "tc" to accept the filter, but doesn''t have any practical effect because ingress can''t do classful stuff. Bye, Peter Surda (Shurdeek) <shurdeek@routehat.org>, ICQ 10236103, +436505122023 -- Press every key to continue. _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
diab
2004-Nov-26 17:25 UTC
Re[2]: simple dual Internet connection setup not sending return packets on correct interface
>> 1) SNAT to the right source address, like >> iptables -A POSTROUTING -j nat -t SNAT [-s from.where or -d to.where]\ >> --to-source source.addr.of.eth0BJM> Surely you mean -t nat -j SNAT? sorry, yeah it was -t nat -j SNAT.. i double checked now :) BJM> But these two iptables rules conflict with each other. If -s BJM> "from.where" is my internal lan and the same in both rules, they are BJM> both trying to do the SNATting of the same packets. In my two rules, I BJM> added a -o <iface> (where <iface> is the interface matching the BJM> source.addr.of.<iface>). yes they are conflicting with each other.. i thought that you could select which connection the packets should be using either based on the address the packets are coming FROM (-s some.ip.on.the.lan) or going TO (-d wan.destination.address.). BJM> I have: BJM> ip rule add from 66.11.173.224 lookup 1 BJM> ip rule add from 24.235.240.15 lookup 2 anyways, you can "name" routing tables in /etc/iproute2/rt_tables then it makes a bit more sense ("ip rule" also displays/uses them so eg. if you name 1 to "abcd" then ip rule add from x.x.x.x lookup abcd also works. iif is the interface packets are coming in (there is also oif).. if it''s not a static ip address it might be convenient not having to use the IP of the connection but the interface. (same goes for the "via XX when you are doing "ip route add default dev XY table N") if you do "man ip" it reads (ip rule add/ip rule del): iif NAME select the incoming device to match. If the interface is loopback, the rule only matches packets originating from this host. This means that you may create separate routing tables for forwarded and local packets and, hence, com pletely segregate them. - diab _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Brian J. Murrell
2004-Nov-26 17:45 UTC
Re: simple dual Internet connection setup not sending return packets on correct interface
On Fri, 2004-11-26 at 18:25 +0100, diab wrote:> yes they are conflicting with each other.. i thought that you could > select which connection the packets should be using either based on > the address the packets are coming FROM (-s some.ip.on.the.lan) or > going TO (-d wan.destination.address.).No. The problem is that outbound reply packets (i.e. a SYN-ACK packet) to incoming packets (i.e. SYN) are being NATted correctly (i.e. they have the correct source address) they are just not being put on the right interface. They are being put on the interface of the default route in the main routing table.> iif is the interface packets are coming in (there is also oif).. if > it''s not a static ip address it might be convenient not having to use > the IP of the connection but the interface. (same goes for the "via > XX when you are doing "ip route add default dev XY table N") > > if you do "man ip" it reads (ip rule add/ip rule del):~sigh~ My man page for "ip" says only: NAME ip - TCP/IP interface configuration and routing utility SYNTAX ip DESCRIPTION This utility allows you to configure your network interfaces in various ways. OPTIONS For the complete command reference please look at the following docu- ment: /usr/share/doc/iproute-2.4.7/ip-cref.ps SEE ALSO ifconfig(8), route(8), netstat(8), arp(8), rarp(8), ipchains(8) AUTHORS Alexey Kuznetsov <kuznet@ms2.inr.ac.ru> and no "/usr/share/doc/iproute-2.4.7/ip-cref.ps" exists.> iif NAME > select the incoming device to match. If the interface is > loopback, the rule only matches packets originating from > this host. This means that you may create separate routing tables for > forwarded and local packets and, hence, com > pletely segregate them.OK. But I don''t know the device to use. That is the *whole point* of the ip rule add (from <iface address> lookup <table>) isn''t it? To select the routing table (and therefore the outbound device) to send the return packets on. Maybe I am completely missing something in your explanation. b.
gypsy
2004-Nov-26 21:27 UTC
Re: simple dual Internet connection setup not sendingreturn packetson correct interface
"Brian J. Murrell" wrote: I could be way off base here, but I (obviously) don''t think so!> On Thu, 2004-11-25 at 21:40 -0800, gypsy wrote: > > > > Guessing from the lack of any mention of KeepState > > KeepState? If you are referring to: > > 52459 2774K ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED > > rules, I have those sprinkled throughout my ruleset where necessary. > The iptables "snippet" I included in my previous message was just that. > Just the relevant portion that does the NATting.No, I am referring to http://www.ssi.bg/~ja/nano.txt which is a MUST READ for you, like it or not.> > in your iptables > > setup, > > Like I said, the RELATED,ESTABLISHED state rules are in there. My full > set of iptables rules is >400. I did not see see a need to post that > fully here. > > > my guess is that you ignored the advice to vist Julian > > Anastasov''s web site. > > No I didn''t ignore it. But what that site is promoting is some kind of > floppy disk based router distribution or something.There is a lot of stuff on Julian''s site, but I see nothing referring to a floppy disk based router. Use the nano.txt URL above.> > Start with this: > > http://www.geocities.com/mctiew/ffw/dual.htm > > I am not looking to replace/rebuild my whole firewall. I simply want to > add a second link to my existing one and have the packets use the > correct interface -- to travel back out the interface from which they > came.That is not why I sent you there...> I don''t want to do load balancing or failover or anything fancy. I want > two interfaces where I use one for all outgoing traffic and the only > time the alternate is used is to send response packets to connections > that come _in_ that interface or for routes that are specifically > directed through that interface via a routing table entry.Yes, you DO want something fancy.> > You should also google LARTC "Finally: A working case of two adsl load > > balance". Read Ron Senykoff''s post "load balance a file download across > > two connections - success!". > > Interesting. Followed a few links too. Looks like a lot of bells and > whistles I am not really looking for (load balancing and failover, etc.) > but there is some hint of indication that there is a patch needed to > make sure NAT uses the right physical interface. Maybe I will go bug > the netfilter guys to see if this is the case.Perhaps you are not looking for bells and whistles, but you certainly need correct routing tables. Netfilter has nothing to do with what you need except that which is contained in nano.txt.> Thanx, > b.gypsy _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
James Lista wrote:> > please, what do those 2 following lines do ? mainly that "drop flowid :1" > ? > > tc qdisc add dev $WAN handle ffff: ingress > tc filter add dev $WAN parent ffff: protocol ip prio 50 u32 match ip src > 0.0.0.0/0 police rate ${DOWNLINK}kbit burst 10k drop flowid :1They rate limit incoming packets. "u32 match ip src 0.0.0.0/0" says to match everything. "police rate ${DOWNLINK}kbit burst 10k" says to rate limit at DOWNLINK kbits per second, but to accept (fairly large) bursts of data up to 10K bytes. "drop flowid :1" says to drop all packets in excess of the police rate. HTH gypsy _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/