I have a network much like that show in:
<http://www.lartc.org/howto/lartc.rpdb.multiple-links.html>
with a few wrinkles. This is with a ptrace patched 2.4.20 kernel.
The wrinkles are that one interface has 14 IPs (.2 to .15) total (via
eth1 and aliasing), the other being a single DHCP-provided IP (eth0).
I have a iptables rule:
/sbin/iptables -t nat -A PREROUTING -i eth1 -d eee.fff.ggg.11 -p tcp --dport
22 -j DNAT --to 192.168.0.2:22
That''s all there is (currently) in the PREROUTING iptable.
POSTROUTING has:
/sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
/sbin/iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
When I connect from the outside to eee.fff.ggg.11:22, I never hear a
reply. Ethereal (on the internal interface, eth2) shows replies, but
looking on the other external interface (eth0) it appears that the
replies are all shooting out of it (the default path). Nothing ever
returns out the eth1 interface as it "should". I have the ip rules
set up as follows:
# ip rule show
0: from all lookup local
32757: from aaa.bbb.ccc.ddd lookup T1
32758: from eee.fff.ggg.2/31 lookup T2 # note the 3 rules are to cover
32759: from eee.fff.ggg.4/30 lookup T2 # the .2 to .15 IP space
32760: from eee.fff.ggg.8/29 lookup T2
32766: from all lookup main
32767: from all lookup default
# ip route show table T1
192.168.0.0/24 dev eth2 scope link
default via aaa.bbb.ccc.1 dev eth0
# ip route show table T2
eee.fff.ggg.1 dev eth1 scope link
eee.fff.ggg.0/28 dev lo scope link
192.168.0.0/24 dev eth2 scope link
default via eee.fff.ggg.1 dev eth1
x# ip route # (default rule)
eee.fff.ggg.1 dev eth1 scope link
eee.fff.ggg.0/28 dev lo scope link
192.168.0.0/24 dev eth2 proto kernel scope link src 192.168.0.1
eee.fff.ggg.0/24 dev eth1 proto kernel scope link src eee.fff.ggg.10
aaa.bbb.ccc.0/23 dev eth0 proto kernel scope link src aaa.bbb.ccc.ddd
default via aaa.bbb.ccc.1 dev eth0
When I connect to the port forwarded address from the outside, it
looks like the returning packets are getting routed _before_ the
source IP is translated (and thus aren''t matching a special rule and
thus get routed according to the default rule). Everything else seems
to be working fine.
Has anyone seen this? Is it a bug or am I just confused?
--
Russell Senior ``I''ve seen every kind of critter God ever made,
seniorr@aracnet.com and I ain''t never seen a meaner, lower, more
stinkin'' yellow hypocrite than
you!''''
-- Burl Ives as Rufus Hennessy
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
In my attempt to figure out what is going haywire, I compiled the
NETFILTER debugging into a kernel and booted it, but I haven''t been
able to figure out where the messages go! I am not seeing anything in
/var/log/kern.log. Hints anyone?
--
Russell Senior ``I''ve seen every kind of critter God ever made,
seniorr@aracnet.com and I ain''t never seen a meaner, lower, more
stinkin'' yellow hypocrite than
you!''''
-- Burl Ives as Rufus Hennessy
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Martin A. Brown
2003-Apr-12 19:08 UTC
Re: multiple uplinks/iptables -t nat -PREROUTING funny
Russell, : I have a network much like that show in: : : <http://www.lartc.org/howto/lartc.rpdb.multiple-links.html> : : with a few wrinkles. This is with a ptrace patched 2.4.20 kernel. : : The wrinkles are that one interface has 14 IPs (.2 to .15) total (via : eth1 and aliasing), the other being a single DHCP-provided IP (eth0). As a side note, after reading your post, Russell, I''m a touch confused why you have so many IPs hosted on your eth1 interface. It strikes me that you really just want packets to arrive (inbound) on eth1. So, you really don''t need to have all these IPs locally hosted on your box. You can simply proxy ARP for the IPs, and ethernet frames with IP packets bound for e.f.g.2-15 will arrive on your eth1. Perhaps you are doing more than that with these 14 IP addresses, but if not, why bother hosting the IPs locally? Suggestion: # for h in $( seq 2 15 ) ; do> arp -s e.f.g.$h -i eth1 -D eth1 pub > donePurely out of curiousity, I wonder if you are doing something else with these IPs? [ snipped iptables rules that look fine ] : When I connect from the outside to eee.fff.ggg.11:22, I never hear a : reply. Ethereal (on the internal interface, eth2) shows replies, but : looking on the other external interface (eth0) it appears that the : replies are all shooting out of it (the default path). Nothing ever : returns out the eth1 interface as it "should". I have the ip rules : set up as follows: [ thank you for the "ip route" and "ip rule" output, which has been snipped to save space ] : When I connect to the port forwarded address from the outside, it : looks like the returning packets are getting routed _before_ the : source IP is translated (and thus aren''t matching a special rule and : thus get routed according to the default rule). Everything else seems : to be working fine. : : Has anyone seen this? Is it a bug or am I just confused? This is not a bug--this is a fact of packet flow through the kernel. See the kernel packet traveling diagram (KPTD) [1] for more details on the sequence of operations. So to answer your question: you must be confused! :) You should try adding just one more rule: # ip rule add from 192.168.0.2 table T2 For a more thorough analysis of this interaction between routing and netfilter, see my advanced routing with multiple Internet connections chapter [2] and/or this informative post from Wes Hodges [3]. -Martin [1] http://www.docum.org/stef.coene/qos/kptd/ [2] http://linux-ip.net/html/adv-multi-internet.html [3] http://lists.netfilter.org/pipermail/netfilter/2001-May/011697.html -- Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Russell Senior
2003-Apr-12 20:04 UTC
Re: multiple uplinks/iptables -t nat -PREROUTING funny
>>>>> "Martin" == Martin A Brown <mabrown-lartc@securepipe.com> writes:Russell> When I connect to the port forwarded address from the Russell> outside, it looks like the returning packets are getting Russell> routed _before_ the source IP is translated (and thus aren''t Russell> matching a special rule and thus get routed according to the Russell> default rule). Everything else seems to be working fine. Russell> Has anyone seen this? Is it a bug or am I just confused? Martin> This is not a bug--this is a fact of packet flow through the Martin> kernel. See the kernel packet traveling diagram (KPTD) [1] Martin> for more details on the sequence of operations. So to answer Martin> your question: you must be confused! :) What that very nice diagram doesn''t show is how the reply packets to DNAT''d connections are handled. The prima facie evidence seems to be that DNAT was in the PREROUTING iptable and "consequently" the reverse translation should occur before routing. That is the source of my confusion. Martin> You should try adding just one more rule: Martin> # ip rule add from 192.168.0.2 table T2 That would "work", but it is kind of messy. What if I have a second DNAT from IF1 that also forwards to 192.168.0.2? It would get complicated in a hurry. All would be solved if the reverse translation just occurred in PREROUTING as seems like it "should". I don''t understand yet why it doesn''t. Perhaps there is a good reason that I just don''t see. Or, maybe there isn''t a good reason, and it should be "fixed". Too soon for me to say. If anyone can point me at some detailed documentation on DNAT or even the relevant bits of the source code, I''d really appreciate it! -- Russell Senior ``I''ve seen every kind of critter God ever made, seniorr@aracnet.com and I ain''t never seen a meaner, lower, more stinkin'' yellow hypocrite than you!'''' -- Burl Ives as Rufus Hennessy _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On Saturday 12 April 2003 02:39, Russell Senior scrawled:> In my attempt to figure out what is going haywire, I compiled the > NETFILTER debugging into a kernel and booted it, but I haven''t been > able to figure out where the messages go! I am not seeing anything in > /var/log/kern.log. Hints anyone?How about /var/log/message? My syslog.conf is set to log *.info (which I suspect NETFILTER debug to be using) messages in /var/log/messages. Ashok - -- - ----------------------------------------------------------------------------- My public key: gpg --recv-keys --keyserver blackhole.pca.dfn.de DCB44F2E http://pgp.mit.edu:11371/pks/lookup?search=N+N+Ashok+%28CSE+MSU%29+%3Cnnashok%40yahoo.com%3E&op=index [Fingerprint: 4D6D B9F3 D2B6 A22A F4E5 5763 4615 E956 DCB4 4F2E ] - ----------------------------------------------------------------------------- "...there is nothing so unnatural as the commonplace." Sir Arthur Conan Doyle in "Adventures of Sherlock Holmes: A Case of Identity" - ----------------------------------------------------------------------------- -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.0.7 (GNU/Linux) iD8DBQE+mQ8jRhXpVty0Ty4RAjYjAKDREmUc3UzrRrM3xoIS69+TjNDrPgCeIEpB M8UxOmYA92sY26KEyoS2d/M=Y24t -----END PGP SIGNATURE----- _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
>>>>> "N" == N N Ashok <nalkunda@cse.msu.edu> writes:Russell> In my attempt to figure out what is going haywire, I compiled Russell> the NETFILTER debugging into a kernel and booted it, but I Russell> haven''t been able to figure out where the messages go! I am Russell> not seeing anything in /var/log/kern.log. Hints anyone? N> How about /var/log/message? My syslog.conf is set to log *.info N> (which I suspect NETFILTER debug to be using) messages in N> /var/log/messages. Nope. I''ve got two that I would think would catch them: kern.* -/var/log/kern.log kern.debug -/var/log/kern.debug but nothing related is showing up. A chronological sorting "ls -lart" in /var/log shows what is relatively active and none of those is showing anything recognizable as netfilter debugging. Is there a /proc/sys/<mumble> switch I need to turn on? -- Russell Senior ``I''ve seen every kind of critter God ever made, seniorr@aracnet.com and I ain''t never seen a meaner, lower, more stinkin'' yellow hypocrite than you!'''' -- Burl Ives as Rufus Hennessy _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Russell Senior
2003-Apr-20 07:42 UTC
Re: multiple uplinks/iptables -t nat -PREROUTING funny
>>>>> "Russell" == Russell Senior <seniorr@aracnet.com> writes:>>>>> "Martin" == Martin A Brown <mabrown-lartc@securepipe.com> writes:Russell> When I connect to the port forwarded address from the Russell> outside, it looks like the returning packets are getting Russell> routed _before_ the source IP is translated (and thus aren''t Russell> matching a special rule and thus get routed according to the Russell> default rule). Everything else seems to be working fine. Russell> Has anyone seen this? Is it a bug or am I just confused? Martin> This is not a bug--this is a fact of packet flow through the Martin> kernel. See the kernel packet traveling diagram (KPTD) [1] Martin> for more details on the sequence of operations. So to answer Martin> your question: you must be confused! :) Russell> What that very nice diagram doesn''t show is how the reply Russell> packets to DNAT''d connections are handled. The prima facie Russell> evidence seems to be that DNAT was in the PREROUTING iptable Russell> and "consequently" the reverse translation should occur Russell> before routing. That is the source of my confusion. Martin> You should try adding just one more rule: Martin> # ip rule add from 192.168.0.2 table T2 Russell> That would "work", but it is kind of messy. What if I have a Russell> second DNAT from IF1 that also forwards to 192.168.0.2? It Russell> would get complicated in a hurry. Russell> All would be solved if the reverse translation just occurred Russell> in PREROUTING as seems like it "should". I don''t understand Russell> yet why it doesn''t. Perhaps there is a good reason that I Russell> just don''t see. Or, maybe there isn''t a good reason, and it Russell> should be "fixed". Too soon for me to say. Just as a followup to this: a relatively clean solution is to mark the reply packets: iptables -t mangle -I PREROUTING -m conntrack --ctstate DNAT --ctorigdst eee.fff.ggg.11 -j MARK --set-mark 2 ip rule add fwmark 2 table T2 and make sure that rp_filter is appropriately off: echo 0 > /proc/sys/net/ipv4/conf/<if>/rp_filter -- Russell Senior ``I''ve seen every kind of critter God ever made, seniorr@aracnet.com and I ain''t never seen a meaner, lower, more stinkin'' yellow hypocrite than you!'''' -- Burl Ives as Rufus Hennessy _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/