I have following setup:
- eth0 is an internal network
- eth1 is an Internet connection (IP = 1.1.1.128, GW=1.1.1.1)
- eth2 is another Internet connection (IP = 2.2.2.128, GW=2.2.2.1)
I would like to masquerade port 80 through eth2, but all other traffic
should be masq''ed through eth1.
My routing configuration:
(default route in main table is 1.1.1.1)
ip rule add fwmark 2 pref 1002 table 666
ip route flush table 666
ip route add default via 2.2.2.1 dev eth3 proto static table 666
ip route flush cache
My firewall configuration:
iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK
--set-mark 2
iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128
iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128
Unfortunately, this does not work. Outgoing packets are fine. Incoming
packets on port 80 are not de-masqueraded and do not reach the internal
hosts.
Also, if I change the ip rule above to be based on the source address
(instead of a mark), connections start working fine.
Here is the output of ''ip rule ls'', to prove that I do have
fwmark compiled:
0: from all lookup local
1002: from all fwmark 2 lookup http
32766: from all lookup main
32767: from all lookup 253
I am wondering if there is some kind of bug related to the interaction
between fwmark and NAT. Any ideas?
Thanks,
Miron Cuperman
This doesn''t seem right:> My firewall configuration: > iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK > --set-mark 2 > iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128 > iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128I''m not the expert, but I don''t think you want to SNAT. You''re marking inbound packets with destination port 80 and then putting rules in the POSTROUTING table to fudge in a different IP address for outbound packets. That doesn''t seem right. It doesn''t redirect inbound packets the way you want. I think you want to DNAT instead of SNAT and forget about marking packets. Set up some PREROUTING rules and DNAT all incoming port 80 stuff over to the interface you want. That should be all you need to do because the connection tracking should take care of getting the reply packets from your internal web server back to where they belong. Verify this with the experts before you do it, but I think I''m right on this one. - Greg Scott -----Original Message----- From: Miron [mailto:miron@hyper.to] Sent: Thursday, December 06, 2001 3:58 AM To: lartc@mailman.ds9a.nl Subject: [LARTC] Masq/route based on port I have following setup: - eth0 is an internal network - eth1 is an Internet connection (IP = 1.1.1.128, GW=1.1.1.1) - eth2 is another Internet connection (IP = 2.2.2.128, GW=2.2.2.1) I would like to masquerade port 80 through eth2, but all other traffic should be masq''ed through eth1. My routing configuration: (default route in main table is 1.1.1.1) ip rule add fwmark 2 pref 1002 table 666 ip route flush table 666 ip route add default via 2.2.2.1 dev eth3 proto static table 666 ip route flush cache My firewall configuration: iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK --set-mark 2 iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128 iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128 Unfortunately, this does not work. Outgoing packets are fine. Incoming packets on port 80 are not de-masqueraded and do not reach the internal hosts. Also, if I change the ip rule above to be based on the source address (instead of a mark), connections start working fine. Here is the output of ''ip rule ls'', to prove that I do have fwmark compiled: 0: from all lookup local 1002: from all fwmark 2 lookup http 32766: from all lookup main 32767: from all lookup 253 I am wondering if there is some kind of bug related to the interaction between fwmark and NAT. Any ideas? Thanks, Miron Cuperman _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/
This is a home setup, not a server setup. We have no servers on our network. The reason we want port 80 on eth2 is because eth2 has more download bandwidth. For other protocols we want eth1, because it has more symmetric bandwidth. Greg Scott wrote:>This doesn''t seem right: > >>My firewall configuration: >> iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK >>--set-mark 2 >> iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128 >> iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128 >> > >I''m not the expert, but I don''t think you want to SNAT. > >You''re marking inbound packets with destination port 80 and then putting >rules in the POSTROUTING table to fudge in a different IP address for >outbound packets. That doesn''t seem right. It doesn''t redirect inbound >packets the way you want. > >I think you want to DNAT instead of SNAT and forget about marking packets. >Set up some PREROUTING rules and DNAT all incoming port 80 stuff over to >the interface you want. That should be all you need to do because the >connection tracking should take care of getting the reply packets from your >internal web server back to where they belong. > >Verify this with the experts before you do it, but I think I''m right on >this one. > >- Greg Scott > > > >-----Original Message----- >From: Miron [mailto:miron@hyper.to] >Sent: Thursday, December 06, 2001 3:58 AM >To: lartc@mailman.ds9a.nl >Subject: [LARTC] Masq/route based on port > > >I have following setup: > >- eth0 is an internal network >- eth1 is an Internet connection (IP = 1.1.1.128, GW=1.1.1.1) >- eth2 is another Internet connection (IP = 2.2.2.128, GW=2.2.2.1) > >I would like to masquerade port 80 through eth2, but all other traffic >should be masq''ed through eth1. > >My routing configuration: > > (default route in main table is 1.1.1.1) > > ip rule add fwmark 2 pref 1002 table 666 > > ip route flush table 666 > ip route add default via 2.2.2.1 dev eth3 proto static table 666 > ip route flush cache > >My firewall configuration: > iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK >--set-mark 2 > iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128 > iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128 > >Unfortunately, this does not work. Outgoing packets are fine. Incoming >packets on port 80 are not de-masqueraded and do not reach the internal >hosts. > >Also, if I change the ip rule above to be based on the source address >(instead of a mark), connections start working fine. > >Here is the output of ''ip rule ls'', to prove that I do have fwmark compiled: > 0: from all lookup local > 1002: from all fwmark 2 lookup http > 32766: from all lookup main > 32767: from all lookup 253 > >I am wondering if there is some kind of bug related to the interaction >between fwmark and NAT. Any ideas? > >Thanks, >Miron Cuperman > > > >_______________________________________________ >LARTC mailing list / LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: >http://ds9a.nl/2.4Routing/ > >_______________________________________________ >LARTC mailing list / LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/ >
> This is a home setup, not a server setup. We have no servers on our > network. The reason we want port 80 on eth2 is because eth2 has more > download bandwidth. For other protocols we want eth1, because it has > more symmetric bandwidth.So anything that comes in from the Internet for port 80, no matter the source, you want the reply to go back out on ETH2. And anything that comes in from the Internet other than port 80, you want those replies to go out ETH1. So the web server process is inside your Linux box? Did I get that much right? Or do I have it backwards? The Linux box is your internal LAN''s default gateway and you want this box to decide which Internet interface to use, based on the destination port your internal client PCs choose? Hadn''t thought about it that way before. - Greg -----Original Message----- From: Miron [mailto:miron@hyper.to] Sent: Thursday, December 06, 2001 1:03 PM To: Greg Scott Cc: lartc@mailman.ds9a.nl Subject: Re: [LARTC] Masq/route based on port>I have following setup: > >- eth0 is an internal network >- eth1 is an Internet connection (IP = 1.1.1.128, GW=1.1.1.1) >- eth2 is another Internet connection (IP = 2.2.2.128, GW=2.2.2.1) > >I would like to masquerade port 80 through eth2, but all other traffic >should be masq''ed through eth1. > >My routing configuration: > > (default route in main table is 1.1.1.1) > > ip rule add fwmark 2 pref 1002 table 666 > > ip route flush table 666 > ip route add default via 2.2.2.1 dev eth3 proto static table 666 > ip route flush cache > >My firewall configuration: > iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK >--set-mark 2 > iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128 > iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128 > >Unfortunately, this does not work. Outgoing packets are fine. Incoming >packets on port 80 are not de-masqueraded and do not reach the internal >hosts. > >Also, if I change the ip rule above to be based on the source address >(instead of a mark), connections start working fine. > >Here is the output of ''ip rule ls'', to prove that I do have fwmarkcompiled:> 0: from all lookup local > 1002: from all fwmark 2 lookup http > 32766: from all lookup main > 32767: from all lookup 253 > >I am wondering if there is some kind of bug related to the interaction >between fwmark and NAT. Any ideas? > >Thanks, >Miron Cuperman
I think we are on the same wavelength, except maybe exactly backwards. ;) I''m trying to provide fast web browsing for clients that are on the internal network (talking to servers on the Internet). Anything from the Internet with source port 80 should go through eth2. Anything else should go through eth1. I did some more investigation, and it seems that the Linux box is dropping all incoming packets arriving into eth2. I can see packets with tcpdump, but the applications and NAT don''t actually get it. It must be getting dropped in the kernel. It seems that this has something to do with the default route pointing to eth1. I don''t understand why the fact that the default route is not pointing to an interface should cause *incoming* packets to be dropped on that interface. Sorry if my post was confusing. Greg Scott wrote:>>This is a home setup, not a server setup. We have no servers on our >>network. The reason we want port 80 on eth2 is because eth2 has more >>download bandwidth. For other protocols we want eth1, because it has >>more symmetric bandwidth. >> > > >So anything that comes in from the Internet for port 80, no matter the >source, you want the reply to go back out on ETH2. And anything that >comes in from the Internet other than port 80, you want those replies to >go out ETH1. So the web server process is inside your Linux box? > >Did I get that much right? Or do I have it backwards? > >The Linux box is your internal LAN''s default gateway and you want this >box to decide which Internet interface to use, based on the destination >port your internal client PCs choose? > >Hadn''t thought about it that way before. > >- Greg > > > > >-----Original Message----- >From: Miron [mailto:miron@hyper.to] >Sent: Thursday, December 06, 2001 1:03 PM >To: Greg Scott >Cc: lartc@mailman.ds9a.nl >Subject: Re: [LARTC] Masq/route based on port > > > >>I have following setup: >> >>- eth0 is an internal network >>- eth1 is an Internet connection (IP = 1.1.1.128, GW=1.1.1.1) >>- eth2 is another Internet connection (IP = 2.2.2.128, GW=2.2.2.1) >> >>I would like to masquerade port 80 through eth2, but all other traffic >>should be masq''ed through eth1. >> >>My routing configuration: >> >> (default route in main table is 1.1.1.1) >> >> ip rule add fwmark 2 pref 1002 table 666 >> >> ip route flush table 666 >> ip route add default via 2.2.2.1 dev eth3 proto static table 666 >> ip route flush cache >> >>My firewall configuration: >> iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK >>--set-mark 2 >> iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128 >> iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128 >> >>Unfortunately, this does not work. Outgoing packets are fine. Incoming >>packets on port 80 are not de-masqueraded and do not reach the internal >>hosts. >> >>Also, if I change the ip rule above to be based on the source address >>(instead of a mark), connections start working fine. >> >>Here is the output of ''ip rule ls'', to prove that I do have fwmark >> >compiled: > >> 0: from all lookup local >> 1002: from all fwmark 2 lookup http >> 32766: from all lookup main >> 32767: from all lookup 253 >> >>I am wondering if there is some kind of bug related to the interaction >>between fwmark and NAT. Any ideas? >> >>Thanks, >>Miron Cuperman >> > >_______________________________________________ >LARTC mailing list / LARTC@mailman.ds9a.nl >http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/ >
Okay, I figured it out. The secret is to set rp_filter to zero on the non-default interface. I guess the kernel thought that the packets were source-routed because of the fact that the interface received packets it could not route. That was frustrating, yet fun... ;) Miron wrote:> I think we are on the same wavelength, except maybe exactly backwards. ;) > > I''m trying to provide fast web browsing for clients that are on the > internal network (talking to servers on the Internet). Anything from > the Internet with source port 80 should go through eth2. Anything > else should go through eth1. > > I did some more investigation, and it seems that the Linux box is > dropping all incoming packets arriving into eth2. I can see packets > with tcpdump, but the applications and NAT don''t actually get it. It > must be getting dropped in the kernel. It seems that this has > something to do with the default route pointing to eth1. I don''t > understand why the fact that the default route is not pointing to an > interface should cause *incoming* packets to be dropped on that > interface. > > Sorry if my post was confusing. > > Greg Scott wrote: > >>> This is a home setup, not a server setup. We have no servers on our >>> network. The reason we want port 80 on eth2 is because eth2 has >>> more download bandwidth. For other protocols we want eth1, because >>> it has more symmetric bandwidth. >>> >> >> >> So anything that comes in from the Internet for port 80, no matter >> the source, you want the reply to go back out on ETH2. And anything >> that >> comes in from the Internet other than port 80, you want those replies to >> go out ETH1. So the web server process is inside your Linux box? >> Did I get that much right? Or do I have it backwards? >> The Linux box is your internal LAN''s default gateway and you want >> this box to decide which Internet interface to use, based on the >> destination port your internal client PCs choose? >> Hadn''t thought about it that way before. >> >> - Greg >> >> >> >> >> -----Original Message----- >> From: Miron [mailto:miron@hyper.to] >> Sent: Thursday, December 06, 2001 1:03 PM >> To: Greg Scott >> Cc: lartc@mailman.ds9a.nl >> Subject: Re: [LARTC] Masq/route based on port >> >> >> >>> I have following setup: >>> >>> - eth0 is an internal network >>> - eth1 is an Internet connection (IP = 1.1.1.128, GW=1.1.1.1) >>> - eth2 is another Internet connection (IP = 2.2.2.128, GW=2.2.2.1) >>> >>> I would like to masquerade port 80 through eth2, but all other >>> traffic should be masq''ed through eth1. >>> >>> My routing configuration: >>> >>> (default route in main table is 1.1.1.1) >>> >>> ip rule add fwmark 2 pref 1002 table 666 >>> >>> ip route flush table 666 >>> ip route add default via 2.2.2.1 dev eth3 proto static table 666 >>> ip route flush cache >>> >>> My firewall configuration: >>> iptables -t mangle -A PREROUTING -p tcp -m tcp --dport 80 -j MARK >>> --set-mark 2 >>> iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to-source 1.1.1.128 >>> iptables -t nat -A POSTROUTING -o eth2 -j SNAT --to-source 2.2.2.128 >>> >>> Unfortunately, this does not work. Outgoing packets are fine. >>> Incoming packets on port 80 are not de-masqueraded and do not reach >>> the internal hosts. >>> >>> Also, if I change the ip rule above to be based on the source >>> address (instead of a mark), connections start working fine. >>> >>> Here is the output of ''ip rule ls'', to prove that I do have fwmark >>> >> compiled: >> >>> 0: from all lookup local >>> 1002: from all fwmark 2 lookup http >>> 32766: from all lookup main >>> 32767: from all lookup 253 >>> >>> I am wondering if there is some kind of bug related to the >>> interaction between fwmark and NAT. Any ideas? >>> >>> Thanks, >>> Miron Cuperman >>>