Hi, I''m currently trying to get a Linux machine to route all traffic coming from a certain UID over a dedicated PPP interface. After going throught the available documentation and experimenting a bit, I settled for the following attempt: # 62.46.87.104 - local PPP address # 195.4.91.104 - PPP peer ip route add 195.3.91.104 dev ppp0 src 62.46.87.104 table special # local for DNS etc ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table special ip route add default via 195.3.91.104 src 62.46.87.104 table special ip rule add fwmark 3 lookup special iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK --set-mark 3 ip route flush cache This seems to work in a way. It correctly sends the packets generated by that user out the ppp0 interface, but they don''t get the correct source address: | sophokles:~# sh -x description.txt | + ip route flush table aonc | + ip route add 195.3.91.103 dev ppp0 src 62.46.86.137 table aonc | + ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table aonc | + ip route add default via 195.3.91.103 src 62.46.86.137 table aonc | + ip rule add fwmark 3 lookup aonc | + iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK | --set-mark 3 | + ip route flush cache | sophokles:~# tcpdump -ni ppp0 port 22 & | [1] 841 | sophokles:~# tcpdump: listening on ppp0 | | sophokles:~# nc iwoars.net 22 | SSH-1.99-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3 | | sophokles:~# su - freenet | freenet@sophokles:~$ nc iwoars.net 22 | 01:25:17.044883 192.168.1.1.32848 > 217.160.110.113.22: SWE | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2056533 | 0,nop,wscale 0> (DF) | 01:25:20.043828 192.168.1.1.32848 > 217.160.110.113.22: SWE | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2059533 | 0,nop,wscale 0> (DF) | 01:25:26.042584 192.168.1.1.32848 > 217.160.110.113.22: SWE | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2065533 | 0,nop,wscale 0> (DF) | | freenet@sophokles:~$ I''ve read on this list that owner-based policy routing is impossible because the routing decision is made before the packet traverses the OUTPUT chain. However, if this is true, then I don''t understand how the packet can go out via the correct interface unless there are separate route lookups to determine the source address and outgoing interface. Could someone who knows please elaborate? I have also tried, unsuccessfully, to just mangle the source address using an iptables SNAT rule, but even though that produces correct network traffic, it seems to break something internally that keeps the TCP handshake from completing: | sophokles:~# iptables -t nat -A POSTROUTING -j SNAT -o ppp0 --to-source | 62.46.86.137 | sophokles:~# su - freenet | freenet@sophokles:~$ nc iwoars.net 22 | 01:30:16.448930 62.46.86.137.32849 > 217.160.110.113.22: SWE | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2356000 | 0,nop,wscale 0> (DF) | 01:30:16.516732 217.160.110.113.22 > 62.46.86.137.32849: S | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss | 1460,sackOK,timestamp 313375234 2356000,nop,wscale 0> (DF) | 01:30:19.448146 62.46.86.137.32849 > 217.160.110.113.22: SWE | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2359000 | 0,nop,wscale 0> (DF) | 01:30:19.518099 217.160.110.113.22 > 62.46.86.137.32849: S | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss | 1460,sackOK,timestamp 313375535 2356000,nop,wscale 0> (DF) | 01:30:19.823023 217.160.110.113.22 > 62.46.86.137.32849: S | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss | 1460,sackOK,timestamp 313375566 2356000,nop,wscale 0> (DF) | [...] ciao, -- [*Thomas Themel*] Since you obviously haven''t lurked enough, [extended contact] we subtly tell lusers to piss off via the words "piss [info provided in] off, luser". [*message header*] - David P. Murphy in the monastery
Hi, christopher cuse (ccuse@manchotnetworks.net) wrote on 2003-10-28:> i had the same issue ... try adding to your iptables: > > iptables --append POSTROUTING --table nat --match mark \ > --mark 0x3 --jump SNAT --to-source 62.46.87.104 > > ip route flush cacheI already tried that (though with a different match):> > I have also tried, unsuccessfully, to just mangle the source address > > using an iptables SNAT rule, but even though that produces correct > > network traffic, it seems to break something internally that keeps the > > TCP handshake from completing: > > > > | sophokles:~# iptables -t nat -A POSTROUTING -j SNAT -o ppp0 --to-source > > | 62.46.86.137 > > | sophokles:~# su - freenet > > | freenet@sophokles:~$ nc iwoars.net 22 > > | 01:30:16.448930 62.46.86.137.32849 > 217.160.110.113.22: SWE > > | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2356000 > > | 0,nop,wscale 0> (DF) > > | 01:30:16.516732 217.160.110.113.22 > 62.46.86.137.32849: S > > | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss > > | 1460,sackOK,timestamp 313375234 2356000,nop,wscale 0> (DF) > > | 01:30:19.448146 62.46.86.137.32849 > 217.160.110.113.22: SWE > > | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2359000 > > | 0,nop,wscale 0> (DF) > > | 01:30:19.518099 217.160.110.113.22 > 62.46.86.137.32849: S > > | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss > > | 1460,sackOK,timestamp 313375535 2356000,nop,wscale 0> (DF) > > | 01:30:19.823023 217.160.110.113.22 > 62.46.86.137.32849: S > > | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss > > | 1460,sackOK,timestamp 313375566 2356000,nop,wscale 0> (DF)So, the packets go out with the correct source address, the SYN-ACK comes back from the target host, but the local IP stack then doesn''t send the last ACK back because it doesn''t feel that these packets belong to a local socket. That is quite understandable, since doing a netstat reveals that the socket believes it''s using 192.168.1.1 as its source address (netstat output doesn''t match tcpdump output from above, of course, because the tcpdump is from yesterday, but I did the same things). | freenet@sophokles:~$ netstat -ant | grep SYN_SENT | tcp 0 1 192.168.1.1:32915 217.160.110.113:22 SYN_SENT Now, with the ugly hacks mounting, I''ve considered something like iptables -A INPUT -i ppp0 -j DNAT --to-destination 192.168.1.1 but of course that doesn''t work. At the core of the problem is still how the socket ever got to be have the local address of 192.168.1.1. I''m pretty sure netcat doesn''t bind to a specific local address, so it must be something in iproute2 behaving odd. ciao, -- [*Thomas Themel*] Frankly, many of you on this list really need to [extended contact] be doused with gasoline and then lit. [info provided in] [*message header*] - Tim May on cypherpunks
Hi Thomas, We have the same problem. ;) You''re right, it doesn''t make any sense. Can anyone elaborate?? My setup is virtually identical to what Thomas has. However, I''m using IPMASQ on my outgoing connection, so I am able to see that the problem exists even without using SNAT. Something is borked...> Thomas Themel themel@iwoars.net > Tue, 28 Oct 2003 01:32:09 +0100 > > Hi, > > I''m currently trying to get a Linux machine to route all traffic coming > from a certain UID over a dedicated PPP interface. After going throught > the available documentation and experimenting a bit, I settled for the > following attempt: > > > # 62.46.87.104 - local PPP address > # 195.4.91.104 - PPP peer > ip route add 195.3.91.104 dev ppp0 src 62.46.87.104 table special > # local for DNS etc > ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table special > ip route add default via 195.3.91.104 src 62.46.87.104 table special > ip rule add fwmark 3 lookup special > iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK > --set-mar= k 3 > ip route flush cache > > This seems to work in a way. It correctly sends the packets generated by > that user out the ppp0 interface, but they don''t get the correct source > address: > > | sophokles:~# sh -x description.txt=20 > | + ip route flush table aon> | + ip route add 195.3.91.103 dev ppp0 src 62.46.86.137 table aonc > | + ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table aonc > | + ip route add default via 195.3.91.103 src 62.46.86.137 table aonc > | + ip rule add fwmark 3 lookup aonc > | + iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK > | --set-mark 3 > | + ip route flush cache > | sophokles:~# tcpdump -ni ppp0 port 22 &=20 > | [1] 841 > | sophokles:~# tcpdump: listening on ppp0 > |=20 > | sophokles:~# nc iwoars.net 22 > | SSH-1.99-OpenSSH_3.4p1 Debian 1:3.4p1-1.woody.3 > |=20 > | sophokles:~# su - freenet > | freenet@sophokles:~$ nc iwoars.net 22 > | 01:25:17.044883 192.168.1.1.32848 > 217.160.110.113.22: SWE > | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2056533 > | 0,nop,wscale 0> (DF) > | 01:25:20.043828 192.168.1.1.32848 > 217.160.110.113.22: SWE > | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2059533 > | 0,nop,wscale 0> (DF) > | 01:25:26.042584 192.168.1.1.32848 > 217.160.110.113.22: SWE > | 1344336467:1344336467(0) win 5840 <mss 1460,sackOK,timestamp 2065533 > | 0,nop,wscale 0> (DF) > |=20 > | freenet@sophokles:~$=20 > > I''ve read on this list that owner-based policy routing is impossible > because the routing decision is made before the packet traverses the > OUTPUT chain. However, if this is true, then I don''t understand how the > packet can go out via the correct interface unless there are separate > route lookups to determine the source address and outgoing interface. > > Could someone who knows please elaborate? > > I have also tried, unsuccessfully, to just mangle the source address > using an iptables SNAT rule, but even though that produces correct > network traffic, it seems to break something internally that keeps the > TCP handshake from completing: > > | sophokles:~# iptables -t nat -A POSTROUTING -j SNAT -o ppp0 > --to-source| 62.46.86.137 > | sophokles:~# su - freenet > | freenet@sophokles:~$ nc iwoars.net 22 > | 01:30:16.448930 62.46.86.137.32849 > 217.160.110.113.22: SWE > | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2356000 > | 0,nop,wscale 0> (DF) > | 01:30:16.516732 217.160.110.113.22 > 62.46.86.137.32849: S > | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss > | 1460,sackOK,timestamp 313375234 2356000,nop,wscale 0> (DF) > | 01:30:19.448146 62.46.86.137.32849 > 217.160.110.113.22: SWE > | 1656968486:1656968486(0) win 5840 <mss 1460,sackOK,timestamp 2359000 > | 0,nop,wscale 0> (DF) > | 01:30:19.518099 217.160.110.113.22 > 62.46.86.137.32849: S > | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss > | 1460,sackOK,timestamp 313375535 2356000,nop,wscale 0> (DF) > | 01:30:19.823023 217.160.110.113.22 > 62.46.86.137.32849: S > | 2293250552:2293250552(0) ack 1656968487 win 32120 <mss > | 1460,sackOK,timestamp 313375566 2356000,nop,wscale 0> (DF) > | [...]_______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hi, Brad Barnett (bb@L8R.net) wrote on 2003-11-01:> We have the same problem. ;) You''re right, it doesn''t make any sense.Yeah, that''s what I feel. Unfortunately, my message to linux-netdev went unnoticed, and I don''t really know of any reference against which to compare whether the current behaviour is intended or not.> Can anyone elaborate?? My setup is virtually identical to what Thomas > has. However, I''m using IPMASQ on my outgoing connection, so I am able to > see that the problem exists even without using SNAT.This is what I do currently, and it seems to work. This is from an ip-up script for my PPP link: ip route flush table aonc ip route add $GWIP dev $PPP_IFACE src $PPP_LOCAL table aonc # local for DNS etc ip route add 192.168.1.0/24 dev eth0 src 192.168.1.1 table aonc # internet via PPP lnk ip route add default via $PPP_REMOTE src $PPP_LOCAL table aonc ip rule add fwmark 3 lookup aonc # owner tagging iptables -t mangle -A OUTPUT -m owner --uid-owner freenet -j MARK --set-mark 3 # SNAT for outgoing packets iptables -A POSTROUTING -t nat -o $PPP_IFACE --match mark --mark 0x03 -j SNAT --to-source $PPP_LOCAL # DNAT for incoming packets iptables -t nat -A PREROUTING -i $PPP_IFACE -d $PPP_LOCAL -j DNAT --to-destination 192.168.1.1 ip route flush cache However, this is feels like a very bad hack - the final DNAT makes the interface unusable for other purposes, and the interaction between the sockets'' addresses and the packets'' addresses are too complicated for my taste. It''s okay for my home network, but I wouldn''t want to run this productively. ciao, -- [*Thomas Themel*] Our national shield displays a lion with two tails. It''s [extended contact] rumoured that it symbolizes the nature of our governments; [info provided in] that there actually were two lions, but one is now stuck [*message header*] deep in the ass of the other one. - Thomas Shaddack _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hi, Thomas Themel (themel@iwoars.net) wrote on 2003-11-01:> # SNAT for outgoing packets > iptables -A POSTROUTING -t nat -o $PPP_IFACE --match mark --mark 0x03 -j SNAT --to-source $PPP_LOCALI''ve been able to do away with the DNAT rule now.> # DNAT for incoming packets > iptables -t nat -A PREROUTING -i $PPP_IFACE -d $PPP_LOCAL -j DNAT --to-destination 192.168.1.1I couldn''t get it to work with just the SNAT rule originally (see original post, the SACKs would be ignored), but I''ve finally figured out why: I had enabled rp_filter on that machine. Quite obviously, enabling rp_filter in combination with policy routing is a bad idea. echo 0 > /proc/sys/net/ipv4/conf/$PPP_IFACE/rp_filter fixed it for me. I still think I shouldn''t need the netfilter SNAT rule, but even enabling NAT with the routing rule (ip rule add fwmark 3 table aonc nat $PPP_REMOTE) doesn''t seem to set the source address of the outgoing packets correctly. ciao, -- [*Thomas Themel*] "If we''re not supposed to eat animals, how come [extended contact] they''re made of meat?" [info provided in] - Treat Carnivores Ethically, [*message header*] in the fuckedcompany.com forums