LARTC - Nov 2005 - Iproute2 src mangling problems

Hi,

I''m having an issue with changing the source address of packets 
origininating from a specific user using a combination of iptables and 
iproute2.  We want to change packets originating from a specific 
service so that they originate from that service''s IP address, so that 
we can move it to a different physical server and have the packets 
appear to come from the same place.

First, I''ll explain what I have set up.

==============
OS: CentOS 4.2 (also tried Fedora Core 4)
IPRoute version: iproute.i386 2.6.9-3 (from yum)
IPTables version: iptables.i386 1.2.11-3.1.RHEL4
Kernel: 2.6.9-11.EL

==============
- We have two IP addresses assigned to this machine using aliases:

[root@tequesta ~]# ip addr show
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
...
    inet xxx.yyy.zzz.39/24 brd xxx.yyy.zzz.255 scope global eth0
    inet xxx.yyy.zzz.16/24 brd xxx.yyy.zzz.255 scope global secondary eth0:0
...

- We''re marking packets that go through iptables with mark 0xf1:

/sbin/iptables -I OUTPUT -t mangle -m owner --uid-owner (uid) -j MARK 
--set-mark 0xf1

- With iproute2, we''re setting packets marked with 0xf1 to go to a 
special table (which has already been created in the proper places):

/sbin/ip rule add fwmark 0xf1 lookup kloostec

- The routing table kloostec looks like this:

xxx.yyy.zzz.0/24 dev eth0  proto static  src xxx.yyy.zzz.16
default via xxx.yyy.zzz.254 dev eth0

- The routing table default looks like this:

xxx.yyy.zzz.0/24 dev eth0  scope link  src xxx.yyy.zzz.39
...
default via xxx.yyy.zzz.254 dev eth0

==============
So, all traffic should appear to originate from .39 except traffic 
flagged with mark 0xf1 (all traffic from user (uid)), which should have 
its source mangled to .16.

However, all traffic appears to originate from .39, regardless of user.

==============
If I modify the routing table default so that it looks like this:

xxx.yyy.zzz.0/24 dev eth0  scope link  src xxx.yyy.zzz.16
...
default via xxx.yyy.zzz.254 dev eth0

Then all traffic appears to originate from .16, as would be expected.

==============
If I modify the routing table kloostec so that the gateway is incorrect:

xxx.yyy.zzz.0/24 dev eth0  proto static  src xxx.yyy.zzz.16
default via xxx.yyy.zzz.100 dev eth0

Then traffic coming from user (uid) that is destined out of subnet 
stops working.  So, I know that the marking and table redirection is 
working correctly.

==============
So, my question is, why are the packets not having their source address 
changed when they are marked 0xf1?  If anyone could answer this, it 
would end days of searching and reading documentation and pulling my 
hair out!

Let me know if you any more details.

Thanks in advance,

Chris Kloosterman


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Chris Kloosterman wrote:
> - We have two IP addresses assigned to this machine using aliases:
> 
> [root@tequesta ~]# ip addr show
> 2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
> ...
>    inet xxx.yyy.zzz.39/24 brd xxx.yyy.zzz.255 scope global eth0
>    inet xxx.yyy.zzz.16/24 brd xxx.yyy.zzz.255 scope global secondary eth0:0
> ...
> 
> - We''re marking packets that go through iptables with mark 0xf1:
> 
> /sbin/iptables -I OUTPUT -t mangle -m owner --uid-owner (uid) -j MARK 
> --set-mark 0xf1
> 
> - With iproute2, we''re setting packets marked with 0xf1 to go to a
> special table (which has already been created in the proper places):
> 
> /sbin/ip rule add fwmark 0xf1 lookup kloostec
> 
> - The routing table kloostec looks like this:
> 
> xxx.yyy.zzz.0/24 dev eth0  proto static  src xxx.yyy.zzz.16
> default via xxx.yyy.zzz.254 dev eth0
> 
> - The routing table default looks like this:
> 
> xxx.yyy.zzz.0/24 dev eth0  scope link  src xxx.yyy.zzz.39
> ...
> default via xxx.yyy.zzz.254 dev eth0
> 
> ==============> 
> So, my question is, why are the packets not having their source address 
> changed when they are marked 0xf1?  If anyone could answer this, it 
> would end days of searching and reading documentation and pulling my 
> hair out!
What source address does the packet have when it hits your mark rule? :)

No, seriously, this is a common misunderstanding, unfortunately what
you''re trying to do is not possible. At the time the source address is
selected there is no packet and as a consequence no netfilter mark, so
marks can''t be uses for this. The only way is to use SNAT.