LARTC - Apr 2002 - Fwmark problem - policy routing does not work.

Well, I''m stating my problem from the beginning:
I have a computer in my LAN with 2 ips (192.168.1.10 and 192.168.1.11), the 
latter one created by "ip addr add" and users on that computer, and
depending
on the uid of the created packets i want those to take another source address.
Of course, it should start with marking the packets with a simple iptables 
rule:

iptables  -t mangle  -A OUTPUT  -m owner --uid-owner 1001 -j MARK --set-mark 1

so, next step should be, to create a new table, so i entered this into 
/etc/iproute2/rt_tables:
1       thilo

so that table 1 should have the identifier "thilo".

Next step is, to enter a rule for this table:
ip rule add fwmark 1 table thilo

and indeed:
router:/etc/init.d# ip rule ls
0:      from all lookup local
32765:  from all fwmark        1 lookup thilo
32766:  from all lookup main
32767:  from all lookup default
router:/etc/init.d#

now, i should only have to enter the route into the table:
ip route add 192.168.1.0/24 dev eth0 src 192.168.1.11 proto kernel

a final "ip route flush cash", and this should be it.
As soon, as i log in as user "thilo" (uid 1001), and i for example
connect
per ssh to 192.168.1.4 (another computer in my lan), it _should_ show me 
logged in from 192.168.1.11, but instead it still shows:
thilo    pts/8        Apr 23 18:37 (192.168.1.10).

The table as well as the route entry are correct, i am sure of this, for i 
have tested removing the fwmark rule and instead added:
ip rule add to 192.168.1.4 table thilo
everything worked then as i have intended, and it showed me logged in from 
192.168.1.11 then.

So it might be, that the Marks are not being set, i checked that too, i 
replaced the 
iptables  -t mangle  -A OUTPUT  -m owner --uid-owner 1001 -j MARK --set-mark 1
with
iptables  -t mangle  -A OUTPUT  -p tcp --dport 22 -j MARK --set-mark 1
- still no success. So I tried placing this rule into the PREROUTING chain 
(for testing though, INPUT should be absolutely correct), and - oh wonder - 
no success either ;)

Another problems checked too: I have enabled MARK based routing in the kernel 
(policy routing, everything switched on ..)

So I only can ask you on this mailing list - anyone got an idea of what 
is/what i did wrong?

- Thilo Schulz

> \> So it might be, that the Marks are not being set, i checked that too,
i
>
> > replaced the
> > iptables  -t mangle  -A OUTPUT  -m owner --uid-owner 1001 -j MARK
> > --set-mark 1 with
> > iptables  -t mangle  -A OUTPUT  -p tcp --dport 22 -j MARK --set-mark 1
> > - still no success. So I tried placing this rule into the PREROUTING
> > chain (for testing though, INPUT should be absolutely correct), and -
oh
> > wonder - no success either ;)
>
> If you do "iptables -L -v -n -t mangle", do the counters
increment ?
>
> Stef
Yes, they do.
I am also sure now, that marking is working:
I have tried the same on another host that uses a gateway to access the 
internet, but has several ips on the net. If i do not enter the route (via 
option) the packets are not reaching their destination, if i am creating 
those as a given user, as all other user it does work. If i give the via 
option in the "ip route add table thilo src ....." command, the
packets are
reaching their destination again - yet even though i have given the source ip 
that should have been used, it is still using the primary one.

Could this be a bug in the routing code?

- Thilo Schulz

> Yes, they do.
> I am also sure now, that marking is working:
> I have tried the same on another host that uses a gateway to access the
> internet, but has several ips on the net. If i do not enter the route (via
> option) the packets are not reaching their destination, if i am creating
> those as a given user, as all other user it does work. If i give the via
> option in the "ip route add table thilo src ....." command, the
packets are
> reaching their destination again - yet even though i have given the source
> ip that should have been used, it is still using the primary one.
>
> Could this be a bug in the routing code?
I can only help you with the marking stuff :(

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.openprojects.net

> I can only help you with the marking stuff :(
Well, any address i can contact to get further information about this?
I''m
pretty much at the end with my latin ...

On Tue, Apr 23, 2002 at 07:23:58PM +0200, Thilo Schulz wrote:
[...]
> I have a computer in my LAN with 2 ips (192.168.1.10 and 192.168.1.11), the
> latter one created by "ip addr add" and users on that computer,
and depending
> on the uid of the created packets i want those to take another source
address.
> Of course, it should start with marking the packets with a simple iptables 
> rule:
> 
> iptables  -t mangle  -A OUTPUT  -m owner --uid-owner 1001 -j MARK
--set-mark 1
[...]
> The table as well as the route entry are correct, i am sure of this, for i 
> have tested removing the fwmark rule and instead added:
> ip rule add to 192.168.1.4 table thilo
> everything worked then as i have intended, and it showed me logged in from 
> 192.168.1.11 then.
Are you sure it showed you as logged in from 192.168.1.11?  I just
tried the same setup here, and with either setup you describe, it
always shows me as having logged in from the primary IP.

In my case, I setup 192.168.1.1 and 192.168.1.2 and tried to connect
to 192.168.1.5.  In both cases above, it showed me as having logged in
from 192.168.1.1.

The counters are incrementing though, so marking is working.
> Another problems checked too: I have enabled MARK based routing in the
kernel
> (policy routing, everything switched on ..)
> 
> So I only can ask you on this mailing list - anyone got an idea of what 
> is/what i did wrong?
Depending on what the results of your second setup above were (ip rule
add to 192.168.1.4 table thilo), I have a theory...

When you add a route that sets a src like:

ip route add table <table> 192.168.1.0/24 src 192.168.1.11 dev eth0

The "src" doesn''t specify the source IP to put in the packet
(it''s not
network address translation, like SNAT in iptables), it just specifies
which local source IP the routing mechanisms should use to determine
where to route the packet.

For example, I''ve got policy routing setup with FreeS/WAN on a gateway
with an internal and external interface, where I do:

# ip rule add table 1 prio 100
# ip route add table 1 <remoteLAN> dev ipsec0 src <localIP>

This forces the box to route all packets to the remote LAN via the
internal interface, rather than the external interface.

However, the packets that show up at the other end don''t contain a
source IP of <localIP> from the table 1 route, rather they contain the
source IP of the client machine that sent them.

This led me to believe that the "src" option only adjusts the way the
routing machinery in the kernel decides where and how to route the
packet, but doesn''t change/rewrite the source address in the packets
themselves.

This makes sense to me as well, because if setup routes for
192.168.1.4 through 192.168.1.11, and send packets from 192.168.1.10,
they''ll have a source address of 192.168.1.10, but get routed through
the 192.168.1.11 interface (which is transparent at 192.168.1.4''s
end).

If you really want the packets to look like they''re coming from
192.168.1.11, why don''t you just use iptables and SNAT them?

--
Adrian Chung (adrian at enfusion-group dot com)
http://www.enfusion-group.com/~adrian
GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17
[toad.enfusion-group.com] up 34 days, 1:10, 17 users

> Are you sure it showed you as logged in from 192.168.1.11?  I just
> tried the same setup here, and with either setup you describe, it
> always shows me as having logged in from the primary IP.
Absolutely sure, it only worked with the "to ipaddress" rule.
> The "src" doesn''t specify the source IP to put in the
packet (it''s not
> network address translation, like SNAT in iptables), it just specifies
> which local source IP the routing mechanisms should use to determine
> where to route the packet.
However, somewhere inthe iproute tools this HAS to be possible ...
> If you really want the packets to look like they''re coming from
> 192.168.1.11, why don''t you just use iptables and SNAT them?
I have thought of that as well, but the socket that has been created should 
only listen to the primary IP, so that the source and destination address do 
not match.

I was going to try it though, and have already compiled a fully NAT able 
kernel (it was not able to previously, because i thought NAT is not necessary)

Also i''m going to contact one of the Policy Routing developers.

Thanks for your help.
- Thilo Schulz

On Fri, Apr 26, 2002 at 10:01:48PM +0200, Thilo Schulz wrote:
[...]
> > The "src" doesn''t specify the source IP to put in
the packet (it''s not
> > network address translation, like SNAT in iptables), it just specifies
> > which local source IP the routing mechanisms should use to determine
> > where to route the packet.
> 
> However, somewhere inthe iproute tools this HAS to be possible ...
I read in the documentation for iproute2 somewhere, sometime that
there was a ''nat'' directive, but I thought I''d also
read that it
wasn''t very well supported, probably didn''t work, and probably
wouldn''t be rewritten to work because netfilter/SNAT already do a much
better job of NAT in general.

[..]
> Also i''m going to contact one of the Policy Routing developers.
Hope you get it working!

--
Adrian Chung (adrian at enfusion-group dot com)
http://www.enfusion-group.com/~adrian
GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17
[toad.enfusion-group.com] up 34 days, 1:40, 17 users

On Fri, 26 Apr 2002, Adrian Chung wrote:
> When you add a route that sets a src like:
>
> ip route add table <table> 192.168.1.0/24 src 192.168.1.11 dev eth0
>
> The "src" doesn''t specify the source IP to put in the
packet (it''s not
> network address translation, like SNAT in iptables), it just specifies
> which local source IP the routing mechanisms should use to determine
> where to route the packet.
Actually, it is more subtle than that. The ''src'' *does*
specify the source
IP to put in the packet *if* the packet doesn''t have a source IP yet.
This
only holds true for packets generated locally.
> For example, I''ve got policy routing setup with FreeS/WAN on a
gateway
> with an internal and external interface, where I do:
>
> # ip rule add table 1 prio 100
> # ip route add table 1 <remoteLAN> dev ipsec0 src <localIP>
>
> This forces the box to route all packets to the remote LAN via the
> internal interface, rather than the external interface.
>
> However, the packets that show up at the other end don''t contain a
> source IP of <localIP> from the table 1 route, rather they contain
the
> source IP of the client machine that sent them.
> This led me to believe that the "src" option only adjusts the way
the
> routing machinery in the kernel decides where and how to route the
> packet, but doesn''t change/rewrite the source address in the
packets
> themselves.
It does not. The ip rule does that. Routing does not mangle packets, unless
the packet is locally generated and incomplete.

Doei, Arthur.

-- 
  /\    / |      arthurvl@sci.kun.nl      | Work like you don''t need
the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there''s nobody
watching

> Actually, it is more subtle than that. The ''src'' *does*
specify the source
> IP to put in the packet *if* the packet doesn''t have a source IP
yet. This
> only holds true for packets generated locally.
Then why does it not work together with the fwmark policer?
> It does not. The ip rule does that. Routing does not mangle packets, unless
> the packet is locally generated and incomplete.
it is generated locally in my case.

- Thilo Schulz

On Sat, Apr 27, 2002 at 10:25:24AM +0200, Arthur van Leeuwen
wrote:
> On Fri, 26 Apr 2002, Adrian Chung wrote:
> 
> > When you add a route that sets a src like:
> >
> > ip route add table <table> 192.168.1.0/24 src 192.168.1.11 dev
eth0
> >
> > The "src" doesn''t specify the source IP to put in
the packet (it''s not
> > network address translation, like SNAT in iptables), it just specifies
> > which local source IP the routing mechanisms should use to determine
> > where to route the packet.
> 
> Actually, it is more subtle than that. The ''src'' *does*
specify the source
> IP to put in the packet *if* the packet doesn''t have a source IP
yet. This
> only holds true for packets generated locally.
Ah okay, that makes sense...  But I think in both our cases the
packets were generated locally, so the ''src'' flag should have
set the
source IP.

Is it possible for the application (telnet in my case) to explicitly
bind to a socket and set it''s source IP?  That could explain why the
rule has no effect since by the time the packet reaches the routing
system, it already has a source IP set.

--
Adrian Chung (adrian at enfusion-group dot com)
http://www.enfusion-group.com/~adrian
GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17
[toad.enfusion-group.com] up 34 days, 17:14, 17 users

On Sat, 27 Apr 2002, Adrian Chung wrote:
> On Sat, Apr 27, 2002 at 10:25:24AM +0200, Arthur van Leeuwen wrote:
> > On Fri, 26 Apr 2002, Adrian Chung wrote:
> >
> > > When you add a route that sets a src like:
> > >
> > > ip route add table <table> 192.168.1.0/24 src 192.168.1.11
dev eth0
> > >
> > > The "src" doesn''t specify the source IP to put
in the packet (it''s not
> > > network address translation, like SNAT in iptables), it just
specifies
> > > which local source IP the routing mechanisms should use to
determine
> > > where to route the packet.
> >
> > Actually, it is more subtle than that. The ''src''
*does* specify the source
> > IP to put in the packet *if* the packet doesn''t have a source
IP yet. This
> > only holds true for packets generated locally.
>
> Ah okay, that makes sense...  But I think in both our cases the
> packets were generated locally, so the ''src'' flag should
have set the
> source IP.
>
> Is it possible for the application (telnet in my case) to explicitly
> bind to a socket and set it''s source IP?  That could explain why
the
> rule has no effect since by the time the packet reaches the routing
> system, it already has a source IP set.
Yes, that is possible.

Doei, Arthur.

-- 
  /\    / |      arthurvl@sci.kun.nl      | Work like you don''t need
the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there''s nobody
watching

Hello,

On Sat, 27 Apr 2002, Thilo Schulz wrote:
> > It does not. The ip rule does that. Routing does not mangle packets,
unless
> > the packet is locally generated and incomplete.
>
> it is generated locally in my case.
	May be your problem is that LOCALOUT is after routing,
the socket already created connected route and is bound to
specific src IP. It is too late to mark packets just to select
different src IP via routing. May be you have to use SNAT to change
the src IP to the desired one. I didn''t tried it and I don''t
know
whether it works. And I see in recent sources some funny things,
for example, ipt_local_hook(MANGLE) calls ip_route_me_harder from
LOCAL_IN but I''m not sure it will hurt your tests. May be you can
ask for new LOCAL_ROUTING hook where MANGLE can select different src
IP for different users before route connection :)))
> - Thilo Schulz
Regards

--
Julian Anastasov <ja@ssi.bg>