LARTC - Apr 2006 - source routing does not work with extra ip addresses

I set up this config:

 +------+
-+ ISP1 +--+
 +------+  |  +-------+
           +--+ linux |
 +------+  |  +-------+
-+ ISP2 +--+
 +------+

No problem. Standard setup with two ISP''s. Both routed subnets. Default
gateway is ISP1. No magic here.

Now I put a server behind the Linux box. I want the server to be
reachable on an /extra/ IP in the routed subnet of ISP2.

 +------+
-+ ISP1 +--+
 +------+  |  +-------+  +-----------------+
           +--+ linux +--+ server 10.0.0.2 |
 +------+  |  +-------+  +-----------------+
-+ ISP2 +--+
 +------+

router ISP2: 1.2.3.1/24
dev ISP2: eth1
Linux box eth1: 1.2.3.2/24
external ip ISP2 for server 10.0.0.2: 1.2.3.3

arp -s 1.2.3.3 aa:bb:cc:dd:ee:ff pub
ip route add 1.2.3.3 via 10.0.0.2
iptables -t nat -A PREROUTING -i eth1 -d 1.2.3.3 -j DNAT --to 10.0.0.2

When pinging 1.2.3.3, the packets get in through eth1 (ok), but the
replies are following the default route through eth0 (wrong)

Even a

ip rule add from 1.2.3.3 lookup table_eth1

doesn''t change this behaviour. It is working ok when I add the address
1.2.3.3 directly to eth1:

ip a a 1.2.3.3 dev eth1

Why is this?

R.

-- 
___________________________________________________________________
It is better to remain silent and be thought a fool, than to speak
aloud and remove all doubt.

+------------------------------------------------------------------+
| Richard Lucassen, Utrecht                                        |
| Public key and email address:                                    |
| http://www.lucassen.org/mail-pubkey.html                         |
+------------------------------------------------------------------+

Hello there,

 : Now I put a server behind the Linux box. I want the server to be 
 : reachable on an /extra/ IP in the routed subnet of ISP2.

Does server have one or two IP addresses?  Best solution?  Use two 
IP addresses on server.

 : router ISP2: 1.2.3.1/24
 : dev ISP2: eth1
 : Linux box eth1: 1.2.3.2/24
 : external ip ISP2 for server 10.0.0.2: 1.2.3.3



 : arp -s 1.2.3.3 aa:bb:cc:dd:ee:ff pub
 : ip route add 1.2.3.3 via 10.0.0.2
 : iptables -t nat -A PREROUTING -i eth1 -d 1.2.3.3 -j DNAT --to 10.0.0.2
 : 
 : When pinging 1.2.3.3, the packets get in through eth1 (ok), but the
 : replies are following the default route through eth0 (wrong)

The problem is routing.  Return packets from your server are handled 
in the main routing table.  There isn''t yet an RPDB entry directing 
traffic from 10.0.0.2 to use table_eth1.  Your RPDB entry looks like 
this:

 : ip rule add from 1.2.3.3 lookup table_eth1

Try changing this (or adding another rule): 

  ip rule add from 10.0.0.2 lookup table_eth1

instead.  Now, your server should have Internet access strictly on 
the link handled by ISP2.

If you would like to handle inbound traffic on both links, then add 
a secondary IP address to your server, and enter another DNAT rule 
which specifies another NAT mapping for the secondary IP.

-Martin

-- 
Martin A. Brown --- Wonderfrog Enterprises --- martin@wonderfrog.net

On Sat, 8 Apr 2006 11:07:00 -0500
"Martin A. Brown" <martin-lartc@wonderfrog.net> wrote:
>  : Now I put a server behind the Linux box. I want the server to be 
>  : reachable on an /extra/ IP in the routed subnet of ISP2.
> 
> Does server have one or two IP addresses?  Best solution?  Use two 
> IP addresses on server.
Hmmm, one for ISP1 and one for ISP2? That would be a nice idea to
workaround this problem :-)
 
>  : When pinging 1.2.3.3, the packets get in through eth1 (ok), but the
>  : replies are following the default route through eth0 (wrong)
> 
> The problem is routing.  Return packets from your server are handled 
> in the main routing table.  There isn''t yet an RPDB entry
directing
> traffic from 10.0.0.2 to use table_eth1.  Your RPDB entry looks like 
> this:
> 
>  : ip rule add from 1.2.3.3 lookup table_eth1
> 
> Try changing this (or adding another rule): 
> 
>   ip rule add from 10.0.0.2 lookup table_eth1
Nope. I already tried that, but no way.
> instead.  Now, your server should have Internet access strictly on 
> the link handled by ISP2.
No. The packets are returned through ISP1.
 
> If you would like to handle inbound traffic on both links, then add 
> a secondary IP address to your server, and enter another DNAT rule 
> which specifies another NAT mapping for the secondary IP.
That''s a very nice idea, but packets keep on entering the wrong table
(default), I think it''s a bug somewhere in the kernel.

It only works when the ip is direct on the external interface of the
Linuxbox, but as soon as 1 tcp port is translated, the return packets
for that translated port get into the wrong (default) table.

Even when using fw marks it doesn''t work. I mark all packets coming
from the servers second ip address with ''1'' and a simple

ip ru a fwmark 1 table t_eth1

should do the job. But no way. Packets keep on getting out through ISP1
(t_eth0).

This is the real test:

10.0.2.1 is the server, 10.0.2.3 is its second ip.
10.0.2.1 = external 10.1.3.100
10.0.2.3 = external 192.168.201.3

# ip r s
192.168.201.3 via 10.0.2.3 dev eth2 
10.1.3.100 via 10.0.2.1 dev eth2 
10.0.2.0/24 dev eth2  proto kernel  scope link  src 10.0.2.2 
192.168.201.0/24 dev eth1  proto kernel  scope link  src 192.168.201.2 
10.1.3.0/24 dev eth0  proto kernel  scope link  src 10.1.3.101 
default via 10.1.3.1 dev eth0


# ip ru s
0:      from all lookup local 
32762:  from all fwmark 0x1 lookup t_eth1 
32764:  from 192.168.201.2 lookup t_eth1 
32765:  from 10.1.3.101 lookup t_eth0 
32766:  from all lookup main 
32767:  from all lookup default


# ip r s t t_eth0
10.0.2.0/24 dev eth2  scope link 
10.1.3.0/24 dev eth0  scope link  src 10.1.3.101 
127.0.0.0/8 dev lo  scope link 
default via 10.1.3.1 dev eth0 



# ip r s t t_eth1
10.0.2.0/24 dev eth2  scope link 
192.168.201.0/24 dev eth1  scope link  src 192.168.201.2 
127.0.0.0/8 dev lo  scope link 
default via 192.168.201.1 dev eth1

Any hints are welcome...

btw: iproute2-ss06011, kernel 2.6.16.2, iptables 1.3.5

R.

-- 
___________________________________________________________________
It is better to remain silent and be thought a fool, than to speak
aloud and remove all doubt.

+------------------------------------------------------------------+
| Richard Lucassen, Utrecht                                        |
| Public key and email address:                                    |
| http://www.lucassen.org/mail-pubkey.html                         |
+------------------------------------------------------------------+
-- 
___________________________________________________________________
It is better to remain silent and be thought a fool, than to speak
aloud and remove all doubt.

+------------------------------------------------------------------+
| Richard Lucassen, Utrecht                                        |
| Public key and email address:                                    |
| http://www.lucassen.org/mail-pubkey.html                         |
+------------------------------------------------------------------+

Hello again,

 : > Does server have one or two IP addresses?  Best solution?  Use two 
 : > IP addresses on server.
 : 
 : Hmmm, one for ISP1 and one for ISP2? That would be a nice idea to
 : workaround this problem :-)

Only way I have done this myself, although I recall somebody else on 
LARTC using connmark with nfmark and/or the ROUTE target to solve 
this problem using only a single IP.  Perhaps the archive will help 
you here....

 : > Try changing this (or adding another rule): 
 : > 
 : >   ip rule add from 10.0.0.2 lookup table_eth1
 : 
 : Nope. I already tried that, but no way.
 : No. The packets are returned through ISP1.
 :  
 : > If you would like to handle inbound traffic on both links, then add 
 : > a secondary IP address to your server, and enter another DNAT rule 
 : > which specifies another NAT mapping for the secondary IP.
 : 
 : That''s a very nice idea, but packets keep on entering the wrong 
 : table (default), I think it''s a bug somewhere in the kernel.

While the kernel certainly has seen bugs before and will see more, I 
hope you don''t mind if I continue to entertain a bit of skepticism 
on this point.  :)

 : It only works when the ip is direct on the external interface of 
 : the Linuxbox, but as soon as 1 tcp port is translated, the return 
 : packets for that translated port get into the wrong (default) 
 : table.
 : 
 : Even when using fw marks it doesn''t work. I mark all packets coming
 : from the servers second ip address with ''1'' and a simple
 :
 : ip ru a fwmark 1 table t_eth1
 : 
 : should do the job. But no way. Packets keep on getting out 
 : through ISP1 (t_eth0).
 : 
 : This is the real test:
 : 
 : 10.0.2.1 is the server, 10.0.2.3 is its second ip.
 : 10.0.2.1 = external 10.1.3.100
 : 10.0.2.3 = external 192.168.201.3

OK, got it!

 : # ip r s
 : 192.168.201.3 via 10.0.2.3 dev eth2 
 : 10.1.3.100 via 10.0.2.1 dev eth2 

[ snipped main and ancillary routing tables except for unusual and
  possibly extraneous routes. ]

Routing tables t_eth0 and t_eth1 look fine, although t_eth0 and main 
should be exactly the same.  I believe your two host routes (for 
192.168.201.3 and 10.1.3.100) are unnecessary and simply complicate 
your scenario.

I still think your problem is in the RPDB and addressing of the 
packet at routing time.  I do not believe (check the KPTD and its 
offspring [0] [1]) that the packet''s source address has yet been 
rewritten.  Think about this, and look at your RPDB:

 : # ip ru s
 : 0:      from all lookup local 
 : 32762:  from all fwmark 0x1 lookup t_eth1 
 : 32764:  from 192.168.201.2 lookup t_eth1 
 : 32765:  from 10.1.3.101 lookup t_eth0 
 : 32766:  from all lookup main 
 : 32767:  from all lookup default

The addresses you have entered are the public side addresses.  When 
the server transmits packets, these packets will have the 10.0.2.1 
and 10.0.2.3 addresses for source addresses.  The RPDB should 
include references to these private addresses instead of the 
addresses available on the public side.

 : btw: iproute2-ss06011, kernel 2.6.16.2, iptables 1.3.5

I hope this helps, and thanks for the detailed listing of your 
configuration.  It''s always helpful.

Best of luck,

-Martin

 [0] http://www.docum.org/docum.org/kptd/
 [1] http://linux-ip.net/nf/nfk-traversal.eps
     http://linux-ip.net/nf/nfk-traversal.png

-- 
Martin A. Brown --- Wonderfrog Enterprises --- martin@wonderfrog.net

On Sat, 8 Apr 2006 15:31:24 -0500
"Martin A. Brown" <martin-lartc@wonderfrog.net> wrote:
> Only way I have done this myself, although I recall somebody else on 
> LARTC using connmark with nfmark and/or the ROUTE target to solve 
> this problem using only a single IP.  Perhaps the archive will help 
> you here....
Ok. I have a working workaround now using fwmarks and a second ip on the
server. That was a very good idea. Thnx! Bu so far so good. Now I''d
like to get it working with 1 ip..
>  : That''s a very nice idea, but packets keep on entering the wrong
>  : table (default), I think it''s a bug somewhere in the kernel.
> 
> While the kernel certainly has seen bugs before and will see more, I 
> hope you don''t mind if I continue to entertain a bit of skepticism
> on this point.  :)
Ok, it''s not a bug in the kernel, it''s a bug in the docs :)

[..]
> Routing tables t_eth0 and t_eth1 look fine, although t_eth0 and main 
> should be exactly the same.  I believe your two host routes (for 
> 192.168.201.3 and 10.1.3.100) are unnecessary and simply complicate 
> your scenario.
Hmm. You''re right. I just need 1 extra table, not two. I just followed
the docs in the lartc-howto, I''ll have a closer look at their example
there. I think that 1 extra table does the same job in that example

http://www.lartc.org/lartc.html#LARTC.RPDB.MULTIPLE-LINKS
> I still think your problem is in the RPDB and addressing of the 
> packet at routing time.  I do not believe (check the KPTD and its 
> offspring [0] [1]) that the packet''s source address has yet been 
> rewritten.  Think about this, and look at your RPDB:
> 
>  : # ip ru s
>  : 0:      from all lookup local 
>  : 32762:  from all fwmark 0x1 lookup t_eth1 
>  : 32764:  from 192.168.201.2 lookup t_eth1 
>  : 32765:  from 10.1.3.101 lookup t_eth0 
>  : 32766:  from all lookup main 
>  : 32767:  from all lookup default
> 
> The addresses you have entered are the public side addresses.  When 
> the server transmits packets, these packets will have the 10.0.2.1 
> and 10.0.2.3 addresses for source addresses.  The RPDB should 
> include references to these private addresses instead of the 
> addresses available on the public side.
Once again you''re right. I accidently commented out the necessary
"ip r f c" (flush cache) in the script, that''s why it
didn''t work
(immediately).

But finally I resolved the problem using the CONNMARK. This is the setup
I''m talking about:

http://www.lucassen.org/divers/ar-test.pdf

I don''t know if this is the right way to do this, but it seems to work
well. I mark all packets coming in to 192.168.201.3 through eth1 with
mark 1:

iptables -t mangle -A PREROUTING -i eth1 -d 192.168.201.3 \
 -j CONNMARK --set-mark 1

I mark the return packets with the same mark:

iptables -t mangle -A PREROUTING -i eth2 -s 10.0.2.1 \
 -j CONNMARK --restore-mark

A simple

ip rule add fwmark 1 table t_eth1
ip r f c

does the rest. I have now 1 ip address on the server and two routes to
the internet. And 1 extra table instead of two ;-)

Thnx for your help,

R.


-- 
___________________________________________________________________
It is better to remain silent and be thought a fool, than to speak
aloud and remove all doubt.

+------------------------------------------------------------------+
| Richard Lucassen, Utrecht                                        |
| Public key and email address:                                    |
| http://www.lucassen.org/mail-pubkey.html                         |
+------------------------------------------------------------------+