LARTC - Jun 2007 - multiple routing tables for internal router programs

Maybe a strange request, I''ll try to explain this as clearer as I can 
(forgive my bad english, please  :-)  ).

I''m setting a linux box as a router. My router uses multiple routing 
tables, so I can address the traffic from specific ip addresses of my 
lan to distinct ISPs providers (specifying a different default gateway 
fo r each table), marking packets with iptables (prerouting marks).

This works with the forwarding traffic (lan-ISPs) that crosses my router.

But how can I reach the same result for programs/services that are 
working INTO the linux box? All I want is that a program (ping, for 
examples, or a VOIP server, better) uses a secondary routing table in 
the same machine. In this mode, I can manipulate route settings for 
different classes of program in my router.

Is it possible?

Thanks

Andrea escribió:
> Maybe a strange request, I''ll try to explain this as clearer as I
can
> (forgive my bad english, please  :-)  ).
>
Está permitido responder en castellano en esta lista?
> I''m setting a linux box as a router. My router uses multiple
routing
> tables, so I can address the traffic from specific ip addresses of my
> lan to distinct ISPs providers (specifying a different default gateway
> fo r each table), marking packets with iptables (prerouting marks).
>
> This works with the forwarding traffic (lan-ISPs) that crosses my router.
>
> But how can I reach the same result for programs/services that are
> working INTO the linux box? All I want is that a program (ping, for
> examples, or a VOIP server, better) uses a secondary routing table in
> the same machine. In this mode, I can manipulate route settings for
> different classes of program in my router.
>
Lo que podés hacer es "marcar" los paquetes mediante iptables -t
mangle
y luego definir reglas (ip rule) para routear cada paquete de acuerdo a
la marca que tenga, por las tablas (ip route) que tengas definidas.

Por ejemplo:
Definís una tabla con su gateway (alguno de tus conexiones), y le ponés
las redes que necesitás sean "conocidas" en la tabla:

ip route add 127.0.0.0/8   dev lo          scope link table 100
ip route add $NET_INTERNA  dev $IF_INTERNA scope link table 100
ip route add $NET_ADSL1    dev $IF_ADSL2   scope link table 100
ip route add $NET_ADSL2    dev $IF_ADSL2   scope link table 100
ip route add default dev $IF_ADSL2 via $GW_ADSL2 table 100


Definís una regla que todo paquete esté marcado con un 1, use esa tabla
de routeo (salga por ese gateway...)

ip rule add fwmark 1 table 100

Y también, marcás con un 1 cada paquete que querés que use esa tabla
(por ejemplo, el tráfico web):

iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 1


Espero te sirva.
Saludos!

Javier.-

On Mon, 11 Jun 2007, Javier Charne wrote:
> Andrea escribió:
>> Maybe a strange request, I''ll try to explain this as clearer
as I can
>> (forgive my bad english, please  :-)  ).
>>
> Está permitido responder en castellano en esta lista?
>> I''m setting a linux box as a router. My router uses multiple
routing
>> tables, so I can address the traffic from specific ip addresses of my
>> lan to distinct ISPs providers (specifying a different default gateway
>> fo r each table), marking packets with iptables (prerouting marks).
>>
>> This works with the forwarding traffic (lan-ISPs) that crosses my
router.
>>
>> But how can I reach the same result for programs/services that are
>> working INTO the linux box? All I want is that a program (ping, for
>> examples, or a VOIP server, better) uses a secondary routing table in
>> the same machine. In this mode, I can manipulate route settings for
>> different classes of program in my router.
>>
>
Any possibility someone could repost this reply in english.
> Lo que podés hacer es "marcar" los paquetes mediante iptables -t
mangle
> y luego definir reglas (ip rule) para routear cada paquete de acuerdo a
> la marca que tenga, por las tablas (ip route) que tengas definidas.
>
> Por ejemplo:
> Definís una tabla con su gateway (alguno de tus conexiones), y le ponés
> las redes que necesitás sean "conocidas" en la tabla:
>
> ip route add 127.0.0.0/8   dev lo          scope link table 100
> ip route add $NET_INTERNA  dev $IF_INTERNA scope link table 100
> ip route add $NET_ADSL1    dev $IF_ADSL2   scope link table 100
> ip route add $NET_ADSL2    dev $IF_ADSL2   scope link table 100
> ip route add default dev $IF_ADSL2 via $GW_ADSL2 table 100
>
>
> Definís una regla que todo paquete esté marcado con un 1, use esa tabla
> de routeo (salga por ese gateway...)
>
> ip rule add fwmark 1 table 100
>
> Y también, marcás con un 1 cada paquete que querés que use esa tabla
> (por ejemplo, el tráfico web):
>
> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 1
Regards,

-- 
Tom Diehl		tdiehl@rogueind.com		Spamtrap address mtd123@rogueind.com
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Tom Diehl escribió:
>>
>
> Any possibility someone could repost this reply in english.
>
Sorry, Tom. My english is really awful.
>> Lo que podés hacer es "marcar" los paquetes mediante iptables
-t mangle
>> y luego definir reglas (ip rule) para routear cada paquete de acuerdo a
>> la marca que tenga, por las tablas (ip route) que tengas definidas.
>>
>> Por ejemplo:
>> Definís una tabla con su gateway (alguno de tus conexiones), y le ponés
>> las redes que necesitás sean "conocidas" en la tabla:
>>
>> ip route add 127.0.0.0/8   dev lo          scope link table 100
>> ip route add $NET_INTERNA  dev $IF_INTERNA scope link table 100
>> ip route add $NET_ADSL1    dev $IF_ADSL2   scope link table 100
>> ip route add $NET_ADSL2    dev $IF_ADSL2   scope link table 100
>> ip route add default dev $IF_ADSL2 via $GW_ADSL2 table 100
>>
>>
>> Definís una regla que todo paquete esté marcado con un 1, use esa tabla
>> de routeo (salga por ese gateway...)
>>
>> ip rule add fwmark 1 table 100
>>
>> Y también, marcás con un 1 cada paquete que querés que use esa tabla
>> (por ejemplo, el tráfico web):
>>
>> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 1
>
> Regards,
>
I was saying Andrea:  Try to define a new routing table, add a chain in
mangle table for tagging packets and add a rule to deliver those packets
to the new route.
Again, I''m sorry.   I didn''t know this is a
"english-only" list.


Saludos!

Javier.-

> I was saying Andrea:  Try to define a new routing table, add a chain in
> mangle table for tagging packets and add a rule to deliver those packets
> to the new route.
> Again, I''m sorry.   I didn''t know this is a
"english-only" list.
Thanks for the reply.

This is the exact way that I used for managing traffic of my lan towards 
     ISPs. But is this mode still valid if I want to manage services 
executed directly in the router?

this rule:

iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 1

capture all (web) traffic that crosses my router. Can I capture only the 
  (web) traffic generated from my router and directed to internet?

Anymore, I don''t need it more: I''ve resolved my problem, the
conflict
between a "ping script" (that I''m writing for multiple
gateway
testing)and servers executed in router too: first version of my script 
sets a default gateway for testing it with ping, now I''ve discovered 
that I can use a specific route involving the gateway without setting 
default gateway, a much better solution.

You have to capture the local packets in OUTPUT chain, not in
PREROUTING.

Well, I have a problem with the ping scripts used for dead gateway
detection, I will post it in another thread.

-----Original Message-----
From: lartc-bounces@mailman.ds9a.nl
[mailto:lartc-bounces@mailman.ds9a.nl] On Behalf Of Andrea
Sent: Tuesday, June 12, 2007 4:00 PM
Cc: lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Re: multiple routing tables for internal router
programs

> I was saying Andrea:  Try to define a new routing table, add a chain
in
> mangle table for tagging packets and add a rule to deliver those
packets
> to the new route.
> Again, I''m sorry.   I didn''t know this is a
"english-only" list.
Thanks for the reply.

This is the exact way that I used for managing traffic of my lan towards

     ISPs. But is this mode still valid if I want to manage services 
executed directly in the router?

this rule:

iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 1

capture all (web) traffic that crosses my router. Can I capture only the

  (web) traffic generated from my router and directed to internet?

Anymore, I don''t need it more: I''ve resolved my problem, the
conflict
between a "ping script" (that I''m writing for multiple
gateway
testing)and servers executed in router too: first version of my script 
sets a default gateway for testing it with ping, now I''ve discovered 
that I can use a specific route involving the gateway without setting 
default gateway, a much better solution.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Andrea wrote:
> This is the exact way that I used for managing traffic of my lan towards 
>     ISPs. But is this mode still valid if I want to manage services 
> executed directly in the router?
> 
> this rule:
> 
> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j MARK --set-mark 1
> 
> capture all (web) traffic that crosses my router. Can I capture only the 
>  (web) traffic generated from my router and directed to internet?
> 
> Anymore, I don''t need it more: I''ve resolved my problem,
the conflict
> between a "ping script" (that I''m writing for multiple
gateway
> testing)and servers executed in router too: first version of my script 
> sets a default gateway for testing it with ping, now I''ve
discovered
> that I can use a specific route involving the gateway without setting 
> default gateway, a much better solution.
> 
It can and can not be done at the same time, depends on what you are 
doing. Normally for bound services you have this:

o	Service is bound to a specific IP 1.2.3.4
o	Its outgoing packet has SRC of 1.2.3.4
o	You mark it in the OUTPUT chain based on that SRC
o	The routing (which occurs after OUTPUT) acts on the MARK

Now what happens when there is no specific binding (you send from 0.0.0.0):

o	Program requests a socket from the kernel, supplying only a DST
o	The kernel consults the _default_ routing table (because it does not 
know any better, there are no marks yet), and _assigns_ a SRC that seems 
the closest to this particular DST
o	Everything else happens as in the scenario above


So depending on what you are doing it might help you or it might drive 
you insane. In your case it plays out nicely - you can request a 
specific interface (what you would do with the ping script), and you are 
guaranteed that packets are going this direction. But if you want to 
_balance_ locally generated traffic - you can not do anything short of 
NATing local connections (ugly), because the routing sort of happens 
before netfilter had a chance to play.

Peter Rabbitson ha scritto:
> o    The routing (which occurs after OUTPUT) acts on the MARK
				   ^
This is the focal point I''m searching for

> Now what happens when there is no specific binding (you send from 0.0.0.0):
[snip]

Very very clear. Thanks very much!!! The only still obscure aspect for 
me is this:

 >you can request a specific interface (what you would do with the ping 
script)

Andrea wrote:
> Very very clear. Thanks very much!!! The only still obscure aspect for 
> me is this:
> 
>  >you can request a specific interface (what you would do with the ping 
> script)
Check the man page of ping, and look for the ''-I'' option. Most
network
testing utilities have this capability in one form or another.By the way 
if you request an _interface_ and not a specific IP, the first IP of the 
interface is taken as listed by `ip addr`

Here is my issue with ping.

When I use -I with ping, the DNS queries for that domain is still sent
out with wrong source address through the interface, and hence, no
reply. This happens in both WAN interfaces.
When I add rules in OUTPUT chain to reroute packets with the unmatching
source address and output interface, things work fine.

When I use IP address instead of URL, everything is fine.

I have applied Julian''s routes patch. 

What could be the problem?

-----Original Message-----
From: lartc-bounces@mailman.ds9a.nl
[mailto:lartc-bounces@mailman.ds9a.nl] On Behalf Of Peter Rabbitson
Sent: Tuesday, June 12, 2007 5:29 PM
To: Andrea
Cc: lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Re: multiple routing tables for internal router
programs

Andrea wrote:
> Very very clear. Thanks very much!!! The only still obscure aspect for
> me is this:
> 
>  >you can request a specific interface (what you would do with the
ping 
> script)
Check the man page of ping, and look for the ''-I'' option. Most
network
testing utilities have this capability in one form or another.By the way

if you request an _interface_ and not a specific IP, the first IP of the

interface is taken as listed by `ip addr`
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Salim S I wrote:
> Here is my issue with ping.
> 
> When I use -I with ping, the DNS queries for that domain is still sent
> out with wrong source address through the interface, and hence, no
> reply. This happens in both WAN interfaces.
> When I add rules in OUTPUT chain to reroute packets with the unmatching
> source address and output interface, things work fine.
> 
> When I use IP address instead of URL, everything is fine.
> 
The problem is ping itself, which uses gethostbyname() which in turn 
does not understand how to bind to specific interfaces etc. Besides 
specifying IP addresses instead of hostnames is much much better IMHO. 
Here is the ping.c snippet:

         while (argc > 0) { 
 

                 target = *argv; 
 

 
 

                 bzero((char *)&whereto, sizeof(whereto)); 
 

                 whereto.sin_family = AF_INET; 
 

                 if (inet_aton(target, &whereto.sin_addr) == 1) { 
 

                         hostname = target; 
 

                         if (argc == 1) 
 

                                 options |= F_NUMERIC; 
 

                 } else { 
 

                         hp = gethostbyname2(target, AF_INET); 
 

                         if (!hp) { 
 

                                 fprintf(stderr, "ping: unknown host 
%s\n", target); 

                                 exit(2); 
 

                         } 
 

                         memcpy(&whereto.sin_addr, hp->h_addr, 4); 
 

                         strncpy(hnamebuf, hp->h_name, sizeof(hnamebuf) 
- 1); 

                         hnamebuf[sizeof(hnamebuf) - 1] = 0; 
 

                         hostname = hnamebuf; 
 

                 } 
 

                 if (argc > 1) 
 

                         route[nroute++] = whereto.sin_addr.s_addr; 
 

                 argc--; 
 

                 argv++; 
 

         }

Peter Rabbitson ha scritto:
> Check the man page of ping, and look for the ''-I'' option.
Most network
> testing utilities have this capability in one form or another.By the way 
> if you request an _interface_ and not a specific IP, the first IP of the 
> interface is taken as listed by `ip addr`
Didn''t know about this option. With this, my (old) script should work 
fine too.

Another lesson learned, thanks :-)

Thanks! I get it now.
But why the src address for the interface is wrong? 
In my case eth2 has a.b.c.d and eth3 has p.q.r.s.

DNS queries going through eth2 has p.q.r.s as src address and those
going through eth3 has a.b.c.d. Something wrong with routing?

I was wondering, how the ping script (to check the lonk status) of
others work id domain name is used.

-----Original Message-----
From: Peter Rabbitson [mailto:rabbit@rabbit.us] 
Sent: Tuesday, June 12, 2007 6:02 PM
To: Salim S I
Cc: ''Andrea''; lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Re: multiple routing tables for internal router
programs

Salim S I wrote:
> Here is my issue with ping.
> 
> When I use -I with ping, the DNS queries for that domain is still sent
> out with wrong source address through the interface, and hence, no
> reply. This happens in both WAN interfaces.
> When I add rules in OUTPUT chain to reroute packets with the
unmatching
> source address and output interface, things work fine.
> 
> When I use IP address instead of URL, everything is fine.
> 
The problem is ping itself, which uses gethostbyname() which in turn 
does not understand how to bind to specific interfaces etc. Besides 
specifying IP addresses instead of hostnames is much much better IMHO. 
Here is the ping.c snippet:

         while (argc > 0) { 
 

                 target = *argv; 
 

 
 

                 bzero((char *)&whereto, sizeof(whereto)); 
 

                 whereto.sin_family = AF_INET; 
 

                 if (inet_aton(target, &whereto.sin_addr) == 1) { 
 

                         hostname = target; 
 

                         if (argc == 1) 
 

                                 options |= F_NUMERIC; 
 

                 } else { 
 

                         hp = gethostbyname2(target, AF_INET); 
 

                         if (!hp) { 
 

                                 fprintf(stderr, "ping: unknown host 
%s\n", target); 

                                 exit(2); 
 

                         } 
 

                         memcpy(&whereto.sin_addr, hp->h_addr, 4); 
 

                         strncpy(hnamebuf, hp->h_name, sizeof(hnamebuf) 
- 1); 

                         hnamebuf[sizeof(hnamebuf) - 1] = 0; 
 

                         hostname = hnamebuf; 
 

                 } 
 

                 if (argc > 1) 
 

                         route[nroute++] = whereto.sin_addr.s_addr; 
 

                 argc--; 
 

                 argv++; 
 

         }

Salim S I wrote:
> Thanks! I get it now.
> But why the src address for the interface is wrong? 
> In my case eth2 has a.b.c.d and eth3 has p.q.r.s.
> 
> DNS queries going through eth2 has p.q.r.s as src address and those
> going through eth3 has a.b.c.d. Something wrong with routing?
Possible. Post full configuration and someone might be able to help.
> I was wondering, how the ping script (to check the lonk status) of
> others work id domain name is used.
Don''t know about others, and I personally use ip addresses :)

My configuration 

root@127.0.0.1:~# ip ru
0:      from all lookup local
32150:  from all lookup main
32201:  from all fwmark 0x200/0x200 lookup wan1_route
32202:  from all fwmark 0x400/0x400 lookup wan2_route
32203:  from all lookup catch_all
32766:  from all lookup main
32767:  from all lookup default

root@127.0.0.1:~# ip ro li ta main
192.168.100.0/24 dev eth0  proto kernel  scope link  src 192.168.100.254
10.20.0.0/24 dev eth2  proto kernel  scope link  src 10.20.0.137
192.168.1.0/24 dev eth10  proto kernel  scope link  src 192.168.1.254
10.2.3.0/24 dev eth3  proto kernel  scope link  src 10.2.3.107
127.0.0.0/8 dev lo  scope link

root@127.0.0.1:~# ip ro li ta wan1_route
default via 10.20.0.1 dev eth2  proto static
root@127.0.0.1:~# ip ro li ta wan2_route
default via 10.2.3.254 dev eth3  proto static

root@127.0.0.1:~# ip ro li ta catch_all
default  proto static
        nexthop via 10.20.0.1  dev eth2 weight 1
        nexthop via 10.2.3.254  dev eth3 weight 1

The catch_all table comes into play only for local packets. All
forwarded packets are marked in mangle PREROUTING, with 0x200 0r 0x400.

If not loadblancing ping script, there maybe other apps using domain
names instead of IP address, they might still fail, right?

The problem happens when one of the link goes down (not the nexthop,but
after that). Then the kernel will pick an interface and wrong src IP for
local packets.


-----Original Message-----
From: Peter Rabbitson [mailto:rabbit@rabbit.us] 
Sent: Tuesday, June 12, 2007 7:24 PM
To: Salim S I
Cc: lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Re: multiple routing tables for internal router
programs

Salim S I wrote:
> Thanks! I get it now.
> But why the src address for the interface is wrong? 
> In my case eth2 has a.b.c.d and eth3 has p.q.r.s.
> 
> DNS queries going through eth2 has p.q.r.s as src address and those
> going through eth3 has a.b.c.d. Something wrong with routing?
Possible. Post full configuration and someone might be able to help.
> I was wondering, how the ping script (to check the lonk status) of
> others work id domain name is used.
Don''t know about others, and I personally use ip addresses :)

I solved it, thought a bit ugly.

Have two more rules now in ip ru

32150:  from all lookup main
32201:  from all fwmark 0x200/0x200 lookup wan1_route
32202:  from all fwmark 0x400/0x400 lookup wan2_route
32203:  from 10.20.0.137 lookup wan1_route
32204:  from 10.2.3.107 lookup wan2_route
32205:  from all lookup catch_all
32766:  from all lookup main

I did not like to include WAN IP anywhere, coz it may be dynamic, but
well, seems like no choice.

And then two rules in OUTPUT chain
Iptables -t mangle -A OUTPUT -o eth2 -j LB1
Iptables -t mangle -A OUTPUT -o eth3 -j LB2

-----Original Message-----
From: lartc-bounces@mailman.ds9a.nl
[mailto:lartc-bounces@mailman.ds9a.nl] On Behalf Of Salim S I
Sent: Wednesday, June 13, 2007 12:08 PM
To: ''Peter Rabbitson''
Cc: lartc@mailman.ds9a.nl
Subject: RE: [LARTC] Re: multiple routing tables for internal router
programs

My configuration 

root@127.0.0.1:~# ip ru
0:      from all lookup local
32150:  from all lookup main
32201:  from all fwmark 0x200/0x200 lookup wan1_route
32202:  from all fwmark 0x400/0x400 lookup wan2_route
32203:  from all lookup catch_all
32766:  from all lookup main
32767:  from all lookup default

root@127.0.0.1:~# ip ro li ta main
192.168.100.0/24 dev eth0  proto kernel  scope link  src 192.168.100.254
10.20.0.0/24 dev eth2  proto kernel  scope link  src 10.20.0.137
192.168.1.0/24 dev eth10  proto kernel  scope link  src 192.168.1.254
10.2.3.0/24 dev eth3  proto kernel  scope link  src 10.2.3.107
127.0.0.0/8 dev lo  scope link

root@127.0.0.1:~# ip ro li ta wan1_route
default via 10.20.0.1 dev eth2  proto static
root@127.0.0.1:~# ip ro li ta wan2_route
default via 10.2.3.254 dev eth3  proto static

root@127.0.0.1:~# ip ro li ta catch_all
default  proto static
        nexthop via 10.20.0.1  dev eth2 weight 1
        nexthop via 10.2.3.254  dev eth3 weight 1

The catch_all table comes into play only for local packets. All
forwarded packets are marked in mangle PREROUTING, with 0x200 0r 0x400.

If not loadblancing ping script, there maybe other apps using domain
names instead of IP address, they might still fail, right?

The problem happens when one of the link goes down (not the nexthop,but
after that). Then the kernel will pick an interface and wrong src IP for
local packets.


-----Original Message-----
From: Peter Rabbitson [mailto:rabbit@rabbit.us] 
Sent: Tuesday, June 12, 2007 7:24 PM
To: Salim S I
Cc: lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Re: multiple routing tables for internal router
programs

Salim S I wrote:
> Thanks! I get it now.
> But why the src address for the interface is wrong? 
> In my case eth2 has a.b.c.d and eth3 has p.q.r.s.
> 
> DNS queries going through eth2 has p.q.r.s as src address and those
> going through eth3 has a.b.c.d. Something wrong with routing?
Possible. Post full configuration and someone might be able to help.
> I was wondering, how the ping script (to check the lonk status) of
> others work id domain name is used.
Don''t know about others, and I personally use ip addresses :)


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

On Thu, Jun 14, 2007 at 11:50:30AM +0800, Salim S I
wrote:
> I solved it, thought a bit ugly.
> 
> Have two more rules now in ip ru
> 
> 32150:  from all lookup main
> 32201:  from all fwmark 0x200/0x200 lookup wan1_route
> 32202:  from all fwmark 0x400/0x400 lookup wan2_route
> 32203:  from 10.20.0.137 lookup wan1_route
> 32204:  from 10.2.3.107 lookup wan2_route
> 32205:  from all lookup catch_all
> 32766:  from all lookup main
> 
> I did not like to include WAN IP anywhere, coz it may be dynamic, but
> well, seems like no choice.
ran into the same problem, I capture the link information at ip-up time for 
ppp/pppoe and dhcp time for cable modem, then I fire off a scrip that pulls 
down all the ip ru & ip ro and builds it from scratch (as well as the 
specialised iptables rules as well).  This should only happen when I loose a 
connection so should be okay

> 
> And then two rules in OUTPUT chain
> Iptables -t mangle -A OUTPUT -o eth2 -j LB1
> Iptables -t mangle -A OUTPUT -o eth3 -j LB2
> 
> -----Original Message-----
> From: lartc-bounces@mailman.ds9a.nl
> [mailto:lartc-bounces@mailman.ds9a.nl] On Behalf Of Salim S I
> Sent: Wednesday, June 13, 2007 12:08 PM
> To: ''Peter Rabbitson''
> Cc: lartc@mailman.ds9a.nl
> Subject: RE: [LARTC] Re: multiple routing tables for internal router
> programs
> 
> My configuration 
> 
> root@127.0.0.1:~# ip ru
> 0:      from all lookup local
> 32150:  from all lookup main
> 32201:  from all fwmark 0x200/0x200 lookup wan1_route
> 32202:  from all fwmark 0x400/0x400 lookup wan2_route
> 32203:  from all lookup catch_all
> 32766:  from all lookup main
> 32767:  from all lookup default
> 
> root@127.0.0.1:~# ip ro li ta main
> 192.168.100.0/24 dev eth0  proto kernel  scope link  src 192.168.100.254
> 10.20.0.0/24 dev eth2  proto kernel  scope link  src 10.20.0.137
> 192.168.1.0/24 dev eth10  proto kernel  scope link  src 192.168.1.254
> 10.2.3.0/24 dev eth3  proto kernel  scope link  src 10.2.3.107
> 127.0.0.0/8 dev lo  scope link
> 
> root@127.0.0.1:~# ip ro li ta wan1_route
> default via 10.20.0.1 dev eth2  proto static
> root@127.0.0.1:~# ip ro li ta wan2_route
> default via 10.2.3.254 dev eth3  proto static
> 
> root@127.0.0.1:~# ip ro li ta catch_all
> default  proto static
>         nexthop via 10.20.0.1  dev eth2 weight 1
>         nexthop via 10.2.3.254  dev eth3 weight 1
> 
> The catch_all table comes into play only for local packets. All
> forwarded packets are marked in mangle PREROUTING, with 0x200 0r 0x400.
> 
> If not loadblancing ping script, there maybe other apps using domain
> names instead of IP address, they might still fail, right?
> 
> The problem happens when one of the link goes down (not the nexthop,but
> after that). Then the kernel will pick an interface and wrong src IP for
> local packets.
> 
> 
> -----Original Message-----
> From: Peter Rabbitson [mailto:rabbit@rabbit.us] 
> Sent: Tuesday, June 12, 2007 7:24 PM
> To: Salim S I
> Cc: lartc@mailman.ds9a.nl
> Subject: Re: [LARTC] Re: multiple routing tables for internal router
> programs
> 
> Salim S I wrote:
> > Thanks! I get it now.
> > But why the src address for the interface is wrong? 
> > In my case eth2 has a.b.c.d and eth3 has p.q.r.s.
> > 
> > DNS queries going through eth2 has p.q.r.s as src address and those
> > going through eth3 has a.b.c.d. Something wrong with routing?
> 
> Possible. Post full configuration and someone might be able to help.
> 
> > I was wondering, how the ping script (to check the lonk status) of
> > others work id domain name is used.
> 
> Don''t know about others, and I personally use ip addresses :)
> 
> 
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
> 
> 
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
> 

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Salim S I wrote:
> I solved it, thought a bit ugly.
> 
Sorry I didn''t answer earlier. Can you post your iptables rules too,
the
routing alone is not sufficient. If your setup is confidential at least 
show all statements that set MARKs one way or another. What you did is 
strange, but it might very well be warranted. Still - depends on your 
existing rules.

The relevant portions are:

root@127.0.0.1:~# iptables -t mangle -L LOC -v
Chain LOC (1 references)
 pkts bytes target     prot opt in     out     source
destination
10125 1152K CONNMARK   all  --  any    any     anywhere
anywhere            CONNMARK restore
   64 12017 LB1        all  --  any    any     anywhere
anywhere            state NEW MARK match 0x0  random 84%
  174 28502 LB2        all  --  any    any     anywhere
anywhere            state NEW MARK match 0x0

root@127.0.0.1:~# iptables -t mangle -L LB1 -v
Chain LB1 (2 references)
 pkts bytes target     prot opt in     out     source
destination
 2350  257K MARK      all  --  any    any     anywhere
anywhere            MARK or 0x200
 2350  257K CONNMARK   all  --  any    any     anywhere
anywhere            CONNMARK save

root@127.0.0.1:~# iptables -t mangle -L LB2 -v
Chain LB2 (2 references)
 pkts bytes target     prot opt in     out     source
destination
 6931 1196K MARK      all  --  any    any     anywhere
anywhere            MARK or 0x400
 6931 1196K CONNMARK   all  --  any    any     anywhere
anywhere            CONNMARK save

root@127.0.0.1:~# iptables -t mangle -L OUTPUT -v
Chain OUTPUT (policy ACCEPT 8358 packets, 1290K bytes)
 pkts bytes target     prot opt in     out     source
destination
 1551  119K LB1        all  --  any    eth2    anywhere
anywhere
 6788 1170K LB2        all  --  any    eth3    anywhere
anywhere

NATing is done with MASQUERADE, not SNAT, I use another MARK for it, but
in essence it is 
-o eth2 -j MASQUEARDE
-o eth3 -j MASQUEARDE

In addition, there are several other MARKs for policy routing. They have
their own routing tables also. But at present, they are all empty.

-----Original Message-----
From: Peter Rabbitson [mailto:rabbit@rabbit.us] 
Sent: Thursday, June 14, 2007 3:27 PM
To: Salim S I
Cc: lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Re: multiple routing tables for internal router
programs

Salim S I wrote:
> I solved it, thought a bit ugly.
> 
Sorry I didn''t answer earlier. Can you post your iptables rules too,
the

routing alone is not sufficient. If your setup is confidential at least 
show all statements that set MARKs one way or another. What you did is 
strange, but it might very well be warranted. Still - depends on your 
existing rules.

I do the same way, from ip-up. But I only change the two concerned
rules. Rest of the things are free from IP.

-----Original Message-----
From: lartc-bounces@mailman.ds9a.nl
[mailto:lartc-bounces@mailman.ds9a.nl] On Behalf Of Alex Samad
Sent: Thursday, June 14, 2007 12:23 PM
To: lartc@mailman.ds9a.nl
Subject: Re: [LARTC] Re: multiple routing tables for internal router
programs

On Thu, Jun 14, 2007 at 11:50:30AM +0800, Salim S I
wrote:
> I solved it, thought a bit ugly.
> 
> Have two more rules now in ip ru
> 
> 32150:  from all lookup main
> 32201:  from all fwmark 0x200/0x200 lookup wan1_route
> 32202:  from all fwmark 0x400/0x400 lookup wan2_route
> 32203:  from 10.20.0.137 lookup wan1_route
> 32204:  from 10.2.3.107 lookup wan2_route
> 32205:  from all lookup catch_all
> 32766:  from all lookup main
> 
> I did not like to include WAN IP anywhere, coz it may be dynamic, but
> well, seems like no choice.
ran into the same problem, I capture the link information at ip-up time
for 
ppp/pppoe and dhcp time for cable modem, then I fire off a scrip that
pulls 
down all the ip ru & ip ro and builds it from scratch (as well as the 
specialised iptables rules as well).  This should only happen when I
loose a 
connection so should be okay

> 
> And then two rules in OUTPUT chain
> Iptables -t mangle -A OUTPUT -o eth2 -j LB1
> Iptables -t mangle -A OUTPUT -o eth3 -j LB2
> 
> -----Original Message-----
> From: lartc-bounces@mailman.ds9a.nl
> [mailto:lartc-bounces@mailman.ds9a.nl] On Behalf Of Salim S I
> Sent: Wednesday, June 13, 2007 12:08 PM
> To: ''Peter Rabbitson''
> Cc: lartc@mailman.ds9a.nl
> Subject: RE: [LARTC] Re: multiple routing tables for internal router
> programs
> 
> My configuration 
> 
> root@127.0.0.1:~# ip ru
> 0:      from all lookup local
> 32150:  from all lookup main
> 32201:  from all fwmark 0x200/0x200 lookup wan1_route
> 32202:  from all fwmark 0x400/0x400 lookup wan2_route
> 32203:  from all lookup catch_all
> 32766:  from all lookup main
> 32767:  from all lookup default
> 
> root@127.0.0.1:~# ip ro li ta main
> 192.168.100.0/24 dev eth0  proto kernel  scope link  src
192.168.100.254
> 10.20.0.0/24 dev eth2  proto kernel  scope link  src 10.20.0.137
> 192.168.1.0/24 dev eth10  proto kernel  scope link  src 192.168.1.254
> 10.2.3.0/24 dev eth3  proto kernel  scope link  src 10.2.3.107
> 127.0.0.0/8 dev lo  scope link
> 
> root@127.0.0.1:~# ip ro li ta wan1_route
> default via 10.20.0.1 dev eth2  proto static
> root@127.0.0.1:~# ip ro li ta wan2_route
> default via 10.2.3.254 dev eth3  proto static
> 
> root@127.0.0.1:~# ip ro li ta catch_all
> default  proto static
>         nexthop via 10.20.0.1  dev eth2 weight 1
>         nexthop via 10.2.3.254  dev eth3 weight 1
> 
> The catch_all table comes into play only for local packets. All
> forwarded packets are marked in mangle PREROUTING, with 0x200 0r
0x400.
> 
> If not loadblancing ping script, there maybe other apps using domain
> names instead of IP address, they might still fail, right?
> 
> The problem happens when one of the link goes down (not the
nexthop,but
> after that). Then the kernel will pick an interface and wrong src IP
for
> local packets.
> 
> 
> -----Original Message-----
> From: Peter Rabbitson [mailto:rabbit@rabbit.us] 
> Sent: Tuesday, June 12, 2007 7:24 PM
> To: Salim S I
> Cc: lartc@mailman.ds9a.nl
> Subject: Re: [LARTC] Re: multiple routing tables for internal router
> programs
> 
> Salim S I wrote:
> > Thanks! I get it now.
> > But why the src address for the interface is wrong? 
> > In my case eth2 has a.b.c.d and eth3 has p.q.r.s.
> > 
> > DNS queries going through eth2 has p.q.r.s as src address and those
> > going through eth3 has a.b.c.d. Something wrong with routing?
> 
> Possible. Post full configuration and someone might be able to help.
> 
> > I was wondering, how the ping script (to check the lonk status) of
> > others work id domain name is used.
> 
> Don''t know about others, and I personally use ip addresses :)
> 
> 
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
> 
> 
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>

Salim S I wrote:
>
> NATing is done with MASQUERADE, not SNAT, I use another MARK for it, but
> in essence it is 
> -o eth2 -j MASQUEARDE
> -o eth3 -j MASQUEARDE
> 
> In addition, there are several other MARKs for policy routing. They have
> their own routing tables also. But at present, they are all empty.
> 
This is the part I definitely do not like. First of all - wht 
SNAT/MASQUERADE _all_ traffic? You should do this for forwarder traffic 
only. Like so:

     iptables -t nat -A POSTROUTING -s 10.0.58.0/24 -j SOURCE_NAT 
 

     iptables -t nat -A POSTROUTING -s 192.168.58.0/24 -j SOURCE_NAT 
 

     iptables -t nat -A POSTROUTING -s 192.168.8.0/24 -j SOURCE_NAT 
 

 
 

     iptables -t nat -A SOURCE_NAT -o $EXTCH -j SNAT --to $EXTCH_IP 
 

     iptables -t nat -A SOURCE_NAT -o $EXTCB -j SNAT --to $EXTCB_IP

Also you mention that there are "other marks" , which means that you 
might very well be overwriting marks as you go. A packet/connection can 
have only _one_ mark value at any time, no more no less (a 0x0 is still 
a mark)

HTH

Salim S I wrote:
>
> NATing is done with MASQUERADE, not SNAT, I use another MARK for it, but
> in essence it is 
> -o eth2 -j MASQUEARDE
> -o eth3 -j MASQUEARDE
> 
> In addition, there are several other MARKs for policy routing. They have
> their own routing tables also. But at present, they are all empty.
> 
This is the part I definitely do not like. First of all - wht 
SNAT/MASQUERADE _all_ traffic? You should do this for forwarder traffic 
only. Like so:

     iptables -t nat -A POSTROUTING -s 10.0.58.0/24 -j SOURCE_NAT 
 

     iptables -t nat -A POSTROUTING -s 192.168.58.0/24 -j SOURCE_NAT 
 

     iptables -t nat -A POSTROUTING -s 192.168.8.0/24 -j SOURCE_NAT 
 

 
 

     iptables -t nat -A SOURCE_NAT -o $EXTCH -j SNAT --to $EXTCH_IP 
 

     iptables -t nat -A SOURCE_NAT -o $EXTCB -j SNAT --to $EXTCB_IP

Also you mention that there are "other marks" , which means that you 
might very well be overwriting marks as you go. A packet/connection can 
have only _one_ mark value at any time, no more no less (a 0x0 is still 
a mark)

HTH

> > NATing is done with MASQUERADE, not SNAT, I use another MARK for it,
but
> > in essence it is
> > -o eth2 -j MASQUEARDE
> > -o eth3 -j MASQUEARDE
> >
> > In addition, there are several other MARKs for policy routing. They
have
> > their own routing tables also. But at present, they are all empty.
> >
> 
> This is the part I definitely do not like. First of all - wht
> SNAT/MASQUERADE _all_ traffic? You should do this for forwarder
traffic
> only. Like so:
Yes, in fact, this is what I do. I mentioned I use MARK for
MASQUERADing, but forgot to elaborate. That particular MARK is set for
forwarded packets only.

> Also you mention that there are "other marks" , which means that
you
> might very well be overwriting marks as you go. A packet/connection
can
> have only _one_ mark value at any time, no more no less (a 0x0 is
still
> a mark)

I use --or-mark in iptables, so that I can use bitwise masks. The
''ip''
tool supports bit masks too.

Salim S I wrote:
> 
>>> NATing is done with MASQUERADE, not SNAT, I use another MARK for
it,
> but
>>> in essence it is
>>> -o eth2 -j MASQUEARDE
>>> -o eth3 -j MASQUEARDE
>>>
>>> In addition, there are several other MARKs for policy routing. They
> have
>>> their own routing tables also. But at present, they are all empty.
>>>
>> This is the part I definitely do not like. First of all - wht
>> SNAT/MASQUERADE _all_ traffic? You should do this for forwarder
> traffic
>> only. Like so:
> 
> Yes, in fact, this is what I do. I mentioned I use MARK for
> MASQUERADing, but forgot to elaborate. That particular MARK is set for
> forwarded packets only.
> 
> 
>> Also you mention that there are "other marks" , which means
that you
>> might very well be overwriting marks as you go. A packet/connection
> can
>> have only _one_ mark value at any time, no more no less (a 0x0 is
> still
>> a mark)
> 
> 
> I use --or-mark in iptables, so that I can use bitwise masks. The
''ip''
> tool supports bit masks too.
> 
Well then you are certainly ahead of the game. Still I would suggest to 
avoid the complexity of bit mask marks - it is rather error prone and is 
pretty hard to maintain, while the same result can usually be achieved 
by other means (like in my SNAT example). As far as your original 
problem goes - it seems like a mark is getting eaten away or is not set 
somewhere in the first place. I have not had any problems like the ones 
you describe.

> -----Original Message-----
> From: lartc-bounces@mailman.ds9a.nl
[mailto:lartc-bounces@mailman.ds9a.nl]
> On Behalf Of Peter Rabbitson
> Sent: Friday, June 15, 2007 2:30 PM
> Cc: lartc@mailman.ds9a.nl
> Subject: Re: [LARTC] Re: multiple routing tables for internal router
> programs
> Well then you are certainly ahead of the game. Still I would suggest
to
> avoid the complexity of bit mask marks - it is rather error prone and
is
> pretty hard to maintain, while the same result can usually be achieved
> by other means (like in my SNAT example). As far as your original
> problem goes - it seems like a mark is getting eaten away or is not
set
> somewhere in the first place. I have not had any problems like the
ones
> you describe.
Those different MARKs are used for policy-routing, load balancing,
firewall, traffic control, virtual server, user-group profiles etc. I
think eventually you may have to use it, warts and all, or find some
other way for integrating all those. :-) I will soon run out of bits, it
seems.

I''ve replaced that multipath rule for local packets with a single
route,
and change it on failovers. No balancing for local traffic, but there
isn''t much local traffic anyway.