LARTC - Apr 2007 - equalize / ecmp not working as expected in 2.6 vs 2.4

Hi,

With kernel 2.4 I was able to use equalize to send our outgoing packets to
two different routers (our isp supports this setup), like this:

ip route add default src ip.a.dd.rr  equalize nexthop via <router1> weight
1
nexthop via <router2> weight 1

The two routes were used equally on a per packet basis, not per flow or per
cached route, but per packet, each line has 800k upload and with that route
we could upload to a single remote host at 1.6mbit.

We replaced the server with a newer one and changed to 2.6 (2.6.20) kernel,
I found that equalize no longer works as expected, it does choose a router
at random but once a single packet has been sent to a remote host the same
route/router is used for all packets going to that remote host. Once the
cached route expires a random route is chosen again, but that is not what we
want.

I had made no changes to the ip route commands, but then I realised that
kernel 2.6.20 has options for multipath (IP: equal cost multipath with
caching support), I enabled that and now our kernel options are:

CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_MULTIPATH_CACHED=y
CONFIG_IP_ROUTE_MULTIPATH_RR=m
CONFIG_IP_ROUTE_MULTIPATH_RANDOM=m
CONFIG_IP_ROUTE_MULTIPATH_WRANDOM=m
CONFIG_IP_ROUTE_MULTIPATH_DRR=m

But even with these options, and default route set as follows:

ip route add default src ip.a.dd.rr mpath rr nexthop via <router1> weight
1
nexthop via <router2> weight 1

The result is the same, a single upload to a remote host only uses 800k
bandwidth on one of the lines, it does not send packets to both lines,
although two uploads to two different hosts will usually make use of both
lines.

It seems to me that the multipath with caching support is broken in 2.6.20?

The exact kernel we use is 2.6.20.4, with that kernel how would you specify
a remote route such that packets going to a remote host are sent 50/50 ratio
to two different routers?

Thanks
Andy

JOSEDV001TAG

Hi Andrew,

I would use a combination with iptables. You should mark the packets, for 
example using average or n-th, and then use ip rules to send half of the 
packets vía one router and the rest to the other router according to the 
marks you set with iptables.
Just a question ¿ dont you have problems with your source IP and the 
returning responses when you are sending packets from one connection over 
multiple routers ? ¿ do you have something like an AS ?


Best regards,
Eric Janz 




Andrew Lyon <andrew.lyon@josims.com> 
Enviado por: lartc-bounces@mailman.ds9a.nl
10/04/2007 14:36

Para
"''lartc@mailman.ds9a.nl''"
<lartc@mailman.ds9a.nl>
cc

Asunto
[LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4






Hi,

With kernel 2.4 I was able to use equalize to send our outgoing packets to
two different routers (our isp supports this setup), like this:

ip route add default src ip.a.dd.rr  equalize nexthop via <router1> weight
1
nexthop via <router2> weight 1

The two routes were used equally on a per packet basis, not per flow or 
per
cached route, but per packet, each line has 800k upload and with that 
route
we could upload to a single remote host at 1.6mbit.

We replaced the server with a newer one and changed to 2.6 (2.6.20) 
kernel,
I found that equalize no longer works as expected, it does choose a router
at random but once a single packet has been sent to a remote host the same
route/router is used for all packets going to that remote host. Once the
cached route expires a random route is chosen again, but that is not what 
we
want.

I had made no changes to the ip route commands, but then I realised that
kernel 2.6.20 has options for multipath (IP: equal cost multipath with
caching support), I enabled that and now our kernel options are:

CONFIG_IP_ROUTE_MULTIPATH=y
CONFIG_IP_ROUTE_MULTIPATH_CACHED=y
CONFIG_IP_ROUTE_MULTIPATH_RR=m
CONFIG_IP_ROUTE_MULTIPATH_RANDOM=m
CONFIG_IP_ROUTE_MULTIPATH_WRANDOM=m
CONFIG_IP_ROUTE_MULTIPATH_DRR=m

But even with these options, and default route set as follows:

ip route add default src ip.a.dd.rr mpath rr nexthop via <router1> weight 
1
nexthop via <router2> weight 1

The result is the same, a single upload to a remote host only uses 800k
bandwidth on one of the lines, it does not send packets to both lines,
although two uploads to two different hosts will usually make use of both
lines.

It seems to me that the multipath with caching support is broken in 
2.6.20?

The exact kernel we use is 2.6.20.4, with that kernel how would you 
specify
a remote route such that packets going to a remote host are sent 50/50 
ratio
to two different routers?

Thanks
Andy

JOSEDV001TAG
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc


--

ADVERTENCIA LEGAL
El contenido de este correo es confidencial y dirigido unicamente a su
destinatario. Para acceder a su clausula de privacidad consulte
http://www.barceloviajes.com/privacy

LEGAL ADVISORY
This message is confidential and intended only for the person or entity to which
it is addressed. In order to read its privacy policy consult it at
http://www.barceloviajes.com/privacy


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

>
>________________________________________
>From: e.janz@barceloviajes.com [mailto:e.janz@barceloviajes.com] 
>Sent: 10 April 2007 15:17
>To: lartc@mailman.ds9a.nl
>Subject: Re: [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4
>
>
>Hi Andrew,
>
>I would use a combination with iptables. You should mark the packets, for
example using average or n-th, and then use ip rules to send half of the
packets vía one router and the rest to the other router according to the
marks you set with iptables.
>Just a question ¿ dont you have problems with your source IP and the
returning responses when you are sending packets from one connection over
multiple routers ? ¿ do you have something like an AS ?
>
>
>Best regards,
>Eric Janz 
>
>
>Andrew Lyon <andrew.lyon@josims.com>
>Enviado por: lartc-bounces@mailman.ds9a.nl
>10/04/2007 14:36
>Para
>"''lartc@mailman.ds9a.nl''"
<lartc@mailman.ds9a.nl>
>cc
>
>Asunto
> [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4


Eric,

Could you give me a example of how to do that? With nth if possible...

It is not common for a ISP to support that sort of setup, but they do
http://aaisp.net.uk/aa/aaisp/multiline.html

Each line has two ips, one for the router and another for the interface on a
linux box or other device, the isp routes a larger /28 down both lines, and
allows packets with source address in the /28 range to be sent through both
lines.

On my linux server I have a routing table for each line with the necessary
routes to make each router ip reachable, and a default route that equalizes
over both router ips, it worked with 2.4 but with 2.6 it seems to be
per-flow instead of per packet.

I can login to a control page app on the ISP website and configure which
lines a given block is routed down, and they also do really good traffic
monitoring etc http://www.aaisp.net.uk/cqm.html

PS. Please reply below original posting, not above!
http://en.wikipedia.org/wiki/Top-posting

Andy


JOSEDV001TAG

Andrew Lyon <andrew.lyon@josims.com> wrote on 10/04/2007 16:34:52:
> >
> >________________________________________
> >From: e.janz@barceloviajes.com [mailto:e.janz@barceloviajes.com] 
> >Sent: 10 April 2007 15:17
> >To: lartc@mailman.ds9a.nl
> >Subject: Re: [LARTC] equalize / ecmp not working as expected in 2.6 vs 
2.4
> >
> >
> >Hi Andrew,
> >
> >I would use a combination with iptables. You should mark the packets, 
for
> example using average or n-th, and then use ip rules to send half of the
> packets vía one router and the rest to the other router according to the
> marks you set with iptables.
> >Just a question ¿ dont you have problems with your source IP and the
> returning responses when you are sending packets from one connection 
over
> multiple routers ? ¿ do you have something like an AS ?
> >
> >
> >Best regards,
> >Eric Janz 
> >
> >
> >Andrew Lyon <andrew.lyon@josims.com>
> >Enviado por: lartc-bounces@mailman.ds9a.nl
> >10/04/2007 14:36
> >Para
> >"''lartc@mailman.ds9a.nl''"
<lartc@mailman.ds9a.nl>
> >cc
> >
> >Asunto
> > [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4
> 
> 
> 
> Eric,
> 
> Could you give me a example of how to do that? With nth if possible...
> 
> It is not common for a ISP to support that sort of setup, but they do
> http://aaisp.net.uk/aa/aaisp/multiline.html
> 
> Each line has two ips, one for the router and another for the interface 
on a
> linux box or other device, the isp routes a larger /28 down both lines, 
and
> allows packets with source address in the /28 range to be sent through 
both
> lines.
> 
> On my linux server I have a routing table for each line with the 
necessary
> routes to make each router ip reachable, and a default route that 
equalizes
> over both router ips, it worked with 2.4 but with 2.6 it seems to be
> per-flow instead of per packet.
> 
> I can login to a control page app on the ISP website and configure which
> lines a given block is routed down, and they also do really good traffic
> monitoring etc http://www.aaisp.net.uk/cqm.html
> 
> PS. Please reply below original posting, not above!
> http://en.wikipedia.org/wiki/Top-posting
> 
> Andy
> 
> 
> JOSEDV001TAG
Hi Andy,

thanks for the info. First of all, in order to use the nth match you need 
to patch your kernel using patch-o-matic.
After that, the nth match should be available. Try something like this:

Supposing that the local traffic is entering into your linux server via 
eth0:

1. Mark the packets using iptables before the routing decision is done:

        iptables -t mangle -A PREROUTING -i eth0 -m nth --every 2 --packet 
0 -j MARK --set-mark 111
        iptables -t mangle -A PREROUTING -i eth0 -m nth --every 2 --packet 
1 -j MARK --set-mark 222

2. Setup some rules to jump to the correct routing tables. In this case I 
will suppose that you are using the tables 111 and 222 ( obviously you can 
use the ones you like )
        ip rule add prio 111 fwmark 111 table 111
        ip rule add prio 222 fwmark 222 table 222
( you can also set the priority of the rules at your convenience )

3. Setup your routing tables ( in this example 111 and 222 ) to reach each 
router as you had with the 2.4 kernel.
        [ ... ]
        ip route add table 111 default via ROUTER1_IP_ADDRESS
        ip route add table 222 default via ROUTER2_IP_ADDRESS


In this case we need no multipath route. Half of all the packets that come 
into eth0 are routed using the 111 table and the rest is routed using the 
222 table thanks to the marks we set.
The problems you are experiencing with the multipath routing are due to 
that the routing decision uses a cache and after a routing decision to a 
destination is done, it would always use the same gateway to reach that 
destination until the routing caché expires.


I hope this helps,
Regards,

Eric Janz




--

ADVERTENCIA LEGAL
El contenido de este correo es confidencial y dirigido unicamente a su
destinatario. Para acceder a su clausula de privacidad consulte
http://www.barceloviajes.com/privacy

LEGAL ADVISORY
This message is confidential and intended only for the person or entity to which
it is addressed. In order to read its privacy policy consult it at
http://www.barceloviajes.com/privacy


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

e.janz@barceloviajes.com wrote:
> thanks for the info. First of all, in order to use the nth match you need 
> to patch your kernel using patch-o-matic.
I think nth is in kernel now as part of the statistic match.

Andy.

Andy Furniss <lists@andyfurniss.entadsl.com> wrote on 10/04/2007 21:29:33:
> e.janz@barceloviajes.com wrote:
> 
> > thanks for the info. First of all, in order to use the nth match you 
need 
> > to patch your kernel using patch-o-matic.
> 
> I think nth is in kernel now as part of the statistic match.
> 
> Andy.
I was searching this yesterday afternoon and could not verify it.
Today I found it in the 2.6.18 kernel''s changelog:

    [NETFILTER]: x_tables: add statistic match

    Add statistic match which is a combination of the nth and random 
matches.

    Signed-off-by: Patrick McHardy <...>
    Signed-off-by: David S. Miller <...>

commit 62b7743483b402f8fb73545d5d487ca714e82766
Author: Patrick McHardy <...>
Date:   Mon May 29 18:20:32 2006 -0700


¿ Does this match help you to solve the problem ?

Regards,
Eric
--

ADVERTENCIA LEGAL
El contenido de este correo es confidencial y dirigido unicamente a su
destinatario. Para acceder a su clausula de privacidad consulte
http://www.barceloviajes.com/privacy

LEGAL ADVISORY
This message is confidential and intended only for the person or entity to which
it is addressed. In order to read its privacy policy consult it at
http://www.barceloviajes.com/privacy


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

>
>
>________________________________________
>From: e.janz@barceloviajes.com [mailto:e.janz@barceloviajes.com] 
>Sent: 11 April 2007 09:01
>To: lartc@mailman.ds9a.nl
>Subject: Re: [LARTC] equalize / ecmp not working as expected in 2.6 vs 2.4
>
>
>Andy Furniss <lists@andyfurniss.entadsl.com> wrote on 10/04/2007
21:29:33:
>
>> e.janz@barceloviajes.com wrote:
>> 
>> > thanks for the info. First of all, in order to use the nth match
you
need 
>> > to patch your kernel using patch-o-matic.
>> 
>> I think nth is in kernel now as part of the statistic match.
>> 
>> Andy.
>
>I was searching this yesterday afternoon and could not verify it.
>Today I found it in the 2.6.18 kernel''s changelog:
>
>    [NETFILTER]: x_tables: add statistic match
>
>    Add statistic match which is a combination of the nth and random
matches.
>
>    Signed-off-by: Patrick McHardy <...>
>    Signed-off-by: David S. Miller <...>
>
>commit 62b7743483b402f8fb73545d5d487ca714e82766
>Author: Patrick McHardy <...>
>Date:   Mon May 29 18:20:32 2006 -0700
>
>
>¿ Does this match help you to solve the problem ?
>
>Regards,
>Eric--
>
>ADVERTENCIA LEGAL
>El contenido de este correo es confidencial y dirigido unicamente a su
destinatario. Para acceder a su clausula de privacidad consulte
http://www.barceloviajes.com/privacy
>
>LEGAL ADVISORY
>This message is confidential and intended only for the person or entity to
which it is addressed. In order to read its privacy policy consult it at
http://www.barceloviajes.com/privacy

Your suggestion pointed me in the right direction, it is now working with
the following setup:

Kernel 2.6.20-gentoo-r4 x86_64
Iptables 1.3.7

Iptables rules:

iptables -t mangle -A OUTPUT -s <localip2> -m statistic --mode nth --every
2
--packet 0 -j MARK --set-mark 111
iptables -t mangle -A OUTPUT -s <localip2> -m statistic --mode nth --every
2
--packet 1 -j MARK --set-mark 222

<localip2> is one of the ip addresses from the /28 range that is routed to
both of our lines, in your example you said to add to PREROUTING, but the
packets are from the box itself so I changed to OUTPUT, so far that has not
caused any problems... any comments on that?

We only want to do per-packet load balanced for some local and some routed
ips, not all of them, some services cannot cope with the out of order
packets that arise from sending outgoing traffic through two different
links.

IP Rules:

ip rule add prio 111 fwmark 111 table ADSLLink1
ip rule add prio 222 fwmark 222 table ADSLLink2

Both ADSLLink1 and ADSLLink2 already existed and contain a default route via
the router for line 1 or line 2, they also have routes for other subnets so
that for example I can ping our routers from my workstation which has a
private IP address, I wont show all the routes as the box has some 10 eth
interfaces and it is very long and confusing, but the important bit is:

ip route show table ADSLLink1 | grep default
default via <router1> dev inet0

ip route show table ADSLLink2 | grep default
default via <router2> dev inet0


This is currently working in combination with the ecmp routes that were
already in place, and that is working very well for us, services that suffer
when there are lots of OOOP''s still get per-flow/cached route load
balanced
over the two lines, and services that can handle a few OOOP''s are
getting
the full benefit of 2 x upload speed.

Iptables also gives me much more fine grained control of the setup, when I
have more time I will be making more improvements.

A final note, I got very confused for a while last night because whenever I
used iptables with -t mangle I got a error like this:

iptables --list -t mangle
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
FATAL: Module ip_tables not found.

<snip>


It turns out that this is a issue with having ip_tables compiled into the
kernel, /sbin/iptables tries to modprobe it regardless of that and then
fails because it is not a module, I believe a fix was posted to netfilter
mailing list, I got rid of the error by making a dummy kernel module with
the name ip_tables, not a nice solution but it does suppress the error.

Many thanks for your help
Andy


JOSEDV001TAG

Hello,

I am trying to implement a Broadcast GRE Tunnel that is described at
this link http://linux-ip.net/gl/ip-tunnels/node9.html but it doesnt
seem to be working.

I am seeing the GRE packets for both networks on both sniffers but the
tcpdump i tun0 doesnt show any of the packets from the remote end
getting there.

Can you tell me if I have something misconfigured or if what I am trying
to do is impossible.

Ive listed all the configuration information under the ascii drawing
 
Basically trying to get routing information from network A to network B
without knowing what the ip address is for remote eth0 is, hence the
multicast address, and keeping the overhead as low as possible.


Thanks.
 

               Router         1      2         Router
+---------+  +--------+     +---+  +---+     +--------+  +---------+
|network A|--| Net-99 |-----|INE|--|INE|-----| Net-77 |--|network B|
+---------+  +--------+  |  +---+  +---+  |  +--------+  +---------+
             eth1  eth0  |  PT CT  CT PT  |  eth0  eth1
                         |                |
                     +-------+        +-------+
                     |sniffer|        |sniffer|
                     +-------+        +-------+



Net99 eth0: 172.16.1.240/24
Net99 eth1: 99.99.99.99/8

Net77 eth0: 172.16.2.240/24
Net77 eth1: 77.77.77.77/8

INE - Inline Network Encryptor

INE1 PT: 172.16.1.1/24
INE1 CT: 10.0.0.1/24

INE2 PT: 172.16.2.1/24
INE2 CT: 10.0.0.2/24

Router Net99
===========
Create Tunnel
-------------

modprobe ip-gre
ip tunnel add tun0 mode gre remote 239.0.0.1 local 172.16.1.240 dev eth0
ip addr add 10.20.1.1/24 dev tun0
ip link set tun0 up
ip link seg gre0 up

=======================================================================
zebra.conf
----------

hostname net99
password itac
enable password itac

interface eth0
  description To KG-175B 789
  ip address 172.16.1.240/24
  multicast

interface eth1
  description To airborne network
  ip address 99.99.99.99/8
  multicast

interface tun0
  description tunnel for ospf
  ip address 10.20.1.1/24
  multicast


ip route 0.0.0.0/0 172.16.1.1

=======================================================================
ospfd.conf
----------

router ospf
  ospf router-id 172.16.1.240
  ospf abr-type cisco
  redistribute connected
  network 10.20.1.0/24 area 0
  network 99.0.0.0/8 area 1
  area 1 stub

=======================================================================
Router Net77
===========
Create Tunnel
-------------

modprobe ip-gre
ip tunnel add tun0 mode gre remote 239.0.0.1 local 172.16.2.240 dev eth0
ip addr add 10.20.1.2/24 dev tun0
ip link set tun0 up
ip link set gre0 up

=======================================================================
zebra.conf
----------

hostname net77
password itac
enable password itac

interface eth0
  description To KG-175E 23969
  ip address 172.16.2.240/24
  multicast

interface eth1
  description To airborne network
  ip address 77.77.77.77/8
  multicast

interface tun0
  description tunnel for ospf
  ip address 10.20.1.2/24
  multicast


ip route 0.0.0.0/0 172.16.1.1

=======================================================================
ospf.conf
---------

router ospf
  ospf router-id 172.16.2.240
  ospf abr-type cisco
  redistribute connected
  network 10.20.1.0/24 area 0
  network 77.0.0.0/8 area 1
  area 1 stub

=======================================================================


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc