Hi,
I have a linux server running kernel 2.6.19 that is connected with 2
seperate 100Mbit links to the same isp:
+---+
+---------------+ | I |
+---------------+
| | | S |
| |
| eth0 --+--------------+ P |
| |
| | | S |
| |
| linux 2.6.19 | | W |========================| ISP
GATEWAY |
| | | I |
| |
| eth1 --+--------------+ T |
| |
| | | C |
| |
+---------------+ | H |
+---------------+
+---+
Both links have their own ip but have the same gateway. The problem is I
can''t seem to get egress traffic load balanced over the 2 nics.
IP config after boot (dhcp from isp)
ip a:
1: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0: <BROADCAST,MULTICAST,NOTRAILERS,UP,10000> mtu 1500 qdisc
pfifo_fast qlen 1000
link/ether 00:00:00:00:00:0f brd ff:ff:ff:ff:ff:ff
inet 10.0.0.110/24 brd 10.0.0.255 scope global eth0
3: eth1: <BROADCAST,MULTICAST,NOTRAILERS,UP,10000> mtu 1500 qdisc
pfifo_fast qlen 1000
link/ether 00:00:00:00:00:ed brd ff:ff:ff:ff:ff:ff
inet 10.0.0.120/24 brd 10.0.0.255 scope global eth1
Default routing table after boot
ip r:
10.0.0.0/24 dev eth0 scope link
10.0.0.0/24 dev eth1 scope link metric 1
127.0.0.0/8 dev lo scope link
default via 10.0.0.1 dev eth0
default via 10.0.0.1 dev eth1 metric 1
I enabled ip_forward and set arp_ignore to 1 for eth0 and eth1 to make
sure the correct nic answers to arp requests.
I tried to get the egress load balancing to work by replacing the above
two default routes with:
ip route add default mpath drr nexthop via 10.0.0.1 dev eth0 weight 1
onlink nexthop via 10.0.0.1 dev eth1 weight 1 onlink
I assumed that with mpath device round robin both nics would be used
more or less equally, but the reality is only one of the nics actually
works and the second nic even stops responding to arp requests.
Am I doing something totally wrong or impossible here or is the device
round robin code not working properly?
On Sat, Jan 13, 2007 at 12:54:24PM +0100, zutph3n@gmail.com wrote:> Hi, > > I have a linux server running kernel 2.6.19 that is connected with 2 > seperate 100Mbit links to the same isp: > > > +---+ > +---------------+ | I | > +---------------+ > | | | S | > | | > | eth0 --+--------------+ P | > | | > | | | S | > | | > | linux 2.6.19 | | W |========================| ISP > GATEWAY | > | | | I | > | | > | eth1 --+--------------+ T | > | | > | | | C | > | | > +---------------+ | H | > +---------------+ > +---+ > > Both links have their own ip but have the same gateway. The problem is I > can''t seem to get egress traffic load balanced over the 2 nics. > > IP config after boot (dhcp from isp) > ip a: > > 1: lo: <LOOPBACK,UP,10000> mtu 16436 qdisc noqueue > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > inet 127.0.0.1/8 brd 127.255.255.255 scope host lo > > 2: eth0: <BROADCAST,MULTICAST,NOTRAILERS,UP,10000> mtu 1500 qdisc > pfifo_fast qlen 1000 > link/ether 00:00:00:00:00:0f brd ff:ff:ff:ff:ff:ff > inet 10.0.0.110/24 brd 10.0.0.255 scope global eth0 > > 3: eth1: <BROADCAST,MULTICAST,NOTRAILERS,UP,10000> mtu 1500 qdisc > pfifo_fast qlen 1000 > link/ether 00:00:00:00:00:ed brd ff:ff:ff:ff:ff:ff > inet 10.0.0.120/24 brd 10.0.0.255 scope global eth1 > > Default routing table after boot > ip r: > > 10.0.0.0/24 dev eth0 scope link > 10.0.0.0/24 dev eth1 scope link metric 1 > 127.0.0.0/8 dev lo scope link > default via 10.0.0.1 dev eth0 > default via 10.0.0.1 dev eth1 metric 1 > > I enabled ip_forward and set arp_ignore to 1 for eth0 and eth1 to make > sure the correct nic answers to arp requests. > > I tried to get the egress load balancing to work by replacing the above > two default routes with: > > ip route add default mpath drr nexthop via 10.0.0.1 dev eth0 weight 1 > onlink nexthop via 10.0.0.1 dev eth1 weight 1 onlink > > I assumed that with mpath device round robin both nics would be used > more or less equally, but the reality is only one of the nics actually > works and the second nic even stops responding to arp requests. > > Am I doing something totally wrong or impossible here or is the device > round robin code not working properly?Curiosity but why use such a setup is your ISP link > 2Gbp/s ? Why not bond if you want HA. why its not round robining. I am going to guess but this line default via 10.0.0.1 dev eth0 costs less to use than default via 10.0.0.1 dev eth1 metric 1 so it should never use the second. I say guess cause I don''t know what the default metric is if you do add one. What you want it to look something like is default proto static metric 5 nexthop via 144.132.144.1 dev vlan2 weight 5 nexthop via 10.20.20.230 dev ppp0 weight 20 There is a link to a howto on the web site that steps out how to set this up Alex> _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc >_______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
On 01/13/07 05:54, zutph3n@gmail.com wrote:> Both links have their own ip but have the same gateway. The problem is I > can''t seem to get egress traffic load balanced over the 2 nics.I don''t know if it is still a problem or not, but I ran in to something very similar a LONG time ago (mid 2.4). Basically what I found was the problem was that (I believe) multi-path routing really is multi gateway routing. I.e. load balancing across two (or more) different gateways. In your case, and the case that I had, all the IPs in the world did not make any difference b/c each path had the same default gateway. My solution at the time was to use UML routers to provide different subnets to the box doing ECMP routing. Each UML basically NATed from the one upstream network to a small downstream private subnet that was unique for each link. This allowed the box doing ECMP to see different gateways. This worked great until I hit a memory limit on connection state. I should probably say that the problem I ran in to was not a problem with ECMP but the number of hosts that I was trying to NAT with the amount of RAM that was on the box. I was able to resolve this by adding RAM to the box to provide a larger Connection State Table. For the record, the box was running a mid 2.4 kernel with a subnet that was the size of 4 class C networks (2048 IPs) on a box that had 256 MB of RAM. I ended up taking the box up to 2 GB of RAM and things have been working GREAT ever sense. I do believe this memory / connection state problem has been resolved long ago. However the system is working and payed for and the client is perfectly happy with what is in place and sees no reason to do any thing with it. (If any one would like more details, just ask.) Grant. . . .
On 01/15/07 15:20, zutph3n@gmail.com wrote:> Wow, that''s a complicated solution. Nicely done:) But I think that''s a > bit too complicated for my setup.... thx for the input anyway.Thanks. Indeed the set up is not simple. You may consider talking with your ISP and seeing if they can assign one of your links an IP on a different subnet. I have found that ISPs that are worth their salt are willing to work with you to help you resolve these types of problems. Grant. . . .
On Mon, Jan 15, 2007 at 06:44:54PM -0600, Grant Taylor wrote:> On 01/15/07 15:20, zutph3n@gmail.com wrote: > >Wow, that''s a complicated solution. Nicely done:) But I think that''s a > >bit too complicated for my setup.... thx for the input anyway. > > Thanks. > > Indeed the set up is not simple. You may consider talking with your ISP > and seeing if they can assign one of your links an IP on a different subnet. > > I have found that ISPs that are worth their salt are willing to work > with you to help you resolve these types of problems. > > > > Grant. . . .something else to look for, because you have 2 nics in the same broadcast domain (http://cactuswax.net/blog/articles/2006/09/arp_ignore.html) explains about arp_ignore. In its default setup you are going to find i nic is going to arp respond for both IP addresses!> _______________________________________________ > LARTC mailing list > LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc >_______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
On 01/16/07 04:01, Andrew Lyon wrote:> Can you suggest how this can be done with 2.6 as we will be upgrading our > server soon and I don''t think ECMP patch exists for 2.6?The patch does not exist because ECMP is in the main line kernel. Networking -> Networking options -> IP: advanced router IP: equal cost multipath Grant. . . .