Hi everybody! I''m trying to implement the load balancing for a LAN with two ISPs. I''ve installed a Suse Linux Enterpise Server 9 with iproute2 for that porpouse. The server have two NICs, one of them is for both the LAN and ISP 1. I''ve setup both NICs with YAST (if I use ip for this, then the whole thing doesn''t work!) and execute the following commands to setup the routing tables: ip route flush cache ip route flush default ip route flush table 1 ip route flush table 2 ip route add 10.1.254.0/24 dev eth0 src 10.1.254.251 ip route add 10.1.1.0/24 dev eth1 src 10.1.1.200 ip route show table main | while read ROUTE ; do ip route add table 1 $ROUTE ; done ip route show table main | while read ROUTE ; do ip route add table 2 $ROUTE ; done ip route add table 1 default via 10.1.254.254 ip route add table 2 default via 10.1.1.254 ip rule add from 10.1.254.251 table 1 ip rule add from 10.1.1.200 table 2 ip route add default equalize nexthop via 10.1.254.254 dev eth0 weight 1 \ nexthop via 10.1.1.254 dev eth1 weight 1 (All of this came from http://linux.lcampino.cl/wiki/index.php/Balanceo_de_Carga#Configurando_la_red (in Spanish)). At this point, things seems to work fine. I browse in the server and watch with iptraf that both NICs have traffic, but not at 50%-50% rate. Now I need to implement the following: - Make load balancing the nearest to 50%-50% rate (Both Internet connections have equal bandwith). - When one of the links goes down, all the traffic redirects to the other inmediately and automatically. Also, when the link is up again, the load balancing restart too. - If posible, when downloading a single big file (i.e. *.exe, *.iso), the packets come from both connections. I''ve read a lot about this, but I''m still confuse and very very lost...! :-))) I''m not sure if I need to adjust some kernel paramters and rebuit it, execute another sequence of commands, apply a kernel patch, etc. I really apreciate if someone could bring me some light in all this, or tell me what documentation, web page or patch do I need to use. Thanks in advanced for all your time and recomendations! P.D. Sorry for my English! _________________________________________________________________ Platica con tus amigos en linea con MSN Messenger http://messenger.t1msn.com.mx/
Vlad, We have also set up a somewhat similar method of load balancing. Our traffic is never a 50-50 split (well 3:2 is how we have it set, but it doesn''t always get close to that), but as the load picks up, it tends to be closer to the actual amount. Dead gateway detection has never worked for us, and one day I''ll probably bother other members of the LARTC group to get some help, but the method that we use is to check the output of the ip neighbor command. Basically, if our two ISPs are 10.1.1.254 and 10.2.2.254, we run a bash script via cron every minute that does a call something like: ETH1 = ip neigh 10.1.1.254 | egrep "REACHABLE|DELAY|PROBE|STALE" -c ETH2 = ip neigh 10.2.2.254 | egrep "REACHABLE|DELAY|PROBE|STALE" -c The neighbor system basically monitors ARP and if it sees a message leave an interface without a reply after something like 3-5 seconds, it moves the interface to DELAY, after another few seconds it moves to PROBE and does an active arp request, and if that fails to work in a few seconds, it becomes INCOMPLETE or FAILED or just simply isn''t listed. If no data is sent either way for a while, the entry can be marked STALE or removed. With the above lines, we get a 1 in the ETH1 or ETH2 variables if the next neighbor is up, and a 0 if not. From there you can use some if scripts to detect if both are up, or if only one is up, which one. In our case, if both are up we clear the default route and then make it something like ip route add default nexthop via 10.1.1.254 dev eth1 weight 1 \ nexthop via 10.2.2.254 dev eth2 weight 1 and if only one is up we clear it and make it : ip route add default nexthop via 10.1.1.254 dev eth1 or ip route add default nexthop via 10.2.2.254 dev eth2 With some additional scripting we can allow this to be overridden, we can set the link to prefer using only one line, but switch to the other if the preferred line fails, and to take input from programs like Nagios to auto-prefer one line or another if ping times get high, etc. In addition, the script remembers the state it was in (so that it only changes the routing table when needed), controls DNS, can flush the DNS cache, and reports status back to Nagios. Once I get all the bugs out and some documentation, I''d be happy to post it to the news group, though you or anyone else can send me an email if you would like to take a look at it before then. In practice, this method usually detects and adjusts outbound connections quickly without user intervention; DNS changes with short TTLS take care of inbound connections. Just be careful... if you don''t have something sending traffic out to your upstream routers (and back) every few minutes, the entry in your ARP table can potentially be removed and thus cause your system to think an unused gateway has failed, or that a recovered gateway is still down. This could be checked with a quick "if ip neigh test fails, ping neighbor 5 times, then test again before making decisions". Running an uptime monitor that pings or does something else to/through the gateway (regardless of default route) also takes care of this. -Will -----Original Message----- From: Vladimir Burciaga Aguilar [mailto:anakinv7@hotmail.com] Sent: Thursday, September 14, 2006 10:25 PM To: lartc@mailman.ds9a.nl Subject: [LARTC] Problem with Load Balancing Hi everybody! I''m trying to implement the load balancing for a LAN with two ISPs. I''ve installed a Suse Linux Enterpise Server 9 with iproute2 for that porpouse. The server have two NICs, one of them is for both the LAN and ISP 1. I''ve setup both NICs with YAST (if I use ip for this, then the whole thing doesn''t work!) and execute the following commands to setup the routing tables: ip route flush cache ip route flush default ip route flush table 1 ip route flush table 2 [snip] _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>We have also set up a somewhat similar method of load balancing. Our >traffic is never a 50-50 split (well 3:2 is how we have it set, but it >doesn''t always get close to that), but as the load picks up, it tends to be >closer to the actual amount.Well, then there is not much to do about this.>Dead gateway detection has never worked for us, and one day I''ll probably >bother other members of the LARTC group to get some help, but the method >that we use is to check the output of the ip neighbor command. Basically, >if our two ISPs are 10.1.1.254 and 10.2.2.254, we run a bash script via >cron >every minute that does a call something like: > >ETH1 = ip neigh 10.1.1.254 | egrep "REACHABLE|DELAY|PROBE|STALE" -c >ETH2 = ip neigh 10.2.2.254 | egrep "REACHABLE|DELAY|PROBE|STALE" -c > >The neighbor system basically monitors ARP and if it sees a message leave >an >interface without a reply after something like 3-5 seconds, it moves the >interface to DELAY, after another few seconds it moves to PROBE and does an >active arp request, and if that fails to work in a few seconds, it becomes >INCOMPLETE or FAILED or just simply isn''t listed. If no data is sent >either >way for a while, the entry can be marked STALE or removed. > >With the above lines, we get a 1 in the ETH1 or ETH2 variables if the next >neighbor is up, and a 0 if not. From there you can use some if scripts to >detect if both are up, or if only one is up, which one. In our case, if >both are up we clear the default route and then make it something like > >ip route add default nexthop via 10.1.1.254 dev eth1 weight 1 \ >nexthop via 10.2.2.254 dev eth2 weight 1 > >and if only one is up we clear it and make it : > >ip route add default nexthop via 10.1.1.254 dev eth1 >or >ip route add default nexthop via 10.2.2.254 dev eth2Ok, William, this looks like what I''m looking for. I''m going to test it and tell you how it works for us. By the way, about the download of a single file between the two conections, do you know if there is a way to do it? Thanks for your help and time and sorry for the delay! _________________________________________________________________ Prodigy/MSN Spaces: Tu espacio en la red http://spaces.msn.com/
To my knowledge, there is no way to download one file from two different connections connected to two different ISPs at the same time. If you are running BGP then you might be able to load balance across the two links, but that would require your upstream providers to allow you to use it, and possibly the purchase of a public AS number an IP address space depending on the setup. If you are doing NAT past this link (IE both of your lines go two the same ISP and same address blocks, but they want to give you 2x 10mb links for 20mb total), then you can look at doing load balancing on layer 2 (Fast EtherChannel, bonding, Link Aggregate Groups, whatever), or creating 2 PPP style links between the computers and using a routing protocol like OSPF, EIGRP (but not on Linux) or something. I believe OSPF does equal cost load balancing, BGP and EIGRP can, I think, do unequal cost load balancing. But either way, I don''t think that''s the solution in your case. The only other option I can think of would be some sort of software that sends every other packet to a different IP or something, which would need to run at the end you are downloading at or maybe at your ISPs, but I can''t think of anything like that. -Will -----Original Message----- From: Vladimir Burciaga Aguilar [mailto:anakinv7@hotmail.com] Sent: Monday, September 18, 2006 12:09 PM To: lartc@mailman.ds9a.nl Subject: RE: [LARTC] Problem with Load Balancing>We have also set up a somewhat similar method of load balancing. Our >traffic is never a 50-50 split (well 3:2 is how we have it set, but it >doesn''t always get close to that), but as the load picks up, it tends to be >closer to the actual amount.[snip] _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>>>>> "William" == William T Mullaney <William> writes:William> To my knowledge, there is no way to download one file William> from two different connections connected to two different William> ISPs at the same time. If you are running BGP then you William> might be able to load balance across the two links, but William> that would require your upstream providers to allow you William> to use it, and possibly the purchase of a public AS William> number an IP address space depending on the setup. If William> you are doing NAT past this link (IE both of your lines William> go two the same ISP and same address blocks, but they William> want to give you 2x 10mb links for 20mb total), then you William> can look at doing load balancing on layer 2 (Fast William> EtherChannel, bonding, Link Aggregate Groups, whatever), William> or creating 2 PPP style links between the computers and William> using a routing protocol like OSPF, EIGRP (but not on William> Linux) or something. I believe OSPF does equal cost load William> balancing, BGP and EIGRP can, I think, do unequal cost William> load balancing. But either way, I don''t think that''s the William> solution in your case. William> The only other option I can think of would be some sort William> of software that sends every other packet to a different William> IP or something, which would need to run at the end you William> are downloading at or maybe at your ISPs, but I can''t William> think of anything like that. Wouldn''t some download manager software that splits the file up into multiple simultaneous downloads do the trick? Agreed, not a single download across multiple ISPs, but definitely a single file across multiple ISPs. Regards, -- Raju -- Raj Mathur raju@kandalaya.org http://kandalaya.org/ GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F It is the mind that moves
Well, if you had a download manager and the system at the other side allowed you to start your transfers in the middle of the file (which isn''t out of the question) that could potentially work. The problem is that as far as I see, there''s nothing to force the second connection onto the second line. It''s been kind of a crap shoot of what line gets more information. In theory you could start the first download stream (and it''s routed to ISP A), then perhaps your email client goes out to check your POP account, so that goes over ISP B. The next connection, the second stream, now goes out over ISP B again. Honestly I don''t know exactly how the equalize command for ip route works, though I would think it says to always use the "less used" connection (perhaps on PPS, BPS, % use, whatever, on a per second, 30 second, minute average?), but in my experience that and the weight options don''t ever get you exactly 50/50 (or whatever you specify) traffic. Things like bit torrent would probably perform better because there are (possibly) many streams for each file, as would having 50 people downloading files vs one. It seems to be just like rolling dice, if you only roll twice you might get two evens or two odds, but if you roll tons of times, you should tend to get a more even distribution. -Will -----Original Message----- From: Raj Mathur [mailto:raju@linux-delhi.org] Sent: Sunday, September 24, 2006 2:49 PM To: lartc@mailman.ds9a.nl Subject: RE: [LARTC] Problem with Load Balancing>>>>> "William" == William T Mullaney <William> writes:William> To my knowledge, there is no way to download one file William> from two different connections connected to two different William> ISPs at the same time. If you are running BGP then you William> might be able to load balance across the two links, but William> that would require your upstream providers to allow you William> to use it, and possibly the purchase of a public AS William> number an IP address space depending on the setup. If William> you are doing NAT past this link (IE both of your lines William> go two the same ISP and same address blocks, but they William> want to give you 2x 10mb links for 20mb total), then you William> can look at doing load balancing on layer 2 (Fast William> EtherChannel, bonding, Link Aggregate Groups, whatever), William> or creating 2 PPP style links between the computers and William> using a routing protocol like OSPF, EIGRP (but not on William> Linux) or something. I believe OSPF does equal cost load William> balancing, BGP and EIGRP can, I think, do unequal cost William> load balancing. But either way, I don''t think that''s the William> solution in your case. William> The only other option I can think of would be some sort William> of software that sends every other packet to a different William> IP or something, which would need to run at the end you William> are downloading at or maybe at your ISPs, but I can''t William> think of anything like that. Wouldn''t some download manager software that splits the file up into multiple simultaneous downloads do the trick? Agreed, not a single download across multiple ISPs, but definitely a single file across multiple ISPs. Regards, -- Raju -- Raj Mathur raju@kandalaya.org http://kandalaya.org/ GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F It is the mind that moves _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc ______________________________________________________________________ This email has been scanned by the MessageLabs Email Security System. For more information please visit http://www.messagelabs.com/email ______________________________________________________________________ _______________________________________________ LARTC mailing list LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
The second connections will problably gets routed though the same
link because of route cache I think.
[]s.
William T Mullaney wrote:>
> Well, if you had a download manager and the system at the other side
> allowed you to start your transfers in the middle of the file (which
> isn''t out of the question) that could potentially work. The
problem
> is that as far as I see, there''s nothing to force the second
> connection onto the second line. It''s been kind of a crap shoot
of
> what line gets more information. In theory you could start the first
> download stream (and it''s routed to ISP A), then perhaps your
email
> client goes out to check your POP account, so that goes over ISP B.
> The next connection, the second stream, now goes out over ISP B
> again. Honestly I don''t know exactly how the equalize command for
ip
> route works, though I would think it says to always use the "less
> used" connection (perhaps on PPS, BPS, % use, whatever, on a per
> second, 30 second, minute average?), but in my experience that and the
> weight options don''t ever get you exactly 50/50 (or whatever you
> specify) traffic.
>
> Things like bit torrent would probably perform better because there
> are (possibly) many streams for each file, as would having 50 people
> downloading files vs one. It seems to be just like rolling dice, if
> you only roll twice you might get two evens or two odds, but if you
> roll tons of times, you should tend to get a more even distribution.
>
> -Will
>
> -----Original Message-----
> From: Raj Mathur [mailto:raju@linux-delhi.org]
> Sent: Sunday, September 24, 2006 2:49 PM
> To: lartc@mailman.ds9a.nl
> Subject: RE: [LARTC] Problem with Load Balancing
>
> >>>>> "William" == William T Mullaney
<William> writes:
>
> William> To my knowledge, there is no way to download one file
> William> from two different connections connected to two different
> William> ISPs at the same time. If you are running BGP then you
> William> might be able to load balance across the two links, but
> William> that would require your upstream providers to allow you
> William> to use it, and possibly the purchase of a public AS
> William> number an IP address space depending on the setup. If
> William> you are doing NAT past this link (IE both of your lines
> William> go two the same ISP and same address blocks, but they
> William> want to give you 2x 10mb links for 20mb total), then you
> William> can look at doing load balancing on layer 2 (Fast
> William> EtherChannel, bonding, Link Aggregate Groups, whatever),
> William> or creating 2 PPP style links between the computers and
> William> using a routing protocol like OSPF, EIGRP (but not on
> William> Linux) or something. I believe OSPF does equal cost load
> William> balancing, BGP and EIGRP can, I think, do unequal cost
> William> load balancing. But either way, I don''t think
that''s the
> William> solution in your case.
>
> William> The only other option I can think of would be some sort
> William> of software that sends every other packet to a different
> William> IP or something, which would need to run at the end you
> William> are downloading at or maybe at your ISPs, but I
can''t
> William> think of anything like that.
>
> Wouldn''t some download manager software that splits the file up
into
> multiple simultaneous downloads do the trick? Agreed, not a single
> download across multiple ISPs, but definitely a single file across
> multiple ISPs.
>
> Regards,
>
> -- Raju
> --
> Raj Mathur raju@kandalaya.org http://kandalaya.org/
> GPG: 78D4 FC67 367F 40E2 0DD5 0FEF C968 D0EF CC68 D17F
> It is the mind that moves
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>
> ______________________________________________________________________
> This email has been scanned by the MessageLabs Email Security System.
> For more information please visit http://www.messagelabs.com/email
> ______________________________________________________________________
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
>