LARTC - Apr 2002 - Load Balance/Redundant Internet Connections

I know this is a question that has been asked a million times over but
here goes.

I have two SDSL lines that I would like to load balance and/or provided
redundant internet access.

Now since these lines come from two different ISPs I have two blocks of
IPs, one block for each line.

Now I have servers that I would like to be able to provided public
access using two different public IP''s. Two public IP for each server,
one from each ISP block, that will be port address translated to a
single internal IP.

This brings about many different problems, one which lies in the way I
do DNS. At the moment if you talk to NS1 you will get public IPs for
each server using ISP1''s SDSL line and public IP block. If you talk to
NS2 you will get public IPs for each server using ISP2''s SDSL line and
public IP block.

Now I have tried to read and experiment with all the examples on the net
I could find including the following links which are the main ones I
came across several times.

http://www.samag.com/documents/s=1824/sam0201h/0201h.htm
http://www.linuxvirtualserver.org/~julian/nano.txt
http://www.linuxvirtualserver.org/~julian/dgd-usage.txt

Now with each of theses I run into a similar issue. That I will get to
in a moment.

I also tried buying a cheap solution, NexLand''s ISBPro800, which in
short does not work. NexLand is aware of the problems I have
experienced, and have been working on a new firmware release for over
two months now. I am not sure if they will get it to work or not.

The only other choices of solutions I can buy are out of my price range
at the moment. One is $5000, and the other starts at $7500.00. Here are
links in order of pricing,
http://www.sysmaster.com/products.htm
http://www.fatpipeinc.com/

Now it is rumored that the second runs a modified flavor of Linux.

So far the closest I have ever been able to get using Linux as a load
balancing/redundant gateway router has some issues when it comes to
routes existing in the route cache table.

If you establish a connection to a server using a public IP from ISP1,
and then go and establish a connection to the same server using a public
IP from ISP2, you cannot until after the routing cache has been flushed.

I tried playing around with the garbage collection and other params that
are supposed to control the frequency that the routing cache is flushed.
I was not able to reach a happy medium.

I was talking with a engineer buddy of mine from Cisco and he said I
needed proper aging to take place in my route cache. So if a new
connection is attempted and an older different route exists in the
routing cache, the older one will be dropped/removed to allow the new
one in and traffic to flow. Makes sense, but I have no clue how to
implement this, if it can be done outside of the kernel itself.

So the bottom line is I can get it to work, providing that each and ever
requests made to the server uses the same public IP from one or the
other ISP. If there is a need to switch to the other line/ip then it
will have to occur once the route has been cleared out of my routers
cache, and possible the requesting machines cache as well.

I am starting to think that this cannot be done properly with the
current state of the linux kernel''s multipath routing. Do I have to buy
a >$5000.00 unit to do this?

It seems that there are allot of people currently, and the more people
switch to broadband I could see it only increasing, trying to do the
same thing I am. So there really needs to be a solution for this. I will
document a working solution if and when I come across one.

Being that a couple companies are providing products that can do what I
am attempting to, it seems this is something that the linux kernel
hackers should look into adding or improving.

If I had the proper knowledge of C/C++ and the linux kernel I would work
on a solution myself. I am trying to envision a shell script or program
outside of the kernel that could help out. But where I run into problems
is for my program to be aware of each and every connection request and
making a decision about the state of or content in the route cache
without depleting the machine of it''s resources.

Is there anybody out there that has been able to get this to work? If
so, how? What I am doing wrong?

I can provide examples and anything needed.

FYI I am currently using the Linux Router release of the 2.2 kernel,
which is the linux router 2.9.8beta and is based on kernel 2.2.19. I think.

I also attempted it with a newer kernel, 2.4.9-31 from Redhat, that was
recompiled by me. Both lead me back to the same problem with the route
cache.

I am getting desperate and about ready to drop $5000.00, although I will
not do it without kicking and screaming.

Please help, anybody, I will try to make it up to you, so it is worth
your time.

Thank you in advance.

-- 
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com

Hello,

On Sun, 28 Apr 2002, William L. Thomson Jr. wrote:
> If you establish a connection to a server using a public IP from ISP1,
> and then go and establish a connection to the same server using a public
> IP from ISP2, you cannot until after the routing cache has been flushed.
	This sounds wrong. The connections from different pub IPs
differ, so they use different cache entries. But it is true when
NAT is used. Then the INTIP->REMOTESERVER is cached.
> I tried playing around with the garbage collection and other params that
> are supposed to control the frequency that the routing cache is flushed.
> I was not able to reach a happy medium.
Can this help?:

echo 5 > /proc/sys/net/ipv4/route/gc_timeout
echo 2 > /proc/sys/net/ipv4/route/gc_interval
> Thank you in advance.
Regards

--
Julian Anastasov <ja@ssi.bg>

Ok,
	I have removed the Nexland ISBPro800, and put our linux router back
into use. My initial testing was not to bad. Not exactly what I was
looking for but it seems to be the closest I have gotten so far.

On Mon, 2002-04-29 at 02:03, Julian Anastasov wrote:
> 	Hello,
> 
> On Sun, 28 Apr 2002, William L. Thomson Jr. wrote:
> > I tried playing around with the garbage collection and other params
that
> > are supposed to control the frequency that the routing cache is
flushed.
> > I was not able to reach a happy medium.
> 
> Can this help?:
> 
> echo 5 > /proc/sys/net/ipv4/route/gc_timeout
> echo 2 > /proc/sys/net/ipv4/route/gc_interval
> 
> --
> Julian Anastasov <ja@ssi.bg>
Thanks Julian, this setting seems to improve performance a little. I
doubt to many people will be bouncing back and forth between the IP''s
from different ISP. It will have more to do with what the DNS lookup''s
return.

DNS not really being a topic for this list, but directly related to the
outcome of what I am attempting to do. So please forgive me asking these
questions here. If you know of a better place please let me know.

Anyway, if anyone reading this could take a few seconds and click on the
following links. Both of my SDSL lines are up and all servers and
services should be available if everything is correct in the linux
router.

Please feel free to email the results of your using the following links
to me. Also please click beyond the initial link just to make sure
further lookups are the same.

Links
http://www.obsidian-studios.com
ftp://ftp.obsidian-studios.com
http://www.elvisplace.com
http://www.suzakstrading.com
 
The others are mail and DB related and will have to be tested out by
clients. These are available to the general public and I am curious to
know if the general public has any problems.

Once again for clarification I am using two SDSL lines as a single
gateway for my internal singles. Hopefully this will allow me to use
both lines. Possible providing redundant service. If I am lucky a simple
form of load balancing, although by load balancing I mean just being
able to use both lines as one. Having the linux router determine the use
of each line and balancing that out is beyond my expectations at the
moment. But would be nice.

So at the time I have this router in place, and if the above links do
not work, it is due to the router''s config.

Thank you very much for anyone taking to time to click and provide
feedback.

Also I am open to comments or suggestions from anyone about my config. 

I do plan to share the results with others so they will not be faced
with the same problems. Much less have to buy a Nexland unit or other
piece of hardware that may or may not work.

-- 
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com

Well after many months, a failed attempt at solving my problem with a
Nexland ISBPro800, I finally now am using both of my SDSL lines
together.

Bliss. :)

This was not an easy process, and I ended up going through just about
every possible scenario until finding the one that worked.

I would not have found the one tha worked had it not been for the
assistance of 

Julian Anastasov

He responded to many direct emails, and was very patient and
cooperative. 

His patches are a must in a situation like this.
http://www.linuxvirtualserver.org/~julian/#routes

Also stick to the examples shown in the nano-how-to
http://www.linuxvirtualserver.org/~julian/nano.txt

What I had problems with and what is a must is that the Linux router
must perform some sort of NAT.

In my situation it was a matter of using 

ipmasqadm portfw 

to do DNAT on my public IP''s

and a 

ipchains -A forward -s w.x.y.z -j MASQ

One the internal network.


I am willing to answer questions and help others out.
That''s just my way of giving back what I have received.
Someone help''s me, I help someone else. 

So others heading down this path, it is possible, but there is only one
way at the present moment to go about it.

Have a nice day, and route on.

-- 
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com