LARTC - May 2002 - Multiple Internet Connection, established connection dropping issues

To begin with I have not applied and am not using the patches provided
by Julian Anastasov.
http://www.linuxvirtualserver.org/~julian/

I do not believe at the moment they apply to the problem I am having. If
they are I will go through the process of applying the patch. They
mostly have to due with what happens when a connection dies. At the
moment both of my connections are live, and I am having other problems.

I am trying to use more than one internet connection and seem to keep
running into the same problem no matter how I address it.

At the moment you can establish a connection using either line without a
problem. After a little while the connection will be dropped as the
linux router attempts to use a different interface. The connection will
only be re-connected after the new route has expired in the route''s
cache. Then the connection can continue on the same interface it was
originally established under for a period of time until the whole thing
repeats itself.

What I would like to happen is if a connection from the outside is
established, the linux router will continue to keep the connection on
the same interface until the user drops the connection. At which time if
the connection is re-established by the user the linux router then could
use a different interface but it should only use the interface that the
connection was established under.

I have made some progress playing around with the gc_interval gc_timeout
and gc_elasticity. 

I also started looking at the code behind the routing and gc logic in
the route.c file in the linux kernel source. It appears that the gc does
not compare which routes to keep or which routes to drop. It just does
sort of a first come first out sweep.

Also it is my assumption from the code and experimentation that the
gc_elasticity effects the amounts of routes which are cleared out by the
gc. So if you decrease the gc_interval and gc_timeout it may still take
a while to clear out the route in cache, by gc 8 routes at a time, the
default. I have increased it gradually to 256 the size of the route
cache, and my gc_interval and gc_timeout appear to be a little more
accurate based on the time it takes in real life to re-establish the
connection.

So aside from the route caching issue I really want to know why in the
middle of a connection the router will stop using one interface and
begin using another.

I have tried both 

ip route default equalize
and just
ip route default

for the multipath route table. 

I have also tried both 

ip route append default table 20 metric 1 proto static
and
ip route append prohibit default table 20 metric 1 proto static

for each interface.

So each interface has a table showing which gateway to use. Those tables
have a higher priority and should be checked/used before the multipath
table. As per examples in both

http://www.linuxvirtualserver.org/~julian/nano.txt
http://www.linuxvirtualserver.org/~julian/dgd-usage.txt

Each of those examples do things a little differently so that alone is
fairly confusing.

Bottom line I am stuck at the moment with it dropping and
re-establishing connections no matter what I do. I have been able to
decrease the time it takes to re-establish, but I believe that also
effects the time the connection will last for before needing to be
re-established.

On the other hand, if a request is sent from inside to the outer world
without having a connection from the outside world previously
established it then could use the multipath route.

I would assume the routing tables, rules, and their priorities would
effect this but it does not seem to do the job.

I also tried to turn off rp_filter, just to see if it could come in one
interface and go out another. Once again no noticeable difference. I did
not try setting the rp_filter to 2, for increased filtering.

Has anybody else experienced this or know of a solution. My gut is
telling me that the logic in the route cache gc might need to be
improved, but I am not sure that is really it or not. Knowing my luck
the solution is right under my nose and I am missing it.

I am about ready to daisy chain the routers and attempt to use both
connections that way if I can get out on both using a single ip as the
gateway.

My deepest gratitude to anyone who can help or come up with a solution.
I am more than willing to pay back the favor in some manor or another.

FYI,
	I am currently using kernel 2.2.19 that comes as part of the old Linux
Router Project version 2.9.8. I have made attempts to compile my own
kernel, but have not had any luck booting off of it due to the use of a
ram disc as the root file system. Basically I wanted to replace the
Linux Router Kernel with mine, but seem to run into problems mounting
the root file system. Errors and problems for another list. If the
version of my kernel is the problem or if I need Julian''s patches that
is the route I will take. Also if it matters I am using a 75mhz pentium
with 24mb ram. Using a 16 mb ram disk, with 8mb left over to use for
memory. It seems to screem, but my gc may be delayed by the processor
not being able to keep up?


-- 
Sincerely,
William L. Thomson Jr.
Obsidian-Studios, Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com
-- 
Sincerely,
William L. Thomson Jr.
Support Group
Obsidian-Studios Inc.
439 Amber Way
Petaluma, Ca. 94952
Phone  707.766.9509
Fax    707.766.8989
http://www.obsidian-studios.com

Hello,

On 23 May 2002, William L. Thomson Jr. wrote:
> I do not believe at the moment they apply to the problem I am having. If
> they are I will go through the process of applying the patch. They
	But you have problems solved only by these patches, read below...
> mostly have to due with what happens when a connection dies. At the
> moment both of my connections are live, and I am having other problems.
> At the moment you can establish a connection using either line without a
> problem. After a little while the connection will be dropped as the
> linux router attempts to use a different interface. The connection will
	With plain kernel this can be true only if you are talking
about masquerade connections.
> only be re-connected after the new route has expired in the
route''s
> cache. Then the connection can continue on the same interface it was
> originally established under for a period of time until the whole thing
> repeats itself.
	Not strictly on the same interface. Note that:

- the routing functionality is dumb, you request route with
specific keys and the result is resolved path (outde, defsrc, gw...)
that is cached for further use.

- in the plain kernel only the sockets correctly provide the same
keys when resolving route, for example:

	- after connect the socket is "connected", i.e. saddr and
	daddr are known and they are always provided when resolving
	route

	- the user can optionally bind the socket to interface,
	this is the only reason the connection to use only one
	interface (in your words)

	- after routing cache expiration (from any kind, caused from
	user, timer expiration or autogenerated flush on route
	change) the sockets resolve again the route providing
	the keys saved for each connection.

	So, the connected socket should not experience any outage
	when the route is resolved after cache entry expiration
	assuming the routing rules do not change.

	The result is that if the routing rules allow using many
	"internet connections" for the particular "keys" you can
	notice that the remote end will not notice that your
	connections use many links/interfaces. It is allowed
	from the routing rules the user defines. So, don''t be
	surprised from the routing rules in the Christoph''s
	nano howto. You must define correct routing rules for
	all networks you are using, some of them use multipath
	routes because it is allowed the specified subnet to
	use multiple gateways (trick used mostly for NAT because
	you can use "internet connections" with distinct IP
	ranges in same multipath route). Note that if some
	ISPs allow spoofing from your side, then you can put
	all such ISPs as alternative in all other routes for
	the other IP ranges that do not have this luck.

- what the plain kernel does not guarantee is that the masquerading
software does not use same/similar route resolving. I.e. the
masq software assumes the route subsystem will always keep the
cache valid and will never forget the cached routes during the
masq connection life. If you reduce the gc_* values you can notice
that this assumption fails very badly with low values, only the
default ones (5mins or so) legitimate the masq software as a
"correct user" of the (multipath) routing because the conns usually
don''t last so much. This is where the patches help: they fix this
behavior
by allowing the masq software to know about the "routing cache
expiration"
problem and to provide the correct routing keys, at least, the interesting
ones - the masq conns are not strictly bound to interface which
is the same default behavior for the connected socket. Why not
to use different interface when the routing rules allow it?
> What I would like to happen is if a connection from the outside is
> established, the linux router will continue to keep the connection on
> the same interface until the user drops the connection. At which time if
	Then use the patches. You can''t control the input interface
or more exactly, the input "internet connection" (at least without
using DNS). What eventually breaks in unpatched kernel is the in->out 
traffic of these connections established from remote hosts (I mean the NAT 
ones).
> the connection is re-established by the user the linux router then could
> use a different interface but it should only use the interface that the
> connection was established under.
	Problem in the routing rules or in the NAT code in unpatched
kernel.
> I have made some progress playing around with the gc_interval gc_timeout
> and gc_elasticity.
	I assume you have progress with the cache expiration but
it should be worse for the stability of the NAT conns in the
unpatched kernel.
> I also started looking at the code behind the routing and gc logic in
> the route.c file in the linux kernel source. It appears that the gc does
> not compare which routes to keep or which routes to drop. It just does
> sort of a first come first out sweep.
	Forget the routing cache, at least, tune it for fast expiration
according to your needs for fast failover but nothing more. The NAT
code and the sockets should not care about routing cache and the
expiration. They can use pointer to the cache entry and can know
that the route is still "valid". Or they can simply resolve route
again, always when needed, by supplying the correct keys.
> So aside from the route caching issue I really want to know why in the
> middle of a connection the router will stop using one interface and
> begin using another.
	May be because the route is not resolved with the right keys.
Is that a NAT conn?
> So each interface has a table showing which gateway to use. Those tables
> have a higher priority and should be checked/used before the multipath
> table. As per examples in both
>
> http://www.linuxvirtualserver.org/~julian/nano.txt
> http://www.linuxvirtualserver.org/~julian/dgd-usage.txt
>
> Each of those examples do things a little differently so that alone is
> fairly confusing.
	Hm, we still don''t know your routes, may be they are
"little different" from the expected ones that should work :)
What is so different in your setup? These routing rules looks
valid, they work. The logic to place the routing rules to
directly connected networks before all routes via gateways is
a recommendation that works in 90% of the cases. It allows
the setup to rely on rp_filter, etc. You must be able to define
the routing settings by relying on the following rules:

- "each interface has a table showing which gateway to use" does
not mean anything. You can attach different IPs from distinct ISP''s
IP ranges to same interface.

- build your routing rules on the "subnet->subnet" assumption,
i.e. how should be routed traffic from one subnet to another
one. Set rp_filter (sometimes other requirement do not allow using 
rp_filter) to keep the traffic symmetric.

- provide "from all to subnet" routes to allow the source address
autoselection mechanism to work when unbound socket talks to
the remote subnet

- always define preferred source IP in the routes and don''t rely
very much on the src IP autoselection mechanism always to select
the right local IP address. The multipath routes used from NAT-ed
subnets usually ignore the preferred source IP (usually in the
patched kernel) because there is a 2nd stage where the src IP
is selected after using the gateway from step 1.
> Bottom line I am stuck at the moment with it dropping and
> re-establishing connections no matter what I do. I have been able to
> decrease the time it takes to re-establish, but I believe that also
> effects the time the connection will last for before needing to be
> re-established.
	The reduced gc timeouts lead to more reroutings.
> On the other hand, if a request is sent from inside to the outer world
> without having a connection from the outside world previously
> established it then could use the multipath route.
	Yes, the multipath route is used if the route with the
supplied keys is not already cached.
> I would assume the routing tables, rules, and their priorities would
> effect this but it does not seem to do the job.
	With the patches applied you can notice that the multipath
route is used only when establishing the NAT conn, then it is not
used because the conn is already bound to public IP and we don''t
have the right to use it. I.e. the multipath route is used only
for outdev/gateway selection, not for selecting src IP. Later
the right src IP for this gateway is selected. Note that if NAT
is implemented so, we can optimize the path to use only one
route lookup for the SNAT-ed traffic because the established
NAT conn has the knowledge that it is bound to specific src IP.
> I also tried to turn off rp_filter, just to see if it could come in one
> interface and go out another. Once again no noticeable difference. I did
> not try setting the rp_filter to 2, for increased filtering.
	2 is not different from 1 in current kernels.
> Has anybody else experienced this or know of a solution. My gut is
> telling me that the logic in the route cache gc might need to be
> improved, but I am not sure that is really it or not. Knowing my luck
> the solution is right under my nose and I am missing it.
	The problems are complex, may be part of them can be solved
with parameter tuning.
> FYI,
> 	I am currently using kernel 2.2.19 that comes as part of the old Linux
> Router Project version 2.9.8. I have made attempts to compile my own
> kernel, but have not had any luck booting off of it due to the use of a
> ram disc as the root file system. Basically I wanted to replace the
> Linux Router Kernel with mine, but seem to run into problems mounting
> the root file system. Errors and problems for another list. If the
> version of my kernel is the problem or if I need Julian''s patches
that
	I can only recommend you to try them, from the listed problems,
the problem with the inaccurate rerouting can be solved. The problems
with the faster failover are independent. You can try another approach:
when your healthchecking script detect "internet connection" failure
you can just flush the route cache and be happy. Then the established
connections that can select another alive path will use it, the
connections that have only one alternative (for example, caused from
only one valid path for the selected IP range) then they block or
are dropped from your router with some IP stack mechanism (ICMP
Dest unreachable or so). The forced cache flush will allow the new
conns not to use cached paths between the local and the remote
hosts and by this way all the new conns will select alive path
and with the help from the socket and the NAT code these paths will
be correctly used and no new/unexpected "interfaces" will be
selected.
> is the route I will take. Also if it matters I am using a 75mhz pentium
> with 24mb ram. Using a 16 mb ram disk, with 8mb left over to use for
> memory. It seems to screem, but my gc may be delayed by the processor
> not being able to keep up?
	Good, but how looks the traffic? KBits, may be MBits? How
many entries you see in the routing cache?

Regards

--
Julian Anastasov <ja@ssi.bg>