LARTC - Oct 2001 - Getting AOL IM client to work with IPTABLES and IPROUTE2 (port forwarding almost)

I''m trying to get my AOL IM''r to work consistenly with 2 cable
modems. It
seems like it says it can''t connect (90% of the time, 10% it works,
just
pure luck) when I have both of the cable modems working together with this
iproute command

ip route default equalize nexthop via ***.***.***.*** dev eth0 nexthop via
***.***.***.*** dev eth2
iptables -A POSTROUTING -t nat -j MASQUERADE -o eth2
iptables -A POSTROUTING -t nat -j MASQUERADE -o eth0
Internet and everything else works fine with that. I need a solution whether
it be some kind of forwarding (port 5190) so that anything received comes
through 1 ethernet address. It might even have to be sent out the same
ethernet address but I''m thinking either one would work if there is
someway
to put a return address on the packet or something. I know AOL im''r
works
100% when the linux box is routing just through 1 cable modem.



Thanks!

David

On Sun, 28 Oct 2001 07:28:53 -0600
"David" <maniacdavid@cableone.net> wrote:
> I''m trying to get my AOL IM''r to work consistenly with 2
cable modems. It
> seems like it says it can''t connect (90% of the time, 10% it
works, just
> pure luck) when I have both of the cable modems working together with this
> iproute command
> 
> ip route default equalize nexthop via ***.***.***.*** dev eth0 nexthop via
> ***.***.***.*** dev eth2
> iptables -A POSTROUTING -t nat -j MASQUERADE -o eth2
> iptables -A POSTROUTING -t nat -j MASQUERADE -o eth0
> Internet and everything else works fine with that. I need a solution
whether
> it be some kind of forwarding (port 5190) so that anything received comes
> through 1 ethernet address. It might even have to be sent out the same
> ethernet address but I''m thinking either one would work if there
is someway
> to put a return address on the packet or something. I know AOL
im''r works
> 100% when the linux box is routing just through 1 cable modem.
I''m a bit surprised that you say that `Internet and everything else
works fine''. I''ve tried this and it did *not* work properly.
Actually,
if I did understand it well, the usage of the `equalize'' argument to
ip makes the selection of a particular interface packet based, while
the omission should make it session base. I have tried all
combinations I''m aware of, including weight''ing of the
nexthops, and
it did not work. HTTP based Internet access will fail with any more or
less elaborated site, as requests will come from more than one
IP. This doesn''t mean that session oriented interface selection
doesn''t work (which I can''t tell for sure); it just means that
certain
subsequent complete user sessions need to use the same IP. This might
be the reason why AIM isn''t working, as it seems to expect always the
same IP from you.

What you could do is trying to configure AIM such that it will always
use only one interface. This doesn''t work for HTTP, specially if there
is a proxy (e.g., squid) which hides where the request actually comes
from (which user and/or which computer on the LAN).

I''ve asked this question in diferent forms more than once, and
didn''t
get more answers than ``if you figure out, please tell me
too''. Unfortunately, no expert on this list bothered to tell me, that
this is plain impossible, or what would be needed to make it
work. (Well, I remember having got one answer, which pointed to
another answer `some months ago'', which I''m not sure to have
found,
and which actually didn''t solve the problem).

There are many, many people out there looking for this to work. Is
there really no knowledgable guy on this list who would dare to give
an definitive answer, wether the simultaneous usage of more than one
independent Internet link is possible, or even better, add such thing
to a FAQ and provide a pointer to it? And giving a reasoning might
also help reduce lots of frustration and lost time.

--
Christoph Simon
datageo@terra.com.br
---
^X^C
q
quit
:q
^C
end
x
exit
ZZ
^D
?
help
NO CARRIER
.

On Sun, 28 Oct 2001 16:56:43 +0000 (GMT)
Julian Anastasov <ja@ssi.bg> wrote:
> On Sun, 28 Oct 2001, Christoph Simon wrote:
> 
> > On Sun, 28 Oct 2001 07:28:53 -0600
> > "David" <maniacdavid@cableone.net> wrote:
> >
> > > I''m trying to get my AOL IM''r to work
consistenly with 2 cable modems. It
> > > seems like it says it can''t connect (90% of the time,
10% it works, just
> > > pure luck) when I have both of the cable modems working together
with this
> > > iproute command
> 
> 	This is a problem for the routing and the way it is used from NAT.
> 
> > > ip route default equalize nexthop via ***.***.***.*** dev eth0
nexthop via
> > > ***.***.***.*** dev eth2
> > > iptables -A POSTROUTING -t nat -j MASQUERADE -o eth2
> > > iptables -A POSTROUTING -t nat -j MASQUERADE -o eth0
> 
> 	I''m not sure Netfilter can work with multipath routes correctly.
> Only TCP can work correctly: rerouting after cache entry expiration, etc.
> 
> 	OTOH, every usage of multipart routes should take care whether
> the different gateways allow traffic with the used source addresses
> to pass through all paths. The key in using multipath routes is teaching
> the NAT code to correctly use the right source addresses when sending
> to each path from the multipath routes, especially when they provide
> you with distinct public IP ranges. If someone thinks that this should
> work by magic, without coding such feature, he is in wrong direction.
When I saw David''s message, I didn''t even think about this (as
in my
case it brakes much earlier). But as AIM is usually very low traffic,
I guess my suggestion to direct it only to one interface was not that
wrong. Unless this is a major AIM gateway, I don''t think any coding
(and getting incompatible with the rest of linux kernels) would not be
worth the efford.
> 	Chris, did you asked the Netfilter gurus for such features? [I
> hope you received my email from October 20]
No, unfortunately, I didn''t receive any answer. This is odd, as
I''m
using email quite frequently, also receiving answers, which makes my
think that my return address should be correct. But knowing my ISP and
Mr. Murphies Law, anything specially interesting has a great
probability to fail.
> > if I did understand it well, the usage of the `equalize''
argument to
> > ip makes the selection of a particular interface packet based, while
> > the omission should make it session base. I have tried all
> 
> 	I''m not sure the equalize patch is into the mainstream kernel.
I''m using only standard kernels, currently 2.4.9 and 2.4.12-acX. As
I''ve tried to use equalize and didn''t get an error message, so
it
should be working. But I have done hundrets of tests, so it is easy
that I oversaw or don''t recall something.
> > combinations I''m aware of, including weight''ing of
the nexthops, and
> > it did not work. HTTP based Internet access will fail with any more or
> > less elaborated site, as requests will come from more than one
> > IP. This doesn''t mean that session oriented interface
selection
> > doesn''t work (which I can''t tell for sure); it just
means that certain
> > subsequent complete user sessions need to use the same IP. This might
> > be the reason why AIM isn''t working, as it seems to expect
always the
> > same IP from you.
> 
> 	IMO, Netfilter should be teached to use destination cache and
> multipath routes. I already have it completely working for Linux 2.2
> masquerade (5 patches) but for 2.4 I''m at 60% (3 of 5 patches):
I''m forced to use 2.4 kernels for not networking, hardware dependant
reasons. So it''s a pitty that the 2.4 patch isn''t finished yet
as i''d
loved to try it, but if you want, I would be gladly willing to help
testing. What are the chances that this will be introduced to the main
stream kernel sources?
> http://www.linuxvirtualserver.org/~julian/
> Under the already incorrect title "Dead Gateway Detection"
I do have a problem understanding this: Let me take a concrete
situation as an example: I have a firewall with three interfaces, one
to the LAN and two to the DSL modems. There is a server behind running
squid. All clients on the LAN are configured to use squid for Internet
access; no forwarding is done beside for the server itself. If there
is a destination cache, all connections to the same server would go
through the same interface. Is this right? There ares some very
important sites which are visited (according to squid reports) by up
to 80% of clients and time (not every day and not always the same, but
it happens frequently). Those sites also use load balancing (the same
address resolves to many IPs). If I understood right, one session is
the set of packets which are exchanged between a connect and a close
statement. From the HTTP point of view (including cookies and IP
tracking) however, a session can be from the first access until the
user explicitly closing of the session, or a timeout of inactivity
happens. I have observed that within an HTTP session to that site, the
destination IP does change (but not within a session in the sense of
connect-close). In this very situation, how can the destination cache
tell that the following connect (i.e., a new tcp session) is part of
an already existing HTTP session or that the proxy is doing this in
name of a different user or workstation on the LAN? Unless I missed
something really important, I would suppose that those 80% would have
to go through the same interface, and that HTTP sessions will break
each time a session is opened on a different destination IP for the
same DNS name. I also can''t imagine that it would help to store in the
destination cache the network addresses rather than destination
addresses, as the IPs corresponding to those sites are not continuous,
and there are others in the middle which exist and would go to
different services and/or sites. OTOH, even if not `load balancing'',
at least it would avoid breaking complex HTTP sessions. For my
understanding, the solution would be an extension to the RELATED
attribute of a packet, similar as for FTP where a secondary connection
is built, although it must be much more difficult.

--
Christoph Simon
datageo@terra.com.br
---
^X^C
q
quit
:q
^C
end
x
exit
ZZ
^D
?
help
NO CARRIER
.

Hello,

	I''m answering to two emails ...

On Sun, 28 Oct 2001, Christoph Simon wrote:
> On Sun, 28 Oct 2001 07:28:53 -0600
> "David" <maniacdavid@cableone.net> wrote:
>
> > I''m trying to get my AOL IM''r to work consistenly
with 2 cable modems. It
> > seems like it says it can''t connect (90% of the time, 10% it
works, just
> > pure luck) when I have both of the cable modems working together with
this
> > iproute command
	This is a problem for the routing and the way it is used from NAT.
> > ip route default equalize nexthop via ***.***.***.*** dev eth0 nexthop
via
> > ***.***.***.*** dev eth2
> > iptables -A POSTROUTING -t nat -j MASQUERADE -o eth2
> > iptables -A POSTROUTING -t nat -j MASQUERADE -o eth0
	I''m not sure Netfilter can work with multipath routes correctly.
Only TCP can work correctly: rerouting after cache entry expiration, etc.

	OTOH, every usage of multipart routes should take care whether
the different gateways allow traffic with the used source addresses
to pass through all paths. The key in using multipath routes is teaching
the NAT code to correctly use the right source addresses when sending
to each path from the multipath routes, especially when they provide
you with distinct public IP ranges. If someone thinks that this should
work by magic, without coding such feature, he is in wrong direction.
> > Internet and everything else works fine with that. I need a solution
whether
> > it be some kind of forwarding (port 5190) so that anything received
comes
> > through 1 ethernet address. It might even have to be sent out the same
> > ethernet address but I''m thinking either one would work if
there is someway
> > to put a return address on the packet or something. I know AOL
im''r works
> > 100% when the linux box is routing just through 1 cable modem.
>
> I''m a bit surprised that you say that `Internet and everything
else
> works fine''. I''ve tried this and it did *not* work
properly. Actually,
	Chris, did you asked the Netfilter gurus for such features? [I
hope you received my email from October 20]
> if I did understand it well, the usage of the `equalize'' argument
to
> ip makes the selection of a particular interface packet based, while
> the omission should make it session base. I have tried all
	I''m not sure the equalize patch is into the mainstream kernel.
> combinations I''m aware of, including weight''ing of the
nexthops, and
> it did not work. HTTP based Internet access will fail with any more or
> less elaborated site, as requests will come from more than one
> IP. This doesn''t mean that session oriented interface selection
> doesn''t work (which I can''t tell for sure); it just means
that certain
> subsequent complete user sessions need to use the same IP. This might
> be the reason why AIM isn''t working, as it seems to expect always
the
> same IP from you.
	IMO, Netfilter should be teached to use destination cache and
multipath routes. I already have it completely working for Linux 2.2
masquerade (5 patches) but for 2.4 I''m at 60% (3 of 5 patches):

http://www.linuxvirtualserver.org/~julian/
Under the already incorrect title "Dead Gateway Detection"

	The difficult part that remains is the changes in NAT for 2.4.
I hope, I''ll have it working in weeks (I''m always busy with
other
stuff). Of course, this period can be reduced if some guru helps me :)
The key features required in NAT are:

- when new NAT connection is created obtain the preferred source address
by looking for the right out device and gateway and use it as masquerade
address

- maintain destination cache entry pointer: when it is obsoleted refresh
it by calling ip_route_output (already done for sockets)
> Christoph Simon
> datageo@terra.com.br
Regards

--
Julian Anastasov <ja@ssi.bg>

Hello,

On Sun, 28 Oct 2001, Christoph Simon wrote:
> > 	OTOH, every usage of multipart routes should take care whether
> > the different gateways allow traffic with the used source addresses
> > to pass through all paths. The key in using multipath routes is
teaching
> > the NAT code to correctly use the right source addresses when sending
> > to each path from the multipath routes, especially when they provide
> > you with distinct public IP ranges. If someone thinks that this should
> > work by magic, without coding such feature, he is in wrong direction.
>
> When I saw David''s message, I didn''t even think about
this (as in my
> case it brakes much earlier). But as AIM is usually very low traffic,
> I guess my suggestion to direct it only to one interface was not that
> wrong. Unless this is a major AIM gateway, I don''t think any
coding
> (and getting incompatible with the rest of linux kernels) would not be
> worth the efford.
	Agreed. I don''t know the David''s setup requirements, whether
the multipath route is suitable (see below why). Multipath is suitable
for any NAT setup, of course, if NAT can use it (after patching).
> > 	Chris, did you asked the Netfilter gurus for such features? [I
> > hope you received my email from October 20]
>
> No, unfortunately, I didn''t receive any answer. This is odd, as
I''m
> using email quite frequently, also receiving answers, which makes my
> think that my return address should be correct. But knowing my ISP and
> Mr. Murphies Law, anything specially interesting has a great
> probability to fail.
	I sent it privately again. Did you received it?
> > > if I did understand it well, the usage of the `equalize''
argument to
> > > ip makes the selection of a particular interface packet based,
while
> > > the omission should make it session base. I have tried all
> >
> > 	I''m not sure the equalize patch is into the mainstream
kernel.
>
> I''m using only standard kernels, currently 2.4.9 and 2.4.12-acX.
As
> I''ve tried to use equalize and didn''t get an error
message, so it
> should be working. But I have done hundrets of tests, so it is easy
> that I oversaw or don''t recall something.
	No, IIRC, it is available only in user space and as a separate
patch. From ip-cref:

"only works if the kernel is patched"

	but for multipath route to universe this shouldn''t make any
sense. Equalize is useful for routes with small number of targets (I never
used it), i.e. for directly connected networks.
> > 	IMO, Netfilter should be teached to use destination cache and
> > multipath routes. I already have it completely working for Linux 2.2
> > masquerade (5 patches) but for 2.4 I''m at 60% (3 of 5
patches):
>
> I''m forced to use 2.4 kernels for not networking, hardware
dependant
> reasons. So it''s a pitty that the 2.4 patch isn''t
finished yet as i''d
> loved to try it, but if you want, I would be gladly willing to help
> testing. What are the chances that this will be introduced to the main
> stream kernel sources?
	When I finish it I will notify you, I''ll be happy if you test
it :) About the mainstream kernels, for now I''m not sure. I have so
much pending stuff for the kernel but the kernel is still in "stable"
state. And may be some of the stuff does not sound very good to the net
gurus :)
> I do have a problem understanding this: Let me take a concrete
> situation as an example: I have a firewall with three interfaces, one
> to the LAN and two to the DSL modems. There is a server behind running
> squid. All clients on the LAN are configured to use squid for Internet
> access; no forwarding is done beside for the server itself. If there
> is a destination cache, all connections to the same server would go
> through the same interface. Is this right? There ares some very
	The destination cache usage handles the right routing once
the connection has source address bound. Even if the multipart route mislead
the socket or the conntracking they can reroute the packet to the
right gateway according to the bound source address. Of course, this
can sound ugly for some gurus but rerouting is needed, you can see
it for the sockets. As result, in some cases, the first input route
does not select the right gateway (you know, bad design, the routing
is not managed from the conntracking). The main problem is that the
routing occurs before we see what is the bound source address for
the NAT-ed connection.
> important sites which are visited (according to squid reports) by up
> to 80% of clients and time (not every day and not always the same, but
> it happens frequently). Those sites also use load balancing (the same
> address resolves to many IPs). If I understood right, one session is
> the set of packets which are exchanged between a connect and a close
> statement. From the HTTP point of view (including cookies and IP
	Yes, your NAT always selects one gateway, may be?
> tracking) however, a session can be from the first access until the
> user explicitly closing of the session, or a timeout of inactivity
> happens. I have observed that within an HTTP session to that site, the
> destination IP does change (but not within a session in the sense of
> connect-close). In this very situation, how can the destination cache
> tell that the following connect (i.e., a new tcp session) is part of
> an already existing HTTP session or that the proxy is doing this in
> name of a different user or workstation on the LAN? Unless I missed
	Once tha path (src->dst) is cached it is used from all
connections that hit this cache entry (same src and dst). About
the HTTP sessions selecting different source address, this is
really a problem where connection persistence is required into the
remote servers. But may be this is a task for the proxy server
always to use specific source address when serving a client. This
is only a beta idea :)

	The destination cache entry has only one purpose: to reroute
the packet to its gateway according to the bound source address (the
masquerade address). Of course, the best implementation can be
discussed but this is what I see for the current netfilter
infrastructure. Such details should be discussed in netfilter mail
list. I know, that there are solutions using fwmarking and they work
but you can''t use the fwmarks for other purposes.

	In any case, when multipart routes are used you should be ready
to send packets with specific source IPs through any path from the
route. But you can see in dgd-usage.txt file in my page that the
multipath is used in different way: to select different gateways for
the masqueraded traffic in loadbalancing way but after that point
alternative routes are used because in the example each gateway
can pass only traffic from one public IP rangeand only the alternative
routes can use different prefsrc. The number of ISPs
that provide you with two or more lines covering same IP range is
small. Most of the users using multipath routes are using it
incorrectly. The current model guarantees the feature works only when
these gateways are reachable through different devices. The most
of the setups I saw are using multipath route NAT-ing to two ADSL
lines reachable through one external device and they expect from such
route to work. From my patches you can see that this is possible
only when the gateway IP address is used as a route lookup key.
> something really important, I would suppose that those 80% would have
> to go through the same interface, and that HTTP sessions will break
> each time a session is opened on a different destination IP for the
> same DNS name. I also can''t imagine that it would help to store in
the
	Yes, when the route cache entry is flushed (5mins by default)
some strange things can happen. This is the reason the NAT connections
should keep a pointer to a destination cache entry and to monitor its
state and to compare it to the route attached to each packet with
the main goal to not send the packet to wrong gateway that does not
relay packets with such source IP address. As result, you have to
see that packets with specific src and dst IP should go to the right
gateway(s).
> destination cache the network addresses rather than destination
> addresses, as the IPs corresponding to those sites are not continuous,
> and there are others in the middle which exist and would go to
> different services and/or sites. OTOH, even if not `load
balancing'',
> at least it would avoid breaking complex HTTP sessions. For my
> understanding, the solution would be an extension to the RELATED
> attribute of a packet, similar as for FTP where a secondary connection
> is built, although it must be much more difficult.
	You are asking some Squid and HTTP questions but the stuff
I''m talking about is Layer 7 independent. May be when I reread your
email I''ll answer again but you first have to understand from my docs
what is possible and what is not, what is extended with these patches.

	To summarize the features the patches give you:

- you should be able to define by using routes what IP range (source
addresses) through which gateway(s) can reach the universe. In most
of the cases it is one gateway per range even if you have many
uplink lines (with distinct ranges in such case).

- the NAT routers can use multipath feature for the packets from
private networks (NAT-ed) to universe by using distinct gateways.
By this way the masqueraded traffic can utilize all uplink lines.
The traffic already bound to specific IP ranges can go only through
its gateway(s). The traffic to your servers (published in DNS
server) can reach your through many lines but you need routes to
return it back through the allowed gateways. Anything else is a
job for your DNS server (failover, etc).

As result, the multipath route is only for the NAT-ed networks, i.e.

ip rule add prio 100 from 192.168.0.0/16 table 100
ip route add default table 100 src ANYSRC \
	nexthop via GW1 dev DEV1 \
	nexthop via GW2 dev DEV2

Then alternative routes are used to the same gateways:

ip rule add prio 200 from 0/0 table 200
ip route append default table 200 via GW1 dev DEV1 src SRC1
ip route append default table 200 via GW2 dev DEV2 src SRC2

The traffic with src IP from public networks is forwarded without
SNAT-ing to the allowed gateway - no multipath routing unless
allowed (I already mentioned that most of the setups use only
one gateway per public range).

Following such rules you can build only with proper routes a good
setup that will allow balancing your uplink lines for the outgoing
traffic. You can see example for such routes in dgd-usage.txt. Of
course, the NAT should be helpful in this direction to always
following the actual routes'' state.
> Christoph Simon
> datageo@terra.com.br

Regards

--
Julian Anastasov <ja@ssi.bg>

Hello,

On Sun, 28 Oct 2001, Christoph Simon wrote:
> I''m forced to use 2.4 kernels for not networking, hardware
dependant
> reasons. So it''s a pitty that the 2.4 patch isn''t
finished yet as i''d
> loved to try it, but if you want, I would be gladly willing to help
> testing. What are the chances that this will be introduced to the main
	I just uploaded the first beta version for the netfilter
and routing changes. You need to apply:

00_static_routes-2.4.12-5.diff
01_alt_routes-2.4.12-5.diff
01_arp_prefsrc-2.4.12-5.diff
05_nf_reroute-2.4.14-5pre1.diff

When you are using multipath routes it is recommended the ARP state
for all gateways to be present (simple ICMP ping checks can refresh
the state, the same is achieved if you have out->in traffic from
these gateways, i.e. when remote clients connect to your servers
through these links).

You can try -j MASQUERADE, -j DNAT, ipchains or ipfwadm masquerade.

-j SNAT is not completed (you will have the same problems as without
these patches)

In dgd-usage.txt you can find the latest info about what is done.
> > http://www.linuxvirtualserver.org/~julian/
> > Under the already incorrect title "Dead Gateway Detection"
> --
> Christoph Simon
> datageo@terra.com.br

Regards

--
Julian Anastasov <ja@ssi.bg>