LARTC - Nov 2000 - Can't use two links on a linux box - correction/addition

I''ve just subscribed to this list, and found this article.  It solved
my problem
perfectly, but he described the solution I was trying as possible, which
I''ve
found is not.

I have one machine running several services including firewall, masq, smtp,
pop3, http, etc.  This is the first Linux computer for the company. 
I''m working
on diversifying their server setup later...

I was reading some old docs that policy routing didn''t work on packets
from
local processes.  I''m sure glad it was wrong. :)

Here''s what my routes looked like:
10.0.0.0 dev eth0  scope link  src 10.0.0.2
63.194.293.210 dev eth1  scope link  src 63.194.293.219
10.1.1.1 dev eth2  scope link  src 10.1.1.2
10.0.0.0/24 dev eth0  scope link
127.0.0.0/8 dev lo  scope link
default
        nexthop via 63.194.293.210  dev eth1 weight 1
        nexthop via 10.1.1.1  dev eth2 weight 1

here''s my ip rule:
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

The default.  Now here''s my problem.  If I connect from the internet to
one of
my tcp services on 10.1.1.2 (which is behind another masq-ing firewall from the
isp for dsl) I would get responses from 63.194.239.202.  As you can see, this is
not good, and kills any tcp traffic.

I added two tables, 40 and 50.
Each has a route to the ISP''s gateway and a default route.

Here''s the new ip rule:
0:      from all lookup local
32764:  from 10.1.1.2 lookup 50
32765:  from 63.194.293.219 lookup 40
32766:  from all lookup main
32767:  from all lookup default

Everything is fine now, thanks to Van''s help.

I just don''t want anyone else to waste their time on what I tried...

HTH

van Leeuwen Wrote:

I''ve quoted your entire e-mail, and my response is at the far bottom.

On Wed, 25 Oct 2000, Gustavo Gibson da Silva wrote:
> Hi,
> I have a linux box connected to two leased lines as follows:
>         _/\__/\_
>        /        \     +--------+        +------------+
>       ( Internet )----+ Router |        | SMTP server|
>        \_  __  _/     +----+---+        +------+-----+
>          \/  \/            | 200.200.200.254   |
>    100.100.100.2\   -+-------+----- ... ---------+--
>     100.100.100.1\   | 200.200.200.1
>               +-+--+--+
>               | Linux |
>               +---+---+    _/\__/\_
>       192.168.1.1 |       /        \
>                  -+------( Intranet )
>                           \_  __  _/
>                             \/  \/
Nice set up. Not all that unfamiliar to me. ;)
> The router came first when there was a small intranet. Now the network
> has grown very much and we were unable to upgrade our link speed because
> the local telco is unable to provide a 128Kbps link. Then we decided to
> install a linux box to act as a proxy server (squid) with a wanpipe card
> and lease a frame-relay link. This new link would have a different IP
> address and there would be two squids running. One would use the old
> link and another the new one. In order to achieve that I first attached
> the outgoing IP address for each squid to the propper network interface.
> Then I followed Adv-Routing HowTo and tried the following:
> # ip rule add from 100.100.100.1 table 200
> # ip route add default via 100.100.100.2 table 200
> # ip route add default via 200.200.200.1 table main
> # ip route flush cache
That looks too darn short.
> I tried some traceroutes with the -s option to check whether this was
> working but apparently it was not. I also tried to create the default
> route via 200.200.200.1 on table 253 but it behaved just the same way.
> Then I created it on a custom table (201) and added a rule for it. Then
> the network was unreachable.
> I tried the same things on kernel 2.2.14 (suse 6.4) and kernel 2.2.16
> (on a modified redhat). I tried also on a machine with two network
> cards. I also played with /proc/sys/net/ipv4/ip_forward.
> The ip command version was ss991023 (suse 6.4) and ss001007 on redhat.
> What am I doing wrong?
The routing. I have set up more than one linux router for the situation
you''re describing.

Let me assume that you want to do equal cost loadbalancing on the
two uplinks to the internet. Let me also assume that the interfaces are
called eth1 and eth2 for the 100.100.100.1 and 200.200.200.1 links
respectively.

First, you want to set up interface-specific routing tables. These are going
to pretend there''s only one single uplink to the internet.

   ip route add 200.200.200.0/24 dev eth2 table 200
   ip route add default via 200.200.200.254 dev eth2 table 200

   ip route add 100.100.100.2 dev eth1 table 100
   ip route add default via 100.100.100.1 dev eth1 table 100

Then you want to set up your ordinary routing. Routing to the gateways is
totally unambiguous, as the ''trouble'' starts at the gateways
(multiple paths to
get to hosts beyond them, after all). So, there''s host- or network
routes to
the gateways in the main table. Note the src hints, so that the kernel will
insert the correct IP address in packets with an unspecified source address
leaving through that interface. This means eventual replies will be sent back
to that interface.

   ip route add 100.100.100.2 src 100.100.100.1 dev eth1
   ip route add 200.200.200.0/24 src 200.200.200.1 dev eth2

Okay, the ordinary static routing is done. However, we don''t have a
default
route yet. Let''s make that a multipath route. That is done as follows:

   ip route add default scope global nexthop via 100.100.100.2 \
                                     nexthop via 200.200.200.254

This is an equal cost multipath default route onto the internet, and it
should provide you with full connectivity and some load balancing. Note that
there is a weight option to the nexthop subcommand of ip route. You might
want check the ip-cref document for that.

Finally, we want to redirect to the specific tables for the case we do have
a source address:

   ip rule add from 100.100.100.1 table 100
   ip rule add from 200.200.200.1 table 200

(Never mind the tables I''ve chosen, they''re there for mnemonic
value only)

Anyway, this should provide you with a routing setup that will function,
will keep the SMTP server you mentioned reachable, and will do some load
balancing over the uplinks. The specific tables (100 and 200) are not 
strictly necessary, but they provide you with the option to do a static
choice of the uplink route to take for certain traffic. This is done by
binding the proxy for that traffic to the corresponding IP address.

Doei, Arthur.

-- 
  /\    /  |             Fight Scientology, See URL: http://xenu.xtdnet.nl/ |
 /__\  /   | Buttons. Lotsa buttons. I like buttons. [Big Dog]              |
/    \/__  | A friend is someone with whom you can dare to Be yourself.     |
 Just Be   +-Arthur van Leeuwen, arthurvl@sci.kun.nl------------------------+

On Thu, 9 Nov 2000, Mike Fedyk wrote:
> I''ve just subscribed to this list, and found this article.  
> It solved my problem perfectly, but he described the solution 
> I was trying as possible, which I''ve found is not.
That is interesting... I can''t see much difference from the setup
you''ve made
with what I''ve described, but for the ordering of the route setup
commands
(which, unfortunately, *is* significant, a detail I forgot when composing
the e-mail).
> I was reading some old docs that policy routing didn''t work on
packets from
> local processes.  I''m sure glad it was wrong. :)
I''ve never seen those docs, fortunately. If I had, I probably
wouldn''t have
tried this... ;)
> Here''s what my routes looked like:
> 10.0.0.0 dev eth0  scope link  src 10.0.0.2
> 63.194.293.210 dev eth1  scope link  src 63.194.293.219
> 10.1.1.1 dev eth2  scope link  src 10.1.1.2
> 10.0.0.0/24 dev eth0  scope link
> 127.0.0.0/8 dev lo  scope link
> default
>         nexthop via 63.194.293.210  dev eth1 weight 1
>         nexthop via 10.1.1.1  dev eth2 weight 1
> 
> here''s my ip rule:
> 0:      from all lookup local
> 32766:  from all lookup main
> 32767:  from all lookup default
> The default.  Now here''s my problem.  If I connect from the
internet to one of
> my tcp services on 10.1.1.2 (which is behind another masq-ing firewall 
> from the isp for dsl) I would get responses from 63.194.239.202.  
> As you can see, this is not good, and kills any tcp traffic.
Yes. TCP only handles multipath traffic if the addresses stay the same on
all the paths. But it''s not really good for most UDP based protocols
either.
> I added two tables, 40 and 50.
> Each has a route to the ISP''s gateway and a default route.
Exactly the right thing to do.
> Here''s the new ip rule:
> 0:      from all lookup local
> 32764:  from 10.1.1.2 lookup 50
> 32765:  from 63.194.293.219 lookup 40
> 32766:  from all lookup main
> 32767:  from all lookup default
> 
> Everything is fine now, thanks to Van''s help.
The name''s Arthur. `van Leeuwen'' is my surname. Yes,
it''s a silly European
thing. Note quite unlike `van Beethoven'' in `Ludwig van
Beethoven''.
> I just don''t want anyone else to waste their time on what I
tried...
This stuff really should be in the HOWTO. Now to find some time to 
put it in...

Doei, Arthur.

-- 
  /\    / |      arthurvl@sci.kun.nl      | Work like you don''t need
the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there''s nobody
watching

Arthur van Leeuwen wrote:
> 
> On Thu, 9 Nov 2000, Mike Fedyk wrote:
> 
> > I''ve just subscribed to this list, and found this article.
> > It solved my problem perfectly, but he described the solution
> > I was trying as possible, which I''ve found is not.
> 
> That is interesting... I can''t see much difference from the setup
you''ve made
> with what I''ve described, but for the ordering of the route setup
commands
> (which, unfortunately, *is* significant, a detail I forgot when composing
> the e-mail).
The setup you described is exactly what worked in the end.

"The specific tables (100 and 200) are not 
strictly necessary, but they provide you with the option to do a static
choice of the uplink route to take for certain traffic."

This is what my reply was for, you really do *need* the extra tables and ip rule
settings.
> > I was reading some old docs that policy routing didn''t work
on packets from
> > local processes.  I''m sure glad it was wrong. :)
> 
> I''ve never seen those docs, fortunately. If I had, I probably
wouldn''t have
> tried this... ;)
> 
http://www.linuxdoc.org/HOWTO/Net-HOWTO/c1431.html
There is a link to http://www.compendium.com.ar/policy-routing.txt

Which was made when 2.1.xx was current.
> Yes. TCP only handles multipath traffic if the addresses stay the same on
> all the paths. But it''s not really good for most UDP based
protocols either.
> 
The only test that worked was icmp ping. ;P
> The name''s Arthur. `van Leeuwen'' is my surname. Yes,
it''s a silly European
> thing. Note quite unlike `van Beethoven'' in `Ludwig van
Beethoven''.
> 
Ok, np.  That was what was in the From: in the archive, so I used that.  Maybe
you should add Arthur to your name in the From: field. "Arthur van
Leeuwen"?

I hope this discussion will help others when it goes into the archive.  It
certainly helped me.  If they actually take the time to check the archive... but
that''s another issue.

HTH
-- 

Mike Fedyk                   "They that can give up essential liberty
Information Systems           to obtain a little temporary safety
Match Mail Productions Inc.   deserve neither liberty nor safety."
mfedyk@matchmail.com                                   Ben Franklin