CentOS - Jan 2010 - routing multiple network cards on a single subnet

I have dealt with machines that have multiple network cards in them
before, but never when they were on the same subnet so this issue has
never come up before.

My problem is that I can only access one IP address at a time.  I
started out using dhcp and found that if I went through the dhcp
song-and-dance then that address became active and the other one was
disabled, and vice versa.

On our local tech mailing list, a couple of the guys advised me that
this is due to a routing issue and, after a bit of googling around I now
understand why that is.

However, I have so far been unable to fix it.

I got rid of dhcp and set up static addresses using
system-config-network.

This machine has three network cards in it, eth0 is 192.168.1.5 and I
use that one to ssh into the box from this computer.  (I really don't
want to lose my ability to connect to eth0; this machine runs headless
on a shelf about 7 feet above the floor and it would be quite an
undertaking to dismantle it and bring it down to hook a monitor and
keyboard to it again.)

The solution to this problem appears to be easier to describe than to
implement, at least for me.  I need to have each network card reply back
on the same interface that it received a request from.

eth1 is 24.89.92.178
eth2 is 24.89.92.180

The gateway for both of these is 24.89.92.1

The suggestion that I got was to add two entries to the end
of /etc/iproute2/rt_tables (which I did with a text editor) and run a
series of ip route commands which set up a custom routing table but I'm
missing something because while the custom routing tables appear to be
getting set up, it's still not working.

I have studied the suggested routing commands and I think I understand
what they are doing and what is supposed to be happening.  But something
is still missing because it's not working.

Here is what I did and what the results are.  If I have missed anything
let me know; this covers what I think is the relevant information  as I
currently understand it.  I would sincerely appreciate any further
advice regarding this situation.  I really would like to know what I am
doing wrong and also why (in the interest of learning something from
this situation).  It's new territory for me.

[root at audio ~]# cat /etc/iproute2/rt_tables 
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
50      access1
60      access2
[root at audio ~]# ip route add 24.89.92.0/24 dev eth1 table access1
[root at audio ~]# ip route add default via 24.89.92.1 table access1
[root at audio ~]# ip rule add from 24.89.92.178/32 lookup access1
[root at audio ~]# 
[root at audio ~]# ip route add 24.89.92.0/24 dev eth2 table access2
[root at audio ~]# ip route add default via 24.89.92.1 table access2
[root at audio ~]# ip rule add from 24.89.92.180/32 lookup access2
[root at audio ~]# ip route show table access2
24.89.92.0/24 dev eth2  scope link 
default via 24.89.92.1 dev eth1 
[root at audio ~]# ip route show table access1
24.89.92.0/24 dev eth1  scope link 
default via 24.89.92.1 dev eth1 
[root at audio ~]# ip route
24.89.92.0/24 dev eth1  proto kernel  scope link  src 24.89.92.178 
24.89.92.0/24 dev eth2  proto kernel  scope link  src 24.89.92.180 
192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.5 
169.254.0.0/16 dev eth2  scope link 
default via 24.89.92.1 dev eth1 

[frankcox at mutt ~]$ ping 24.89.92.178
PING 24.89.92.178 (24.89.92.178) 56(84) bytes of data.
64 bytes from 24.89.92.178: icmp_seq=1 ttl=50 time=92.2 ms
64 bytes from 24.89.92.178: icmp_seq=2 ttl=50 time=96.2 ms
64 bytes from 24.89.92.178: icmp_seq=3 ttl=50 time=91.0 ms

--- 24.89.92.178 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 91.023/93.193/96.263/2.245 ms
[frankcox at mutt ~]$ ping 24.89.92.180
PING 24.89.92.180 (24.89.92.180) 56(84) bytes of data.

--- 24.89.92.180 ping statistics ---
6 packets transmitted, 0 received, 100% packet loss, time 5000ms

Incidentally, it is my current understanding that anything that I do
with an "ip route" command will go away on a reboot, therefore if I
somehow screw up the routing on this box completely all I have to do is
reboot it and I'll be back to what I had before.  Which is not a bad
thing at the moment.  Once I have this nailed down should I put the "ip
route" commands into /etc/rc.local?  Or is there a better place?


-- 
MELVILLE THEATRE ~ Melville Sask ~ http://www.melvilletheatre.com

I can offer one tiny bit of help ...

On Wed, Jan 20, 2010 at 11:27 AM, Frank Cox <theatre at sasktel.net>
wrote:
> Incidentally, it is my current understanding that anything that I do
> with an "ip route" command will go away on a reboot, therefore if
I
> somehow screw up the routing on this box completely all I have to do is
> reboot it and I'll be back to what I had before. ?Which is not a bad
> thing at the moment. ?Once I have this nailed down should I put the
"ip
> route" commands into /etc/rc.local? ?Or is there a better place?
man iptables-save

-- 
-Bob

Frank Cox wrote on Wed, 20 Jan 2010 10:27:29 -0600:
> I got rid of dhcp and set up static addresses using
> system-config-network.
Can't help you on the routing "back" issue. Just wanted to remind
you that
you can assign static IP addresses via DHCP to specific MAC addresses. 
That might be easier to maintain than assign static addresses on each 
machine locally.

Kai

-- 
Get your web at Conactive Internet Services: http://www.conactive.com

On Wed, 2010-01-20 at 10:27 -0600, Frank Cox wrote:
> 
> My problem is that I can only access one IP address at a time.  I
> started out using dhcp and found that if I went through the dhcp
> song-and-dance then that address became active and the other one was
> disabled, and vice versa.
I'm starting to wonder if the simplest solution to this is to punt.

If I put a $40 router between eth2 and the big scary world, then eth2
could become 192.168.whatever.whatever, and then this routing issue
would go away on its own and it could still talk to the outside world
(and vice versa) on its IP address from Access.

I assume, based on the fact that I have never encountered this before on
machines with multiple ethernet cards that were on different subnets.

Or would this still not work as it should?
-- 
MELVILLE THEATRE ~ Melville Sask ~ http://www.melvilletheatre.com

On Wed, 2010-01-20 at 10:27 -0600, Frank Cox wrote:
> My problem is that I can only access one IP address at a time.  I
> started out using dhcp and found that if I went through the dhcp
> song-and-dance then that address became active and the other one was
> disabled, and vice versa.
The solution has been found, thanks to one of the guys on our local tech
mailing list (who is a truly wonderful and extremely knowedgable and
helpful person).

For the benefit of anyone else who might have occasion to do this, here
is the complete solution.

My IP addresses are eth1=24.89.92.178, eth2=24.89.92.180
Gateway for both is 24.89.92.1

First, add two lines to /etc/iproute2/rt_tables, so it looks like this:

#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
50      access1
60      access2

Then add the following routes:

ip route add 24.89.92.0/24 dev eth1 table access1
ip route add default via 24.89.92.1 dev eth1 table access1
ip rule add from 24.89.92.178/32 lookup access1

ip route add 24.89.92.0/24 dev eth2 table access2
ip route add default via 24.89.92.1 dev eth2 table access2
ip rule add from 24.89.92.180/32 lookup access2

> 
-- 
MELVILLE THEATRE ~ Melville Sask ~ http://www.melvilletheatre.com

Frank Cox wrote:
> I have dealt with machines that have multiple network cards in them
> before, but never when they were on the same subnet so this issue has
> never come up before.
>
> My problem is that I can only access one IP address at a time.  I
> started out using dhcp and found that if I went through the dhcp
> song-and-dance then that address became active and the other one was
> disabled, and vice versa.
>
> On our local tech mailing list, a couple of the guys advised me that
> this is due to a routing issue and, after a bit of googling around I now
> understand why that is.
>
> However, I have so far been unable to fix it.
>
> I got rid of dhcp and set up static addresses using
> system-config-network.
>
> This machine has three network cards in it, eth0 is 192.168.1.5 and I
> use that one to ssh into the box from this computer.  (I really don't
> want to lose my ability to connect to eth0; this machine runs headless
> on a shelf about 7 feet above the floor and it would be quite an
> undertaking to dismantle it and bring it down to hook a monitor and
> keyboard to it again.)
>
> The solution to this problem appears to be easier to describe than to
> implement, at least for me.  I need to have each network card reply back
> on the same interface that it received a request from.
>
> eth1 is 24.89.92.178
> eth2 is 24.89.92.180
>
> The gateway for both of these is 24.89.92.1
>
> The suggestion that I got was to add two entries to the end
> of /etc/iproute2/rt_tables (which I did with a text editor) and run a
> series of ip route commands which set up a custom routing table but I'm
> missing something because while the custom routing tables appear to be
> getting set up, it's still not working.
>
> I have studied the suggested routing commands and I think I understand
> what they are doing and what is supposed to be happening.  But something
> is still missing because it's not working.
>
> Here is what I did and what the results are.  If I have missed anything
> let me know; this covers what I think is the relevant information  as I
> currently understand it.  I would sincerely appreciate any further
> advice regarding this situation.  I really would like to know what I am
> doing wrong and also why (in the interest of learning something from
> this situation).  It's new territory for me.
>
> [root at audio ~]# cat /etc/iproute2/rt_tables 
> #
> # reserved values
> #
> 255     local
> 254     main
> 253     default
> 0       unspec
> #
> # local
> #
> #1      inr.ruhep
> 50      access1
> 60      access2
> [root at audio ~]# ip route add 24.89.92.0/24 dev eth1 table access1
> [root at audio ~]# ip route add default via 24.89.92.1 table access1
> [root at audio ~]# ip rule add from 24.89.92.178/32 lookup access1
> [root at audio ~]# 
> [root at audio ~]# ip route add 24.89.92.0/24 dev eth2 table access2
> [root at audio ~]# ip route add default via 24.89.92.1 table access2
> [root at audio ~]# ip rule add from 24.89.92.180/32 lookup access2
> [root at audio ~]# ip route show table access2
> 24.89.92.0/24 dev eth2  scope link 
> default via 24.89.92.1 dev eth1 
> [root at audio ~]# ip route show table access1
> 24.89.92.0/24 dev eth1  scope link 
> default via 24.89.92.1 dev eth1 
> [root at audio ~]# ip route
> 24.89.92.0/24 dev eth1  proto kernel  scope link  src 24.89.92.178 
> 24.89.92.0/24 dev eth2  proto kernel  scope link  src 24.89.92.180 
> 192.168.1.0/24 dev eth0  proto kernel  scope link  src 192.168.1.5 
> 169.254.0.0/16 dev eth2  scope link 
> default via 24.89.92.1 dev eth1 
>
> [frankcox at mutt ~]$ ping 24.89.92.178
> PING 24.89.92.178 (24.89.92.178) 56(84) bytes of data.
> 64 bytes from 24.89.92.178: icmp_seq=1 ttl=50 time=92.2 ms
> 64 bytes from 24.89.92.178: icmp_seq=2 ttl=50 time=96.2 ms
> 64 bytes from 24.89.92.178: icmp_seq=3 ttl=50 time=91.0 ms
>
> --- 24.89.92.178 ping statistics ---
> 3 packets transmitted, 3 received, 0% packet loss, time 2001ms
> rtt min/avg/max/mdev = 91.023/93.193/96.263/2.245 ms
> [frankcox at mutt ~]$ ping 24.89.92.180
> PING 24.89.92.180 (24.89.92.180) 56(84) bytes of data.
>
> --- 24.89.92.180 ping statistics ---
> 6 packets transmitted, 0 received, 100% packet loss, time 5000ms
>
> Incidentally, it is my current understanding that anything that I do
> with an "ip route" command will go away on a reboot, therefore if
I
> somehow screw up the routing on this box completely all I have to do is
> reboot it and I'll be back to what I had before.  Which is not a bad
> thing at the moment.  Once I have this nailed down should I put the
"ip
> route" commands into /etc/rc.local?  Or is there a better place?
>
>
>   
Hi
This Article should be exactly what you need

http://www.linuxjournal.com/node/7291/print

On Wednesday 20 January 2010 13:57, Frank Cox wrote:
>  On Wed, 2010-01-20 at 09:50 -0800, R-Elists wrote:
>  > ummm, why do the two different networks need an IP on the same
>  > subnet ?
>
>  I have had a number of people ask me why I want this arrangement, where
>  I have two modems on a single outbound subnet.
>
>  This is (going to be) a server with limited upload bandwidth.  By having
>  two outbound connections, I can use a round robin dns entry to share the
>  load between the two connections and increase my capacity.
Please be aware that DNS was not designed to do what you are doing.  Yes it 
will do a round-robin but is not connection aware.  Lose a link and you lose 
half of the connections even though one link is still active.


-- 

Regards
Robert

Linux User #296285
http://counter.li.org