CentOS - Jul 2009 - Multiple Internet facing Nics - Gateway issue

I have a server with 4 nics.  Two are using different internet connections,
both with static IP's, and two are connected to our private network.

The two internet facing nics seem to be battling over the gateway
designation.  Which ever I designate as the gateway the other stops
responding to incoming traffic.  I need both to listen to inbound traffic.
One for our main web page and the other for another web server.  I found
some information of Iproute2 but that did not seem to solve my issues.

Here are the config files:

ifcfg-eth2
DEVICE=eth2
BOOTPROTO=static
HWADDR=00:24:E8:52:92:8E
ONBOOT=yes
IPADDR=xx.yy.51.45
NETMASK=255.255.255.252
GATEWAY=xx.yy.51.46

ifcfg-eth3
DEVICE=eth3
BOOTPROTO=static
HWADDR=00:24:E8:52:92:90
ONBOOT=yes
IPADDR=aa.bb.166.2
NETMASK=255.255.255.224
GATEWAY=aa.bb.166.1

network
NETWORKING=yes
NETWORKING_IPV6=no
HOSTNAME=heritage01
GATEWAY=xx.yy.51.46
GATEWAYDEV=eth2

rt_tables
#
# reserved values
#
255     local
254     main
253     default
0       unspec
#
# local
#
#1      inr.ruhep
200 Cable
201 T1
iproute2 routes
ip route add xx.yy.51.44/30 dev eth2 src xx.yy.51.46 table Cable
ip route add default via xx.yy.51.46
ip route add xx.yy.51.44/30 dev eth2 src xx.yy.51.46
ip route add default via xx.yy.51.46
ip rule add from xx.yy.51.46 table Cable
ip route add aa.bb.166.0/27 dev eth3 src aa.bb.165.2 table T1
ip route add default via aa.bb.166.1
ip route add aa.bb.166.0/27 dev eth3 src aa.bb.166.2
ip rule add from aa.bb.166.2 table T1
ip route flush cache

Other people seem to be using this configuration successfully but I seem to
be missing something important.

Any help is greatly appreciated!!!!!!!  Thanks!
-------------- next part --------------
An HTML attachment was scrubbed...
URL:
<http://lists.centos.org/pipermail/centos/attachments/20090701/171814ba/attachment-0003.html>

On Wed, 1 Jul 2009, Doug Coats wrote:
> I have a server with 4 nics.? Two are using different internet connections,
both with static IP's, and two
> are connected to our private network.
> ?
> The two internet facing nics seem to be battling over the gateway
designation.? Which ever I designate as
> the gateway the other stops responding to incoming traffic.? I need both to
listen to inbound traffic.? One
> for our main web page and the other for another web server.? I found some
information of Iproute2 but that
> did not seem to solve my issues.
> ?
> Here are the config files:
> ?
> ifcfg-eth2
> DEVICE=eth2
> BOOTPROTO=static
> HWADDR=00:24:E8:52:92:8E
> ONBOOT=yes
> IPADDR=xx.yy.51.45
> NETMASK=255.255.255.252
> GATEWAY=xx.yy.51.46
> ?
> ifcfg-eth3
> DEVICE=eth3
> BOOTPROTO=static
> HWADDR=00:24:E8:52:92:90
> ONBOOT=yes
> IPADDR=aa.bb.166.2
> NETMASK=255.255.255.224
> GATEWAY=aa.bb.166.1
> ?
> network
> NETWORKING=yes
> NETWORKING_IPV6=no
> HOSTNAME=heritage01
> GATEWAY=xx.yy.51.46
> GATEWAYDEV=eth2
remove the gateway from /etc/sysconfig/network, reboot and try again.

-- 
James A. Peltier
Systems Analyst (FASNet), VIVARIUM Technical Director
HPC Coordinator
Simon Fraser University - Burnaby Campus
Phone   : 778-782-6573
Fax     : 778-782-3045
E-Mail  : jpeltier at sfu.ca
Website : http://www.fas.sfu.ca | http://vivarium.cs.sfu.ca
           http://blogs.sfu.ca/people/jpeltier
MSN     : subatomic_spam at hotmail.com

The point of the HPC scheduler is to
keep everyone equally unhappy.

Doug Coats schrieb:
> I have a server with 4 nics.  Two are using different internet connections,
> both with static IP's, and two are connected to our private network.
> 
> The two internet facing nics seem to be battling over the gateway
> designation.  Which ever I designate as the gateway the other stops
> responding to incoming traffic.  I need both to listen to inbound traffic.
> One for our main web page and the other for another web server.  I found
> some information of Iproute2 but that did not seem to solve my issues.
> 
> Here are the config files:
> 
> ifcfg-eth2
> DEVICE=eth2
> BOOTPROTO=static
> HWADDR=00:24:E8:52:92:8E
> ONBOOT=yes
> IPADDR=xx.yy.51.45
> NETMASK=255.255.255.252
> GATEWAY=xx.yy.51.46
> 
> ifcfg-eth3
> DEVICE=eth3
> BOOTPROTO=static
> HWADDR=00:24:E8:52:92:90
> ONBOOT=yes
> IPADDR=aa.bb.166.2
> NETMASK=255.255.255.224
> GATEWAY=aa.bb.166.1
> 
> network
> NETWORKING=yes
> NETWORKING_IPV6=no
> HOSTNAME=heritage01
> GATEWAY=xx.yy.51.46
> GATEWAYDEV=eth2
> 
> rt_tables
> #
> # reserved values
> #
> 255     local
> 254     main
> 253     default
> 0       unspec
> #
> # local
> #
> #1      inr.ruhep
> 200 Cable
> 201 T1
> iproute2 routes
> ip route add xx.yy.51.44/30 dev eth2 src xx.yy.51.46 table Cable
> ip route add default via xx.yy.51.46
> ip route add xx.yy.51.44/30 dev eth2 src xx.yy.51.46
> ip route add default via xx.yy.51.46
> ip rule add from xx.yy.51.46 table Cable
> ip route add aa.bb.166.0/27 dev eth3 src aa.bb.165.2 table T1
> ip route add default via aa.bb.166.1
> ip route add aa.bb.166.0/27 dev eth3 src aa.bb.166.2
> ip rule add from aa.bb.166.2 table T1
> ip route flush cache
> 
> Other people seem to be using this configuration successfully but I seem to
> be missing something important.
> 
> Any help is greatly appreciated!!!!!!!  Thanks!
The GATEWAY parameter within the ifcfg-<ethX> configuration files tries
to set the default gateway. There can be just 1 default gateway for a
system. So, the interface which comes up first will set the route and
the other has no effect.

Remove the GATEWAY entry from all ifcfg-<ethX> files and place the
GATEWAY parameter into /etc/sysconfig/network only.

Running different networks you must can set additional routings by
route-<ethX> configuration files inside /etc/sysconfig/network-scripts/.

Given your default gateway is set as xx.yy.51.46 for the network
configured on eth2, you set to use a gateway aa.bb.166.1 for other
destinations by

ip route add targetnet/CIDR via aa.bb.166.1 dev eth3

Please see

http://www.centos.org/docs/5/html/5.1/Deployment_Guide/s1-networkscripts-static-routes.html

You multiple uplink topic is documented in this howto

http://lartc.org/howto/lartc.rpdb.multiple-links.html


Alexander

>
> ip rule add from xx.yy.51.46 table Cable
> ip rule add from aa.bb.166.2 table T1
>
>
> Other people seem to be using this configuration successfully but I 
> seem to be missing something important.
>  
Change the above rules to:

ip rule add from xx.yy.51.46 to default lookup Cable

ip rule add from aa.bb.166.2 to default lookup T1