Hello I''m Xisco, I am new in the mailing list, I''ve been readin lots of posts seeking an answer to my questions, I hope you can help. This is the situation: L R I O eth0--------62.42.XX.XX (assigned by dhcp) (gw assigned 62.42.a.b) N U eth1--------62.42.YY.YY (assigned by dhcp) (gw assigned 62.42.a.b) U T eth2--------62.43.ZZ.ZZ (assigned by dhcp) (gw assigned 62.43.c.d) X E eth3--------172.31.0.254 (manually assigned) (this is the gateway for the LAN) R eth3--------192.168.0.6 (manually assigned) (this is a routers subnet) Im using iproute2 + fwmark + iptables mark + one table per device with one different gw each) I classify the kind of traffic classifying it in bulk traffic and on-line traffic. As of the nature of the different applications that are run in this place (we are a lot of friends with a place where we play on-line games, just for fun :) ). I do port forwarding through different gateways, I have 5 operating gateways having traffic all the time, One link for p2p, another for ftp downloads, and the rest for playing quake, warcraft, and whatever game is good, I love linux and Im trying to make all them appreciate this fantastic OS ;) Everything worked fine till we get the modem cable attached to eth1, this modem is from the same ISP than eth0''s, and unfortunately, this ISP assigns to eth0 and eth1 the same gateway (62.42.a.b). I''ve been reading arp flux problem with this kind of troubles, and also the hidden patch but I couldn''t make it work. This is the whole config: EXTIF1="eth0" EXTIF2="eth1" EXTIF3="eth2" MEDIF="eth3" INTIF="eth4" GW1 = Gateway of the first cmodem GW2 = Gateway of the second cmodem Gw3 = Gateway of the third cmodem IP1 = Public IP of eth0 IP2 = Public IP of eth1 IP3 = Public IP of eth2 Create the tables for each route: echo 255 local > /etc/iproute2/rt_tables echo 254 main >> /etc/iproute2/rt_tables echo 253 default >> /etc/iproute2/rt_tables echo 0 unspec >> /etc/iproute2/rt_tables echo 200 cablemodem1 >> /etc/iproute2/rt_tables echo 201 router1 >> /etc/iproute2/rt_tables echo 202 router2 >> /etc/iproute2/rt_tables echo 203 router3 >> /etc/iproute2/rt_tables echo 204 balanceo >> /etc/iproute2/rt_tables echo 205 cablemodem2 >> /etc/iproute2/rt_tables echo 206 cablemodem3 >> /etc/iproute2/rt_tables Turn off reverse path filtering echo "0" > /proc/sys/net/ipv4/conf/eth0/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth1/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth2/rp_filter echo "0" > /proc/sys/net/ipv4/conf/eth3/rp_filter Messing with the arp flux problem (I suppose thats my problem) echo "0" > /proc/sys/net/ipv4/conf/eth0/hidden echo "0" > /proc/sys/net/ipv4/conf/eth1/hidden echo "1" > /proc/sys/net/ipv4/conf/eth0/arp_filter echo "1" > /proc/sys/net/ipv4/conf/eth1/arp_filter Assign fwmark to each table ip rule add fwmark 1 table router1 ip rule add fwmark 2 table router2 ip rule add fwmark 3 table router3 ip rule add fwmark 4 table balanceo ip rule add fwmark 5 table cablemodem1 ip rule add fwmark 6 table cablemodem2 ip rule add fwmark 7 table cablemodem3 Assign route for each table ip route add table balanceo eql nexthop via $GW2 dev $EXTIF2 weight 1 nexthop via $GW1 dev $EXTIF1 weight 1 ip route add default via 192.168.0.1 dev $MEDIF src 192.168.0.6 table router1 ip route add default via 192.168.0.2 dev $MEDIF src 192.168.0.6 table router2 ip route add default via 192.168.0.3 dev $MEDIF src 192.168.0.6 table router3 ip route add default via $GW1 dev $EXTIF1 src $IP1 table cablemodem1 ip route add default via $GW2 dev $EXTIF2 src $IP2 table cablemodem2 ip route add default via $GW3 dev $EXTIF3 src $IP3 table cablemodem3 We ensure each interface will put it''s own IP ip route add $GW1 dev $EXTIF1 src $IP1 ip route add $GW2 dev $EXTIF2 src $IP2 ip route add $GW3 dev $EXTIF3 src $IP3 We assign each gw as HOST for those routing tables ip rule add from $GW1 table cablemodem1 ip rule add from $GW2 table cablemodem2 ip rule add from $GW3 table cablemodem3 I think that it''s "a grosso modo" my config The result of route -n: Kernel IP routing table Destination Gateway Genmask Flags Metric Ref Use Iface 62.43.c.d 0.0.0.0 255.255.255.255 UH 0 0 0 eth2 62.42.a.b 0.0.0.0 255.255.255.255 UH 0 0 0 eth0 192.168.0.0 0.0.0.0 255.255.255.248 U 0 0 0 eth3 172.31.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth4 62.42.x.y 0.0.0.0 255.255.248.0 U 0 0 0 eth0 62.42.x.y 0.0.0.0 255.255.248.0 U 0 0 0 eth1 62.43.t.z 0.0.0.0 255.255.240.0 U 0 0 0 eth2 0.0.0.0 62.42.a.b 0.0.0.0 UG 0 0 0 eth0 The problem resides in eth0 and eth1, the get exactly the same config, just with different IP address. xerver:~# ip neigh sh 62.42.a.b dev eth0 lladdr 00:05:31:f8:17:c0 nud stale 62.43.c.d dev eth1 lladdr 00:05:31:f8:17:c0 nud stale But take a look at this, in the arp cache, the phisical address of the gw is the same for both gw. This is possible because eth0, eth1 and eth2 are from the same ISP. Maybe this isp aliases ip on interfaces for traffic management... (i don''t know). There are a lot of question without answers, I couldn''f find them: Should I remove the dafult gateway? If I should, how could I make the router itself to reach internet? Maybe marking locally generated packets and assigning a route to that fwmark? I tried this, but no success. When I run this in my script ip ro add 62.42.a.b dev eth2 src 62.42.yy.yy RTNETLINK answers: File exists Ok, thats, because both devices use the same gateway, and so, I suppose a shouldn''t univocally bind that address to any? Correct? I don''t know if I''m missing something, Hope u can help me. And lots of thanks for reading until here ;) Xisco Fernandez _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Martin A. Brown
2004-Mar-03 05:36 UTC
Re: Several interfaces same gateway. Arp flux problem? Any idea?
Xisco,
: L R
: I O eth0 - 62.42.XX.XX (assigned by dhcp) (gw assigned 62.42.a.b)
: N U eth1 - 62.42.YY.YY (assigned by dhcp) (gw assigned 62.42.a.b)
: U T eth2 - 62.43.ZZ.ZZ (assigned by dhcp) (gw assigned 62.43.c.d)
: X E eth3 - 172.31.0.254 (manually assigned) (this is the gateway for the LAN)
: R eth3 - 192.168.0.6 (manually assigned) (this is a routers subnet)
:
: Im using iproute2 + fwmark + iptables mark + one table per device with
: one different gw each) I classify the kind of traffic classifying it in
: bulk traffic and on-line traffic. As of the nature of the different
: applications that are run in this place (we are a lot of friends with a
: place where we play on-line games, just for fun :) ). I do port
: forwarding through different gateways, I have 5 operating gateways
: having traffic all the time, One link for p2p, another for ftp
: downloads, and the rest for playing quake, warcraft, and whatever game
: is good, I love linux and Im trying to make all them appreciate this
: fantastic OS ;)
Linux? What''s Linux?
: Everything worked fine till we get the modem cable attached to eth1,
: this modem is from the same ISP than eth0''s, and unfortunately, this
: ISP assigns to eth0 and eth1 the same gateway (62.42.a.b). I''ve been
: reading arp flux problem with this kind of troubles, and also the
: hidden patch but I couldn''t make it work.
To begin, you say "Everything worked fine till...", but then you
don''t
really describe the troubles you are experiencing. I get some idea that
the problem has to do with your control of ARP (from your description),
but you never tell us outright what your problem is.
So here are my two lead-in questions:
Is all of your traffic going out to the ISP over one link?
Is all of your traffic coming back from the ISP over one link?
And then I''ll respond inline with some thoughts about the rest of the
config.
: This is the whole config:
[ rearranged ]
: EXTIF1="eth0" IP1 = Pub IP GW1 = Gateway of the first cmodem
: EXTIF2="eth1" IP2 = Pub IP GW2 = Gateway of the second cmodem
: EXTIF3="eth2" IP3 = Pub IP Gw3 = Gateway of the third cmodem
: MEDIF="eth3"
: INTIF="eth4"
:
: Create the tables for each route:
[ snip ] This looks good.
: Turn off reverse path filtering
[ snip ] Also a good idea for any policy routing.
: Messing with the arp flux problem (I suppose thats my problem)
: echo "0" > /proc/sys/net/ipv4/conf/eth0/hidden
: echo "0" > /proc/sys/net/ipv4/conf/eth1/hidden
I think you want exactly the opposite. For the hidden sysctl, you should set
/p/s/n/i/c/eth{0,1}/hidden to a non-zero value. Try this:
echo "1" > /proc/sys/net/ipv4/conf/eth0/hidden
echo "1" > /proc/sys/net/ipv4/conf/eth1/hidden
: echo "1" > /proc/sys/net/ipv4/conf/eth0/arp_filter
: echo "1" > /proc/sys/net/ipv4/conf/eth1/arp_filter
You could use arp_filter here, but be forewarned, "sane" operation of
arp_filter assumes that you have "sane" routing tables, which you,
dear sir,
most definitely do not have. I think it''d be best to just leave
arp_filter
off, and let the hidden sysctl do the work. See my discussion of
arp_filter for more detail. [0]
: Assign fwmark to each table
[ snip ]
OK, so you must be marking your traffic with iptables....this is fine.
: Assign route for each table
:
: ip route add table balanceo eql nexthop via $GW2 dev $EXTIF2 weight 1 \
: nexthop via $GW1 dev $EXTIF1 weight 1
I am uncertain of the effects of this equalizing route on your entire
configuration. Perhaps another list member can remark intelligently on this.
All of the below makes sense.
: ip route add default via 192.168.0.1 dev $MEDIF src 192.168.0.6 table router1
: ip route add default via 192.168.0.2 dev $MEDIF src 192.168.0.6 table router2
: ip route add default via 192.168.0.3 dev $MEDIF src 192.168.0.6 table router3
: ip route add default via $GW1 dev $EXTIF1 src $IP1 table cablemodem1
: ip route add default via $GW2 dev $EXTIF2 src $IP2 table cablemodem2
: ip route add default via $GW3 dev $EXTIF3 src $IP3 table cablemodem3
:
: We ensure each interface will put it''s own IP
: We assign each gw as HOST for those routing tables
: I think that it''s "a grosso modo" my config
[ snip ]
Providing "route -n" output after all of the above isn''t
really necessary,
although "ip rule show" and "ip route show table
$EACH_TABLE" wouldn''t be a
bad idea, although a summary is better if possible.
: The problem resides in eth0 and eth1, the get exactly the same config, just
: with different IP address.
:
: xerver:~# ip neigh sh
:
: 62.42.a.b dev eth0 lladdr 00:05:31:f8:17:c0 nud stale
: 62.43.c.d dev eth1 lladdr 00:05:31:f8:17:c0 nud stale
:
: But take a look at this, in the arp cache, the phisical address of the
: gw is the same for both gw. This is possible because eth0, eth1 and
: eth2 are from the same ISP.
: Maybe this isp aliases ip on interfaces for traffic management... (i
: don''t know).
There are a plethora of reasons for which an ISP might have the same
hardware address mapping for a given IP address, not the least significant
of which could be that they are all handled on a single device. Really,
this is immaterial to your machine.
By the time the kernel is looking up the link layer address of the gateway
IP, it already knows the output interface. This is why for purposes of
sending packets (frames), you can have the same IP and/or link layer
address occur on multiple interfaces, and there''s no problem.
Your problem is answering for IP1 only on eth0 and IP2 only on eth1. This
is exactly the reason for the hidden patch.
: There are a lot of question without answers, I couldn''f find them:
:
: Should I remove the dafult gateway?
Which one? You have many default routes, one in each routing table.
Try:
for rtable in cablemodem{1,2,3} ; do
echo -e "\n[ -- $rtable -- ]"
ip route show table $rtable
done
: If I should, how could I make the router itself to reach internet?
Conventional rules for source address selection apply. [1]
: Maybe marking locally generated packets and assigning a route to that
: fwmark? I tried this, but no success.
Your lack of success is not surprising.
Locally generated packets and marking for output route selection has been
discussed many times on this list. There are probably other solutions,
but it seems that the netfilter -j ROUTE patch-o-matic target is the ideal
solution to this problem. [2]
: When I run this in my script
:
: ip ro add 62.42.a.b dev eth2 src 62.42.yy.yy
: RTNETLINK answers: File exists
Probably because the route already exists, created by the kernel when the
device came up. I wouldn''t panic at all about this.
: Ok, thats, because both devices use the same gateway, and so, I suppose
: a shouldn''t univocally bind that address to any? Correct?
I''m not sure what you mean.
Best of luck!
-Martin
[0] http://linux-ip.net/html/ether-arp.html#ether-arp-flux
[1] http://linux-ip.net/gl/ip-cref/node155.html
[2] http://www.netfilter.org/patch-o-matic/pom-extra.html#pom-extra-ROUTE
--
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/