All -
I have what I think is a very simple problem but the solution has eluded me.
I have three Token Ring interfaces in one box and I want to load balance
outgoing traffic across the NICs. The system is a RH 6.2 base with 2.2.18
kernel and current RH errata. Based on the Adv-Routing HOWTO I have the
proper stuff configured into the kernel already.
I''m using DNS to do cheap and easy round robin style connections to the
box.
[scottrus@linux scottrus]$ host ftp3.linux.ibm.com
ftp3.linux.ibm.com has address 9.37.145.124
ftp3.linux.ibm.com has address 9.37.145.120
ftp3.linux.ibm.com has address 9.37.145.127
ip address show
1: lo: <LOOPBACK,UP> mtu 3924 qdisc noqueue
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 brd 127.255.255.255 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 100
link/ether 00:60:94:3f:15:63 brd ff:ff:ff:ff:ff:ff
inet 10.200.1.33/24 brd 10.200.1.255 scope global eth0
3: tr0: <BROADCAST,MULTICAST,UP> mtu 2000 qdisc pfifo_fast qlen 100
link/tr 00:20:35:ff:09:66 brd ff:ff:ff:ff:ff:ff
inet 9.37.145.124/20 brd 9.37.159.255 scope global tr0
4: tr1: <BROADCAST,MULTICAST,UP> mtu 2000 qdisc pfifo_fast qlen 100
link/tr 00:60:94:b2:19:dd brd ff:ff:ff:ff:ff:ff
inet 9.37.145.120/20 brd 9.37.159.255 scope global tr1
5: tr2: <BROADCAST,MULTICAST,UP> mtu 2000 qdisc pfifo_fast qlen 100
link/tr 00:60:94:b2:19:6d brd ff:ff:ff:ff:ff:ff
inet 9.37.145.127/20 brd 9.37.159.255 scope global tr2
All trX interfaces are on the same subnet and use the same default gateway
of 9.37.144.1. The eth0 interface is a private network. Traffic from the
public Token Ring network should never cross onto the private eth0 network
or visversa.
I''m looking for one of two possible solutions:
1) Incoming requests are responded to on the same interface they came in on.
For example an ftp connection coming in on tr1 has all TX / RX packets stick
to tr1.
2) All output packets are load balanced across the tr0, tr1, tr2
interfaces regardless of which tr interface they came in on. (This is my
preferred solution.)
What''s currently happening is requests come in on tr0, tr1, tr2 but
outgoing
traffic is being sent back only via tr0, the default route.
[scottrus@linux scottrus]$ ip route show
10.200.1.33 dev eth0 scope link
9.37.145.120 dev tr1 scope link
9.37.145.127 dev tr2 scope link
9.37.145.124 dev tr0 scope link
10.200.1.0/24 dev eth0 proto kernel scope link src 10.200.1.33
9.37.144.0/20 dev tr0 proto kernel scope link src 9.37.145.124
9.37.144.0/20 dev tr1 proto kernel scope link src 9.37.145.120
9.37.144.0/20 dev tr2 proto kernel scope link src 9.37.145.127
10.200.0.0/16 via 10.200.1.1 dev eth0
127.0.0.0/8 dev lo scope link
default via 9.37.144.1 dev tr2
default via 9.37.144.1 dev tr1
default via 9.37.144.1 dev tr0
As you can see I''ve tried simply adding default routes for each token
ring
interface and I think this is a step in the right direction but I''m
missing
something.
*sigh* I thought I knew it all, how wrong I was. I have to say all this is
rather cool and beyond this initial problem I know I''m going to have
play
with it more just for fun. The very thought of being able to twist these
packets into my own demented idea of how they should be handled is just to
tempting to ignore. Baby steps, must remember to take baby steps.
1) Understand what the heck I''ve been reading all day.
2) Get the routes on ftp3 working so users stop yelling at me :)
3) Read more, play with iproute2, watch mailing list, help out. :) :)
Any help would be great. Examples would be better of course. Thanks much!
--
Regards,
Scott Russell (scottrus@raleigh.ibm.com)
Linux Technology Center, System Admin, RHCE.
On Fri, Mar 16, 2001 at 09:06:18PM -0500, Scott Russell wrote:> I''m using DNS to do cheap and easy round robin style connections to the box. > > [scottrus@linux scottrus]$ host ftp3.linux.ibm.com > ftp3.linux.ibm.com has address 9.37.145.124 > ftp3.linux.ibm.com has address 9.37.145.120 > ftp3.linux.ibm.com has address 9.37.145.127DNS is wildly underestimated as a loadbalancing/distribution device. It works very well, especially if you set your TTL pretty low.> 1) Incoming requests are responded to on the same interface they came in on. > For example an ftp connection coming in on tr1 has all TX / RX packets stick > to tr1.This is possible, I think.> As you can see I''ve tried simply adding default routes for each token ring > interface and I think this is a step in the right direction but I''m missing > something.Add source routing, as described in http://ds9a.nl/2.4Routing/HOWTO//cvs/2.4routing/output/2.4routing-4.html#ss4.1 Route packets with a source of 9.37.145.124 to trX, .120 to trY and 127 to trZ.> Any help would be great. Examples would be better of course. Thanks much!# ip rule add from 9.37.145.124 table 124 # ip rule add from 9.37.145.120 table 120 # ip rule add from 9.37.145.127 table 127 # ip route add default via 9.37.145.1 dev tr0 table 124 # ip route add default via 9.37.145.1 dev tr1 table 120 # ip route add default via 9.37.145.1 dev tr2 table 127 # ip route flush cache Something like that should work. I messed up the ip addresses I think, but hey :-) Regards, bert -- http://www.PowerDNS.com Versatile DNS Services Trilab The Technology People ''SYN! .. SYN|ACK! .. ACK!'' - the mating call of the internet
Thanks for the quick reply. Based on reading the HOWTO examples I thought this was what I wanted to do and did some quick tests similar to what you outlined below. My tests failed so I assumed I was missing the obvious and posted to the list. Well, I followed the examples below and ran the following commands in order: ip rule add from 9.37.145.124 table 124 ip rule add from 9.37.145.120 table 120 ip rule add from 9.37.145.127 table 127 ip route add default via 9.37.144.1 dev tr0 table 124 ip route add default via 9.37.144.1 dev tr1 table 120 ip route add default via 9.37.144.1 dev tr2 table 127 ip route flush cache A bit of back ground: tr0 = 9.37.145.124/20 tr1 = 9.37.145.120/20 tr2 = 9.37.145.127/20 default GW = 9.37.144.1 So based on the above rules I would expect connections that originate on tr2 to stay on tr2. To test this I ftped to the system using the IP of tr2 and started receiving a file. I then shut down interface tr0 with /sbin/ifdown tr0. When I did so my file xfer stopped as well. When tr0 was brought back up the xfer resumed. Bringing down tr2 and tr1 had no effect on the transfer at all. (Expected since most likely the ftpd opened the ftp-data connection on tr0 instead of tr2 as I wanted.) Here''s what my rules look like after running the above commands: [scottrus@linux scottrus]$ ip rule show 0: from all lookup local 32763: from 9.37.145.127 lookup 127 32764: from 9.37.145.120 lookup 120 32765: from 9.37.145.124 lookup 124 32766: from all lookup main 32767: from all lookup 253 Since the rules 23763-65 all appear before the main table I would expect them to take precedence over anything in the main table. Is this correct? So here''s what my main routing table looks like after running the above rules: [scottrus@linux scottrus]$ ip route 10.200.1.33 dev eth0 scope link 9.37.145.120 dev tr1 scope link 9.37.145.127 dev tr2 scope link 9.37.145.124 dev tr0 scope link 10.200.1.0/24 dev eth0 proto kernel scope link src 10.200.1.33 9.37.144.0/20 dev tr0 proto kernel scope link src 9.37.145.124 9.37.144.0/20 dev tr1 proto kernel scope link src 9.37.145.120 9.37.144.0/20 dev tr2 proto kernel scope link src 9.37.145.127 10.200.0.0/16 via 10.200.1.1 dev eth0 127.0.0.0/8 dev lo scope link default via 9.37.144.1 dev tr2 default via 9.37.144.1 dev tr1 default via 9.37.144.1 dev tr0 I expect that the rules I inserted are being ignored (and hence the tables the rules point to are ignored as well). This means that the primary routing table is being used and seems to explain why things are going out tr0 by default. Am I off base on what I''m seeing going on? I think I also have ECMP (Equal Cost MultiPath) compiled into my kernel, is that affecting things as well? Again, thanks for the help. -- Scott On Sat, Mar 17, 2001 at 12:45:43PM +0100, bert hubert wrote:> On Fri, Mar 16, 2001 at 09:06:18PM -0500, Scott Russell wrote: > > > I''m using DNS to do cheap and easy round robin style connections to the box. > > > > [scottrus@linux scottrus]$ host ftp3.linux.ibm.com > > ftp3.linux.ibm.com has address 9.37.145.124 > > ftp3.linux.ibm.com has address 9.37.145.120 > > ftp3.linux.ibm.com has address 9.37.145.127 > > DNS is wildly underestimated as a loadbalancing/distribution device. It > works very well, especially if you set your TTL pretty low. > > > 1) Incoming requests are responded to on the same interface they came in on. > > For example an ftp connection coming in on tr1 has all TX / RX packets stick > > to tr1. > > This is possible, I think. > > > As you can see I''ve tried simply adding default routes for each token ring > > interface and I think this is a step in the right direction but I''m missing > > something. > > Add source routing, as described in > http://ds9a.nl/2.4Routing/HOWTO//cvs/2.4routing/output/2.4routing-4.html#ss4.1 > > Route packets with a source of 9.37.145.124 to trX, .120 to trY and 127 to > trZ. > > > Any help would be great. Examples would be better of course. Thanks much! > > # ip rule add from 9.37.145.124 table 124 > # ip rule add from 9.37.145.120 table 120 > # ip rule add from 9.37.145.127 table 127 > > # ip route add default via 9.37.145.1 dev tr0 table 124 > # ip route add default via 9.37.145.1 dev tr1 table 120 > # ip route add default via 9.37.145.1 dev tr2 table 127 > # ip route flush cache > > Something like that should work. I messed up the ip addresses I think, but > hey :-) > > Regards, > > bert > > -- > http://www.PowerDNS.com Versatile DNS Services > Trilab The Technology People > ''SYN! .. SYN|ACK! .. ACK!'' - the mating call of the internet > > _______________________________________________ > LARTC mailing list / LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://ds9a.nl/2.4Routing/-- Regards, Scott Russell (scottrus@raleigh.ibm.com) Linux Technology Center, System Admin, RHCE. T/L 441-9289 / External 919-543-9289 http://bzimage.raleigh.ibm.com/webcam
On Sun, 18 Mar 2001, Scott Russell wrote:> default via 9.37.144.1 dev tr2 > default via 9.37.144.1 dev tr1 > default via 9.37.144.1 dev tr0Interesting. This doesn''t seem to be a multipath route. Did you add the multipath route with the nexthop commands? Doei, Arthur. -- /\ / | arthurvl@sci.kun.nl | Work like you don''t need the money /__\ / | A friend is someone with whom | Love like you have never been hurt / \/__ | you can dare to be yourself | Dance like there''s nobody watching
Umm, no they''re not :) These statements are inserted as straight default routes into the main routing table. What would a command for multipath route look like? If it is what I think it is last time I tried to add one I received an error in the process. If I add a multipath route I assume i should add it to the main table and that I should remove the three default routes shown in your last email? -- Scott On Mon, Mar 19, 2001 at 01:40:52PM +0100, Arthur van Leeuwen wrote:> On Sun, 18 Mar 2001, Scott Russell wrote: > > > default via 9.37.144.1 dev tr2 > > default via 9.37.144.1 dev tr1 > > default via 9.37.144.1 dev tr0 > > Interesting. This doesn''t seem to be a multipath route. Did you add the > multipath route with the nexthop commands? > > Doei, Arthur. > > -- > /\ / | arthurvl@sci.kun.nl | Work like you don''t need the money > /__\ / | A friend is someone with whom | Love like you have never been hurt > / \/__ | you can dare to be yourself | Dance like there''s nobody watching-- Regards, Scott Russell (scottrus@raleigh.ibm.com) Linux Technology Center, System Admin, RHCE. T/L 441-9289 / External 919-543-9289 http://bzimage.raleigh.ibm.com/webcam
On Mon, 19 Mar 2001, Scott Russell wrote:> Umm, no they''re not :) These statements are inserted as straight default > routes into the main routing table. What would a command for multipath route > look like? If it is what I think it is last time I tried to add one I > received an error in the process.ip route add default scope global nexthop via 9.37.144.1 dev tr2 \ nexthop via 9.37.144.1 dev tr1 \ nexthop via 9.37.144.1 dev tr0> If I add a multipath route I assume i should add it to the main table and > that I should remove the three default routes shown in your last email?Yes. Doei, Arthur. -- /\ / | arthurvl@sci.kun.nl | Work like you don''t need the money /__\ / | A friend is someone with whom | Love like you have never been hurt / \/__ | you can dare to be yourself | Dance like there''s nobody watching