Hey all, I''m testing out Julian''s patches, (2.4.16-6/Dec
14th), and
noticing some good, and weird results.
I followed the Nano-HOWTO, and can''t say that I understand all of it,
but I''m slowly trying to get a grasp. :)
Right now, my rules look like this:
[root@wolverine linux]# ip rule ls
0: from all lookup local
50: from all lookup 50
201: from 216.187.106.224 lookup 201
202: from 24.112.58.139 lookup 202
222: from all lookup 222
32766: from all lookup main
32767: from all lookup default
[root@wolverine linux]# ip route show table 50
216.187.106.128/25 dev eth2 proto kernel scope link src 216.187.106.224
192.62.100.0/24 dev eth0 proto kernel scope link src 192.62.100.1
24.112.58.0/23 dev eth1 proto kernel scope link src 24.112.58.139
[root@wolverine linux]# ip route show table 201
default via 216.187.106.252 dev eth2 proto static src 216.187.106.224
prohibit default proto static metric 1
[root@wolverine linux]# ip route show table 202
default via 24.112.58.139 dev eth1 proto static src 24.112.58.139
prohibit default proto static metric 1
[root@wolverine linux]# ip route show table 222
24.0.0.0/8 proto static
nexthop via 24.112.58.1 dev eth1 weight 2
nexthop via 216.187.106.252 dev eth2 weight 1
default proto static
nexthop via 216.187.106.252 dev eth2 weight 2
nexthop via 24.112.58.1 dev eth1 weight 1
If I unplug the DSL modem that''s connected to eth2, after about a
minute or two, all clients start to be routed through the cable modem
on eth1.
However, I then decided I wanted to try and add a multipath route to
24.0.0.0/8 that uses eth1 in preference to eth2.
I find though, that no matter what, even after flushing the route
cache, some requests for 24.x.x.x addresses are still sent via eth2
even though eth1 is up and alive.
Is this supported?
I''m not sure what other information might be helpful, let me know if
any.
--
Adrian Chung (adrian at enfusion-group dot com)
http://www.enfusion-group.com/~adrian
GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17
[rogue.enfusion-group.com] 1:13am up 133 days, 17:12, 2 users
On Fri, 14 Dec 2001 01:17:27 -0500 Adrian Chung <adrian@enfusion-group.com> wrote:> I followed the Nano-HOWTO, and can''t say that I understand all of it, > but I''m slowly trying to get a grasp. :)This probably means, that I didn''t understand that part neither. Maybe you want to tell me which parts you didn''t understand so I can try to add a comment.> I find though, that no matter what, even after flushing the route > cache, some requests for 24.x.x.x addresses are still sent via eth2 > even though eth1 is up and alive.Julian released yesterday v6 of his patch, which seems to address just this problem. I never noticed, because once the setup is done, I never change it. -- Christoph Simon ciccio@kiosknet.com.br --- ^X^C q quit :q ^C end x exit ZZ ^D ? help .
Hello, Adrian Chung wrote:> 50: from all lookup 50Adrian, check ip rule 50, the target should be main. Try all paths with ip route get ... Regards -- Julian Anastasov <ja@ssi.bg>
On Fri, Dec 14, 2001 at 01:45:43PM +0200, Julian Anastasov wrote:> > 50: from all lookup 50 > > Adrian, check ip rule 50, the target should be main. Try > all paths with ip route get ...Okay, my mistake... I modified rule 50 so that the target is now the ''main'' table which the kernel seems to populate. My rules look like this: [root@wolverine /]# ip rule ls 0: from all lookup local 50: from all lookup main 201: from 216.187.106.224 lookup 201 202: from 24.112.58.139 lookup 202 222: from all lookup 222 32766: from all lookup main 32767: from all lookup default And main is: [root@wolverine /]# ip route show table main 192.62.100.1 dev eth0 scope link 24.112.58.139 dev eth1 scope link 216.187.106.224 dev eth2 scope link 216.187.106.128/25 dev eth2 proto kernel scope link src 216.187.106.224 192.62.100.0/24 dev eth0 proto kernel scope link src 192.62.100.1 24.112.58.0/23 dev eth1 proto kernel scope link src 24.112.58.139 127.0.0.0/8 dev lo scope link The other rules are unchanged from my previous mail. The weird thing is, even with this rule: [root@wolverine /]# ip route show table 222 24.0.0.0/8 proto static nexthop via 24.112.58.1 dev eth1 weight 2 nexthop via 216.187.106.252 dev eth2 weight 1 default proto static nexthop via 216.187.106.252 dev eth2 weight 2 nexthop via 24.112.58.1 dev eth1 weight 1 I still get the following output: [root@wolverine /]# ip route get from 192.62.100.1 to 24.2.9.58 24.2.9.58 from 192.62.100.1 via 24.112.58.1 dev eth1 cache mtu 1500 advmss 1460 [root@wolverine /]# ip route get from 192.62.100.1 to 24.2.9.59 24.2.9.59 from 192.62.100.1 via 24.112.58.1 dev eth1 cache mtu 1500 advmss 1460 [root@wolverine /]# ip route get from 192.62.100.1 to 24.2.9.60 24.2.9.60 from 192.62.100.1 via 24.112.58.1 dev eth1 cache mtu 1500 advmss 1460 [root@wolverine /]# ip route get from 192.62.100.1 to 24.1.2.3 24.1.2.3 from 192.62.100.1 via 24.112.58.1 dev eth1 cache mtu 1500 advmss 1460 [root@wolverine /]# ip route get from 192.62.100.1 to 24.112.58.7 24.112.58.7 from 192.62.100.1 dev eth1 cache mtu 1500 advmss 1460 [root@wolverine /]# ip route get from 192.62.100.1 to 24.100.155.45 24.100.155.45 from 192.62.100.1 via 216.187.106.252 dev eth2 cache mtu 1500 advmss 1460 The last one is going out via eth2, instead of eth1, even though eth1 is still up? My IPtables rules are set up as: /sbin/iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE /sbin/iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE If that makes any difference? -- Adrian Chung (adrian at enfusion-group dot com) http://www.enfusion-group.com/~adrian GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17 [rogue.enfusion-group.com] 9:57am up 134 days, 1:56, 2 users
On Fri, Dec 14, 2001 at 09:09:22AM -0200, Christoph Simon wrote:> On Fri, 14 Dec 2001 01:17:27 -0500 > Adrian Chung <adrian@enfusion-group.com> wrote: > > > I followed the Nano-HOWTO, and can''t say that I understand all of it, > > but I''m slowly trying to get a grasp. :) > > This probably means, that I didn''t understand that part neither. Maybe > you want to tell me which parts you didn''t understand so I can try to > add a comment.I didn''t have trouble following the HOWTO, it was easy to follow. I''m just not sure I understand the reasoning for having to do certain steps, but that''s not necessarily the goal of your HOWTO. Should I post my confusions to the list? Or offlist to you personally?> > I find though, that no matter what, even after flushing the route > > cache, some requests for 24.x.x.x addresses are still sent via eth2 > > even though eth1 is up and alive. > > Julian released yesterday v6 of his patch, which seems to address just > this problem. I never noticed, because once the setup is done, I never > change it.Yeah, that''s what I''m using. I''m still not having any luck, but it might likely be my fault. -- Adrian Chung (adrian at enfusion-group dot com) http://www.enfusion-group.com/~adrian GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17 [rogue.enfusion-group.com] up 134 days, 2:00, 2 users
On Fri, Dec 14, 2001 at 10:00:34AM -0500, Adrian Chung wrote:> The weird thing is, even with this rule: > > [root@wolverine /]# ip route show table 222 > 24.0.0.0/8 proto static > nexthop via 24.112.58.1 dev eth1 weight 2 > nexthop via 216.187.106.252 dev eth2 weight 1 > default proto static > nexthop via 216.187.106.252 dev eth2 weight 2 > nexthop via 24.112.58.1 dev eth1 weight 1[...]> [root@wolverine /]# ip route get from 192.62.100.1 to 24.112.58.7 > 24.112.58.7 from 192.62.100.1 dev eth1 > cache mtu 1500 advmss 1460 > [root@wolverine /]# ip route get from 192.62.100.1 to 24.100.155.45 > 24.100.155.45 from 192.62.100.1 via 216.187.106.252 dev eth2 > cache mtu 1500 advmss 1460 > > The last one is going out via eth2, instead of eth1, even though eth1 > is still up?Actually, I should add that even regular routes that don''t go through 24.0.0.0/8 are affected, and don''t seem to be using the "weights" properly... This is from a linux client that is behind the gateway, showing some consecutive traceroutes. I removed the 24.0.0.0/8 rule from table 222 above, and ran ''ip route flush cache'': adrian@toad:[~] $ traceroute mail.e-smith.com traceroute to mail.e-smith.com (64.26.145.90), 30 hops max, 38 byte packets 1 wolverine (192.62.100.1) 0.499 ms 0.235 ms 0.211 ms 2 ott-fe0.istop.com (216.187.106.252) 24.974 ms 7.367 ms 7.573 ms 3 tor-fe1.istop.com (216.187.106.1) 23.876 ms 18.536 ms 20.356 ms This traversed eth2, which is good... adrian@toad:[~] $ traceroute www.shore.net traceroute to www.shore.net (207.244.124.202), 30 hops max, 38 byte packets 1 wolverine (192.62.100.1) 0.453 ms 0.233 ms 0.212 ms 2 10.21.172.1 (10.21.172.1) 2.808 ms 23.383 ms 6.237 ms 3 216.197.153.101 (216.197.153.101) 22.176 ms 2.000 ms 4.217 ms 4 bb2-pos7-0.rdc2.on.home.net (216.197.153.97) 10.625 ms 9.390 ms 7.087 ms This traversed eth1, which shouldn''t happen if weight is 2 for eth2, right? adrian@toad:[~] $ traceroute 24.2.9.58 traceroute to 24.2.9.58 (24.2.9.58), 30 hops max, 38 byte packets 1 wolverine (192.62.100.1) 0.494 ms 0.295 ms 0.220 ms 2 ott-fe0.istop.com (216.187.106.252) 26.908 ms 7.888 ms 11.428 ms 3 tor-fe1.istop.com (216.187.106.1) 32.210 ms 21.756 ms 20.785 ms This went out over eth2 again... adrian@toad:[~] $ traceroute 24.2.9.59 traceroute to 24.2.9.59 (24.2.9.59), 30 hops max, 38 byte packets 1 wolverine (192.62.100.1) 0.591 ms 0.277 ms 0.220 ms 2 10.21.172.1 (10.21.172.1) 26.389 ms 1.614 ms 3.975 ms 3 216.197.153.101 (216.197.153.101) 5.120 ms 6.755 ms 17.854 ms 4 bb2-pos7-0.rdc2.on.home.net (216.197.153.97) 10.264 ms 18.565 ms 16.043 ms 5 10.0.185.21 (10.0.185.21) 10.566 ms 19.751 ms !H This went out over the right interface (eth1). -- Adrian Chung (adrian at enfusion-group dot com) http://www.enfusion-group.com/~adrian GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17 [rogue.enfusion-group.com] up 134 days, 2:05, 2 users
On Fri, 14 Dec 2001 10:02:50 -0500 Adrian Chung <adrian@enfusion-group.com> wrote:> I didn''t have trouble following the HOWTO, it was easy to follow. I''m > just not sure I understand the reasoning for having to do certain > steps, but that''s not necessarily the goal of your HOWTO.Well, up to a certain point it is, as at least some understanding is necessary to apply this setup to different situations. For instance, in your last two messages you seem to have tried to test the load balancing with individual cases. As I wrote, this is not possible; you need a certain amount of traffic to see it. The reason is a rather complex interaction of the routing cache and the connection tracking. Give the thing full load, and you''ll see. Look at this: eth0: RX bytes:628147759 (599.0 Mb) TX bytes:143581725 (136.9 Mb) eth2: RX bytes:717176203 (683.9 Mb) TX bytes:149120155 (142.2 Mb) We had a power failure yesterday in the afternoon, so numbers are still small. I don''t use weights and the nominal capacity of both lines is the same. You see, balancing is not 100% fair, but it''s more or less close. For eth2 it certainly makes a difference to download 683 MB or 1282 MB. Also, I didn''t check the logfiles, and it''s possible that there was a failure in eth0 (or in eth2 or in both). So this doesn''t necessarily represent the balancing. But I''m used to numbers in about this ratio.> Should I post my confusions to the list? Or offlist to you personally?Probably it''s better to the list. I''m far from being an expert, and we all can learn something. Also, right now, I''m very busy and will not be able to modify that howto too soon.> Yeah, that''s what I''m using. I''m still not having any luck, but it > might likely be my fault.-- Christoph Simon ciccio@kiosknet.com.br --- ^X^C q quit :q ^C end x exit ZZ ^D ? help .
On Fri, Dec 14, 2001 at 01:30:27PM -0200, Christoph Simon wrote:> On Fri, 14 Dec 2001 10:02:50 -0500 > Adrian Chung <adrian@enfusion-group.com> wrote: > > > I didn''t have trouble following the HOWTO, it was easy to follow. I''m > > just not sure I understand the reasoning for having to do certain > > steps, but that''s not necessarily the goal of your HOWTO. > > Well, up to a certain point it is, as at least some understanding is > necessary to apply this setup to different situations. For instance, > in your last two messages you seem to have tried to test the load > balancing with individual cases. As I wrote, this is not possible; you > need a certain amount of traffic to see it. The reason is a ratherActually, after I wrote my last couple of messages, I was thinking about it more, and I think it''s a misunderstanding that I have. Your HOWTO is mainly geared towards load balancing, and failover when one of the links goes down. I''d like to do that, but I''d like to mainly use one link, and only fall back (fail over) to the second link when the first goes down. I tried to represent that using a greater weight on the first link, but as I think it says in your HOWTO, this merely results in the kernel adding more routes for the first link, and inevitably, I''m guessing some traffic will still be routed through the second link when the first is up. My two links are through different ISP''s, and I want to be able to use both ISP''s news, and proxy servers. In order to do this, I need to make sure that when accessing ISP1''s news server, I always connect via the same link. Load balancing defeats this, by sometimes routing to ISP1''s news server via ISP2, and then I get denied access because I don''t have a source IP on ISP1''s network.> Probably it''s better to the list. I''m far from being an expert, and we > all can learn something. Also, right now, I''m very busy and will not > be able to modify that howto too soon.A couple of things I''m unsure about: 1) is there a way to only use the second link if and when the first goes down, instead of load balancing via both links all the time? 2) is it possible to implement source-based policy routing but fall back to the second link when the preferred gateway is down? Does any of this make sense? -- Adrian Chung (adrian at enfusion-group dot com) http://www.enfusion-group.com/~adrian GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17 [rogue.enfusion-group.com] up 134 days, 2:44, 2 users
On Fri, 14 Dec 2001 10:52:28 -0500 Adrian Chung <adrian@enfusion-group.com> wrote:> Actually, after I wrote my last couple of messages, I was thinking > about it more, and I think it''s a misunderstanding that I have. Your > HOWTO is mainly geared towards load balancing, and failover when one > of the links goes down. > > I''d like to do that, but I''d like to mainly use one link, and only > fall back (fail over) to the second link when the first goes down.You are right, this is a quite different situation. It worked for me like that before installing Julian''s patches. I had no multipath route, but explicit routes to the gateways and never more than one default route. Then I had a daemon running which ping''ed the gateways and would delete the default route to put the other. But this is loosing the bandwidth of a line which is being payed, so I was unhappy with that solution. Unfortunately, I don''t know which is the best routing setup for your case. Maybe somebody else on this list can give you a hint.> My two links are through different ISP''s, and I want to be able to use > both ISP''s news, and proxy servers. In order to do this, I need to > make sure that when accessing ISP1''s news server, I always connect via > the same link. > > Load balancing defeats this, by sometimes routing to ISP1''s news > server via ISP2, and then I get denied access because I don''t have a > source IP on ISP1''s network.You should be able to add an explicit host route in table main. That is hit before the multipath route is queried. You do know the IP of that ISP''s news server, do you?> 1) is there a way to only use the second link if and when the first > goes down, instead of load balancing via both links all the time?As I said above, the contrary was the aim of my setup. I would imagine that the solution is not using a multipath route but just two default routes with different priority. If the first is working, it should be used always; if it goes down, the second would be used. And if the first comes up again, thanks to Julian''s patches, it should be used again. But note, that for this last step to work, I think you still need to ping/arping the gateways, such that there is a chance to detect that the failing link is up again.> 2) is it possible to implement source-based policy routing but fall > back to the second link when the preferred gateway is down?I think it is, using two default routes with different priority.> Does any of this make sense?Unless you have to pay for traffic on one link and not on the other, I''m not sure why you would want to leave a link unused. Beside that, I think it does make sense. -- Christoph Simon ciccio@kiosknet.com.br --- ^X^C q quit :q ^C end x exit ZZ ^D ? help .
On Fri, Dec 14, 2001 at 02:15:50PM -0200, Christoph Simon wrote:> On Fri, 14 Dec 2001 10:52:28 -0500 > Adrian Chung <adrian@enfusion-group.com> wrote:[...]> > My two links are through different ISP''s, and I want to be able to use > > both ISP''s news, and proxy servers. In order to do this, I need to > > make sure that when accessing ISP1''s news server, I always connect via > > the same link. > > > > Load balancing defeats this, by sometimes routing to ISP1''s news > > server via ISP2, and then I get denied access because I don''t have a > > source IP on ISP1''s network. > > You should be able to add an explicit host route in table main. That > is hit before the multipath route is queried. You do know the IP > of that ISP''s news server, do you?Yes, although at one point I had noticed that almost everything for 24.0.0.0/8 was faster via the second link, but everything else was faster through the first link. So I was trying to avoid having to add individual host entries in the main table. But if I add the entire 24.0.0.0/8 network into the main table, if the second link goes down, I can''t get to anything on that network at all. In any case, it works just fine adding a host route in the main table, and that''s probably the best solution, since there aren''t that many hosts that I connect to that require a certain source IP. This way as well, since ISP2''s servers require an IP from their pool, if my second link goes down, there''s no point in routing to that server from the first link, since their ACL''s deny me access anyways. It''s a good solution, just a little more legwork on my part enumerating all of the servers I need to talk to. Thanks!> > 1) is there a way to only use the second link if and when the first > > goes down, instead of load balancing via both links all the time? > > As I said above, the contrary was the aim of my setup. I would imagine > that the solution is not using a multipath route but just two default > routes with different priority. If the first is working, it should be > used always; if it goes down, the second would be used. And if the > first comes up again, thanks to Julian''s patches, it should be used > again. But note, that for this last step to work, I think you still > need to ping/arping the gateways, such that there is a chance to > detect that the failing link is up again.What do you mean by "two default routes with different priority"? Do you mean different ''ip rule prio'' priorities? Or something else? That sounds like it would be viable, I''m just not sure how I do that. :)> > Does any of this make sense? > > Unless you have to pay for traffic on one link and not on the other, > I''m not sure why you would want to leave a link unused. Beside that, I > think it does make sense.Actually, I''d rather not leave it unused, but before these patches, I couldn''t get multipath routing/load balancing to work at all. :) Now, I don''t mind load balancing across both, as long as specific ISP servers which need to see a certain source IP get reached via the right link, but from above, I now can do that. -- Adrian Chung (adrian at enfusion-group dot com) http://www.enfusion-group.com/~adrian GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17 [rogue.enfusion-group.com] up 134 days, 3:57, 3 users
On Fri, Dec 14, 2001 at 02:15:50PM -0200, Christoph Simon wrote:> On Fri, 14 Dec 2001 10:52:28 -0500 > Adrian Chung <adrian@enfusion-group.com> wrote: > > > Actually, after I wrote my last couple of messages, I was thinking > > about it more, and I think it''s a misunderstanding that I have. Your > > HOWTO is mainly geared towards load balancing, and failover when one > > of the links goes down. > > > > I''d like to do that, but I''d like to mainly use one link, and only > > fall back (fail over) to the second link when the first goes down.Since these are Julian''s patches (from what I can tell), he might be the best person to ask, but, how hard would it be to do: $ ip route add default table 100 nexthop via aaa.bbb.ccc.ddd dev eth2 weight 1 nexthop via bbb.ccc.ddd.eee dev eth1 weight 0 And have the first link used fulltime, and the second (eth1) link only used when the first is "dead". Is "weight 0" used for anything right now, or even possible? -- Adrian Chung (adrian at enfusion-group dot com) http://www.enfusion-group.com/~adrian GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17 [rogue.enfusion-group.com] up 134 days, 4:07, 4 users
Hello, On Fri, 14 Dec 2001, Adrian Chung wrote:> > > I''d like to do that, but I''d like to mainly use one link, and only > > > fall back (fail over) to the second link when the first goes down. > > Since these are Julian''s patches (from what I can tell), he might be > the best person to ask, but, how hard would it be to do: > > $ ip route add default table 100 nexthop via aaa.bbb.ccc.ddd dev eth2 > weight 1 nexthop via bbb.ccc.ddd.eee dev eth1 weight 0 > > And have the first link used fulltime, and the second (eth1) link only > used when the first is "dead".You can mix a list of unipath or multipath routes into alternative routes, see my docs. One of the alternative routes is used until it fails. If you need different algorithm you have to create script to handle it.> Is "weight 0" used for anything right now, or even possible?Weight 0 is not allowed, it will need changing too much code. If you can call ip route change, then you can call del+add and to apply the new routes.> Adrian Chung (adrian at enfusion-group dot com) > http://www.enfusion-group.com/~adrian > GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17 > [rogue.enfusion-group.com] up 134 days, 4:07, 4 usersRegards -- Julian Anastasov <ja@ssi.bg>
On Fri, Dec 14, 2001 at 07:33:25PM +0200, Julian Anastasov wrote:> On Fri, 14 Dec 2001, Adrian Chung wrote:[...]> > And have the first link used fulltime, and the second (eth1) link only > > used when the first is "dead". > > You can mix a list of unipath or multipath routes into > alternative routes, see my docs. One of the alternative routes is used > until it fails. If you need different algorithm you have to create script > to handle it.Ah! I obviously didn''t pay close enough attention to the documentation. I''ll give this a shot. So, should I be able to do something like this (I''ll test later): ip route add default via aaa.bbb.ccc.ddd dev eth1 table 100 ip route add default via bbb.ccc.ddd.eee dev eth2 table 100 And eth1 will be used until failure?> > Is "weight 0" used for anything right now, or even possible? > > Weight 0 is not allowed, it will need changing too much code. > If you can call ip route change, then you can call del+add and to > apply the new routes.Makes sense, and it''s not necessary with your alternative routes implementation. I''ll re-read the docs. :) Thanks for the help. BTW, any idea whether your patches are going to make it into 2.4 proper? -- Adrian Chung (adrian at enfusion-group dot com) http://www.enfusion-group.com/~adrian GPG Fingerprint: C620 C8EA 86BA 79CC 384C E7BE A10C 353B 919D 1A17 [rogue.enfusion-group.com] 3:16pm up 134 days, 7:15, 4 users
Hello, On Fri, 14 Dec 2001, Adrian Chung wrote:> > You can mix a list of unipath or multipath routes into > > alternative routes, see my docs. One of the alternative routes is used > > until it fails. If you need different algorithm you have to create script > > to handle it. > > Ah! I obviously didn''t pay close enough attention to the > documentation. I''ll give this a shot. > > So, should I be able to do something like this (I''ll test later): > > ip route add default via aaa.bbb.ccc.ddd dev eth1 table 100 > ip route add default via bbb.ccc.ddd.eee dev eth2 table 100 > > And eth1 will be used until failure?Yes, while aaa.bbb.ccc.ddd is alive> BTW, any idea whether your patches are going to make it into 2.4 > proper?May be no, may be parts of them, I don''t know.> Adrian Chung (adrian at enfusion-group dot com)Regards -- Julian Anastasov <ja@ssi.bg>