Hi, http://lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-links.html I am not sure who wrote this part or what it was based upon, but since I am working a lot longer now with ip rules, I think I want to add some stuff: The example 4.2.1 refers to the picture above, and does a plain ip rule add from .... table .... The problem with the exampe is that if you connect from the inside (local network) to your if1 ip or if2 ip, that in this example the replies to the local-network are going out if1 or if2... That is not what you want. If we carefully study the ip rule set, we see the first number. This number is the priority, and if you use this priority number in your rule command, it will insert it in the rule list after the rule with the same or lower priority. I am using this to differentiate between known routing, and default routing. So first we set up the link local routing (the things you can reach directly). Actually you don''t have to do a thing for that, except setting up the interface. List your local routing to understand: ip route show table main Then we put the default routing into table default. And between the main and the default rule we put rules that differentiate the default routing per provider. example: ############################################################ # set up table main by upping the interfaces ifup eth0 # local net ifup eth1 # provider 1 ifup eth2 # provider 2 # Now set up the default routes in a failover fashion, with the # most important route first(f.i. gw-provider1): ip route add default via gw-provider1 table default # Secondary route when the first gw fails: ip route append default via gw-provider2 table default # (The append is needed because else the routes will clash) # So we have a table with two default routes wich will failover # for eachother (takes about 10 minutes in default config) # We now have a simple failover, which for most of us will not # work, since most providers will have src-ip filtering. # We are going to fix that now: ip route add default via gw-provider1 table rt-provider1 ip route add default via gw-provider2 table rt-provider2 # We now have 2 tables each with a single different default gw. # They are not used, that is what we are going to solve now: ip rule add from ip-eth1 table rt-provider1 prio 32766 ip rule add from ip-eth2 table rt-provider2 prio 32766 # That''s it. ############################################################ So let''s think about it, and look at it: ard@erwin(slave):~$ /sbin/ip rule list 0: from all lookup local 32766: from all lookup main 32766: from <someip> lookup <sometable> 32767: from all lookup default This is not what we did above, but it is a rule list from a working environment (does www.telegraaf.nl ring a bell?). What we should have seen was this: /sbin/ip rule list 0: from all lookup local 32766: from all lookup main 32766: from <ip-eth1> lookup rt-provider1 32766: from <ip-eth2> lookup rt-provider2 32767: from all lookup default The main difference with the example in the document is: - We do *not* have a default route in main - We *have* default routes in the default table - We have rules *after* main, not before main So what is the catch: The only catch is that if you do not have point-to-point connections with your provider, but a /24 for example, then requests coming in from provider2 for the ip-eth1, will go out from your eth2 and not from your eth1. This might be a problem if your /24 is filtered by your ISP. The solution to that is the essence of this story: move the calling of your default route tables from the rules to the last possible moment. So to fix the catch you get two more routing tables: (With a provider3 added for clarity) ############################################################ ip route add net-provider2/24 via gw-provider1 table use-gw-provider1 ip route add net-provider3/24 via gw-provider1 table use-gw-provider1 ip rule add from ip-eth1 table use-gw-provider1 prio 32765 ip route add net-provider1/24 via gw-provider2 table use-gw-provider2 ip route add net-provider3/24 via gw-provider2 table use-gw-provider2 ip rule add from ip-eth2 table use-gw-provider2 prio 32765 ip route add net-provider1/24 via gw-provider3 table use-gw-provider3 ip route add net-provider2/24 via gw-provider3 table use-gw-provider3 ip rule add from ip-eth3 table use-gw-provider3 prio 32765 ############################################################ I hope this makes some sense, and I hope it also is clear that this is only needed for the link-local network of your provider only if it is filtered! Next thing: I was talking about failover earlier: If a gateway is not available (ie, it does not reply to arps), linux will think it is dead within a few minutes, and use the other gateway. But only if it reaches the default table. It reaches that, when it does not have a clue of the outgoing src-ip yet. So if an application makes a connection to a website, and the first gateway is considered dead, it will connect to the website using the second gateway, and therefore bind to ip-eth2. Last thing: this failover thingie can also be a "loadbalanced" thingie as explained in "4.2.2 Load balancing". However: due to bugs in the equalizeing code, I recommend against it. Somewhere inside the kernel it cannot clearly come up with a route, which results in a lot of "cannot happen 777". Next to that: the usage counts of the devices are not correctly incremented and decremented. You have to be very careful and craft an extra non-multipath route before, then remove the existing multi-path route, before bringing down a network device. Else it ends up in an endless "device still in use, waiting". And you will not be able to use the device anymore until you reset some sense into the machine... -- begin ILOVEYOU.VBS 666 <ard@telegraafnet.nl> Telegraaf Elektronische Media Real geeks don''t get viruses end
Arthur van Leeuwen
2002-Jul-05 18:13 UTC
Re: "Bug" in howto 4.2.1 Split access and other advice
On Fri, 5 Jul 2002, Ard van Breemen wrote:> Hi, > http://lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-links.html > I am not sure who wrote this part or what it was based upon, but > since I am working a lot longer now with ip rules, I think I want > to add some stuff:The stuff that is in the HOWTO was designed and tested back in 1999. Oh, and I am the author. :)> The example 4.2.1 refers to the picture above, and does a plain > ip rule add from .... table .... > The problem with the exampe is that if you connect from the > inside (local network) to your if1 ip or if2 ip, that in this > example the replies to the local-network are going out if1 or > if2... That is not what you want.True. That is indeed a bug. Never saw it in actual practice though: you *should*not* connect to the external IP addresses of your router from the internal network... for various security reasons and such. But you are right. [snip] Whoa, that was large. I''m not sure I entirely follow you though. The *point* of the extra routing tables is that they take precedence over the default routing tables... Doei, Arthur. -- /\ / | arthurvl@sci.kun.nl | Work like you don''t need the money /__\ / | A friend is someone with whom | Love like you have never been hurt / \/__ | you can dare to be yourself | Dance like there''s nobody watching
Laurens van Alphen
2002-Jul-05 18:25 UTC
RE: "Bug" in howto 4.2.1 Split access and other advice
Hi, What is still unclear to me is when the http://www.linuxvirtualserver.org/~julian/#routes patches are needed. What do they do exactly? Thanks in advance, -- Laurens van Alphen -----Original Message----- From: Ard van Breemen [mailto:ard@telegraafnet.nl] Sent: vrijdag 5 juli 2002 19:32 To: lartc@mailman.ds9a.nl Cc: HOWTO@ds9a.nl Subject: [LARTC] "Bug" in howto 4.2.1 Split access and other advice Hi, http://lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-links.ht ml I am not sure who wrote this part or what it was based upon, but since I am working a lot longer now with ip rules, I think I want to add some stuff: The example 4.2.1 refers to the picture above, and does a plain ip rule add from .... table .... The problem with the exampe is that if you connect from the inside (local network) to your if1 ip or if2 ip, that in this example the replies to the local-network are going out if1 or if2... That is not what you want. If we carefully study the ip rule set, we see the first number. This number is the priority, and if you use this priority number in your rule command, it will insert it in the rule list after the rule with the same or lower priority. I am using this to differentiate between known routing, and default routing. So first we set up the link local routing (the things you can reach directly). Actually you don''t have to do a thing for that, except setting up the interface. List your local routing to understand: ip route show table main Then we put the default routing into table default. And between the main and the default rule we put rules that differentiate the default routing per provider. example: ############################################################ # set up table main by upping the interfaces ifup eth0 # local net ifup eth1 # provider 1 ifup eth2 # provider 2 # Now set up the default routes in a failover fashion, with the # most important route first(f.i. gw-provider1): ip route add default via gw-provider1 table default # Secondary route when the first gw fails: ip route append default via gw-provider2 table default # (The append is needed because else the routes will clash) # So we have a table with two default routes wich will failover # for eachother (takes about 10 minutes in default config) # We now have a simple failover, which for most of us will not # work, since most providers will have src-ip filtering. # We are going to fix that now: ip route add default via gw-provider1 table rt-provider1 ip route add default via gw-provider2 table rt-provider2 # We now have 2 tables each with a single different default gw. # They are not used, that is what we are going to solve now: ip rule add from ip-eth1 table rt-provider1 prio 32766 ip rule add from ip-eth2 table rt-provider2 prio 32766 # That''s it. ############################################################ So let''s think about it, and look at it: ard@erwin(slave):~$ /sbin/ip rule list 0: from all lookup local 32766: from all lookup main 32766: from <someip> lookup <sometable> 32767: from all lookup default This is not what we did above, but it is a rule list from a working environment (does www.telegraaf.nl ring a bell?). What we should have seen was this: /sbin/ip rule list 0: from all lookup local 32766: from all lookup main 32766: from <ip-eth1> lookup rt-provider1 32766: from <ip-eth2> lookup rt-provider2 32767: from all lookup default The main difference with the example in the document is: - We do *not* have a default route in main - We *have* default routes in the default table - We have rules *after* main, not before main So what is the catch: The only catch is that if you do not have point-to-point connections with your provider, but a /24 for example, then requests coming in from provider2 for the ip-eth1, will go out from your eth2 and not from your eth1. This might be a problem if your /24 is filtered by your ISP. The solution to that is the essence of this story: move the calling of your default route tables from the rules to the last possible moment. So to fix the catch you get two more routing tables: (With a provider3 added for clarity) ############################################################ ip route add net-provider2/24 via gw-provider1 table use-gw-provider1 ip route add net-provider3/24 via gw-provider1 table use-gw-provider1 ip rule add from ip-eth1 table use-gw-provider1 prio 32765 ip route add net-provider1/24 via gw-provider2 table use-gw-provider2 ip route add net-provider3/24 via gw-provider2 table use-gw-provider2 ip rule add from ip-eth2 table use-gw-provider2 prio 32765 ip route add net-provider1/24 via gw-provider3 table use-gw-provider3 ip route add net-provider2/24 via gw-provider3 table use-gw-provider3 ip rule add from ip-eth3 table use-gw-provider3 prio 32765 ############################################################ I hope this makes some sense, and I hope it also is clear that this is only needed for the link-local network of your provider only if it is filtered! Next thing: I was talking about failover earlier: If a gateway is not available (ie, it does not reply to arps), linux will think it is dead within a few minutes, and use the other gateway. But only if it reaches the default table. It reaches that, when it does not have a clue of the outgoing src-ip yet. So if an application makes a connection to a website, and the first gateway is considered dead, it will connect to the website using the second gateway, and therefore bind to ip-eth2. Last thing: this failover thingie can also be a "loadbalanced" thingie as explained in "4.2.2 Load balancing". However: due to bugs in the equalizeing code, I recommend against it. Somewhere inside the kernel it cannot clearly come up with a route, which results in a lot of "cannot happen 777". Next to that: the usage counts of the devices are not correctly incremented and decremented. You have to be very careful and craft an extra non-multipath route before, then remove the existing multi-path route, before bringing down a network device. Else it ends up in an endless "device still in use, waiting". And you will not be able to use the device anymore until you reset some sense into the machine... --
Hi, I was rereading the pages in the howto about multiple ISP''s and I also found some strange stuff in it. Maybe you can create a patch for Bert to update the howto ? Stef PS. Het wordt hier echt wel een lage-landen onderonsje :) English : lot of low-country (Belgium, The Netherlands) people on the list. On Friday 05 July 2002 19:31, Ard van Breemen wrote:> Hi, > http://lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-links.html > I am not sure who wrote this part or what it was based upon, but > since I am working a lot longer now with ip rules, I think I want > to add some stuff: > The example 4.2.1 refers to the picture above, and does a plain > ip rule add from .... table .... > The problem with the exampe is that if you connect from the > inside (local network) to your if1 ip or if2 ip, that in this > example the replies to the local-network are going out if1 or > if2... That is not what you want. > > If we carefully study the ip rule set, we see the first number. > This number is the priority, and if you use this priority number > in your rule command, it will insert it in the rule list after the > rule with the same or lower priority. > > I am using this to differentiate between known routing, and > default routing. > > So first we set up the link local routing (the things you can > reach directly). Actually you don''t have to do a thing for that, > except setting up the interface. > List your local routing to understand: > ip route show table main > > Then we put the default routing into table default. > And between the main and the default rule we put rules that > differentiate the default routing per provider. > > example: > ############################################################ > # set up table main by upping the interfaces > ifup eth0 # local net > ifup eth1 # provider 1 > ifup eth2 # provider 2 > > # Now set up the default routes in a failover fashion, with the > # most important route first(f.i. gw-provider1): > ip route add default via gw-provider1 table default > # Secondary route when the first gw fails: > ip route append default via gw-provider2 table default > # (The append is needed because else the routes will clash) > # So we have a table with two default routes wich will failover > # for eachother (takes about 10 minutes in default config) > > # We now have a simple failover, which for most of us will not > # work, since most providers will have src-ip filtering. > # We are going to fix that now: > ip route add default via gw-provider1 table rt-provider1 > ip route add default via gw-provider2 table rt-provider2 > # We now have 2 tables each with a single different default gw. > # They are not used, that is what we are going to solve now: > > ip rule add from ip-eth1 table rt-provider1 prio 32766 > ip rule add from ip-eth2 table rt-provider2 prio 32766 > > > # That''s it. > ############################################################ > > So let''s think about it, and look at it: > ard@erwin(slave):~$ /sbin/ip rule list > 0: from all lookup local > 32766: from all lookup main > 32766: from <someip> lookup <sometable> > 32767: from all lookup default > > This is not what we did above, but it is a rule list from a > working environment (does www.telegraaf.nl ring a bell?). > > What we should have seen was this: > /sbin/ip rule list > 0: from all lookup local > 32766: from all lookup main > 32766: from <ip-eth1> lookup rt-provider1 > 32766: from <ip-eth2> lookup rt-provider2 > 32767: from all lookup default > > The main difference with the example in the document is: > - We do *not* have a default route in main > - We *have* default routes in the default table > - We have rules *after* main, not before main > > So what is the catch: > The only catch is that if you do not have point-to-point > connections with your provider, but a /24 for example, then > requests coming in from provider2 for the ip-eth1, will go out > from your eth2 and not from your eth1. This might be a problem if > your /24 is filtered by your ISP. > The solution to that is the essence of this story: move the > calling of your default route tables from the rules to the last > possible moment. > So to fix the catch you get two more routing tables: > (With a provider3 added for clarity) > ############################################################ > ip route add net-provider2/24 via gw-provider1 table use-gw-provider1 > ip route add net-provider3/24 via gw-provider1 table use-gw-provider1 > ip rule add from ip-eth1 table use-gw-provider1 prio 32765 > > ip route add net-provider1/24 via gw-provider2 table use-gw-provider2 > ip route add net-provider3/24 via gw-provider2 table use-gw-provider2 > ip rule add from ip-eth2 table use-gw-provider2 prio 32765 > > ip route add net-provider1/24 via gw-provider3 table use-gw-provider3 > ip route add net-provider2/24 via gw-provider3 table use-gw-provider3 > ip rule add from ip-eth3 table use-gw-provider3 prio 32765 > ############################################################ > > I hope this makes some sense, and I hope it also is clear that > this is only needed for the link-local network of your provider > only if it is filtered! > > > Next thing: I was talking about failover earlier: > If a gateway is not available (ie, it does not reply to arps), > linux will think it is dead within a few minutes, and use the > other gateway. But only if it reaches the default table. > It reaches that, when it does not have a clue of the outgoing > src-ip yet. So if an application makes a connection to a website, > and the first gateway is considered dead, it will connect to the > website using the second gateway, and therefore bind to ip-eth2. > > > Last thing: this failover thingie can also be a "loadbalanced" > thingie as explained in "4.2.2 Load balancing". > However: due to bugs in the equalizeing code, I recommend against > it. Somewhere inside the kernel it cannot clearly come up with a > route, which results in a lot of "cannot happen 777". > Next to that: the usage counts of the devices are not correctly > incremented and decremented. You have to be very careful and craft > an extra non-multipath route before, then remove the existing > multi-path route, before bringing down a network device. Else it > ends up in an endless "device still in use, waiting". And you > will not be able to use the device anymore until you reset some > sense into the machine...-- stef.coene@docum.org "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.openprojects.net
Arthur van Leeuwen
2002-Jul-05 18:45 UTC
RE: "Bug" in howto 4.2.1 Split access and other advice
On Fri, 5 Jul 2002, Laurens van Alphen wrote:> What is still unclear to me is when the > http://www.linuxvirtualserver.org/~julian/#routes patches are needed. > What do they do exactly?They add actual dead gateway detection (so that failover in fact does fail over... ;)), alternative routes (so that if multiple routes are defined only the active ones will actually be used), static routes (so that routes don''t get deleted by the kernel if the correspondign interface goes down... thus automatically reusing it when it comes back up), and MASQUERADING support for multiple gateways. Doei, Arthur. -- /\ / | arthurvl@sci.kun.nl | Work like you don''t need the money /__\ / | A friend is someone with whom | Love like you have never been hurt / \/__ | you can dare to be yourself | Dance like there''s nobody watching
Arthur van Leeuwen
2002-Jul-05 18:47 UTC
Re: "Bug" in howto 4.2.1 Split access and other advice
On Fri, 5 Jul 2002, Stef Coene wrote:> Hi, > > I was rereading the pages in the howto about multiple ISP''s and I also found > some strange stuff in it. Maybe you can create a patch for Bert to update > the howto ?Could you be more precies? :) Doei, Arthur. (Yeah, the type is intentional :)) -- /\ / | arthurvl@sci.kun.nl | Work like you don''t need the money /__\ / | A friend is someone with whom | Love like you have never been hurt / \/__ | you can dare to be yourself | Dance like there''s nobody watching
On Friday 05 July 2002 20:47, Arthur van Leeuwen wrote:> On Fri, 5 Jul 2002, Stef Coene wrote: > > Hi, > > > > I was rereading the pages in the howto about multiple ISP''s and I also > > found some strange stuff in it. Maybe you can create a patch for Bert to > > update the howto ? > > Could you be more precies? :)export CVSROOT=:pserver:anon@outpost.ds9a.nl:/var/cvsroot cvs login CVS password: [enter ''cvs'' (without ''s)] cvs co 2.4routing password = cvs vi 2.4routing/2.4routing.sgml :wq cvs diff -u > patch uuencode patch patch | mail -s "HOWTO update" HOWTO@ds9a.nl :) Stef
Arthur van Leeuwen
2002-Jul-05 19:05 UTC
Re: "Bug" in howto 4.2.1 Split access and other advice
On Fri, 5 Jul 2002, Stef Coene wrote:> On Friday 05 July 2002 20:47, Arthur van Leeuwen wrote: > > On Fri, 5 Jul 2002, Stef Coene wrote: > > > Hi, > > > > > > I was rereading the pages in the howto about multiple ISP''s and I also > > > found some strange stuff in it. Maybe you can create a patch for Bert to > > > update the howto ? > > > > Could you be more precies? :)> export CVSROOT=:pserver:anon@outpost.ds9a.nl:/var/cvsroot > cvs login > CVS password: [enter ''cvs'' (without ''s)] > cvs co 2.4routing > password = cvs > vi 2.4routing/2.4routing.sgml > :wq > cvs diff -u > patch > uuencode patch patch | mail -s "HOWTO update" HOWTO@ds9a.nlOn what stuff you found strange, I meant... not on the process of sending in a patch for the HOWTO... Doei, Arthur. (Jeez, nerds... :)) -- /\ / | arthurvl@sci.kun.nl | Work like you don''t need the money /__\ / | A friend is someone with whom | Love like you have never been hurt / \/__ | you can dare to be yourself | Dance like there''s nobody watching
On Friday 05 July 2002 21:05, Arthur van Leeuwen wrote:> On Fri, 5 Jul 2002, Stef Coene wrote: > > On Friday 05 July 2002 20:47, Arthur van Leeuwen wrote: > > > On Fri, 5 Jul 2002, Stef Coene wrote: > > > > Hi, > > > > > > > > I was rereading the pages in the howto about multiple ISP''s and I > > > > also found some strange stuff in it. Maybe you can create a patch > > > > for Bert to update the howto ? > > > > > > Could you be more precies? :) > > > > export CVSROOT=:pserver:anon@outpost.ds9a.nl:/var/cvsroot > > cvs login > > CVS password: [enter ''cvs'' (without ''s)] > > cvs co 2.4routing > > password = cvs > > vi 2.4routing/2.4routing.sgml > > > > :wq > > > > cvs diff -u > patch > > uuencode patch patch | mail -s "HOWTO update" HOWTO@ds9a.nl > > On what stuff you found strange, I meant... not on the process of sending > in a patch for the HOWTO...quoted from my post above :> > > > I was rereading the pages in the howto about multiple ISP''s and Iquoted from the subject Re: [LARTC] "Bug" in howto 4.2.1 Split access and other advice So I was talking about chapter 4.2 : http://www.lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-links.html> Doei, Arthur. (Jeez, nerds... :))A real nerd takes a plane to Canada to talk about traffic shaping :) Yesterday a colleague had a problem. He had a redhat based firewall and installed NT4 on it. After first reboot, lilo was still booting a linux kernel even after fdisk''ing the disk. He couldn''t remove the tux-virus :) Stef --
Arthur van Leeuwen
2002-Jul-05 19:39 UTC
Re: "Bug" in howto 4.2.1 Split access and other advice
On Fri, 5 Jul 2002, Stef Coene wrote:> On Friday 05 July 2002 21:05, Arthur van Leeuwen wrote: > > On Fri, 5 Jul 2002, Stef Coene wrote: > > > On Friday 05 July 2002 20:47, Arthur van Leeuwen wrote: > > > > On Fri, 5 Jul 2002, Stef Coene wrote: > > > > > Hi, > > > > > > > > > > I was rereading the pages in the howto about multiple ISP''s and I > > > > > also found some strange stuff in it. Maybe you can create a patch > > > > > for Bert to update the howto ? > > > > > > > > Could you be more precies? :) > > > > > > export CVSROOT=:pserver:anon@outpost.ds9a.nl:/var/cvsroot > > > cvs login > > > CVS password: [enter ''cvs'' (without ''s)] > > > cvs co 2.4routing > > > password = cvs > > > vi 2.4routing/2.4routing.sgml > > > > > > :wq > > > > > > cvs diff -u > patch > > > uuencode patch patch | mail -s "HOWTO update" HOWTO@ds9a.nl> > On what stuff you found strange, I meant... not on the process of sending > > in a patch for the HOWTO... > quoted from my post above : > > > > > I was rereading the pages in the howto about multiple ISP''s and I > quoted from the subject > Re: [LARTC] "Bug" in howto 4.2.1 Split access and other advice> So I was talking about chapter 4.2 : > http://www.lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-links.htmlYeah, that was quite clear. But what did you find particularly strange, other than the style of writing (for which I''m *not* going to apologize :))?> > Doei, Arthur. (Jeez, nerds... :))> A real nerd takes a plane to Canada to talk about traffic shaping :)No, a real nerd takes a plane to Canada to listen to others talk about traffic shaping. >:)> Yesterday a colleague had a problem. He had a redhat based firewall and > installed NT4 on it. After first reboot, lilo was still booting a linux > kernel even after fdisk''ing the disk. He couldn''t remove the tux-virus :)Yes? The disk probably wasn''t fully wiped by the NT install, nor was the bootblock cleared. The first will have happened if the kernel was nowhere near the low end of the disk, the second is one of the lesser known issues with Microsoft''s fdisk: it will only reset the MBR if you use fdisk /mbr . Doei, Arthur. (Not surprised in the least) -- /\ / | arthurvl@sci.kun.nl | Work like you don''t need the money /__\ / | A friend is someone with whom | Love like you have never been hurt / \/__ | you can dare to be yourself | Dance like there''s nobody watching
> Yeah, that was quite clear. But what did you find particularly strange, > other than the style of writing (for which I''m *not* going to apologize > :))?<staf classie=newbie> - All the variables are fine to read, but it''s nicer to have an example with real numbers. - "One creates two additional routing tables, say T1 and T2. These are added in /etc/iproute2/rt_tables" How do you do this ? - "ip route add $P1_NET dev $IF1 src $IP1 table T1" blablabla "ip route add $P1_NET dev $IF1 src $IP1" Why adding allmost the same route ???? - what''s the right syntax for $P1_NET ???? </staf> I''m not a newbie (anymore, at least that''s my opinion), I know what I can do with the tc command, but ip is sometimes a mistery to me. If I have some time, I will reread the chapter with the mind of a ip-newbie (no brain adaption needed) to try to update the chapter. But my mind is already on holiday, my body follows sunday :) At least for the next 2 weeks. Stef -- stef.coene@docum.org "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.openprojects.net
Julian Anastasov
2002-Jul-05 23:31 UTC
Re: "Bug" in howto 4.2.1 Split access and other advice
Hello, On Fri, 5 Jul 2002, Ard van Breemen wrote:> I am using this to differentiate between known routing, and > default routing.Or more correctly, to specify the path between each two subnets, the more specific rules and routes before the others.> # (The append is needed because else the routes will clash) > # So we have a table with two default routes wich will failover > # for eachother (takes about 10 minutes in default config)You can add note about tuning this timeout with gc_interval, gc_timeout and friends.> So what is the catch: > The only catch is that if you do not have point-to-point > connections with your provider, but a /24 for example, then > requests coming in from provider2 for the ip-eth1, will go out > from your eth2 and not from your eth1. This might be a problem if > your /24 is filtered by your ISP.This should be covered from sharing /24 subnet with the provider - the link-local networks. The most usual setups are: 1. shared /30 => routes via peer gateway 2. shared pubnet, ISP/we have one IP from pubnet => link-local routes, gateway is onlink The 2nd case can be split again in 2 cases according to the pubnet owner (ISP has the pubnet or the pubnet is yours). I.e. this explains where is the subnet, on internal or on external interface. Other alternatives?> Last thing: this failover thingie can also be a "loadbalanced" > thingie as explained in "4.2.2 Load balancing". > However: due to bugs in the equalizeing code, I recommend againstYes, equalize does not work if the ISPs filter by src. This flag is simply not in the kernel, only in user space.> it. Somewhere inside the kernel it cannot clearly come up with a > route, which results in a lot of "cannot happen 777".Fixed in 2.4.19pre8, SMP race.> Next to that: the usage counts of the devices are not correctly > incremented and decremented. You have to be very careful and craft > an extra non-multipath route before, then remove the existing > multi-path route, before bringing down a network device. Else it > ends up in an endless "device still in use, waiting". And you > will not be able to use the device anymore until you reset some > sense into the machine...This snafu is also fixed in 2.4.19pre8, it is even killed: the multipath route is deleted due to SMP locking problems. Regards -- Julian Anastasov <ja@ssi.bg>
Ard van Breemen
2002-Jul-08 11:22 UTC
Re: "Bug" in howto 4.2.1 Split access and other advice
On Fri, Jul 05, 2002 at 08:13:53PM +0200, Arthur van Leeuwen wrote:> On Fri, 5 Jul 2002, Ard van Breemen wrote: > > http://lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-links.html > > I am not sure who wrote this part or what it was based upon, but > > since I am working a lot longer now with ip rules, I think I want > > to add some stuff: > The stuff that is in the HOWTO was designed and tested back in 1999. > Oh, and I am the author. :)Ok... I would have written the same example, so I was not sure on who''s experience it was based upon. It was not meant as a "the author is stupid", but more like "do I know the author...". I''ve told this example also to many people (before I even heard about the lartc. I usually do not read HOWTO''s or stuff like that), because it was the same setup I was using at home. But as experience evolves, I now know it is not ok.> > The example 4.2.1 refers to the picture above, and does a plain > > ip rule add from .... table .... > > The problem with the exampe is that if you connect from the > > inside (local network) to your if1 ip or if2 ip, that in this > > example the replies to the local-network are going out if1 or > > if2... That is not what you want. > > True. That is indeed a bug. Never saw it in actual practice though: you > *should*not* connect to the external IP addresses of your router from > the internal network... for various security reasons and such. But you are > right.Hmmmm, to the linux kernel, an IP address is not really interface bound, so everybody should be able to connect to any ip address on the router. My filters are usually only based on interface instead of ip addresses. Usually rp_filter will do the remaining work. So I see no harm in connecting to the "external" ip addressess. (Quoted, since they are not really external or completely bound to an interface, you can always arp for them on another interface..., eh..., if rp_filter allows that of-course.)> > [snip] > > Whoa, that was large. I''m not sure I entirely follow you though. > The *point* of the extra routing tables is that they take precedence > over the default routing tables...-----------^^^^^^^ That''s exactly my point: default routes make the kernel go "hey I found the route, so I do not have to search anymore", so they should be *after* the normal routing, but *before* the big catchall default route. Anything else not being a default route, should of course go before the normal routing. I like the way Julian describes it: " Or more correctly, to specify the path between each two subnets, the more specific rules and routes before the others." So, eventually we will get a good description and a good practices guide. -- begin ILOVEYOU.VBS 666 <ard@telegraafnet.nl> Telegraaf Elektronische Media Real geeks don''t get viruses end
Ard van Breemen
2002-Jul-08 11:54 UTC
Re: "Bug" in howto 4.2.1 Split access and other advice
On Fri, Jul 05, 2002 at 11:31:53PM +0000, Julian Anastasov wrote:> On Fri, 5 Jul 2002, Ard van Breemen wrote: > > I am using this to differentiate between known routing, and > > default routing. > Or more correctly, to specify the path between > each two subnets, the more specific rules and routes before the > others.Yes... I like. Although too complex for the users that do not know what we are talking about.> > # (The append is needed because else the routes will clash) > > # So we have a table with two default routes wich will failover > > # for eachother (takes about 10 minutes in default config) > You can add note about tuning this timeout with > gc_interval, gc_timeout and friends.Or refer to the page, ehhhh, oops... I remember Bert asking me to write one up :(...> > So what is the catch: > > The only catch is that if you do not have point-to-point > > connections with your provider, but a /24 for example, then > > requests coming in from provider2 for the ip-eth1, will go out > > from your eth2 and not from your eth1. This might be a problem if > > your /24 is filtered by your ISP.> This should be covered from sharing /24 subnet with > the provider - the link-local networks. > > The most usual setups are: > > 1. shared /30 => routes via peer gateway > > 2. shared pubnet, ISP/we have one IP from pubnet => link-local > routes, gateway is onlink > > The 2nd case can be split again in 2 cases according to the pubnet > owner (ISP has the pubnet or the pubnet is yours). I.e. this > explains where is the subnet, on internal or on external > interface.Once again, I like it. This time it is also not complex.> Other alternatives?I think that covers everything: peer-to-peer, and subnets... Is there something else I did not know about?> > Last thing: this failover thingie can also be a "loadbalanced" > > thingie as explained in "4.2.2 Load balancing". > > However: due to bugs in the equalizeing code, I recommend against > Yes, equalize does not work if the ISPs filter by src. > This flag is simply not in the kernel, only in user space.Well, it *does* work in this framework: we only need the multipath code to select a route, and therefore a source ip. Once we got the source ip, we will not hit the multipath route anymore due to the rules... And even if we did, there is a good chance it is still cached :).> > it. Somewhere inside the kernel it cannot clearly come up with a > > route, which results in a lot of "cannot happen 777". > Fixed in 2.4.19pre8, SMP race.Cool!> > Next to that: the usage counts of the devices are not correctly > > incremented and decremented. You have to be very careful and craft > > an extra non-multipath route before, then remove the existing > > multi-path route, before bringing down a network device. Else it > > ends up in an endless "device still in use, waiting". And you > > will not be able to use the device anymore until you reset some > > sense into the machine... > This snafu is also fixed in 2.4.19pre8, it is even killed: > the multipath route is deleted due to SMP locking problems.Do you mean we cannot have multipaths anymore? Or that the fix was to remove the multipath route? -- begin ILOVEYOU.VBS 666 <ard@telegraafnet.nl> Telegraaf Elektronische Media Real geeks don''t get viruses end
Julian Anastasov
2002-Jul-08 12:15 UTC
Re: "Bug" in howto 4.2.1 Split access and other advice
Hello, On Mon, 8 Jul 2002, Ard van Breemen wrote:> > Or more correctly, to specify the path between > > each two subnets, the more specific rules and routes before the > > others. > Yes... I like. Although too complex for the users that do not > know what we are talking about.Yes, I usually can''t write good docs for the user''s eyes :)> > Yes, equalize does not work if the ISPs filter by src. > > This flag is simply not in the kernel, only in user space. > Well, it *does* work in this framework: we only need the > multipath code to select a route, and therefore a source ip. Once > we got the source ip, we will not hit the multipath route > anymore due to the rules... And even if we did, there is a good > chance it is still cached :).Yes, the route cache is the only place that remembers these associations. But it is dangerous for the NAT conns, one IP added and some of the connections can die selecting wrong src IP after the cache is autoflushed.> > This snafu is also fixed in 2.4.19pre8, it is even killed: > > the multipath route is deleted due to SMP locking problems. > Do you mean we cannot have multipaths anymore? Or that the fix > was to remove the multipath route?Yes, the kernel removes the whole multipath route if one device from a path is unregistered. Even without such action from the kernel, the user is obligated to delete this multipath route because if we register the same device name, the nexthop will not switch from dead to alive. So, for 2.4 (I''m not sure how the behaviour will change in 2.5 later) the kernel removes the multipaths but note that we (the users) always have to recreate the multipath routes with unregistered devices. Regards -- Julian Anastasov <ja@ssi.bg>
Dear Ard: All your mails seem to come with an attachment I LOVE YOU.VBS 666.dat. Can you check this out lest you end up spreading virus all over? If I''m wrong, forgive me. Mohan -----Original Message----- From: lartc-admin@mailman.ds9a.nl [mailto:lartc-admin@mailman.ds9a.nl] On Behalf Of Ard van Breemen Sent: 08 July, 2002 4:52 PM To: Arthur van Leeuwen Cc: lartc@mailman.ds9a.nl Subject: Re: [LARTC] "Bug" in howto 4.2.1 Split access and other advice On Fri, Jul 05, 2002 at 08:13:53PM +0200, Arthur van Leeuwen wrote:> On Fri, 5 Jul 2002, Ard van Breemen wrote: > > http://lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-link > > s.html > > I am not sure who wrote this part or what it was based upon, but > > since I am working a lot longer now with ip rules, I think I want > > to add some stuff: > The stuff that is in the HOWTO was designed and tested back in 1999. > Oh, and I am the author. :)Ok... I would have written the same example, so I was not sure on who''s experience it was based upon. It was not meant as a "the author is stupid", but more like "do I know the author...". I''ve told this example also to many people (before I even heard about the lartc. I usually do not read HOWTO''s or stuff like that), because it was the same setup I was using at home. But as experience evolves, I now know it is not ok.> > The example 4.2.1 refers to the picture above, and does a plain ip > > rule add from .... table .... The problem with the exampe is that if> > you connect from the inside (local network) to your if1 ip or if2 > > ip, that in this example the replies to the local-network are going > > out if1 or if2... That is not what you want. > > True. That is indeed a bug. Never saw it in actual practice though: > you > *should*not* connect to the external IP addresses of your router from > the internal network... for various security reasons and such. But youare> right.Hmmmm, to the linux kernel, an IP address is not really interface bound, so everybody should be able to connect to any ip address on the router. My filters are usually only based on interface instead of ip addresses. Usually rp_filter will do the remaining work. So I see no harm in connecting to the "external" ip addressess. (Quoted, since they are not really external or completely bound to an interface, you can always arp for them on another interface..., eh..., if rp_filter allows that of-course.)> > [snip] > > Whoa, that was large. I''m not sure I entirely follow you though. The > *point* of the extra routing tables is that they take precedence over > the default routing tables...-----------^^^^^^^ That''s exactly my point: default routes make the kernel go "hey I found the route, so I do not have to search anymore", so they should be *after* the normal routing, but *before* the big catchall default route. Anything else not being a default route, should of course go before the normal routing. I like the way Julian describes it: " Or more correctly, to specify the path between each two subnets, the more specific rules and routes before the others." So, eventually we will get a good description and a good practices guide. --
''Ard van Breemen''
2002-Jul-08 13:50 UTC
Re: "Bug" in howto 4.2.1 Split access and other advice
On Mon, Jul 08, 2002 at 05:46:02PM +0530, S Mohan wrote:> All your mails seem to come with an attachment I LOVE YOU.VBS 666.dat. > Can you check this out lest you end up spreading virus all over? If I''m > wrong, forgive me.You are not the first one accusing me of that. This is my .signature indented with a # because some mailers (If I am correct only outlook has this wrong behaviour) have problems with plaintext remarks, and show them as attachments, although they really are not. .signature altered for outlook users: -- # begin ILOVEYOU.VBS 666 # <ard@telegraafnet.nl> Telegraaf Elektronische Media # Real geeks don''t get viruses # end
''Ard van Breemen'' wrote:>You are not the first one accusing me of that. >This is my .signature indented with a # because some mailers (If >I am correct only outlook has this wrong behaviour) have problems >with plaintext remarks, and show them as attachments, although >they really are not. > >It is a great bug for most of my clients; sometimes they even get attachments in their Inbox which they cannot open / double-click :-). Ah well, I''m glad to see that my (Linux-based) spam/virus filter didn''t pick it up incorrectly either. -- Michael T. Babcock CTO, FibreSpeed Ltd.