LARTC - Jul 2002 - "Bug" in howto 4.2.1 Split access and other advice

Hi,
http://lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-links.html
I am not sure who wrote this part or what it was based upon, but
since I am working a lot longer now with ip rules, I think I want
to add some stuff:
The example 4.2.1 refers to the picture above, and does a plain
ip rule add from .... table ....
The problem with the exampe is that if you connect from the
inside (local network) to your if1 ip or if2 ip, that in this
example the replies to the local-network are going out if1 or
if2... That is not what you want.

If we carefully study the ip rule set, we see the first number.
This number is the priority, and if you use this priority number
in your rule command, it will insert it in the rule list after the 
rule with the same or lower priority.

I am using this to differentiate between known routing, and
default routing.

So first we set up the link local routing (the things you can
reach directly). Actually you don''t have to do a thing for that,
except setting up the interface.
List your local routing to understand:
ip route show table main

Then we put the default routing into table default.
And between the main and the default rule we put rules that
differentiate the default routing per provider.

example:
############################################################
# set up table main by upping the interfaces
ifup eth0   # local net
ifup eth1   # provider 1
ifup eth2   # provider 2

# Now set up the default routes in a failover fashion, with the
# most important route first(f.i. gw-provider1):
ip route add default via gw-provider1    table default
# Secondary route when the first gw fails:
ip route append default via gw-provider2 table default
# (The append is needed because else the routes will clash)
# So we have a table with two default routes wich will failover
# for eachother (takes about 10 minutes in default config)

# We now have a simple failover, which for most of us will not
# work, since most providers will have src-ip filtering.
# We are going to fix that now:
ip route add default via gw-provider1 table rt-provider1
ip route add default via gw-provider2 table rt-provider2
# We now have 2 tables each with a single different default gw.
# They are not used, that is what we are going to solve now:

ip rule add from ip-eth1 table rt-provider1 prio 32766
ip rule add from ip-eth2 table rt-provider2 prio 32766


# That''s it.
############################################################

So let''s think about it, and look at it:
ard@erwin(slave):~$ /sbin/ip rule list
0:      from all lookup local 
32766:  from all lookup main 
32766:  from <someip> lookup <sometable>
32767:  from all lookup default 

This is not what we did above, but it is a rule list from a
working environment (does www.telegraaf.nl ring a bell?).

What we should have seen was this:
/sbin/ip rule list
0:      from all lookup local
32766:  from all lookup main
32766:  from <ip-eth1> lookup rt-provider1
32766:  from <ip-eth2> lookup rt-provider2
32767:  from all lookup default

The main difference with the example in the document is:
- We do *not* have a default route in main
- We *have* default routes in the default table
- We have rules *after* main, not before main

So what is the catch:
The only catch is that if you do not have point-to-point
connections with your provider, but a /24 for example, then
requests coming in from provider2 for the ip-eth1, will go out
from your eth2 and not from your eth1. This might be a problem if
your /24 is filtered by your ISP.
The solution to that is the essence of this story: move the
calling of your default route tables from the rules to the last
possible moment.
So to fix the catch you get two more routing tables:
(With a provider3 added for clarity)
############################################################
ip route add net-provider2/24 via gw-provider1 table use-gw-provider1
ip route add net-provider3/24 via gw-provider1 table use-gw-provider1
ip rule add from ip-eth1 table use-gw-provider1 prio 32765

ip route add net-provider1/24 via gw-provider2 table use-gw-provider2
ip route add net-provider3/24 via gw-provider2 table use-gw-provider2
ip rule add from ip-eth2 table use-gw-provider2 prio 32765

ip route add net-provider1/24 via gw-provider3 table use-gw-provider3
ip route add net-provider2/24 via gw-provider3 table use-gw-provider3
ip rule add from ip-eth3 table use-gw-provider3 prio 32765
############################################################

I hope this makes some sense, and I hope it also is clear that
this is only needed for the link-local network of your provider
only if it is filtered!


Next thing: I was talking about failover earlier:
If a gateway is not available (ie, it does not reply to arps),
linux will think it is dead within a few minutes, and use the
other gateway. But only if it reaches the default table.
It reaches that, when it does not have a clue of the outgoing
src-ip yet. So if an application makes a connection to a website,
and the first gateway is considered dead, it will connect to the
website using the second gateway, and therefore bind to ip-eth2.


Last thing: this failover thingie can also be a "loadbalanced"
thingie as explained in "4.2.2 Load balancing".
However: due to bugs in the equalizeing code, I recommend against
it. Somewhere inside the kernel it cannot clearly come up with a
route, which results in a lot of "cannot happen 777".
Next to that: the usage counts of the devices are not correctly
incremented and decremented. You have to be very careful and craft
an extra non-multipath route before, then remove the existing
multi-path route, before bringing down a network device. Else it
ends up in an endless "device still in use, waiting". And you
will not be able to use the device anymore until you reset some
sense into the machine...
-- 
begin  ILOVEYOU.VBS 666
<ard@telegraafnet.nl> Telegraaf Elektronische Media
Real geeks don''t get viruses
end

On Fri, 5 Jul 2002, Ard van Breemen wrote:
> Hi,
> http://lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-links.html
> I am not sure who wrote this part or what it was based upon, but
> since I am working a lot longer now with ip rules, I think I want
> to add some stuff:
The stuff that is in the HOWTO was designed and tested back in 1999.
Oh, and I am the author. :)
> The example 4.2.1 refers to the picture above, and does a plain
> ip rule add from .... table ....
> The problem with the exampe is that if you connect from the
> inside (local network) to your if1 ip or if2 ip, that in this
> example the replies to the local-network are going out if1 or
> if2... That is not what you want.
True. That is indeed a bug. Never saw it in actual practice though: you
*should*not* connect to the external IP addresses of your router from
the internal network... for various security reasons and such. But you are
right.

[snip]

Whoa, that was large. I''m not sure I entirely follow you though.
The *point* of the extra routing tables is that they take precedence
over the default routing tables...

Doei, Arthur.

-- 
  /\    / |      arthurvl@sci.kun.nl      | Work like you don''t need
the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there''s nobody
watching

Hi,

What is still unclear to me is when the
http://www.linuxvirtualserver.org/~julian/#routes patches are needed.
What do they do exactly?

Thanks in advance,

--
Laurens van Alphen

-----Original Message-----
From: Ard van Breemen [mailto:ard@telegraafnet.nl] 
Sent: vrijdag 5 juli 2002 19:32
To: lartc@mailman.ds9a.nl
Cc: HOWTO@ds9a.nl
Subject: [LARTC] "Bug" in howto 4.2.1 Split access and other advice


Hi,
http://lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-links.ht
ml
I am not sure who wrote this part or what it was based upon, but since I
am working a lot longer now with ip rules, I think I want to add some
stuff: The example 4.2.1 refers to the picture above, and does a plain
ip rule add from .... table .... The problem with the exampe is that if
you connect from the inside (local network) to your if1 ip or if2 ip,
that in this example the replies to the local-network are going out if1
or if2... That is not what you want.

If we carefully study the ip rule set, we see the first number. This
number is the priority, and if you use this priority number in your rule
command, it will insert it in the rule list after the 
rule with the same or lower priority.

I am using this to differentiate between known routing, and default
routing.

So first we set up the link local routing (the things you can reach
directly). Actually you don''t have to do a thing for that, except
setting up the interface. List your local routing to understand: ip
route show table main

Then we put the default routing into table default.
And between the main and the default rule we put rules that
differentiate the default routing per provider.

example: ############################################################
# set up table main by upping the interfaces
ifup eth0   # local net
ifup eth1   # provider 1
ifup eth2   # provider 2

# Now set up the default routes in a failover fashion, with the # most
important route first(f.i. gw-provider1):
ip route add default via gw-provider1    table default
# Secondary route when the first gw fails:
ip route append default via gw-provider2 table default
# (The append is needed because else the routes will clash)
# So we have a table with two default routes wich will failover # for
eachother (takes about 10 minutes in default config)

# We now have a simple failover, which for most of us will not # work,
since most providers will have src-ip filtering. # We are going to fix
that now: ip route add default via gw-provider1 table rt-provider1 ip
route add default via gw-provider2 table rt-provider2 # We now have 2
tables each with a single different default gw. # They are not used,
that is what we are going to solve now:

ip rule add from ip-eth1 table rt-provider1 prio 32766
ip rule add from ip-eth2 table rt-provider2 prio 32766


# That''s it.
############################################################

So let''s think about it, and look at it:
ard@erwin(slave):~$ /sbin/ip rule list
0:      from all lookup local 
32766:  from all lookup main 
32766:  from <someip> lookup <sometable>
32767:  from all lookup default 

This is not what we did above, but it is a rule list from a working
environment (does www.telegraaf.nl ring a bell?).

What we should have seen was this:
/sbin/ip rule list
0:      from all lookup local
32766:  from all lookup main
32766:  from <ip-eth1> lookup rt-provider1
32766:  from <ip-eth2> lookup rt-provider2
32767:  from all lookup default

The main difference with the example in the document is:
- We do *not* have a default route in main
- We *have* default routes in the default table
- We have rules *after* main, not before main

So what is the catch:
The only catch is that if you do not have point-to-point connections
with your provider, but a /24 for example, then requests coming in from
provider2 for the ip-eth1, will go out from your eth2 and not from your
eth1. This might be a problem if your /24 is filtered by your ISP. The
solution to that is the essence of this story: move the calling of your
default route tables from the rules to the last possible moment. So to
fix the catch you get two more routing tables: (With a provider3 added
for clarity)
############################################################
ip route add net-provider2/24 via gw-provider1 table use-gw-provider1 ip
route add net-provider3/24 via gw-provider1 table use-gw-provider1 ip
rule add from ip-eth1 table use-gw-provider1 prio 32765

ip route add net-provider1/24 via gw-provider2 table use-gw-provider2 ip
route add net-provider3/24 via gw-provider2 table use-gw-provider2 ip
rule add from ip-eth2 table use-gw-provider2 prio 32765

ip route add net-provider1/24 via gw-provider3 table use-gw-provider3 ip
route add net-provider2/24 via gw-provider3 table use-gw-provider3 ip
rule add from ip-eth3 table use-gw-provider3 prio 32765
############################################################

I hope this makes some sense, and I hope it also is clear that this is
only needed for the link-local network of your provider only if it is
filtered!


Next thing: I was talking about failover earlier:
If a gateway is not available (ie, it does not reply to arps), linux
will think it is dead within a few minutes, and use the other gateway.
But only if it reaches the default table. It reaches that, when it does
not have a clue of the outgoing src-ip yet. So if an application makes a
connection to a website, and the first gateway is considered dead, it
will connect to the website using the second gateway, and therefore bind
to ip-eth2.


Last thing: this failover thingie can also be a "loadbalanced" thingie
as explained in "4.2.2 Load balancing".
However: due to bugs in the equalizeing code, I recommend against it.
Somewhere inside the kernel it cannot clearly come up with a route,
which results in a lot of "cannot happen 777". Next to that: the usage
counts of the devices are not correctly incremented and decremented. You
have to be very careful and craft an extra non-multipath route before,
then remove the existing multi-path route, before bringing down a
network device. Else it ends up in an endless "device still in use,
waiting". And you will not be able to use the device anymore until you
reset some sense into the machine...
--

Hi,

I was rereading the pages in the howto about multiple ISP''s and I also
found
some strange stuff in it.  Maybe you can create a patch for Bert to update 
the howto ?

Stef

PS.  Het wordt hier echt wel een lage-landen onderonsje :)
English : lot of low-country (Belgium, The Netherlands) people on the list.

On Friday 05 July 2002 19:31, Ard van Breemen wrote:
> Hi,
> http://lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-links.html
> I am not sure who wrote this part or what it was based upon, but
> since I am working a lot longer now with ip rules, I think I want
> to add some stuff:
> The example 4.2.1 refers to the picture above, and does a plain
> ip rule add from .... table ....
> The problem with the exampe is that if you connect from the
> inside (local network) to your if1 ip or if2 ip, that in this
> example the replies to the local-network are going out if1 or
> if2... That is not what you want.
>
> If we carefully study the ip rule set, we see the first number.
> This number is the priority, and if you use this priority number
> in your rule command, it will insert it in the rule list after the
> rule with the same or lower priority.
>
> I am using this to differentiate between known routing, and
> default routing.
>
> So first we set up the link local routing (the things you can
> reach directly). Actually you don''t have to do a thing for that,
> except setting up the interface.
> List your local routing to understand:
> ip route show table main
>
> Then we put the default routing into table default.
> And between the main and the default rule we put rules that
> differentiate the default routing per provider.
>
> example:
> ############################################################
> # set up table main by upping the interfaces
> ifup eth0   # local net
> ifup eth1   # provider 1
> ifup eth2   # provider 2
>
> # Now set up the default routes in a failover fashion, with the
> # most important route first(f.i. gw-provider1):
> ip route add default via gw-provider1    table default
> # Secondary route when the first gw fails:
> ip route append default via gw-provider2 table default
> # (The append is needed because else the routes will clash)
> # So we have a table with two default routes wich will failover
> # for eachother (takes about 10 minutes in default config)
>
> # We now have a simple failover, which for most of us will not
> # work, since most providers will have src-ip filtering.
> # We are going to fix that now:
> ip route add default via gw-provider1 table rt-provider1
> ip route add default via gw-provider2 table rt-provider2
> # We now have 2 tables each with a single different default gw.
> # They are not used, that is what we are going to solve now:
>
> ip rule add from ip-eth1 table rt-provider1 prio 32766
> ip rule add from ip-eth2 table rt-provider2 prio 32766
>
>
> # That''s it.
> ############################################################
>
> So let''s think about it, and look at it:
> ard@erwin(slave):~$ /sbin/ip rule list
> 0:      from all lookup local
> 32766:  from all lookup main
> 32766:  from <someip> lookup <sometable>
> 32767:  from all lookup default
>
> This is not what we did above, but it is a rule list from a
> working environment (does www.telegraaf.nl ring a bell?).
>
> What we should have seen was this:
> /sbin/ip rule list
> 0:      from all lookup local
> 32766:  from all lookup main
> 32766:  from <ip-eth1> lookup rt-provider1
> 32766:  from <ip-eth2> lookup rt-provider2
> 32767:  from all lookup default
>
> The main difference with the example in the document is:
> - We do *not* have a default route in main
> - We *have* default routes in the default table
> - We have rules *after* main, not before main
>
> So what is the catch:
> The only catch is that if you do not have point-to-point
> connections with your provider, but a /24 for example, then
> requests coming in from provider2 for the ip-eth1, will go out
> from your eth2 and not from your eth1. This might be a problem if
> your /24 is filtered by your ISP.
> The solution to that is the essence of this story: move the
> calling of your default route tables from the rules to the last
> possible moment.
> So to fix the catch you get two more routing tables:
> (With a provider3 added for clarity)
> ############################################################
> ip route add net-provider2/24 via gw-provider1 table use-gw-provider1
> ip route add net-provider3/24 via gw-provider1 table use-gw-provider1
> ip rule add from ip-eth1 table use-gw-provider1 prio 32765
>
> ip route add net-provider1/24 via gw-provider2 table use-gw-provider2
> ip route add net-provider3/24 via gw-provider2 table use-gw-provider2
> ip rule add from ip-eth2 table use-gw-provider2 prio 32765
>
> ip route add net-provider1/24 via gw-provider3 table use-gw-provider3
> ip route add net-provider2/24 via gw-provider3 table use-gw-provider3
> ip rule add from ip-eth3 table use-gw-provider3 prio 32765
> ############################################################
>
> I hope this makes some sense, and I hope it also is clear that
> this is only needed for the link-local network of your provider
> only if it is filtered!
>
>
> Next thing: I was talking about failover earlier:
> If a gateway is not available (ie, it does not reply to arps),
> linux will think it is dead within a few minutes, and use the
> other gateway. But only if it reaches the default table.
> It reaches that, when it does not have a clue of the outgoing
> src-ip yet. So if an application makes a connection to a website,
> and the first gateway is considered dead, it will connect to the
> website using the second gateway, and therefore bind to ip-eth2.
>
>
> Last thing: this failover thingie can also be a "loadbalanced"
> thingie as explained in "4.2.2 Load balancing".
> However: due to bugs in the equalizeing code, I recommend against
> it. Somewhere inside the kernel it cannot clearly come up with a
> route, which results in a lot of "cannot happen 777".
> Next to that: the usage counts of the devices are not correctly
> incremented and decremented. You have to be very careful and craft
> an extra non-multipath route before, then remove the existing
> multi-path route, before bringing down a network device. Else it
> ends up in an endless "device still in use, waiting". And you
> will not be able to use the device anymore until you reset some
> sense into the machine...
-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.openprojects.net

On Fri, 5 Jul 2002, Laurens van Alphen wrote:
> What is still unclear to me is when the
> http://www.linuxvirtualserver.org/~julian/#routes patches are needed.
> What do they do exactly?
They add actual dead gateway detection (so that failover in fact does fail
over... ;)), alternative routes (so that if multiple routes are defined only
the active ones will actually be used), static routes (so that routes
don''t
get deleted by the kernel if the correspondign interface goes down... thus
automatically reusing it when it comes back up), and MASQUERADING support for
multiple gateways.

Doei, Arthur.

-- 
  /\    / |      arthurvl@sci.kun.nl      | Work like you don''t need
the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there''s nobody
watching

On Fri, 5 Jul 2002, Stef Coene wrote:
> Hi,
>
> I was rereading the pages in the howto about multiple ISP''s and I
also found
> some strange stuff in it.  Maybe you can create a patch for Bert to update
> the howto ?
Could you be more precies? :)

Doei, Arthur. (Yeah, the type is intentional :))

-- 
  /\    / |      arthurvl@sci.kun.nl      | Work like you don''t need
the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there''s nobody
watching

On Friday 05 July 2002 20:47, Arthur van Leeuwen wrote:
> On Fri, 5 Jul 2002, Stef Coene wrote:
> > Hi,
> >
> > I was rereading the pages in the howto about multiple ISP''s
and I also
> > found some strange stuff in it.  Maybe you can create a patch for Bert
to
> > update the howto ?
>
> Could you be more precies? :)
export CVSROOT=:pserver:anon@outpost.ds9a.nl:/var/cvsroot
cvs login
CVS password: [enter ''cvs'' (without ''s)]
cvs co 2.4routing
   password = cvs
vi 2.4routing/2.4routing.sgml
  :wq
cvs diff -u > patch
uuencode patch patch | mail -s "HOWTO update"  HOWTO@ds9a.nl

:)

Stef

On Fri, 5 Jul 2002, Stef Coene wrote:
> On Friday 05 July 2002 20:47, Arthur van Leeuwen wrote:
> > On Fri, 5 Jul 2002, Stef Coene wrote:
> > > Hi,
> > >
> > > I was rereading the pages in the howto about multiple
ISP''s and I also
> > > found some strange stuff in it.  Maybe you can create a patch for
Bert to
> > > update the howto ?
> >
> > Could you be more precies? :)
> export CVSROOT=:pserver:anon@outpost.ds9a.nl:/var/cvsroot
> cvs login
> CVS password: [enter ''cvs'' (without ''s)]
> cvs co 2.4routing
>    password = cvs
> vi 2.4routing/2.4routing.sgml
>   :wq
> cvs diff -u > patch
> uuencode patch patch | mail -s "HOWTO update"  HOWTO@ds9a.nl
On what stuff you found strange, I meant... not on the process of sending in
a patch for the HOWTO...

Doei, Arthur. (Jeez, nerds... :))

-- 
  /\    / |      arthurvl@sci.kun.nl      | Work like you don''t need
the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there''s nobody
watching

On Friday 05 July 2002 21:05, Arthur van Leeuwen wrote:
> On Fri, 5 Jul 2002, Stef Coene wrote:
> > On Friday 05 July 2002 20:47, Arthur van Leeuwen wrote:
> > > On Fri, 5 Jul 2002, Stef Coene wrote:
> > > > Hi,
> > > >
> > > > I was rereading the pages in the howto about multiple
ISP''s and I
> > > > also found some strange stuff in it.  Maybe you can create a
patch
> > > > for Bert to update the howto ?
> > >
> > > Could you be more precies? :)
> >
> > export CVSROOT=:pserver:anon@outpost.ds9a.nl:/var/cvsroot
> > cvs login
> > CVS password: [enter ''cvs'' (without ''s)]
> > cvs co 2.4routing
> >    password = cvs
> > vi 2.4routing/2.4routing.sgml
> >
> >   :wq
> >
> > cvs diff -u > patch
> > uuencode patch patch | mail -s "HOWTO update"  HOWTO@ds9a.nl
>
> On what stuff you found strange, I meant... not on the process of sending
> in a patch for the HOWTO...
quoted from my post above :
> > > > I was rereading the pages in the howto about multiple
ISP''s and I
quoted from the subject
Re: [LARTC] "Bug" in howto 4.2.1 Split access and other advice

So I was talking about chapter 4.2 :
http://www.lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-links.html
> Doei, Arthur. (Jeez, nerds... :))
A real nerd takes a plane to Canada to talk about traffic shaping :)

Yesterday a colleague had a problem.  He had a redhat based firewall and 
installed NT4 on it.  After first reboot, lilo was still booting a linux 
kernel even after fdisk''ing the disk.  He couldn''t remove the
tux-virus :)

Stef

--

On Fri, 5 Jul 2002, Stef Coene wrote:
> On Friday 05 July 2002 21:05, Arthur van Leeuwen wrote:
> > On Fri, 5 Jul 2002, Stef Coene wrote:
> > > On Friday 05 July 2002 20:47, Arthur van Leeuwen wrote:
> > > > On Fri, 5 Jul 2002, Stef Coene wrote:
> > > > > Hi,
> > > > >
> > > > > I was rereading the pages in the howto about multiple
ISP''s and I
> > > > > also found some strange stuff in it.  Maybe you can
create a patch
> > > > > for Bert to update the howto ?
> > > >
> > > > Could you be more precies? :)
> > >
> > > export CVSROOT=:pserver:anon@outpost.ds9a.nl:/var/cvsroot
> > > cvs login
> > > CVS password: [enter ''cvs'' (without
''s)]
> > > cvs co 2.4routing
> > >    password = cvs
> > > vi 2.4routing/2.4routing.sgml
> > >
> > >   :wq
> > >
> > > cvs diff -u > patch
> > > uuencode patch patch | mail -s "HOWTO update" 
HOWTO@ds9a.nl
> > On what stuff you found strange, I meant... not on the process of
sending
> > in a patch for the HOWTO...
> quoted from my post above :
> > > > > I was rereading the pages in the howto about multiple
ISP''s and I
> quoted from the subject
> Re: [LARTC] "Bug" in howto 4.2.1 Split access and other advice
> So I was talking about chapter 4.2 :
>
http://www.lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-links.html
Yeah, that was quite clear. But what did you find particularly strange,
other than the style of writing (for which I''m *not* going to apologize
:))?
> > Doei, Arthur. (Jeez, nerds... :))
> A real nerd takes a plane to Canada to talk about traffic shaping :)
No, a real nerd takes a plane to Canada to listen to others talk about
traffic shaping. >:)
> Yesterday a colleague had a problem.  He had a redhat based firewall and
> installed NT4 on it.  After first reboot, lilo was still booting a linux
> kernel even after fdisk''ing the disk.  He couldn''t remove
the tux-virus :)
Yes? The disk probably wasn''t fully wiped by the NT install, nor was
the
bootblock cleared. The first will have happened if the kernel was nowhere
near the low end of the disk, the second is one of the lesser known issues
with Microsoft''s fdisk: it will only reset the MBR if you use fdisk
/mbr .

Doei, Arthur. (Not surprised in the least)

-- 
  /\    / |      arthurvl@sci.kun.nl      | Work like you don''t need
the money
 /__\  /  | A friend is someone with whom | Love like you have never been hurt
/    \/__ | you can dare to be yourself   | Dance like there''s nobody
watching

> Yeah, that was quite clear. But what did you find particularly strange,
> other than the style of writing (for which I''m *not* going to
apologize
> :))?
<staf classie=newbie>
- All the variables are fine to read, but it''s nicer to have an example
with
real numbers.
- "One creates two additional routing tables, say T1 and T2. These are
added
in /etc/iproute2/rt_tables"  How do you do this ?
- "ip route add $P1_NET dev $IF1 src $IP1 table T1"
blablabla
"ip route add $P1_NET dev $IF1 src $IP1"
Why adding allmost the same route ????
- what''s the right syntax for $P1_NET ????
</staf>

I''m not a newbie (anymore, at least that''s my opinion), I know
what I can do
with the tc command, but ip is sometimes a mistery to me.  If I have some 
time, I will reread the chapter with the mind of a ip-newbie (no brain 
adaption needed) to try to update the chapter.  
But my mind is already on holiday, my body follows sunday :)  At least for 
the next 2 weeks.

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.openprojects.net

Hello,

On Fri, 5 Jul 2002, Ard van Breemen wrote:
> I am using this to differentiate between known routing, and
> default routing.
	Or more correctly, to specify the path between
each two subnets, the more specific rules and routes before the
others.
> # (The append is needed because else the routes will clash)
> # So we have a table with two default routes wich will failover
> # for eachother (takes about 10 minutes in default config)
	You can add note about tuning this timeout with
gc_interval, gc_timeout and friends.
> So what is the catch:
> The only catch is that if you do not have point-to-point
> connections with your provider, but a /24 for example, then
> requests coming in from provider2 for the ip-eth1, will go out
> from your eth2 and not from your eth1. This might be a problem if
> your /24 is filtered by your ISP.
	This should be covered from sharing /24 subnet with
the provider - the link-local networks.

	The most usual setups are:

1. shared /30 => routes via peer gateway

2. shared pubnet, ISP/we have one IP from pubnet => link-local
routes, gateway is onlink

The 2nd case can be split again in 2 cases according to the pubnet
owner (ISP has the pubnet or the pubnet is yours). I.e. this
explains where is the subnet, on internal or on external
interface.

Other alternatives?
> Last thing: this failover thingie can also be a "loadbalanced"
> thingie as explained in "4.2.2 Load balancing".
> However: due to bugs in the equalizeing code, I recommend against
	Yes, equalize does not work if the ISPs filter by src.
This flag is simply not in the kernel, only in user space.
> it. Somewhere inside the kernel it cannot clearly come up with a
> route, which results in a lot of "cannot happen 777".
	Fixed in 2.4.19pre8, SMP race.
> Next to that: the usage counts of the devices are not correctly
> incremented and decremented. You have to be very careful and craft
> an extra non-multipath route before, then remove the existing
> multi-path route, before bringing down a network device. Else it
> ends up in an endless "device still in use, waiting". And you
> will not be able to use the device anymore until you reset some
> sense into the machine...
	This snafu is also fixed in 2.4.19pre8, it is even killed:
the multipath route is deleted due to SMP locking problems.

Regards

--
Julian Anastasov <ja@ssi.bg>

On Fri, Jul 05, 2002 at 08:13:53PM +0200, Arthur van Leeuwen
wrote:
> On Fri, 5 Jul 2002, Ard van Breemen wrote:
> >
http://lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-links.html
> > I am not sure who wrote this part or what it was based upon, but
> > since I am working a lot longer now with ip rules, I think I want
> > to add some stuff:
> The stuff that is in the HOWTO was designed and tested back in 1999.
> Oh, and I am the author. :)
Ok... I would have written the same example, so I was not sure on
who''s experience it was based upon. It was not meant as a "the
author is stupid", but more like "do I know the author...".
I''ve told this example also to many people (before I even heard
about the lartc. I usually do not read HOWTO''s or stuff like
that), because it was the same setup I was using at home. But as
experience evolves, I now know it is not ok.
> > The example 4.2.1 refers to the picture above, and does a plain
> > ip rule add from .... table ....
> > The problem with the exampe is that if you connect from the
> > inside (local network) to your if1 ip or if2 ip, that in this
> > example the replies to the local-network are going out if1 or
> > if2... That is not what you want.
> 
> True. That is indeed a bug. Never saw it in actual practice though: you
> *should*not* connect to the external IP addresses of your router from
> the internal network... for various security reasons and such. But you are
> right.
Hmmmm, to the linux kernel, an IP address is not really interface
bound, so everybody should be able to connect to any ip address
on the router. My filters are usually only based on interface instead
of ip addresses. Usually rp_filter will do the remaining work.
So I see no harm in connecting to the "external" ip addressess.
(Quoted, since they are not really external or completely bound to an
interface, you can always arp for them on another interface...,
eh..., if rp_filter allows that of-course.)
> 
> [snip]
> 
> Whoa, that was large. I''m not sure I entirely follow you though.
> The *point* of the extra routing tables is that they take precedence
> over the default routing tables...
-----------^^^^^^^
That''s exactly my point: default routes make the kernel go "hey I
found the route, so I do not have to search anymore", so they
should be *after* the normal routing, but *before* the big
catchall default route.
Anything else not being a default route, should of course go
before the normal routing.

I like the way Julian describes it:
"        Or more correctly, to specify the path between
each two subnets, the more specific rules and routes before the
others."

So, eventually we will get a good description and a good
practices guide.

-- 
begin  ILOVEYOU.VBS 666
<ard@telegraafnet.nl> Telegraaf Elektronische Media
Real geeks don''t get viruses
end

On Fri, Jul 05, 2002 at 11:31:53PM +0000, Julian Anastasov
wrote:
> On Fri, 5 Jul 2002, Ard van Breemen wrote:
> > I am using this to differentiate between known routing, and
> > default routing.
> 	Or more correctly, to specify the path between
> each two subnets, the more specific rules and routes before the
> others.
Yes... I like. Although too complex for the users that do not
know what we are talking about.
> > # (The append is needed because else the routes will clash)
> > # So we have a table with two default routes wich will failover
> > # for eachother (takes about 10 minutes in default config)
> 	You can add note about tuning this timeout with
> gc_interval, gc_timeout and friends.
Or refer to the page, ehhhh, oops... I remember Bert asking me to
write one up :(...
> > So what is the catch:
> > The only catch is that if you do not have point-to-point
> > connections with your provider, but a /24 for example, then
> > requests coming in from provider2 for the ip-eth1, will go out
> > from your eth2 and not from your eth1. This might be a problem if
> > your /24 is filtered by your ISP.
> 	This should be covered from sharing /24 subnet with
> the provider - the link-local networks.
> 
> 	The most usual setups are:
> 
> 1. shared /30 => routes via peer gateway
> 
> 2. shared pubnet, ISP/we have one IP from pubnet => link-local
> routes, gateway is onlink
> 
> The 2nd case can be split again in 2 cases according to the pubnet
> owner (ISP has the pubnet or the pubnet is yours). I.e. this
> explains where is the subnet, on internal or on external
> interface.
Once again, I like it. This time it is also not complex.
> Other alternatives?
I think that covers everything: peer-to-peer, and subnets... Is
there something else I did not know about?
> > Last thing: this failover thingie can also be a
"loadbalanced"
> > thingie as explained in "4.2.2 Load balancing".
> > However: due to bugs in the equalizeing code, I recommend against
> 	Yes, equalize does not work if the ISPs filter by src.
> This flag is simply not in the kernel, only in user space.
Well, it *does* work in this framework: we only need the
multipath code to select a route, and therefore a source ip. Once
we got the source ip, we will not hit the multipath route
anymore due to the rules... And even if we did, there is a good
chance it is still cached :).
> > it. Somewhere inside the kernel it cannot clearly come up with a
> > route, which results in a lot of "cannot happen 777".
> 	Fixed in 2.4.19pre8, SMP race.
Cool!
> > Next to that: the usage counts of the devices are not correctly
> > incremented and decremented. You have to be very careful and craft
> > an extra non-multipath route before, then remove the existing
> > multi-path route, before bringing down a network device. Else it
> > ends up in an endless "device still in use, waiting". And
you
> > will not be able to use the device anymore until you reset some
> > sense into the machine...
> 	This snafu is also fixed in 2.4.19pre8, it is even killed:
> the multipath route is deleted due to SMP locking problems.
Do you mean we cannot have multipaths anymore? Or that the fix
was to remove the multipath route?

-- 
begin  ILOVEYOU.VBS 666
<ard@telegraafnet.nl> Telegraaf Elektronische Media
Real geeks don''t get viruses
end

Hello,

On Mon, 8 Jul 2002, Ard van Breemen wrote:
> > 	Or more correctly, to specify the path between
> > each two subnets, the more specific rules and routes before the
> > others.
> Yes... I like. Although too complex for the users that do not
> know what we are talking about.
	Yes, I usually can''t write good docs for the user''s eyes :)
> > 	Yes, equalize does not work if the ISPs filter by src.
> > This flag is simply not in the kernel, only in user space.
> Well, it *does* work in this framework: we only need the
> multipath code to select a route, and therefore a source ip. Once
> we got the source ip, we will not hit the multipath route
> anymore due to the rules... And even if we did, there is a good
> chance it is still cached :).
	Yes, the route cache is the only place that remembers
these associations. But it is dangerous for the NAT conns, one
IP added and some of the connections can die selecting wrong src
IP after the cache is autoflushed.
> > 	This snafu is also fixed in 2.4.19pre8, it is even killed:
> > the multipath route is deleted due to SMP locking problems.
> Do you mean we cannot have multipaths anymore? Or that the fix
> was to remove the multipath route?
	Yes, the kernel removes the whole multipath route if
one device from a path is unregistered. Even without such
action from the kernel, the user is obligated to delete this
multipath route because if we register the same device name,
the nexthop will not switch from dead to alive. So, for 2.4 (I''m
not sure how the behaviour will change in 2.5 later) the
kernel removes the multipaths but note that we (the users)
always have to recreate the multipath routes with unregistered
devices.

Regards

--
Julian Anastasov <ja@ssi.bg>

Dear Ard:

All your mails seem to come with an attachment I LOVE YOU.VBS 666.dat.
Can you check this out lest you end up spreading virus all over? If I''m
wrong, forgive me.

Mohan

-----Original Message-----
From: lartc-admin@mailman.ds9a.nl [mailto:lartc-admin@mailman.ds9a.nl]
On Behalf Of Ard van Breemen
Sent: 08 July, 2002 4:52 PM
To: Arthur van Leeuwen
Cc: lartc@mailman.ds9a.nl
Subject: Re: [LARTC] "Bug" in howto 4.2.1 Split access and other
advice


On Fri, Jul 05, 2002 at 08:13:53PM +0200, Arthur van Leeuwen
wrote:
> On Fri, 5 Jul 2002, Ard van Breemen wrote:
> > http://lartc.org/HOWTO//cvs/2.4routing/html/lartc.rpdb.multiple-link
> > s.html
> > I am not sure who wrote this part or what it was based upon, but
> > since I am working a lot longer now with ip rules, I think I want
> > to add some stuff:
> The stuff that is in the HOWTO was designed and tested back in 1999. 
> Oh, and I am the author. :)
Ok... I would have written the same example, so I was not sure on who''s
experience it was based upon. It was not meant as a "the author is
stupid", but more like "do I know the author...". I''ve
told this example
also to many people (before I even heard about the lartc. I usually do
not read HOWTO''s or stuff like that), because it was the same setup I
was using at home. But as experience evolves, I now know it is not
ok.
> > The example 4.2.1 refers to the picture above, and does a plain ip 
> > rule add from .... table .... The problem with the exampe is that if
> > you connect from the inside (local network) to your if1 ip or if2 
> > ip, that in this example the replies to the local-network are going 
> > out if1 or if2... That is not what you want.
> 
> True. That is indeed a bug. Never saw it in actual practice though: 
> you
> *should*not* connect to the external IP addresses of your router from
> the internal network... for various security reasons and such. But you
are
> right.
Hmmmm, to the linux kernel, an IP address is not really interface bound,
so everybody should be able to connect to any ip address on the router.
My filters are usually only based on interface instead of ip addresses.
Usually rp_filter will do the remaining work. So I see no harm in
connecting to the "external" ip addressess. (Quoted, since they are
not
really external or completely bound to an interface, you can always arp
for them on another interface..., eh..., if rp_filter allows that
of-course.)
> 
> [snip]
> 
> Whoa, that was large. I''m not sure I entirely follow you though.
The
> *point* of the extra routing tables is that they take precedence over 
> the default routing tables...
-----------^^^^^^^
That''s exactly my point: default routes make the kernel go "hey I
found
the route, so I do not have to search anymore", so they should be
*after* the normal routing, but *before* the big catchall default route.
Anything else not being a default route, should of course go before the
normal routing.

I like the way Julian describes it:
"        Or more correctly, to specify the path between
each two subnets, the more specific rules and routes before the others."

So, eventually we will get a good description and a good practices
guide.

--

On Mon, Jul 08, 2002 at 05:46:02PM +0530, S Mohan wrote:
> All your mails seem to come with an attachment I LOVE YOU.VBS 666.dat.
> Can you check this out lest you end up spreading virus all over? If
I''m
> wrong, forgive me.
You are not the first one accusing me of that.
This is my .signature indented with a # because some mailers (If
I am correct only outlook has this wrong behaviour) have problems
with plaintext remarks, and show them as attachments, although
they really are not.

.signature altered for outlook users:
-- 
# begin  ILOVEYOU.VBS 666
# <ard@telegraafnet.nl> Telegraaf Elektronische Media
# Real geeks don''t get viruses
# end

''Ard van Breemen'' wrote:
>You are not the first one accusing me of that.
>This is my .signature indented with a # because some mailers (If
>I am correct only outlook has this wrong behaviour) have problems
>with plaintext remarks, and show them as attachments, although
>they really are not.
>  
>
It is a great bug for most of my clients; sometimes they even get 
attachments in their Inbox which they cannot open / double-click :-).

Ah well, I''m glad to see that my (Linux-based) spam/virus filter
didn''t
pick it up incorrectly either.
-- 
Michael T. Babcock
CTO, FibreSpeed Ltd.