LARTC - Nov 2002 - Multipath Routing Question with Public networks

Hi there;

I''m planning to implement multipath routing across two ISP in the next
few weeks, but
before going further, I have a routing question which is puzzling me :)

Here is the setup:

10.0.0.0/8    172.16.0.0/16
 ISP1        ISP2
  |            |
  |            |
 _|____________|_
 [Linux Firewall]
  |            |
__|_____       |
     |         |
     |       __|_____
     |        | 
_____|________|__________
[Internal Public Network]

All machines are linux based. (Firewall is kernel 2.2, public servers are either
2.2 or
2.4)

Linux firewall has 4 networks interfaces, all of them using public
IP''s; 2 routed through
ISP1, others 2 routed through ISP2. 
From the firewall itself, we can do multipath routing over both ISP''s
without problems.
In this network topology, we don''t use NAT. ALL IPs are publics, and
belongs to either
ISP1 or ISP2 network. So that all servers in the Internal Public Network are
reachable
from the Internet, through both links. 
All servers do have 2 network adapters, using public IPs belonging to ISP1 &
ISP2
networks. 
On All servers, we setup multipath default route, so that they can use both link
as well.

Let''s say we have Server A in our internal public network, with 2
network adapters, one
using 10.0.0.1 (routed through ISP1), other using 172.16.0.1 (routed through
ISP2)

Assuming that rp_filter is configured correctly on our firewall, what happens
when a
client want to reach Server A using 10.0.0.1 ? What path will be used for the
replies,
with what source IP ?

I assume that if rp_filter is configured correctly, return path do not matter
(since we
don''t do NAT), but I''m worried about the SRC IP beeing choosed
for the reply. Because, if
the kernel choose the src IP according to the output default route beeing
choosed, then
half clients<->servers sessions will just break.

So, basically my question is, what rules decide of the SRC IP to be used in a
reply
packet on a system with several default route through different network
interfaces ?

Has anyone already experiencing such setup ?

Thanks In advance !
Regards,

Vincent.

---
Vincent Jaussaud
Kelkoo - Security Manager / Networks & Systems Administration
AIM Nick: portsentry
---

 
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hello,

On Sat, 16 Nov 2002, Vincent Jaussaud wrote:
> On All servers, we setup multipath default route, so that they can use both
link as well.
	That means they know which link is alive or it does not
matter? :)
> Let''s say we have Server A in our internal public network, with 2
network adapters, one
> using 10.0.0.1 (routed through ISP1), other using 172.16.0.1 (routed
through ISP2)
>
> Assuming that rp_filter is configured correctly on our firewall, what
happens when a
> client want to reach Server A using 10.0.0.1 ? What path will be used for
the replies,
> with what source IP ?
	The server uses the 10.0.0.1 as source when resolving
route for the reply. Then it depends on the routing rules.
> I assume that if rp_filter is configured correctly, return path do not
matter (since we
> don''t do NAT), but I''m worried about the SRC IP beeing
choosed for the reply. Because, if
> the kernel choose the src IP according to the output default route beeing
choosed, then
> half clients<->servers sessions will just break.
	I''m sure you need correct routes on firewall and all internal
hosts, for example:

ip rule add prio ... from pubnet_X to 0/0 table table_for_ISP_X
ip rule add prio ... from pubnet_Y to 0/0 table table_for_ISP_Y
ip rule add prio ... from 0/0 to 0/0 table your_multipath_route

	Of course, the internal hosts use the proper firewall
IP as gateway.

	That is all, traffic from specific pub IP should use only
its gateway. You can expect rp_filter drops in firewall if
the internal servers select wrong NIC by using multipath
route for all route resolutions. The multipath route should be
used only for source address autoselection:

- originating connections without bind()
- selecting masquerade address (for NAT)
- etc

	I.e. it is a bad idea to use only multipath route.

	Internal servers creating outgoing connections after
using bind() to specific pubip should not reach the multipath route.
> So, basically my question is, what rules decide of the SRC IP to be used in
a reply
> packet on a system with several default route through different network
interfaces ?
	The transport and the application decide what source IP
to put in the reply. Then they decide how to call the routing.
The right thing to do when addresses to both ends are known is
to feed the routing with saddr and daddr. If callers use 0.0.0.0
as saddr when resolving routes, they will hit the multipath
route which is bad.
> Has anyone already experiencing such setup ?
	Not exactly, but everything is in the details :)
> Vincent.
Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

> Hello,
> 
Hi julian !
> On Sat, 16 Nov 2002, Vincent Jaussaud wrote:
> 
> > On All servers, we setup multipath default route, so that they can use
both link as well.
> 
> 	That means they know which link is alive or it does not
> matter? :)
> 
Well, at this time, it doesn''t matter yet :-)
Anyway, it would be best if they could, but I''m afraid that all
internal
servers will have to run your patch in order for this to work. And I''m
talking
about 20 production servers. 
So, I can''t easily patch them & reboot :)

On top of this, defaults gateways for internal servers are unlikely to become
unreachable, since they are on the firewall itself.
Gateways which are likely to colapse, are the ones used by the firewall, eg
ISP1 & ISP2 routers, and this will never be noticed by the internal servers
since they are not directly connected to them. Am I wrong ?
> > Let''s say we have Server A in our internal public network,
with 2 network
adapters, one
> > using 10.0.0.1 (routed through ISP1), other using 172.16.0.1 (routed
through ISP2)
> >
> > Assuming that rp_filter is configured correctly on our firewall, what
happens when a
> > client want to reach Server A using 10.0.0.1 ? What path will be used
for
the replies,
> > with what source IP ?
> 
> 	The server uses the 10.0.0.1 as source when resolving
> route for the reply. Then it depends on the routing rules.
That''s good news. So it means that if my routing rules are setup
correctly, it
will work. I was afraid of having some packets sent back using a wrong SRC IP :)

I assume however that this will work only for usual protocol such as http,
smtp, ssh... I''ll have to find a workaround for protocols like FTP, if
any.
Any idea ?
> 
> > I assume that if rp_filter is configured correctly, return path do not
matter (since we
> > don''t do NAT), but I''m worried about the SRC IP
beeing choosed for the
reply. Because, if
> > the kernel choose the src IP according to the output default route
beeing
choosed, then
> > half clients<->servers sessions will just break.
> 
> 	I''m sure you need correct routes on firewall and all internal
> hosts, for example:
> 
> ip rule add prio ... from pubnet_X to 0/0 table table_for_ISP_X
> ip rule add prio ... from pubnet_Y to 0/0 table table_for_ISP_Y
> ip rule add prio ... from 0/0 to 0/0 table your_multipath_route
> 
> 	Of course, the internal hosts use the proper firewall
> IP as gateway.
> 
Yes, of course. Firewall already have routing policy setup this way, and all
internal servers will be reconfigured in a similar way.
> 	That is all, traffic from specific pub IP should use only
> its gateway. You can expect rp_filter drops in firewall if
> the internal servers select wrong NIC by using multipath
> route for all route resolutions. The multipath route should be
> used only for source address autoselection:
> 
Ok, this is not a problem. 
But, just beeing curious, why would rp_filter drop such packets ?
I mean, we don''t really care what link is beeing used for a reply, as
soon as
the SRC IP & DST IP are correct. It''s likely that ISP1 & ISP2
router won''t do
source address validation anyway. Am I wrong ?
> - originating connections without bind()
> - selecting masquerade address (for NAT)
> - etc
> 
> 	I.e. it is a bad idea to use only multipath route.
> 
> 	Internal servers creating outgoing connections after
> using bind() to specific pubip should not reach the multipath route.
> 
Ok. I''ll ensure that multipath route is beeing used only if SRC IP
isn''t set,
on all internal servers.
> > So, basically my question is, what rules decide of the SRC IP to be
used
in a reply
> > packet on a system with several default route through different
network
interfaces ?
> 
> 	The transport and the application decide what source IP
> to put in the reply. Then they decide how to call the routing.
> The right thing to do when addresses to both ends are known is
> to feed the routing with saddr and daddr. If callers use 0.0.0.0
> as saddr when resolving routes, they will hit the multipath
> route which is bad.
> 
I''m not sure to understand this point, especially "to feed the
routing with
saddr and daddr."
I know we can instruct the routing to use a specific saddr, but what about daddr
?

> > Has anyone already experiencing such setup ?
> 
> 	Not exactly, but everything is in the details :)
Once again, you''re saving me :-)
Thanks a lot.
Regards,
Vincent.
> 
> > Vincent.
> 
> Regards
> 
> --
> Julian Anastasov <ja@ssi.bg>

--
Kelkoo.com: http://www.kelkoo.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hello,

On Sun, 17 Nov 2002, Vincent Jaussaud Mailing Listes wrote:
> On top of this, defaults gateways for internal servers are unlikely to
become
> unreachable, since they are on the firewall itself.
> Gateways which are likely to colapse, are the ones used by the firewall, eg
> ISP1 & ISP2 routers, and this will never be noticed by the internal
servers
> since they are not directly connected to them. Am I wrong ?
	Yes, this is a problem, job for user space tools to change
the routing settings on failure.
> > 	The server uses the 10.0.0.1 as source when resolving
> > route for the reply. Then it depends on the routing rules.
> That''s good news. So it means that if my routing rules are setup
correctly, it
> will work. I was afraid of having some packets sent back using a wrong SRC
IP :)
>
> I assume however that this will work only for usual protocol such as http,
> smtp, ssh... I''ll have to find a workaround for protocols like
FTP, if any.
> Any idea ?
	Should not happen for TCP servers but sometimes the UDP servers
are not smart enough when used on multihomed servers. See below.
> But, just beeing curious, why would rp_filter drop such packets ?
	Firewall with rp_filter set on internal interfaces
expects the traffic to come from the right internal interface (I
assume you have the two pubnets configured on different internal
interfaces). There is no such problem if the internal interfaces
do not use rp_filter.
> I mean, we don''t really care what link is beeing used for a reply,
as soon as
> the SRC IP & DST IP are correct. It''s likely that ISP1 &
ISP2 router won''t do
> source address validation anyway. Am I wrong ?
	If the ISPs allow spoofing then while the links are alive
there is no problem, it comes when some ISP fails. We should stop
using its addresses in this case.
> > 	The transport and the application decide what source IP
> > to put in the reply. Then they decide how to call the routing.
> > The right thing to do when addresses to both ends are known is
> > to feed the routing with saddr and daddr. If callers use 0.0.0.0
> > as saddr when resolving routes, they will hit the multipath
> > route which is bad.
> >
> I''m not sure to understand this point, especially "to feed
the routing with
> saddr and daddr."
> I know we can instruct the routing to use a specific saddr, but what about
daddr ?
	daddr is always used.

	Some examples (of course, there are other route keys used,
not shown here):

- TCP connect() for unbound socket uses saddr=0.0.0.0 daddr=REMOTE_IP.
The routing then returns the best source IP to use for this connection
after creating a connected route in the routing cache.

- TCP connect() after bind() uses saddr=LOCAL_IP daddr=REMOTE_IP

- TCP listener uses saddr=LOCAL_IP daddr=REMOTE_IP when replying to
SYN

- UDP can also avoid using 0.0.0.0 as saddr if the socket is bound
or when IP_PKTINFO contains local IP information. If the app does
not take steps to inform the kernel that this socket is bound
to some local IP when sending the packet then 0.0.0.0 is used
as src IP for the route lookup (ignoring the fact that this
UDP packet has known saddr in iphdr). So, it depends both on
transport and on app to feed the routing with the right keys.
> Vincent.
Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

On Sun, 2002-11-17 at 21:43, Julian Anastasov wrote:
> 	Hello,
> 
Hi !
> On Sun, 17 Nov 2002, Vincent Jaussaud Mailing Listes wrote:
> 
> Yes, this is a problem, job for user space tools to change
> the routing settings on failure.
Ok, I think I can manage to write some scripts to manage the routing
rules, depending on the state of the links.
> 	Should not happen for TCP servers but sometimes the UDP servers
> are not smart enough when used on multihomed servers. See below.
Ok. If all TCP Servers behaves correctly, then it''s all I need.
> 
> 	Firewall with rp_filter set on internal interfaces
> expects the traffic to come from the right internal interface (I
> assume you have the two pubnets configured on different internal
> interfaces). There is no such problem if the internal interfaces
> do not use rp_filter.
Right. And disabling rp_filter might open a security hole; so I''ll
ensure traffic always go through the right interface.
> 
> > I mean, we don''t really care what link is beeing used for a
reply, as soon as
> > the SRC IP & DST IP are correct. It''s likely that ISP1
& ISP2 router won''t do
> > source address validation anyway. Am I wrong ?
> 
> 	If the ISPs allow spoofing then while the links are alive
> there is no problem, it comes when some ISP fails. We should stop
> using its addresses in this case.
> 
Right.
> 	daddr is always used.
> 
> 	Some examples (of course, there are other route keys used,
> not shown here):
> 
> - TCP connect() for unbound socket uses saddr=0.0.0.0 daddr=REMOTE_IP.
> The routing then returns the best source IP to use for this connection
> after creating a connected route in the routing cache.
What do you mean by "unbound socket" ?
> - TCP connect() after bind() uses saddr=LOCAL_IP daddr=REMOTE_IP
> 
> - TCP listener uses saddr=LOCAL_IP daddr=REMOTE_IP when replying to
> SYN
> 
> - UDP can also avoid using 0.0.0.0 as saddr if the socket is bound
> or when IP_PKTINFO contains local IP information. If the app does
> not take steps to inform the kernel that this socket is bound
> to some local IP when sending the packet then 0.0.0.0 is used
> as src IP for the route lookup (ignoring the fact that this
> UDP packet has known saddr in iphdr). So, it depends both on
> transport and on app to feed the routing with the right keys.
> 
Ok. Seems like I''ll have to make some heavy testing. :)

Thanks again.
Vincent.
> > Vincent.
> 
> Regards
> 
> --
> Julian Anastasov <ja@ssi.bg>
> 
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
-- 
Vincent Jaussaud <tatooin@kelkoo.com>
Kelkoo.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Hello,

On 18 Nov 2002, Vincent Jaussaud wrote:
> Right. And disabling rp_filter might open a security hole; so I''ll
	For internal interfaces rp_filter is optional.
> ensure traffic always go through the right interface.
> > - TCP connect() for unbound socket uses saddr=0.0.0.0 daddr=REMOTE_IP.
> > The routing then returns the best source IP to use for this connection
> > after creating a connected route in the routing cache.
> What do you mean by "unbound socket" ?
	socket(), connect(), i.e. when there is no bind() to local addr.

Regards

--
Julian Anastasov <ja@ssi.bg>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

On Mon, 2002-11-18 at 12:59, Julian Anastasov wrote:
> > ensure traffic always go through the right interface.
> > > - TCP connect() for unbound socket uses saddr=0.0.0.0
daddr=REMOTE_IP.
> > > The routing then returns the best source IP to use for this
connection
> > > after creating a connected route in the routing cache.
> > What do you mean by "unbound socket" ?
> 
> 	socket(), connect(), i.e. when there is no bind() to local addr.
> 
> Regards
Thanks julian !

I''ll let you know how it goes.

Thanks again.
Regards,
Vincent.
> 
> --
> Julian Anastasov <ja@ssi.bg>
> 
> _______________________________________________
> LARTC mailing list / LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
-- 
Vincent Jaussaud <tatooin@kelkoo.com>
Kelkoo.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/