Adam Niedzwiedzki
2008-Jan-08 05:54 UTC
Shorewall and LVS-NAT (via fwmark) nat''d machines can''t access the outside world directly
Hi guys,
I''m not sure where to post for help on this one, shorewall or lvs,
I''ll
start with shorewall (only cause Tom is a gun at this stuff, and is polite
enough to tell me to bugger off to the LVS list if I''m posting in the
wrong
one ;)
I have a single box that is my router/firewall/LVS.
Internet -- eth0 - router/firewall - eth1 --- internal lan
|
eth2 LVS-NAT setup
With LVS setup as LVS-NAT, everything works a treat as in load balancing
from the internet is spread across my 2 servers that are connected to eth2.
LVS seems to handling the NAT/masq just nicely, and plays nice with
shorewall via using fwmark.
BUT, my "realservers" (192.168.1.x connected via eth2) can''t
access anything
externally say dns lookups, routing mail, whatever, ie anything originating
from the box apart from web traffic which LVS-NAT is handling. (which I
figured is right, but I don''t know how to "fix" it so they
can).
This is the issue, how can I setup shorewall to allow the
"realservers"
access to the internet, if it is shorewall that I should be trying to make
this happen with.
Should I MASQ eth2 in shorewall? Will this then break LVS-NAT doing the masq
on the incoming stuff?
PART B: to all this is maybe changing LVS to DR (direct routing), so that my
machines are connected via eth1 with 202.45.102.x ip''s etc, but I
can''t seem
to see if that will play nice with shorewall, I read something about a patch
needed for connecting tracking (but I thought the fwmark got around all
that), not going here unless I can''t get my NAT''d machines to
talk
externally :s....
202.45.102.90 is the virtual IP added to eth1 via LVS
*snipped* setup files..
/etc/shorewall/zones
fw firewall
net ipv4
loc ipv4
lvs ipv4
/etc/shorewall/interfaces
net eth0 detect
loc eth1 detect
lvs eth2 detect routeback
/etc/shorewall/policy
lvs net ACCEPT
fw lvs ACCEPT
lvs fw ACCEPT
/etc/shorewall/tcrules
1 0.0.0.0/0 202.45.102.90 tcp 80
/etc/shorewall/rules
ACCEPT all fw:202.45.102.90 tcp 80
My keepalived config is very simple
virtual_server fwmark 1 {
delay_loop 6
lb_algo rr
lb_kind NAT
protocol TCP
real_server 192.168.1.10 80 {
weight 1
TCP_CHECK {
connect_timeout 3
connect_port 80
}
}
real_server 192.168.1.11 80 {
weight 1
TCP_CHECK {
connect_timeout 3
connect_port 80
}
}
}
Other stuff
firewall# ip addr
3: eth0: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:40:d0:43:b7:cc brd ff:ff:ff:ff:ff:ff
inet 202.45.103.86/30 brd 202.45.103.87 scope global eth0
4: eth1: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:0e:0c:d0:71:bc brd ff:ff:ff:ff:ff:ff
inet 202.45.102.1/25 brd 202.45.102.127 scope global eth1
inet 202.45.102.90/32 scope global eth1
inet 202.45.102.91/32 scope global eth1
inet 202.45.102.92/32 scope global eth1
inet 202.45.102.93/32 scope global eth1
5: eth2: <BROADCAST,MULTICAST,UP> mtu 1500 qdisc pfifo_fast qlen 1000
link/ether 00:40:d0:43:b7:cd brd ff:ff:ff:ff:ff:ff
inet 192.168.1.1/24 brd 192.168.1.255 scope global eth2
firewall# ip route
202.45.103.84/30 dev eth0 proto kernel scope link src 202.45.103.86
202.45.102.0/25 dev eth1 proto kernel scope link src 202.45.102.1
192.168.1.0/24 dev eth2 proto kernel scope link src 192.168.1.1
default via 202.45.103.85 dev eth0 proto zebra equalize
Cheers
Adam
-------------------------------------------------------------------------
Check out the new SourceForge.net Marketplace.
It''s the best place to buy or sell services for
just about anything Open Source.
http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
Tom Eastep
2008-Jan-08 06:07 UTC
Re: Shorewall and LVS-NAT (via fwmark) nat''d machines can''t access the outside world directly
Adam Niedzwiedzki wrote:> > This is the issue, how can I setup shorewall to allow the "realservers" > access to the internet, if it is shorewall that I should be trying to make > this happen with. > > Should I MASQ eth2 in shorewall?Yes. That or run a proxy on the Shorewall box.> Will this then break LVS-NAT doing the masq on the incoming stuff?I shouldn''t think so.> > PART B: to all this is maybe changing LVS to DR (direct routing), so that my > machines are connected via eth1 with 202.45.102.x ip''s etc, but I can''t seem > to see if that will play nice with shorewall, I read something about a patch > needed for connecting tracking (but I thought the fwmark got around all > that), not going here unless I can''t get my NAT''d machines to talk > externally :s....You''ve exhausted my limited knowledge of LVS. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
Nathan Gibbs
2008-Jan-08 22:37 UTC
Re: Shorewall and LVS-NAT (via fwmark) nat''d machines can''t access the outside world directly
* Tom Eastep wrote:> Adam Niedzwiedzki wrote: > >> >> This is the issue, how can I setup shorewall to allow the "realservers" >> access to the internet, if it is shorewall that I should be trying to >> make >> this happen with. >> >> Should I MASQ eth2 in shorewall? > > Yes. That or run a proxy on the Shorewall box.Yep> >> Will this then break LVS-NAT doing the masq on the incoming stuff? > > I shouldn''t think so. >No, thats what we do. Shorewall masqing all other interfaces / internal nets through our internet interface. LVS masqing incoming traffic from the internet to our realservers. The only trouble we had was forgetting to put an ACCEPT rule in net to fw for the ports handled by LVS. :-) Other than that, it has worked solid for over two years. We haven''t even had to mess with fwmark. ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
Adam Niedzwiedzki
2008-Jan-08 23:14 UTC
Re: Shorewall and LVS-NAT (via fwmark) nat''d machines can''t access the outside world directly
Hi guys, Ok I went to masq the LVS interface and realised I "think" I have an issue.. This machine IS my router AS well as my firewall and my load balancer... Internet -- eth0 - router/firewall - eth1 --- internal lan | eth2 LVS-NAT setup Hence eth0 is connected to my upstream, eth1 isn''t masq''d it''s routed and eth2 is my LVS NIC (which is handled by LVS) (which I want to masq) I''m sure I''ve missed something simple. /etc/shorewall/masq #INTERFACE SUBNET ADDRESS PROTO PORT(S) IPSEC eth1 eth2 *snipped* setup files.. /etc/shorewall/zones fw firewall net ipv4 loc ipv4 lvs ipv4 /etc/shorewall/interfaces net eth0 detect loc eth1 detect lvs eth2 detect routeback /etc/shorewall/policy lvs net ACCEPT fw lvs ACCEPT lvs fw ACCEPT LVS has access to net via the policy file, but after restarting shorewall, my machines still can''t get "out" to the internet. Cheers Ad -----Original Message----- From: shorewall-users-bounces@lists.sourceforge.net [mailto:shorewall-users-bounces@lists.sourceforge.net] On Behalf Of Nathan Gibbs Sent: Wednesday, 9 January 2008 9:37 AM To: Shorewall Users Subject: Re: [Shorewall-users] Shorewall and LVS-NAT (via fwmark) nat''d machines can''t access the outside world directly * Tom Eastep wrote:> Adam Niedzwiedzki wrote: > >> >> This is the issue, how can I setup shorewall to allow the "realservers" >> access to the internet, if it is shorewall that I should be trying to >> make >> this happen with. >> >> Should I MASQ eth2 in shorewall? > > Yes. That or run a proxy on the Shorewall box.Yep> >> Will this then break LVS-NAT doing the masq on the incoming stuff? > > I shouldn''t think so. >No, thats what we do. Shorewall masqing all other interfaces / internal nets through our internet interface. LVS masqing incoming traffic from the internet to our realservers. The only trouble we had was forgetting to put an ACCEPT rule in net to fw for the ports handled by LVS. :-) Other than that, it has worked solid for over two years. We haven''t even had to mess with fwmark. ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace _______________________________________________ Shorewall-users mailing list Shorewall-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/shorewall-users ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
Tom Eastep
2008-Jan-08 23:48 UTC
Re: Shorewall and LVS-NAT (via fwmark) nat''d machines can''t access the outside world directly
Adam Niedzwiedzki wrote:> Hi guys, > > Ok I went to masq the LVS interface and realised I "think" I have an issue.. > > This machine IS my router AS well as my firewall and my load balancer... > > Internet -- eth0 - router/firewall - eth1 --- internal lan > | > eth2 LVS-NAT setup > > Hence eth0 is connected to my upstream, > eth1 isn''t masq''d it''s routed and eth2 is my LVS NIC (which is handled by > LVS) (which I want to masq) > > I''m sure I''ve missed something simple. > > /etc/shorewall/masq > #INTERFACE SUBNET ADDRESS PROTO PORT(S) > IPSEC > eth1 eth2---- Wrong interface.> > *snipped* setup files.. > /etc/shorewall/zones > fw firewall > net ipv4 > loc ipv4 > lvs ipv4 > > /etc/shorewall/interfaces > net eth0 detect------------ -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
Adam Niedzwiedzki
2008-Jan-08 23:54 UTC
Re: Shorewall and LVS-NAT (via fwmark) nat''d machines can''t access the outside world directly
Ahh ok, I''m confusing myself :( If I put an entry in the /etc/shorewall/nat do I have to setup /etc/shorewall/masq The machine/s behind LVS will need to connect via an External IP other then the router/firewall one... Hence why I masq behind eth1 Remember this machine is my router as well (eth0 has a /30 with my upstream) eth1 is my /25 Cheers Ad -----Original Message----- From: shorewall-users-bounces@lists.sourceforge.net [mailto:shorewall-users-bounces@lists.sourceforge.net] On Behalf Of Tom Eastep Sent: Wednesday, 9 January 2008 10:49 AM To: adstar@genis-x.com; Shorewall Users Subject: Re: [Shorewall-users] Shorewall and LVS-NAT (via fwmark) nat''d machines can''t access the outside world directly Adam Niedzwiedzki wrote:> Hi guys, > > Ok I went to masq the LVS interface and realised I "think" I have anissue..> > This machine IS my router AS well as my firewall and my load balancer... > > Internet -- eth0 - router/firewall - eth1 --- internal lan > | > eth2 LVS-NAT setup > > Hence eth0 is connected to my upstream, > eth1 isn''t masq''d it''s routed and eth2 is my LVS NIC (which is handled by > LVS) (which I want to masq) > > I''m sure I''ve missed something simple. > > /etc/shorewall/masq > #INTERFACE SUBNET ADDRESS PROTO PORT(S) > IPSEC > eth1 eth2---- Wrong interface.> > *snipped* setup files.. > /etc/shorewall/zones > fw firewall > net ipv4 > loc ipv4 > lvs ipv4 > > /etc/shorewall/interfaces > net eth0 detect------------ -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
Tom Eastep
2008-Jan-08 23:59 UTC
Re: Shorewall and LVS-NAT (via fwmark) nat''d machines can''t access the outside world directly
Adam Niedzwiedzki wrote:> Ahh ok, > > I''m confusing myself :( > If I put an entry in the /etc/shorewall/nat do I have to setup > /etc/shorewall/masq > The machine/s behind LVS will need to connect via an External IP other then > the router/firewall one... > > Hence why I masq behind eth1 > > Remember this machine is my router as well (eth0 has a /30 with my upstream) > eth1 is my /25 >If you want the hosts on eth2 to use a different external IP address, you put that address in the ADDRESS column of the masq file entry. This isn''t brain surgery.... -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
Tom Eastep
2008-Jan-09 03:24 UTC
Re: Shorewall and LVS-NAT (via fwmark) nat''d machines can''t access the outside world directly
Tom Eastep wrote:> Adam Niedzwiedzki wrote: >> Ahh ok, >> I''m confusing myself :( >> If I put an entry in the /etc/shorewall/nat do I have to setup >> /etc/shorewall/masq The machine/s behind LVS will need to connect via >> an External IP other then >> the router/firewall one... >> >> Hence why I masq behind eth1 >> >> Remember this machine is my router as well (eth0 has a /30 with my >> upstream) >> eth1 is my /25 >> > > If you want the hosts on eth2 to use a different external IP address, > you put that address in the ADDRESS column of the masq file entry. >Your statement ''Hence why I masq behind eth1'' indicates that you may not be viewing the relationship between the host, the interfaces and the addresses properly. Lets say that a Linux system has IP addresses IP1, IP2, IP3 and IP4 and that it has interfaces IF1, IF2, and IF3. The way that I mentally picture this system is like this: __________ | IF1 | _______________|__________|______________ | | | | | |___ | IP1 | | | IP2 | | | IP3 |IF2| | IP4 | | | |___| | | | | |_________________________________________| | | | IF3 | ---------- This view emphasizes the fact that the IP addresses belong to the *host* and not to the Interfaces. In the Linux default mode of operation, an ARP ''who-has'' request for any of the addresses received on any of the (ethernet) interfaces, will be responded to with the MAC address of that interface. Each IP address is configured on an interface but the address<->interface relationship only really important in two cases: a) when the system is sending a packet that doesn''t have an address (the local client has bound its sending socket to the 0 address) b) in MASQUERADE when you are letting the system pick the source address to use for some outgoing packets. So for traffic leaving the system on IF1, you can pick any of the IP addresses (IP1-4) as the SNAT source address provided that responses from the target host with that destination IP address will be routed back to this system. In fact, you can use the IP address of any host accessed via IF2 or IF3 if that address meets the criteria that traffic sent to that address from the recipient will be routed back to this system. HTH -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace
Adam Niedzwiedzki
2008-Jan-09 04:07 UTC
Re: Shorewall and LVS-NAT (via fwmark) nat''d machines can''t access the outside world directly
Thank you Tom, That has cleared everything up for me. I was "tying" the IP''s to there specific interfaces, and getting bogged down in details..... or as the classic phrase goes "Step back and look at the bigger picture" which you clearly illustrated for me. Thank you again Tom Cheers Adam -----Original Message----- From: shorewall-users-bounces@lists.sourceforge.net [mailto:shorewall-users-bounces@lists.sourceforge.net] On Behalf Of Tom Eastep Sent: Wednesday, 9 January 2008 2:25 PM To: Shorewall Users Subject: Re: [Shorewall-users] Shorewall and LVS-NAT (via fwmark) nat''d machines can''t access the outside world directly Tom Eastep wrote:> Adam Niedzwiedzki wrote: >> Ahh ok, >> I''m confusing myself :( >> If I put an entry in the /etc/shorewall/nat do I have to setup >> /etc/shorewall/masq The machine/s behind LVS will need to connect via >> an External IP other then the router/firewall one... >> >> Hence why I masq behind eth1 >> >> Remember this machine is my router as well (eth0 has a /30 with my >> upstream) >> eth1 is my /25 >> > > If you want the hosts on eth2 to use a different external IP address, > you put that address in the ADDRESS column of the masq file entry. >Your statement ''Hence why I masq behind eth1'' indicates that you may not be viewing the relationship between the host, the interfaces and the addresses properly. Lets say that a Linux system has IP addresses IP1, IP2, IP3 and IP4 and that it has interfaces IF1, IF2, and IF3. The way that I mentally picture this system is like this: __________ | IF1 | _______________|__________|______________ | | | | | |___ | IP1 | | | IP2 | | | IP3 |IF2| | IP4 | | | |___| | | | | |_________________________________________| | | | IF3 | ---------- This view emphasizes the fact that the IP addresses belong to the *host* and not to the Interfaces. In the Linux default mode of operation, an ARP ''who-has'' request for any of the addresses received on any of the (ethernet) interfaces, will be responded to with the MAC address of that interface. Each IP address is configured on an interface but the address<->interface relationship only really important in two cases: a) when the system is sending a packet that doesn''t have an address (the local client has bound its sending socket to the 0 address) b) in MASQUERADE when you are letting the system pick the source address to use for some outgoing packets. So for traffic leaving the system on IF1, you can pick any of the IP addresses (IP1-4) as the SNAT source address provided that responses from the target host with that destination IP address will be routed back to this system. In fact, you can use the IP address of any host accessed via IF2 or IF3 if that address meets the criteria that traffic sent to that address from the recipient will be routed back to this system. HTH -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key ------------------------------------------------------------------------- Check out the new SourceForge.net Marketplace. It''s the best place to buy or sell services for just about anything Open Source. http://ad.doubleclick.net/clk;164216239;13503038;w?http://sf.net/marketplace