Hello all,
I am running shorewall-4.5.0.1 on CentOS 6.4 and having trouble getting certain
internal IP addresses to NAT out via certain interfaces. This is complicated by
the fact that I am using two different providers.
First, my providers file (boiletplate comment lines removed):
pbb 1 4 main eth1 207.71.189.129
track,balance
vbb 2 5 main eth2 217.240.176.1
track,balance
Then my masq file:
eth1 10.0.2.32/32 207.71.189.254 # mail server
eth1 10.0.2.0/24 207.71.189.130 # everything else
my tcrules:
# Per the providers file, traffic marked 4 goes out PBB while traffic marked 5
goes out VBB.
# Default everything out of PBB. Should eventually change this to VBB.
4 10.0.0.0/8 0.0.0.0/0
# All of this goes out VBB.
5 10.0.2.37 0.0.0.0/0 # post
5 10.0.2.8 0.0.0.0/0 # util1
5 10.0.2.48 0.0.0.0/0 # ftp
5 10.0.2.106 0.0.0.0/0 # rezaspider
5 10.0.2.111 0.0.0.0/0 # spider1-eth0:1
5 10.0.2.112 0.0.0.0/0 # spider1-eth0:2
5 10.0.2.113 0.0.0.0/0 # spider1-eth0:3
5 10.0.2.114 0.0.0.0/0 # spider1-eth0:4
And my rules file:
# Let the many spider1 interfaces access the outside for spidering
ACCEPT dmz:10.0.2.110 vbb tcp http
ACCEPT dmz:10.0.2.110 vbb tcp https
ACCEPT dmz:10.0.2.111 vbb tcp http
ACCEPT dmz:10.0.2.111 vbb tcp https
ACCEPT dmz:10.0.2.112 vbb tcp http
ACCEPT dmz:10.0.2.112 vbb tcp https
ACCEPT dmz:10.0.2.113 vbb tcp http
ACCEPT dmz:10.0.2.113 vbb tcp https
I''m ultimately trying to get any traffic from 10.0.2.111 to go out
217.240.176.67 and traffic from 10.0.2.112 to go out 217.240.176.68 etc. With
this config I cannot source a connection from 10.0.2.111 to any outside IP
address:
[root@spider1 ~]# curl --interface eth0:1 http://ifconfig.me
curl: (7) couldn''t connect to host
[root@spider1 ~]# /sbin/ifconfig eth0:1
eth0:1 Link encap:Ethernet HWaddr 00:16:3E:0D:15:21
inet addr:10.0.2.111 Bcast:10.0.2.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
Interrupt:23
What am I doing wrong here? I am somewhat confused on whether this sort of
masq/NAT is to be done through the masq file or the tcrules file. The first
throught is to try to do this through the masq file but the shorewall-masq
manpage says:
Warning
If you have more than one ISP link, adding entries to this file will not
force connections to go out through a particular link. You must use entries
in shorewall-rtrules[1](5) or PREROUTING entries in shorewall-tcrules[2](5)
to do that.
So that is what I am trying to do. Does this mean that the masq file serves no
purpose at all in a multi-ISP setup such as I have?
Which is preferred, rtrules or tcrules? I''m going with tcrules for now
since
that is where I''m setting my traffic with mark 4 which sends it out the
"pbb"
provider.
--
Tracy Reed
------------------------------------------------------------------------------
Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more!
Discover the easy way to master current and previous Microsoft technologies
and advance your career. Get an incredible 1,500+ hours of step-by-step
tutorial videos with LearnDevNow. Subscribe today and save!
http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk
I''ve got a few bucks available for a really good Shorewall consultant since I haven''t yet been able to figure this one out myself... On Tue, Sep 03, 2013 at 11:49:22AM PDT, Tracy Reed spake thusly:> Hello all, > > I am running shorewall-4.5.0.1 on CentOS 6.4 and having trouble getting certain > internal IP addresses to NAT out via certain interfaces. This is complicated by > the fact that I am using two different providers. > > First, my providers file (boiletplate comment lines removed): > > pbb 1 4 main eth1 207.71.189.129 track,balance > vbb 2 5 main eth2 217.240.176.1 track,balance > > Then my masq file: > > eth1 10.0.2.32/32 207.71.189.254 # mail server > eth1 10.0.2.0/24 207.71.189.130 # everything else > > my tcrules: > > # Per the providers file, traffic marked 4 goes out PBB while traffic marked 5 goes out VBB. > # Default everything out of PBB. Should eventually change this to VBB. > 4 10.0.0.0/8 0.0.0.0/0 > # All of this goes out VBB. > 5 10.0.2.37 0.0.0.0/0 # post > 5 10.0.2.8 0.0.0.0/0 # util1 > 5 10.0.2.48 0.0.0.0/0 # ftp > 5 10.0.2.106 0.0.0.0/0 # rezaspider > 5 10.0.2.111 0.0.0.0/0 # spider1-eth0:1 > 5 10.0.2.112 0.0.0.0/0 # spider1-eth0:2 > 5 10.0.2.113 0.0.0.0/0 # spider1-eth0:3 > 5 10.0.2.114 0.0.0.0/0 # spider1-eth0:4 > > And my rules file: > # Let the many spider1 interfaces access the outside for spidering > ACCEPT dmz:10.0.2.110 vbb tcp http > ACCEPT dmz:10.0.2.110 vbb tcp https > ACCEPT dmz:10.0.2.111 vbb tcp http > ACCEPT dmz:10.0.2.111 vbb tcp https > ACCEPT dmz:10.0.2.112 vbb tcp http > ACCEPT dmz:10.0.2.112 vbb tcp https > ACCEPT dmz:10.0.2.113 vbb tcp http > ACCEPT dmz:10.0.2.113 vbb tcp https > > I''m ultimately trying to get any traffic from 10.0.2.111 to go out > 217.240.176.67 and traffic from 10.0.2.112 to go out 217.240.176.68 etc. With > this config I cannot source a connection from 10.0.2.111 to any outside IP > address: > > [root@spider1 ~]# curl --interface eth0:1 http://ifconfig.me > curl: (7) couldn''t connect to host > [root@spider1 ~]# /sbin/ifconfig eth0:1 > eth0:1 Link encap:Ethernet HWaddr 00:16:3E:0D:15:21 > inet addr:10.0.2.111 Bcast:10.0.2.255 Mask:255.255.255.0 > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > Interrupt:23 > > What am I doing wrong here? I am somewhat confused on whether this sort of > masq/NAT is to be done through the masq file or the tcrules file. The first > throught is to try to do this through the masq file but the shorewall-masq > manpage says: > > Warning > If you have more than one ISP link, adding entries to this file will not > force connections to go out through a particular link. You must use entries > in shorewall-rtrules[1](5) or PREROUTING entries in shorewall-tcrules[2](5) > to do that. > > So that is what I am trying to do. Does this mean that the masq file serves no > purpose at all in a multi-ISP setup such as I have? > > Which is preferred, rtrules or tcrules? I''m going with tcrules for now since > that is where I''m setting my traffic with mark 4 which sends it out the "pbb" > provider. > > -- > Tracy Reed> ------------------------------------------------------------------------------ > Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! > Discover the easy way to master current and previous Microsoft technologies > and advance your career. Get an incredible 1,500+ hours of step-by-step > tutorial videos with LearnDevNow. Subscribe today and save! > http://pubads.g.doubleclick.net/gampad/clk?id=58040911&iu=/4140/ostg.clktrk> _______________________________________________ > Shorewall-users mailing list > Shorewall-users@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/shorewall-users-- Tracy Reed, RHCE Digital signature attached for your safety. Copilotco PCI/HIPAA/SOX Compliant Secure Hosting 866-MY-COPILOT x101 http://copilotco.com ------------------------------------------------------------------------------ Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk
On 5 Sep 2013, at 22:40, Tracy Reed <treed@ultraviolet.org> wrote:> I''ve got a few bucks available for a really good Shorewall consultant since I > haven''t yet been able to figure this one out myself... > > On Tue, Sep 03, 2013 at 11:49:22AM PDT, Tracy Reed spake thusly: >> Hello all, >> >> I am running shorewall-4.5.0.1 on CentOS 6.4 and having trouble getting certain >> internal IP addresses to NAT out via certain interfaces. This is complicated by >> the fact that I am using two different providers. >> >> First, my providers file (boiletplate comment lines removed): >> >> pbb 1 4 main eth1 207.71.189.129 track,balance >> vbb 2 5 main eth2 217.240.176.1 track,balance >> >> Then my masq file: >> >> eth1 10.0.2.32/32 207.71.189.254 # mail server >> eth1 10.0.2.0/24 207.71.189.130 # everything else >> >> my tcrules: >> >> # Per the providers file, traffic marked 4 goes out PBB while traffic marked 5 goes out VBB. >> # Default everything out of PBB. Should eventually change this to VBB. >> 4 10.0.0.0/8 0.0.0.0/0 >> # All of this goes out VBB. >> 5 10.0.2.37 0.0.0.0/0 # post >> 5 10.0.2.8 0.0.0.0/0 # util1 >> 5 10.0.2.48 0.0.0.0/0 # ftp >> 5 10.0.2.106 0.0.0.0/0 # rezaspider >> 5 10.0.2.111 0.0.0.0/0 # spider1-eth0:1 >> 5 10.0.2.112 0.0.0.0/0 # spider1-eth0:2 >> 5 10.0.2.113 0.0.0.0/0 # spider1-eth0:3 >> 5 10.0.2.114 0.0.0.0/0 # spider1-eth0:4 >> >> And my rules file: >> # Let the many spider1 interfaces access the outside for spidering >> ACCEPT dmz:10.0.2.110 vbb tcp http >> ACCEPT dmz:10.0.2.110 vbb tcp https >> ACCEPT dmz:10.0.2.111 vbb tcp http >> ACCEPT dmz:10.0.2.111 vbb tcp https >> ACCEPT dmz:10.0.2.112 vbb tcp http >> ACCEPT dmz:10.0.2.112 vbb tcp https >> ACCEPT dmz:10.0.2.113 vbb tcp http >> ACCEPT dmz:10.0.2.113 vbb tcp https >> >> I''m ultimately trying to get any traffic from 10.0.2.111 to go out >> 217.240.176.67 and traffic from 10.0.2.112 to go out 217.240.176.68 etc. With >> this config I cannot source a connection from 10.0.2.111 to any outside IP >> address: >> >> [root@spider1 ~]# curl --interface eth0:1 http://ifconfig.me >> curl: (7) couldn''t connect to host >> [root@spider1 ~]# /sbin/ifconfig eth0:1 >> eth0:1 Link encap:Ethernet HWaddr 00:16:3E:0D:15:21 >> inet addr:10.0.2.111 Bcast:10.0.2.255 Mask:255.255.255.0 >> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 >> Interrupt:23 >> >> What am I doing wrong here? I am somewhat confused on whether this sort of >> masq/NAT is to be done through the masq file or the tcrules file. The first >> throught is to try to do this through the masq file but the shorewall-masq >> manpage says: >> >> Warning >> If you have more than one ISP link, adding entries to this file will not >> force connections to go out through a particular link. You must use entries >> in shorewall-rtrules[1](5) or PREROUTING entries in shorewall-tcrules[2](5) >> to do that. >> >> So that is what I am trying to do. Does this mean that the masq file serves no >> purpose at all in a multi-ISP setup such as I have?I believe that the point of this warning is that masq entries alone do not control choice of outgoing interface - what they do is configure source NAT *given* an output interface. So I think you need to add a masq entry for eth2 so that traffic exiting that interface gets snat-ted as required.>> >> Which is preferred, rtrules or tcrules? I''m going with tcrules for now since >> that is where I''m setting my traffic with mark 4 which sends it out the "pbb" >> provider.Pass. The implication is that your existing config here is working, though - otherwise it would either work from all source IPs or none.>> >> -- >> Tracy Reed > >Dominic ------------------------------------------------------------------------------ Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk
On Thu, Sep 5, 2013 at 2:40 PM, Tracy Reed <treed@ultraviolet.org> wrote:> I''ve got a few bucks available for a really good Shorewall consultant > since I > haven''t yet been able to figure this one out myself... > > On Tue, Sep 03, 2013 at 11:49:22AM PDT, Tracy Reed spake thusly: > > Hello all, > > > > I am running shorewall-4.5.0.1 on CentOS 6.4 and having trouble getting > certain > > internal IP addresses to NAT out via certain interfaces. This is > complicated by > > the fact that I am using two different providers. > > > > First, my providers file (boiletplate comment lines removed): > > > > pbb 1 4 main eth1 207.71.189.129 > track,balance > > vbb 2 5 main eth2 217.240.176.1 > track,balance > > > > Then my masq file: > > > > eth1 10.0.2.32/32 207.71.189.254 # mail server > > eth1 10.0.2.0/24 207.71.189.130 # everything > else > > > > my tcrules: > > > > # Per the providers file, traffic marked 4 goes out PBB while traffic > marked 5 goes out VBB. > > # Default everything out of PBB. Should eventually change this to VBB. > > 4 10.0.0.0/8 0.0.0.0/0 > > # All of this goes out VBB. > > 5 10.0.2.37 0.0.0.0/0 # post > > 5 10.0.2.8 0.0.0.0/0 # util1 > > 5 10.0.2.48 0.0.0.0/0 # ftp > > 5 10.0.2.106 0.0.0.0/0 # rezaspider > > 5 10.0.2.111 0.0.0.0/0 # spider1-eth0:1 > > 5 10.0.2.112 0.0.0.0/0 # spider1-eth0:2 > > 5 10.0.2.113 0.0.0.0/0 # spider1-eth0:3 > > 5 10.0.2.114 0.0.0.0/0 # spider1-eth0:4 > > > > And my rules file: > > # Let the many spider1 interfaces access the outside for spidering > > ACCEPT dmz:10.0.2.110 vbb tcp http > > ACCEPT dmz:10.0.2.110 vbb tcp https > > ACCEPT dmz:10.0.2.111 vbb tcp http > > ACCEPT dmz:10.0.2.111 vbb tcp https > > ACCEPT dmz:10.0.2.112 vbb tcp http > > ACCEPT dmz:10.0.2.112 vbb tcp https > > ACCEPT dmz:10.0.2.113 vbb tcp http > > ACCEPT dmz:10.0.2.113 vbb tcp https > > > > I''m ultimately trying to get any traffic from 10.0.2.111 to go out > > 217.240.176.67 and traffic from 10.0.2.112 to go out 217.240.176.68 etc. > With > > this config I cannot source a connection from 10.0.2.111 to any outside > IP > > address: > > > > [root@spider1 ~]# curl --interface eth0:1 http://ifconfig.me > > curl: (7) couldn''t connect to host > > [root@spider1 ~]# /sbin/ifconfig eth0:1 > > eth0:1 Link encap:Ethernet HWaddr 00:16:3E:0D:15:21 > > inet addr:10.0.2.111 Bcast:10.0.2.255 Mask:255.255.255.0 > > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > > Interrupt:23 > > > > What am I doing wrong here? I am somewhat confused on whether this sort > of > > masq/NAT is to be done through the masq file or the tcrules file. The > first > > throught is to try to do this through the masq file but the > shorewall-masq > > manpage says: > > > > Warning > > If you have more than one ISP link, adding entries to this file will > not > > force connections to go out through a particular link. You must use > entries > > in shorewall-rtrules[1](5) or PREROUTING entries in > shorewall-tcrules[2](5) > > to do that. > > > > So that is what I am trying to do. Does this mean that the masq file > serves no > > purpose at all in a multi-ISP setup such as I have? > > > > Which is preferred, rtrules or tcrules? I''m going with tcrules for now > since > > that is where I''m setting my traffic with mark 4 which sends it out the > "pbb" > > provider. > > > > -- > > Tracy Reed > > > Traffic control can be implemented in a few ways, YMMV but my experienceis that tcrules marks the packets. masq changes the address when exiting an interface. what you are missing is entries in rtrules to decide where the packets go. I''ve extracted from my configuration the following example: rtrules: The following states the 192.168.1.0/24 subnet routes via provider "blkmDP" with a priority of 26101 192.168.1.0/24 - blkmDP 26101 providers: This provider has an IP of 10.3.11.64 from my perspective gw is 10.3.11.1 blkmDP 4 103 - eth1.9:10.3.11.64 10.3.11.1 loose masq: everything from 192.168.1.0/24 going via eth1.9 should become 10.3.11.64 eth1.9 192.168.1.0/24 10.3.11.64 tcrules: nothing interesting here. empty I believe there are some flags in the shorewall.conf that may change the above behaviour though, so maybe relevant: TCTC_ENABLED=Yes TC_EXPERT=No CLEAR_TC=Yes WIDE_TC_MARKS=No ------------------------------------------------------------------------------ Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk
On 9/3/2013 11:49 AM, Tracy Reed wrote:> Hello all, > > I am running shorewall-4.5.0.1 on CentOS 6.4 and having trouble getting certain > internal IP addresses to NAT out via certain interfaces. This is complicated by > the fact that I am using two different providers. > > First, my providers file (boiletplate comment lines removed): > > pbb 1 4 main eth1 207.71.189.129 track,balance > vbb 2 5 main eth2 217.240.176.1 track,balance > > Then my masq file: > > eth1 10.0.2.32/32 207.71.189.254 # mail server > eth1 10.0.2.0/24 207.71.189.130 # everything elseYou have no masq entry for eth2?> > my tcrules: > > # Per the providers file, traffic marked 4 goes out PBB while traffic marked 5 goes out VBB. > # Default everything out of PBB. Should eventually change this to VBB. > 4 10.0.0.0/8 0.0.0.0/0 > # All of this goes out VBB. > 5 10.0.2.37 0.0.0.0/0 # post > 5 10.0.2.8 0.0.0.0/0 # util1 > 5 10.0.2.48 0.0.0.0/0 # ftp > 5 10.0.2.106 0.0.0.0/0 # rezaspider > 5 10.0.2.111 0.0.0.0/0 # spider1-eth0:1 > 5 10.0.2.112 0.0.0.0/0 # spider1-eth0:2 > 5 10.0.2.113 0.0.0.0/0 # spider1-eth0:3 > 5 10.0.2.114 0.0.0.0/0 # spider1-eth0:4What is your setting of MARK_IN_FORWARD_CHAIN in shorewall.conf? If it is ''Yes'', then you need to add '':P'' to the mark values.> > And my rules file: > # Let the many spider1 interfaces access the outside for spidering > ACCEPT dmz:10.0.2.110 vbb tcp http > ACCEPT dmz:10.0.2.110 vbb tcp https > ACCEPT dmz:10.0.2.111 vbb tcp http > ACCEPT dmz:10.0.2.111 vbb tcp https > ACCEPT dmz:10.0.2.112 vbb tcp http > ACCEPT dmz:10.0.2.112 vbb tcp https > ACCEPT dmz:10.0.2.113 vbb tcp http > ACCEPT dmz:10.0.2.113 vbb tcp https > > I''m ultimately trying to get any traffic from 10.0.2.111 to go out > 217.240.176.67 and traffic from 10.0.2.112 to go out 217.240.176.68 etc. With > this config I cannot source a connection from 10.0.2.111 to any outside IP > address: > > [root@spider1 ~]# curl --interface eth0:1 http://ifconfig.me > curl: (7) couldn''t connect to host > [root@spider1 ~]# /sbin/ifconfig eth0:1 > eth0:1 Link encap:Ethernet HWaddr 00:16:3E:0D:15:21 > inet addr:10.0.2.111 Bcast:10.0.2.255 Mask:255.255.255.0 > UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 > Interrupt:23 > > What am I doing wrong here?Is the default gateway set correctly on spider?> I am somewhat confused on whether this sort of > masq/NAT is to be done through the masq file or the tcrules file. The first > throught is to try to do this through the masq file but the shorewall-masq > manpage says: > > Warning > If you have more than one ISP link, adding entries to this file will not > force connections to go out through a particular link. You must use entries > in shorewall-rtrules[1](5) or PREROUTING entries in shorewall-tcrules[2](5) > to do that. > > So that is what I am trying to do. Does this mean that the masq file serves no > purpose at all in a multi-ISP setup such as I have?No. If traffic with a private source IP address exits on an Internet interface, then the masq entry for that interface rewrites the source address so that return traffic can be routed correctly back to your Shorewall box.> > Which is preferred, rtrules or tcrules? I''m going with tcrules for now since > that is where I''m setting my traffic with mark 4 which sends it out the "pbb" > provider.I always use rtrules when they are sufficient. So if you are selecting your provider based solely on source IP address, then i like rtrules. -Tom -- Tom Eastep \ When I die, I want to go like my Grandfather who Shoreline, \ died peacefully in his sleep. Not screaming like Washington, USA \ all of the passengers in his car http://shorewall.net \________________________________________________ ------------------------------------------------------------------------------ Learn the latest--Visual Studio 2012, SharePoint 2013, SQL 2012, more! Discover the easy way to master current and previous Microsoft technologies and advance your career. Get an incredible 1,500+ hours of step-by-step tutorial videos with LearnDevNow. Subscribe today and save! http://pubads.g.doubleclick.net/gampad/clk?id=58041391&iu=/4140/ostg.clktrk
On Thu, Sep 05, 2013 at 04:43:41PM PDT, Tom Eastep spake thusly:> On 9/3/2013 11:49 AM, Tracy Reed wrote: > > Then my masq file: > > > > eth1 10.0.2.32/32 207.71.189.254 # mail server > > eth1 10.0.2.0/24 207.71.189.130 # everything else > > You have no masq entry for eth2?No, I did not at the time. But now I see that I do need such an entry.> What is your setting of MARK_IN_FORWARD_CHAIN in shorewall.conf? If it > is ''Yes'', then you need to add '':P'' to the mark values.It is No.> Is the default gateway set correctly on spider?Yes.> No. If traffic with a private source IP address exits on an Internet > interface, then the masq entry for that interface rewrites the source > address so that return traffic can be routed correctly back to your > Shorewall box.So I see...> I always use rtrules when they are sufficient. So if you are selecting > your provider based solely on source IP address, then i like rtrules.Between your comments and those of Dominic Benson and Lee Brown I now have a complete understanding of the problem and what I was doing wrong. I added the appropriate masq and rtrules entries and now it works perfectly! Thank you all! And thank you Tom for an awesome firewall configuration system. I have been using Shorewall for around 10 years now! -- Tracy Reed ------------------------------------------------------------------------------ How ServiceNow helps IT people transform IT departments: 1. Consolidate legacy IT systems to a single system of record for IT 2. Standardize and globalize service processes across IT 3. Implement zero-touch automation to replace manual, redundant tasks http://pubads.g.doubleclick.net/gampad/clk?id=51271111&iu=/4140/ostg.clktrk