Hi all, I''m having a problem with multiple internet providers. According to all the things I have found so far it should not be that uncomplicated to set it up - however, I''m having some problems getting it to work. Firsth the scenario - I''m having 2 Internet providers, one fast with 32 static IP''s and one slow with dynamic IP''s. Internally we are using a private net which is set up for 32 ip''s. Now, we have to map all 32 static ip''s to the private IP''s so all machines can be reached externally through the static net. The default route is suppose to be the slow ISP. So far, I have found that I need to set up the routing using marking, and then set up the rules so it work accordingly. But this appears to just work with static routing, not when one of the providers is dynamic! Does anyone have some suggestions for what to do? /Kim _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Kim, : I''m having a problem with multiple internet providers. According to all : the things I have found so far it should not be that uncomplicated to : set it up - however, I''m having some problems getting it to work. You are correct. It is not uncomplicated. It is also not difficult. :) : Firsth the scenario - I''m having 2 Internet providers, one fast with 32 : static IP''s and one slow with dynamic IP''s. Internally we are using a : private net which is set up for 32 ip''s. Now, we have to map all 32 : static ip''s to the private IP''s so all machines can be reached : externally through the static net. The default route is suppose to be : the slow ISP. : : So far, I have found that I need to set up the routing using marking, : and then set up the rules so it work accordingly. That''s one approach, documented here: http://linux-ip.net/html/adv-multi-internet.html This only works if you are not trying to perform link load balancing. If that is your intended goal, you''ll need to check out the Nano-HOWTO (Hey Julian--this appears to be missing right now): http://www.linuxvirtualserver.org/~julian/nano.txt : But this appears to just work with static routing, not when one of the : providers is dynamic! It doesn''t matter that the IPs are dynamic, just that you can alter the affected routing table when you get a new dynamic IP. If you are using rp-pppoe, you''ll want to perform your additional routing table manipulations in ip-up.local. And also, unless you have a routing daemon, you are using static routing. A dynamic IP address is simply that--your network-connected machine is still performing static routing, though it has a dynamic address. Since you have a dynamic IP on one interface, you may wish to use the -j MASQUERADE target instead of the -j SNAT target. : Does anyone have some suggestions for what to do? Do the above suggestions help? -Martin -- Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hello, On Tue, 18 Mar 2003, Martin A. Brown wrote:> that is your intended goal, you''ll need to check out the Nano-HOWTO (Hey > Julian--this appears to be missing right now): > > http://www.linuxvirtualserver.org/~julian/nano.txtThe source site is: http://www.ssi.bg/~ja/ The old URL is still maintained but as mirror, not under my control, may be it is now back as redirect (http://www.linuxvirtualserver.org/~julian/) to the primary site. Regards -- Julian Anastasov <ja@ssi.bg> _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Tuesday 18 March 2003 17:47, Martin A. Brown wrote:> That''s one approach, documented here: > > http://linux-ip.net/html/adv-multi-internet.html >Hi Martin, Thanks for the help, I got started but have a strange problem. According to the adv-multi-internet.html doc, I should set up a couple of simple rules, first the IP routing, which seems to be running smoothely, secondly, I have to set up the iptables rules as well, this is also fairly straightforward. Now, the interesting (or actually not!) things begin, my box gets packets in, which is suppose to be routed, I can DNAT them, but after that they simply disappear - they never reach the forward chain!!! Is there a reason why? /Kim _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hi there Kim, : Thanks for the help, I got started but have a strange problem. : According to the adv-multi-internet.html doc, I should set up a couple : of simple rules, first the IP routing, which seems to be running : smoothely, secondly, I have to set up the iptables rules as well, this : is also fairly straightforward. Does that mean you have had success passing packets over multiple links? Or perhaps no success yet? : Now, the interesting (or actually not!) things begin, Sometimes, interesting is not good. But, at least, interesting is the way to understanding.... : my box gets packets in, which is suppose to be routed, I can DNAT them, : but after that they simply disappear - they never reach the forward : chain!!! Is there a reason why? Sounds like a routing problem. Post "ip rule show", and "ip route show table $TABLEID" for your main routing table, and the other routing tables, and if you can, a textual description of the network, or a simple ASCII network map. -Martin -- Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Kim,
Well, I *think* the solution is a simple sysctl twiddle (see close to the
bottom), but I left my reasoning and research intact on the off-chance
that my method would benefit you or posterity reviewing this thread..
iptables mangle thoughts
- - - - - - - - - - - - -
I have line broken the iptables output, but I''m curious to know if you
are
seeing any activity in these particular chains:
# iptables -t mangle -nvL FORWARD
<snip>
0 0 MARK all -- * eth1 0.0.0.0/0
0.0.0.0/0 MARK set 0x1
0 0 MARK all -- * * 212.202.108.160/27 \
0.0.0.0/0 MARK set 0x1
0 0 MARK all -- * * 0.0.0.0/0 \
212.202.108.160/27 MARK set 0x1
</snip>
I expect that the counter on the first of these three chains is going up,
because this is the chain through which all of your packets leaving eth1
will pass. This might lead you to believe that the chain was performing
some useful function--but since the routing decision has already been
made, your packet is getting marked after the routing decision has been
made. I''m not sure about my diagnosis on this one, however....
I don''t think the other two are doing anything. I think you
will probably want to use 192.168.1.160/27 here instead of the public
network IPs.
Since your problem (for split access to the Internet via multiple
providers) is correctly routing the outbound packets, you really only need
to mark the outbound packets. You don''t need to distinguish the
inbound
packets because connection tracking will take care of restoring the
internal IP address.
So, could you perform all of the marking in the PREROUTING chain?
for internalnet in $ITX_NET $DMZ_NET ; do
iptables -t mangle -I PREROUTING -s $internalnet \
-j MARK --set-mark $FWMARK
done
I don''t see why you''d choose the FORWARD chain in the mangle
table for
this marking. I''d suggest the PREROUTING chain in the mangle table--at
least that way, you know the fwmark is available for ANY routing decision.
: > Sounds like a routing problem.
Sounds like I was speculating without enough information. :( Everything
looks perfect in your routing tables and RPDB!
<snipped your routing tables and RPDB>
: I''m also attaching the firewall script, which both generates the
: routing table and sets up the iptable rules.
Ah, the joys of digging through mountains of iptables rules. At least
it''s better than ipchains! Here''s what I think you are seeing
- outbound packet from 192.168.1.161
- marked and rewritten to appear to the Internet from 212.202.108.161
- ACK returns, accepted on eth1
- connection tracking rewrites destination IP to 192.168.1.161
- you lose packet!
: I can see packets running out, and getting an ACK back. From logging I
: can see the packet getting into my DNAT, but then it stops!
Do we concur?
Question 1: Why do you say that the packet is hitting the DNAT?
Shouldn''t the packet be handled by the connection tracking
mechanism?
Question 2: Did you try tcpdump''ing on eth2? Are packets showing up
on
the wire with the wrong destination address?
Question 3: Did you try defining the minimal required iptables rules to
allow one single IP to reach the ''net via this split
access
technique?
After all of the above, I was reviewing your script one more time, and
discovered the rp_filter sysctl....
I think your rp_filter sysctl is disagreeing with your multiple uplink
paths.
What happens when you change nothing of the above, and change this?
# IP Spoof protection
if [ -f /proc/sys/net/ipv4/conf/all/rp_filter ]; then
echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter
fi
To
# IP Spoof protection
if [ -f /proc/sys/net/ipv4/conf/all/rp_filter ]; then
for i in $( ls -1 /proc/sys/net/ipv4/conf ) ; do
echo 0 > /proc/sys/net/ipv4/conf/$i/rp_filter
done
fi
Reverse path filtering is very handy in some situations as a very basic
antispoof measure, but when you have multiple paths to the same network,
it''s important to turn off this spoof protection.
For those following along on this thread, see the section on rp_filter in
the IP Sysctl tutorial:
http://ipsysctl-tutorial.frozentux.net/
http://ipsysctl-tutorial.frozentux.net/ipsysctl-tutorial.html#AEN616
Oskar Andreasson''s cautionary note mocks me gently after an hour
puzzling
about this problem! :)
: I hope that you with this information can give me some hints. The net
: setup is 2 internal and two external. The rules should appear from the
: firewall script.
I think this is what your network looks like:
+------------------+ 212.202.108.160/27 +--------+
192.168.1.160/28 --+eth2 eth1+-- 192.168.0.0/24 ------+ router |
$DMZ_NET | | $QSC_NET +--------+
| linux |
| | +--------+
192.168.1.176/28 --+eth3 eth0+-- 192.168.141.160/27 --+ router |
$ITX_NET +------------------+ (Arcor) +--------+
Good luck, Kim,
-Martin
--
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hi Martin, First, just to clarify, all my packet destined for the internal net via our QSC provider was hitting the DNAT chain, and then lost. All packets going the other way was passing the FORWARD chain and then lost. So my problem seemed to be just after the incoming packet has arrived or before it leaves. I have tracked packets using extensive logging (and multiple changes in my firewall script) as well as tcpdump on all relevant interfaces in my system. The packet looks great when it comes in, correct information, etc. But never reaches the destination on the box. I have not played much with the firewall script after I got your reply, my ADSL died and I only just got it back up! But changing the rp_filter from one to zero seems to work, as I can track packets hitting my FORWARD chain now. So, right now I seem to be on the right track - I think the general problem with setting up something like this is that all relevant information is usually scattered over many places with mostly only fragments put together. But now now, Thanks Martin - you have been a great help :-) /Kim On Saturday 22 March 2003 06:57, Martin A. Brown wrote:> Kim, > > Well, I *think* the solution is a simple sysctl twiddle (see close to the > bottom), but I left my reasoning and research intact on the off-chance > that my method would benefit you or posterity reviewing this thread.. > > iptables mangle thoughts > - - - - - - - - - - - - - > I have line broken the iptables output, but I''m curious to know if you are > seeing any activity in these particular chains: > > # iptables -t mangle -nvL FORWARD > <snip> > 0 0 MARK all -- * eth1 0.0.0.0/0 > 0.0.0.0/0 MARK set 0x1 > 0 0 MARK all -- * * 212.202.108.160/27 \ > 0.0.0.0/0 MARK set 0x1 > 0 0 MARK all -- * * 0.0.0.0/0 \ > 212.202.108.160/27 MARK set 0x1 > </snip> > > I expect that the counter on the first of these three chains is going up, > because this is the chain through which all of your packets leaving eth1 > will pass. This might lead you to believe that the chain was performing > some useful function--but since the routing decision has already been > made, your packet is getting marked after the routing decision has been > made. I''m not sure about my diagnosis on this one, however.... > > I don''t think the other two are doing anything. I think you > will probably want to use 192.168.1.160/27 here instead of the public > network IPs. > > Since your problem (for split access to the Internet via multiple > providers) is correctly routing the outbound packets, you really only need > to mark the outbound packets. You don''t need to distinguish the inbound > packets because connection tracking will take care of restoring the > internal IP address. > > So, could you perform all of the marking in the PREROUTING chain? > > for internalnet in $ITX_NET $DMZ_NET ; do > iptables -t mangle -I PREROUTING -s $internalnet \ > -j MARK --set-mark $FWMARK > done > > I don''t see why you''d choose the FORWARD chain in the mangle table for > this marking. I''d suggest the PREROUTING chain in the mangle table--at > least that way, you know the fwmark is available for ANY routing decision. > > : > Sounds like a routing problem. > > Sounds like I was speculating without enough information. :( Everything > looks perfect in your routing tables and RPDB! > > <snipped your routing tables and RPDB> > > : I''m also attaching the firewall script, which both generates the > : routing table and sets up the iptable rules. > > Ah, the joys of digging through mountains of iptables rules. At least > it''s better than ipchains! Here''s what I think you are seeing > > - outbound packet from 192.168.1.161 > - marked and rewritten to appear to the Internet from 212.202.108.161 > - ACK returns, accepted on eth1 > - connection tracking rewrites destination IP to 192.168.1.161 > - you lose packet! > > : I can see packets running out, and getting an ACK back. From logging I > : can see the packet getting into my DNAT, but then it stops! > > Do we concur? > > Question 1: Why do you say that the packet is hitting the DNAT? > Shouldn''t the packet be handled by the connection tracking > mechanism? > Question 2: Did you try tcpdump''ing on eth2? Are packets showing up on > the wire with the wrong destination address? > Question 3: Did you try defining the minimal required iptables rules to > allow one single IP to reach the ''net via this split access > technique? > > After all of the above, I was reviewing your script one more time, and > discovered the rp_filter sysctl.... > > I think your rp_filter sysctl is disagreeing with your multiple uplink > paths. > > What happens when you change nothing of the above, and change this? > > # IP Spoof protection > if [ -f /proc/sys/net/ipv4/conf/all/rp_filter ]; then > echo 1 > /proc/sys/net/ipv4/conf/all/rp_filter > fi > > To > > # IP Spoof protection > if [ -f /proc/sys/net/ipv4/conf/all/rp_filter ]; then > for i in $( ls -1 /proc/sys/net/ipv4/conf ) ; do > echo 0 > /proc/sys/net/ipv4/conf/$i/rp_filter > done > fi > > Reverse path filtering is very handy in some situations as a very basic > antispoof measure, but when you have multiple paths to the same network, > it''s important to turn off this spoof protection. > > For those following along on this thread, see the section on rp_filter in > the IP Sysctl tutorial: > > http://ipsysctl-tutorial.frozentux.net/ > http://ipsysctl-tutorial.frozentux.net/ipsysctl-tutorial.html#AEN616 > > Oskar Andreasson''s cautionary note mocks me gently after an hour puzzling > about this problem! :) > > : I hope that you with this information can give me some hints. The net > : setup is 2 internal and two external. The rules should appear from the > : firewall script. > > I think this is what your network looks like: > > +------------------+ 212.202.108.160/27 +--------+ > 192.168.1.160/28 --+eth2 eth1+-- 192.168.0.0/24 ------+ router | > $DMZ_NET | | $QSC_NET +--------+ > > | linux | > | > | | +--------+ > > 192.168.1.176/28 --+eth3 eth0+-- 192.168.141.160/27 --+ router | > $ITX_NET +------------------+ (Arcor) +--------+ > > Good luck, Kim, > > -Martin_______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
: But changing the rp_filter from one to zero seems to work, as I can : track packets hitting my FORWARD chain now. Excellent. I''m quite glad to hear it. : So, right now I seem to be on the right track - I think the general : problem with setting up something like this is that all relevant : information is usually scattered over many places with mostly only : fragments put together. But now now, Thanks Martin - you have been a : great help :-) Well, that''s what my documentation attempts to remedy--but never can. Such a dynamic target is very hard to adequately document, but that will not prevent me from trying. Regardless your experience with my section on multiple Internet connections points out a now-obvious deficiency in the section on multiple uplinks. I will modify the section to include a similar cautionary note about the rp_filter sysctl. Thanks for letting me know it''s working for you, -Martin -- Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
On Saturday 22 March 2003 17:33, you wrote:> : But changing the rp_filter from one to zero seems to work, as I can > : track packets hitting my FORWARD chain now. > > Excellent. I''m quite glad to hear it. > > : So, right now I seem to be on the right track - I think the general > : problem with setting up something like this is that all relevant > : information is usually scattered over many places with mostly only > : fragments put together. But now now, Thanks Martin - you have been a > : great help :-) > > Well, that''s what my documentation attempts to remedy--but never can. > Such a dynamic target is very hard to adequately document, but that will > not prevent me from trying. Regardless your experience with my section on > multiple Internet connections points out a now-obvious deficiency in the > section on multiple uplinks. I will modify the section to include a > similar cautionary note about the rp_filter sysctl. > > Thanks for letting me know it''s working for you, >Well, not so fast ... I''m still having some problems, but I fear that they will be very hard to solve! We have 2 connections, our cheap (Arcor, dynamic IP) and our expensive (QSC) with 32 public addresses. Since my company is also linked up to a second company using a set of private addresses, we have to use the 192.168.1.160/27 net internally, with a DHCP server. Most of our traffic must go via the Arcor connection, while all mail will go via the QSC together with SSH to specific machines. For incoming traffic, all addresses in our 32 public IP numbers, must be routed 1-1 for the internal net, so it is possible to make external connections to either mail, web, ssh, etc. on internal machines. Now my question is, is it possible to make a destinction for packets coming from our internal nets, whether it is part of a connection from QSC or from Arcor? So far, I am lost when it comes to ideas and plans... It appears like SNAT is impossible as I don''t know whether it will correctly translate the packet back, and MASQUERADE doesn''t seem like the solution either! Using MASQUERADING, it is possible to have traffic running normally via the Arcor net, but once we wish to include QSC in the calculation - I keep hitting my head on the wall. /Kim _______________________________________________ LARTC mailing list / LARTC@mailman.ds9a.nl http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/
Hello again Kim,
So, this is your network, correct?
+------------------+ 212.202.108.160/27 +--------+
192.168.1.160/28 --+eth2 eth1+-- 192.168.0.0/24 ------+ router |
$DMZ_NET | | $QSC_NET +--------+
| linux |
| | +--------+
192.168.1.176/28 --+eth3 eth0+-- 192.168.141.160/27 --+ router |
$ITX_NET +------------------+ (Arcor) +--------+
: Well, not so fast ... I''m still having some problems, but I fear
that
: they will be very hard to solve!
Maybe...but maybe not....
: We have 2 connections, our cheap (Arcor, dynamic IP) and our expensive
: (QSC) with 32 public addresses. Since my company is also linked up to
: a second company using a set of private addresses, we have to use the
: 192.168.1.160/27 net internally, with a DHCP server.
: Most of our traffic must go via the Arcor connection, while
So, in this case, I''d make Arcor the default route in table main, as
you
have done.
: all mail will go via the QSC together with SSH to specific machines.
So, MAIL=25 (but could be anything):
for internalnet in $ITX_NET $DMZ_NET ; do
iptables -t mangle -I PREROUTING -p tcp -s $internalnet \
--dport $MAIL -j MARK --set-mark $FWMARK
for ssh_server in $SSH_HOSTS ; do
iptables -t mangle -I PREROUTING -p tcp -s $internalnet \
-d $ssh_server --dport 22 -j MARK --set-mark $FWMARK
done # -- end of ssh_server loop
done # -- end of internalnet loop
: For incoming traffic, all addresses in our 32 public IP numbers, must
: be routed 1-1 for the internal net, so it is possible to make external
: connections to either mail, web, ssh, etc. on internal machines.
No problems...as long as the connections are initiated from the outside,
connection tracking will do the work of transforming addresses for you.
This is one of the big benefits of using netfilter--the connection
tracking mechanisms can make complex scenarios much easier. You should be
able to use the MASQUERADE target on the outbound connections through
Arcor, which gets you connection tracking for anything initiated from
either of your inside networks.
I would, however, recommend adding the following to your RPDB (I assume
you will continue your convention--using an fwmark corresponding to the
table ID of the routing table):
# ip rule add from 212.202.108.160/27 table 1
This should ensure that traffic with an outgoing IP in this range is
transmitted via the QSC default route.
: Now my question is, is it possible to make a destinction for packets
: coming from our internal nets, whether it is part of a connection from
: QSC or from Arcor? So far, I am lost when it comes to ideas and
: plans... It appears like SNAT is impossible as I don''t know whether
it
: will correctly translate the packet back, and MASQUERADE doesn''t
seem
: like the solution either!
:
: Using MASQUERADING, it is possible to have traffic running normally via
: the Arcor net, but once we wish to include QSC in the calculation - I
: keep hitting my head on the wall.
Don''t hit your head on the wall, Kim--this is linux!
Let us know if the above suggestion(s) help you get split access working
correctly.
-Martin
--
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/