Hello,
I''m new to this list. I spent the last week reading docs and trying
various ideas, annoying my users at work in the process.
Our setup seems to be very common; I installed a firewall linux machine
(RH72 with a 2.4.17 kernel with the HTB patches applied)
This firewall has :
- eth0 as an internal LAN device, IP 192.168.1.254
- eth1 as the first external device, IP 192.168.254.253
connected to an ADSL cisco router (with int. IP 192.168.254.254)
- eth2 as the second external device
connected to a cable modem
getting it''s IP information from DHCP, but according to the provider
I
will always get the same address.
Now, I implemented stuff from the nano.txt document, and for outgoing
connections most things seem to work.
I set up /sbin/ifup-local to remove the default gateway route added by
dhcpcd so that after boot, the firewall machine at least works over the
ADSL connection.
Then I started writing the firewall script.
I start by applying the iptables rules for statefulness (are these
necessary ? exactly what do they do). I removed the interface
configuration commands, since that is handled by redhat.
Then I remove the default route, and add the three tables which together
implement the load balancing.
For outgoing connections, this mostly works : I can tell from traceroutes
that I get alternating outgoing gateways.
Now for the problems I''m having :
* before, when only using the ADSL as gateway, I could ssh to other boxes
on the internet without problems. With the new setup, when I ssh to one
of them (and the route goes over the second interface), the connection
hangs at the moment ssh starts up the X port forwarding. I suppose this
is because (IIRC) ssh tries to set up a connection from that box to my
current machine, which somehow fails. If the route happens to go over the
first interface, everything is ok.
* When trying to access the firewall from the outside, connections only
get established when coming in over the ADSL interface. When coming in
over the cable interface, the connection hangs, indicating the route back
is failing. This seems to me like another symptom of the same problem as
the other.
So here is a set of questions ;) You knew this was coming ...
a) nano.txt only mentions outgoing connections. Does this document apply
to incoming connections as well or not ? Should it work as outlined there,
should I infer different iptables and ip rules to handle incoming traffic,
or does it work in another way entirely ?
b) Since I don''t have a default gateway and the gateway alternation
works
on outgoing routes, I suppose that my gateway setup is correct. So the
fact that it cannot make incoming connections over eth2 is not due to eth1
being the default gateway as was the case before.
But what else could cause this behaviour ? Is it possible I might have my
SNAT/MASQUERADE set up wrong to get this effect ?
c) do I need to apply julian''s patches in order for this basic setup
(incoming traffic on both interfaces) to work ? It is my understanding
from browsing through the archive that, for this basic functionality,
it''s
not necessary. I will of course apply these patches later on to have
gateway failure detection, but my question is if applying these patches
now or not will have any effect on my current setup.
Here is a list of output of various commands :
[root@columbus root]# ip rule
0: from all lookup local
50: from all lookup main
201: from 192.168.254.0/24 lookup 201
202: from 62.205.66.0/23 lookup 202
222: from all lookup 222
32766: from all lookup main
32767: from all lookup 253
[root@columbus root]# ip route list table main
192.168.4.0/24 dev eth0 proto kernel scope link src 192.168.4.254
192.168.1.0/24 dev eth0 scope link
192.168.254.0/24 dev eth1 proto kernel scope link src 192.168.254.253
62.205.66.0/23 dev eth2 proto kernel scope link src 62.205.66.243
127.0.0.0/8 dev lo scope link
(I have to note here that using redhat''s network configuration
initialized
the 192.168.254.0/24 to be "scope link" only, so no proto kernel and
no
src addresss. I thought that this might have been wrong so I changed it
manually but it had no effect as far as I could tell)
[root@columbus root]# ip route list table 201
default via 192.168.254.254 dev eth1 proto static src 192.168.254.253
prohibit default proto static metric 1
[root@columbus root]# ip route list table 202
default via 62.205.67.254 dev eth2 proto static src 62.205.66.243
prohibit default proto static metric 1
default proto static
nexthop via 192.168.254.254 dev eth1 weight 1
nexthop via 62.205.67.254 dev eth2 weight 1
[root@columbus root]# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
keep_state all -- anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
keep_state all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
keep_state all -- anywhere anywhere
Chain keep_state (3 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere state
RELATED,ESTABLISHED
RETURN all -- anywhere anywhere
[root@columbus root]# iptables -L -t nat
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
keep_state all -- anywhere anywhere
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
keep_state all -- anywhere anywhere
MASQUERADE all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
keep_state all -- anywhere anywhere
Chain keep_state (3 references)
target prot opt source destination
ACCEPT all -- anywhere anywhere state
RELATED,ESTABLISHED
RETURN all -- anywhere anywhere
I hope this is enough information to help me debug the situation. Any
help is MUCH appreciated.
Thanks in advance,
Thomas
--
The Dave/Dina Project : future TV today ! - http://davedina.apestaart.org/
<-*- -*->
I should dress you up in pearl
Finest silk to touch your skin
Don''t know how to write a love song
But don''t leave
<-*- thomas@apestaart.org -*->
URGent, the best radio on the Internet - 24/7 ! - http://urgent.rug.ac.be/
Hi Folks, I just installed here cacti (http://www.raxnet.net/) which is a very good rrd frontend. I am wondering how to make a script to get data from QoS setups to monitor the usage and behavior of the links. Cheers, -- Luiz Felipe Ceglia - Staff TereNet lceglia@terenet.com.br - +55-21-9135-3679
On Thursday 21 March 2002 20:01, Luiz Felipe Ceglia wrote:> Hi Folks, > > I just installed here cacti (http://www.raxnet.net/) which is a very > good rrd frontend. > > I am wondering how to make a script to get data from QoS setups to > monitor the usage and behavior of the links.I wrote my own scripts. An example can be found on http://home.docum.org/stef.coene/qos/gui/rrd.html I can help you to make the script more usefull to you if you want. Stef -- stef.coene@docum.org "Using Linux as bandwidth manager" http://www.docum.org/ #lartc @ irc.openprojects.net
Hi, thanks for your quick response ;)> > Then I started writing the firewall script. > > I start by applying the iptables rules for statefulness (are these > > necessary ? exactly what do they do). I removed the interface > > They are independent from the routing stuff.Ok, just out of curiosity, should I reread the docs to find out what they do help for ? It was not really clear to me on the first couple of reads.> > * before, when only using the ADSL as gateway, I could ssh to other boxes > > on the internet without problems. With the new setup, when I ssh to one > > of them (and the route goes over the second interface), the connection > > hangs at the moment ssh starts up the X port forwarding. I suppose this > > Some users report for this problem with openssh where > the TOS changes in established state cause using a different > route (selected from the multipath scheduler). The new cached > route differs in the TOS field and so to a new gw/outdev. In > short, the problem is that the multipath scheduler when used > for NAT is used for all packets, not only at connection setup. > The details are explained in the docs.OK, I''ll read them again. But basically you''re saying it''s NOT because the X port forwarding opens up a second connection ? Am I wrong in my assumption about how ssh X port forwarding works, or is this not an issue here ?> > * When trying to access the firewall from the outside, connections only > > get established when coming in over the ADSL interface. When coming in > > over the cable interface, the connection hangs, indicating the route back > > is failing. This seems to me like another symptom of the same problem as > > the other. > > May be, to summarize, the rule is: "the plain kernel > _only seems_ to work correctly for setups with NAT and multipath > routes".So you''re saying it looks like it works right, but it doesn''t ? Hm, ok. So are the times when it doesn''t work totally random, or is there some logic in it ?> You should apply the patches if you expect the router > correctly to NAT the packets when using multipath route. I hope > the ssh problem will disappear because there the multipath scheduler > selects new route only for the first packet in each connection, > the established connections are considered bound to the masquerade > address for which usually we don''t have multipath route.Meanwhile, I recompiled my kernel with the patches and at first glance I get the same behaviour. I will look into it some more tomorrow when I''m back at work. What I wanted to ask re: masquerading is, if I add an iptables rule to do masquerading without specifying a device, is that ok ? or should I have one for each device specifically ? The nano doc says that for fixed IP addresses you need an SNAT rule, while for PPPoE devices you need MASQUERADE. Which should I be using for a DHCP device ?> > from browsing through the archive that, for this basic functionality, it''s > > not necessary. I will of course apply these patches later on to have > > gateway failure detection, but my question is if applying these patches > > now or not will have any effect on my current setup. > > I hope there will be effectOn first inspection, no. Is there some way I can debug incoming packets ? What else can I give as feedback ? Thanks again for your help, it is much appreciated. Thomas -- The Dave/Dina Project : future TV today ! - http://davedina.apestaart.org/ <-*- -*-> I''m alive. I can tell because of the pain. <-*- thomas@apestaart.org -*-> URGent, the best radio on the Internet - 24/7 ! - http://urgent.rug.ac.be/
Hello, On Thu, 21 Mar 2002, Thomas Vander Stichele wrote:> Then I started writing the firewall script. > I start by applying the iptables rules for statefulness (are these > necessary ? exactly what do they do). I removed the interfaceThey are independent from the routing stuff.> configuration commands, since that is handled by redhat. > Then I remove the default route, and add the three tables which together > implement the load balancing. > > For outgoing connections, this mostly works : I can tell from traceroutes > that I get alternating outgoing gateways. > > Now for the problems I''m having : > > * before, when only using the ADSL as gateway, I could ssh to other boxes > on the internet without problems. With the new setup, when I ssh to one > of them (and the route goes over the second interface), the connection > hangs at the moment ssh starts up the X port forwarding. I suppose thisSome users report for this problem with openssh where the TOS changes in established state cause using a different route (selected from the multipath scheduler). The new cached route differs in the TOS field and so to a new gw/outdev. In short, the problem is that the multipath scheduler when used for NAT is used for all packets, not only at connection setup. The details are explained in the docs.> is because (IIRC) ssh tries to set up a connection from that box to my > current machine, which somehow fails. If the route happens to go over the > first interface, everything is ok. > > * When trying to access the firewall from the outside, connections only > get established when coming in over the ADSL interface. When coming in > over the cable interface, the connection hangs, indicating the route back > is failing. This seems to me like another symptom of the same problem as > the other.May be, to summarize, the rule is: "the plain kernel _only seems_ to work correctly for setups with NAT and multipath routes".> So here is a set of questions ;) You knew this was coming ... > > a) nano.txt only mentions outgoing connections. Does this document apply > to incoming connections as well or not ? Should it work as outlined there,Yes, the routing is considered symmetric.> should I infer different iptables and ip rules to handle incoming traffic, > or does it work in another way entirely ?No, it will work only by using the routing rules.> b) Since I don''t have a default gateway and the gateway alternation works > on outgoing routes, I suppose that my gateway setup is correct. So the > fact that it cannot make incoming connections over eth2 is not due to eth1 > being the default gateway as was the case before. > But what else could cause this behaviour ? Is it possible I might have my > SNAT/MASQUERADE set up wrong to get this effect ? > > c) do I need to apply julian''s patches in order for this basic setup > (incoming traffic on both interfaces) to work ? It is my understandingYou should apply the patches if you expect the router correctly to NAT the packets when using multipath route. I hope the ssh problem will disappear because there the multipath scheduler selects new route only for the first packet in each connection, the established connections are considered bound to the masquerade address for which usually we don''t have multipath route.> from browsing through the archive that, for this basic functionality, it''s > not necessary. I will of course apply these patches later on to have > gateway failure detection, but my question is if applying these patches > now or not will have any effect on my current setup.I hope there will be effect> Here is a list of output of various commands :look good> (I have to note here that using redhat''s network configuration initialized > the 192.168.254.0/24 to be "scope link" only, so no proto kernel and no > src addresss. I thought that this might have been wrong so I changed it > manually but it had no effect as far as I could tell)In any case make sure the preferred source IP is valid or autoselected to be such. The routing rules look good, I didn''t checked the iptables> I hope this is enough information to help me debug the situation. Any > help is MUCH appreciated. > > Thanks in advance, > ThomasRegards -- Julian Anastasov <ja@ssi.bg>
Hello, On Fri, 22 Mar 2002, Thomas Vander Stichele wrote:> > They are independent from the routing stuff. > > Ok, just out of curiosity, should I reread the docs to find out what they:)> do help for ? It was not really clear to me on the first couple of reads.> OK, I''ll read them again. But basically you''re saying it''s NOT because > the X port forwarding opens up a second connection ? Am I wrong in my > assumption about how ssh X port forwarding works, or is this not an issue > here ?No, there are issues where packets from one connection can use different paths and that causes problems on NAT.> > May be, to summarize, the rule is: "the plain kernel > > _only seems_ to work correctly for setups with NAT and multipath > > routes". > > So you''re saying it looks like it works right, but it doesn''t ? Hm, ok. > So are the times when it doesn''t work totally random, or is there some > logic in it ?Nothing is random :) The problem comes when the cached route entry expires (/proc/sys/net/ipv4/route/gc_timeout) or the cache is flushed as result of a user command such as adding/deleting IP address or flushing the routes explicitly with "ip route cache flush". The result is clear: the routing cache forgets the path and the NAT code does not care.> > You should apply the patches if you expect the router > > correctly to NAT the packets when using multipath route. I hope > > the ssh problem will disappear because there the multipath scheduler > > selects new route only for the first packet in each connection, > > the established connections are considered bound to the masquerade > > address for which usually we don''t have multipath route. > > Meanwhile, I recompiled my kernel with the patches and at first glance I > get the same behaviour. I will look into it some more tomorrow when I''m > back at work. What I wanted to ask re: masquerading is, if I add an > iptables rule to do masquerading without specifying a device, is that ok ? > or should I have one for each device specifically ?If you use SNAT target then you must have one for each device. The MASQUERADE target inherits the masquerade address from the resolved route so specifying the output device is optional.> The nano doc says that for fixed IP addresses you need an SNAT rule, while > for PPPoE devices you need MASQUERADE. Which should I be using for a DHCP > device ?Not a must. The only differences between MASQUERADE and SNAT are: - SNAT specifies the masquerade address, MASQUERADE uses the preferred src ip from the route - For MASQUERADE all masquerade connections are removed on device down event (sometimes useful, sometimes not) So, nothing can stop you to use MASQUERADE for static IPs or SNAT for dynamic IPs, it depends when you create the NAT rules.> > > from browsing through the archive that, for this basic functionality, it''s > > > not necessary. I will of course apply these patches later on to have > > > gateway failure detection, but my question is if applying these patches > > > now or not will have any effect on my current setup. > > > > I hope there will be effect > > On first inspection, no. Is there some way I can debug incoming packets ? > What else can I give as feedback ?You have to be able to analyze with tcpdump what is going on.> ThomasRegards -- Julian Anastasov <ja@ssi.bg>
Hello Julian,> > So you''re saying it looks like it works right, but it doesn''t ? Hm, ok. > > So are the times when it doesn''t work totally random, or is there some > > logic in it ? > > Nothing is random :) The problem comes when the cached > route entry expires (/proc/sys/net/ipv4/route/gc_timeout) or the > cache is flushed as result of a user command such as adding/deleting > IP address or flushing the routes explicitly with "ip route cache flush". > The result is clear: the routing cache forgets the path and the NAT > code does not care.Hm, OK. I checked the value on my system and it''s set to 300, which IIRC would mean 5 minutes (it''s in seconds, right ?). The ssh connection set up takes only about 2 seconds. Isn''t it highly unlikely that the cache would have a timeout in that interval, *reliably*, every time ?> > > I hope there will be effect > > > > On first inspection, no. Is there some way I can debug incoming packets ? > > What else can I give as feedback ? > > You have to be able to analyze with tcpdump what is going on.OK, I used both iptraf and tcpdump to check what is going on, and I find something very odd. Traffic does indeed go out on two interfaces, as traceroutes and output of last on the boxes I ssh to show. But using both tcpdump and iptraf, I only see non-tcp data going over eth2. None of the tcp-connections show up on eth2. They do on eth1. The only difference I can see is that ifconfig shows eth2 to be "UP BROADCAST NOTRAILERS RUNNING", while eth1 is "UP BROADCAST RUNNING MULTICAST". Other than that, the total received traffic is roughly the same on both interfaces (which is weird, since I still cannot connect from the outside over eth2), while the transmitted data for eth1 is 10 times larger than for eth2. I''m not sure if this is because somehow traffic going out over eth2 is not "registered" right (as seen by tcpdump and iptraf) or because of something else. I''m not that experienced using tcpdump, so I can''t tell. Also, I don''t know enough about the lesser-used output from ifconfig since I never needed it before ;) Is there something that could explain this weird behaviour ? Or are there some hints or guides to use tcpdump correctly in debugging this particular problem ? All of this, btw, is done with my patched kernel including your patches (just to make sure ;) ) Thanks in advance, Thomas -- The Dave/Dina Project : future TV today ! - http://davedina.apestaart.org/ <-*- -*-> "You don''t need a girlfriend, you need a maid !" "Isn''t that the same thing ?" "Uh uh, baby, you''re in the wrong century." <-*- thomas@apestaart.org -*-> URGent, the best radio on the Internet - 24/7 ! - http://urgent.rug.ac.be/
Hello, On Fri, 22 Mar 2002, Thomas Vander Stichele wrote:> Hm, OK. I checked the value on my system and it''s set to 300, which IIRC > would mean 5 minutes (it''s in seconds, right ?). The ssh connection set > up takes only about 2 seconds. Isn''t it highly unlikely that the cache > would have a timeout in that interval, *reliably*, every time ?With the patched kernel it is 0 seconds: on the first packet. With unpatched kernel the multipath route usage takes long, the whole connection life.> OK, I used both iptraf and tcpdump to check what is going on, and I find > something very odd. Traffic does indeed go out on two interfaces, as > traceroutes and output of last on the boxes I ssh to show. But using both > tcpdump and iptraf, I only see non-tcp data going over eth2. None of the > tcp-connections show up on eth2. They do on eth1.I don''t remember whether you have script that does active gateway monitoring because the patched kernel does only passive detection on route resolution, explained in nano.txt. I''m not sure whether only the first alive nexthop from the multipath route is used in your case. Check with "ip neigh" the status of the both gateways, they should be in reachable state. May be I''ll tune soon the detection to treat more states as valid. Currently, only the gateways in reachable state are considered as valid.> The only difference I can see is that ifconfig shows eth2 to be > "UP BROADCAST NOTRAILERS RUNNING", while eth1 is "UP BROADCAST RUNNING > MULTICAST". Other than that, the total received traffic is roughly the > same on both interfaces (which is weird, since I still cannot connect from > the outside over eth2), while the transmitted data for eth1 is 10 times > larger than for eth2. I''m not sure if this is because somehow traffic > going out over eth2 is not "registered" right (as seen by tcpdump and > iptraf) or because of something else. I''m not that experienced using > tcpdump, so I can''t tell. Also, I don''t know enough about the lesser-used > output from ifconfig since I never needed it before ;)The old 2.4 kernels don''t show correctly the IP addresses for the NAT-ed packets in tcpdump (copy-on-write problem) but the device should be correct. I''m not sure, may be 2.4.17 has these fixes.> Is there something that could explain this weird behaviour ? Or are there > some hints or guides to use tcpdump correctly in debugging this particular > problem ?It is enough to see the addresses. With a healthchecking script you should not see problems.> Thanks in advance, > ThomasRegards -- Julian Anastasov <ja@ssi.bg>
Hi, I feel stupid in having to admit this, but it seems my ISP is at fault. They set up the wrong type of connection. We were supposed to have a static IP address with all ports enabled. The installers only installed the modem and handed me some docs, but they are for the "lite" version. So my guess is that I''m on the standard DHCP which serves regular customers, instead of using a static IP address (which they failed to provide, so I have to call them for that). Also, my guess is they block incoming traffic < 1024 (as do some other calbe operators). I could get connections to work with higher port numbers on the second interface. So, sorry for trying your patience ;) I should''ve realized something was amiss because of being on a DHCP server. Thanks again for your kind help, Thomas> -- > Julian Anastasov <ja@ssi.bg> > > _______________________________________________ > LARTC mailing list / LARTC@mailman.ds9a.nl > http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/ >-- The Dave/Dina Project : future TV today ! - http://davedina.apestaart.org/ <-*- -*-> I was about to have my first sexual experience and I wasn''t even one of the players... <-*- thomas@apestaart.org -*-> URGent, the best radio on the Internet - 24/7 ! - http://urgent.rug.ac.be/