Hi I''m setting up a firewall based on Shorewall 2.2 from Debian stable (sarge). It seems to work just fine, but I have a weird thing happening when using ping and other ICMP traffic. The box is a 2.8 GHz with 1 GB of memory. It has 3 gigabit ethernet adapters, and I''m not loading it with a lot of traffic at this point. When shutting down Shorewall I have a ping latency at around 0.1 ms from my local network to the firewall, but as soon as I enable Shorewall the latency goes up to about 25-30 ms. However, if I traceroute through the firewall to some other host on the internet it replies quickly in less than 0.2 ms. To illustrate: $ ping -c 10 -q 10.0.0.8 PING 10.0.0.8 (10.0.0.8): 56 data bytes --- 10.0.0.8 ping statistics --- 10 packets transmitted, 10 packets received, 0% packet loss round-trip min/avg/max = 26.5/34.7/100.0 ms Traceroute to my ISP: $ traceroute -I www.webpartner.dk traceroute to www.webpartner.dk (195.184.96.72), 30 hops max, 40 byte packets 1 10.0.0.8 (10.0.0.8) 0.221 ms 0.186 ms 0.238 ms 2 213.173.237.225 (213.173.237.225) 16.764 ms 14.519 ms 14.098 ms 3 213.173.240.90 (213.173.240.90) 20.972 ms 24.657 ms 19.967 ms 4 213.173.240.89 (213.173.240.89) 22.925 ms 34.553 ms 22.194 ms 5 195.184.96.72 (195.184.96.72) 24.843 ms 9.660 ms 10.103 ms 213.173.237.225 is my router to the internet, it''s not the world''s fastest router, but still faster than ñwhat the above traceroute shows. If I ping it through another firewall I get: $ ping -c 10 -q 213.173.237.225 PING 213.173.237.225 (213.173.237.225) 56(84) bytes of data. --- 213.173.237.225 ping statistics --- 10 packets transmitted, 10 received, 0% packet loss, time 9012ms rtt min/avg/max/mdev = 1.300/1.832/4.290/0.903 ms I see the same pattern even if I cut the rules down to only permitting ping. Does anyone have a clue as to what''s happening? I''m using a newly compiled 2.6.14 kernel, but saw the same behavior with an older 2.6.8-2 kernel. -- Jacob ------------------------------------------------------- SF.Net email is sponsored by: Tame your development challenges with Apache''s Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
On Friday 11 November 2005 04:45, Jacob Bunk Nielsen wrote:> I see the same pattern even if I cut the rules down to only permitting > ping. Does anyone have a clue as to what''s happening? I''m using a > newly compiled 2.6.14 kernel, but saw the same behavior with an older > 2.6.8-2 kernel.I have not seen this behavior before. Do you have large blacklists? If so, have you tried setting BLACKLISTNEWONLY=Yes in shorewall.conf? -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
On Friday 11 November 2005 08:10, Tom Eastep wrote:> On Friday 11 November 2005 04:45, Jacob Bunk Nielsen wrote: > > I see the same pattern even if I cut the rules down to only permitting > > ping. Does anyone have a clue as to what''s happening? I''m using a > > newly compiled 2.6.14 kernel, but saw the same behavior with an older > > 2.6.8-2 kernel. > > I have not seen this behavior before. Do you have large blacklists? If so, > have you tried setting BLACKLISTNEWONLY=Yes in shorewall.conf? >Also, are you using the QUEUE target or policy? -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
Hi Tom Eastep <teastep@shorewall.net> writes:> On Friday 11 November 2005 04:45, Jacob Bunk Nielsen wrote: > >> I see the same pattern even if I cut the rules down to only permitting >> ping. Does anyone have a clue as to what''s happening? I''m using a >> newly compiled 2.6.14 kernel, but saw the same behavior with an older >> 2.6.8-2 kernel. > > I have not seen this behavior before. Do you have large blacklists? If so, > have you tried setting BLACKLISTNEWONLY=Yes in shorewall.conf?No, I don''t have any blacklists at all. BLACKLISTNEWONLY is already set to yes. This is my configuration: # egrep -v "^#|^$" /etc/shorewall/shorewall.conf LOGFILE=/var/log/messages LOGFORMAT="Shorewall:%s:%s:" LOGTAGONLY=No LOGRATELOGBURSTLOGALLNEWBLACKLIST_LOGLEVELLOGNEWNOTSYN=info MACLIST_LOG_LEVEL=info TCP_FLAGS_LOG_LEVEL=info RFC1918_LOG_LEVEL=info SMURF_LOG_LEVEL=info BOGON_LOG_LEVEL=info LOG_MARTIANS=Yes IPTABLESPATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/bin:/usr/local/sbin SHOREWALL_SHELL=/bin/sh SUBSYSLOCK="" STATEDIR=/var/lib/shorewall MODULESDIRCONFIG_PATH=/etc/shorewall:/usr/share/shorewall RESTOREFILEFW=fw IP_FORWARDING=On ADD_IP_ALIASES=Yes ADD_SNAT_ALIASES=No RETAIN_ALIASES=No TC_ENABLED=No CLEAR_TC=Yes MARK_IN_FORWARD_CHAIN=No CLAMPMSS=No ROUTE_FILTER=Yes DETECT_DNAT_IPADDRS=No MUTEX_TIMEOUT=60 NEWNOTSYN=No ADMINISABSENTMINDED=Yes BLACKLISTNEWONLY=Yes DELAYBLACKLISTLOAD=No MODULE_SUFFIXDISABLE_IPV6=Yes BRIDGING=No DYNAMIC_ZONES=No PKTTYPE=Yes DROPINVALID=Yes RFC1918_STRICT=No MACLIST_TTLBLACKLIST_DISPOSITION=DROP MACLIST_DISPOSITION=REJECT TCP_FLAGS_DISPOSITION=DROP It seems as if the kernel uses a lot of CPU time dealing with pings. Can it be some sort of kernel configuration parameter I have set incorrectly? It''s the first time I''ve tried Shorewall, but I''ve used iptables before, and never seen anything like this. I have also tried disabling all masquerading, but that didn''t help. -- Jacob ------------------------------------------------------- SF.Net email is sponsored by: Tame your development challenges with Apache''s Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
Hi Tom Eastep <teastep@shorewall.net> writes:> On Friday 11 November 2005 08:10, Tom Eastep wrote: >> On Friday 11 November 2005 04:45, Jacob Bunk Nielsen wrote: >> > I see the same pattern even if I cut the rules down to only permitting >> > ping. Does anyone have a clue as to what''s happening? I''m using a >> > newly compiled 2.6.14 kernel, but saw the same behavior with an older >> > 2.6.8-2 kernel. >> >> I have not seen this behavior before. Do you have large blacklists? If so, >> have you tried setting BLACKLISTNEWONLY=Yes in shorewall.conf? > > Also, are you using the QUEUE target or policy?I''m not using the QUEUE target, but I am using policies. As far as I can see from the iptables rules created policies doesn''t really mean the same thing as the policy on a chain in iptables, but will just insert a rule at the end of certain chains. This seems to work fine. -- Jacob ------------------------------------------------------- SF.Net email is sponsored by: Tame your development challenges with Apache''s Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
On Friday 11 November 2005 09:19, Jacob Bunk Nielsen wrote:> > As far as I can see from the iptables rules created policies doesn''t > really mean the same thing as the policy on a chain in iptables, but > will just insert a rule at the end of certain chains. This seems to > work fine.Netfilter doesn''t allow a policy to be specified on a user-defined chain so Shorewall must enforce its policies using rules. If you''ll post the output of ''shorewall status'' as a compressed attachment, I''ll take a look at your configuration. Can''t promise anything though since Shorewall doesn''t do anything special with ICMP. There are entries in /proc/sys/net/ipv4 that affect ICMP latency. On my gateway (which doesn''t exhibit high ICMP latency), these are set as follows: gateway:/etc/shorewall# cat /proc/sys/net/ipv4/icmp_ratelimit 250 gateway:/etc/shorewall# cat /proc/sys/net/ipv4/icmp_ratemask 6168 Again, Shorewall doesn''t alter those settings so given that your firewall behaves normally until ''shorewall start'', I wouldn''t think that those settings are involved. If you want to see everything that ''shorewall start'' does *besides* configure Netfilter, look at /var/lib/shorewall/restore-base -- that file contains the commands other than ''iptables'' that Shorewall executed the last time it was started. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
Hi Tom Eastep <teastep@shorewall.net> writes:> On Friday 11 November 2005 09:19, Jacob Bunk Nielsen wrote: > >> As far as I can see from the iptables rules created policies doesn''t >> really mean the same thing as the policy on a chain in iptables, but >> will just insert a rule at the end of certain chains. This seems to >> work fine. > > Netfilter doesn''t allow a policy to be specified on a user-defined chain so > Shorewall must enforce its policies using rules. > > If you''ll post the output of ''shorewall status'' as a compressed attachment, > I''ll take a look at your configuration. Can''t promise anything though since > Shorewall doesn''t do anything special with ICMP.I wasn''t sure if the list would allow me to post attached files, so I''ve made it available at (compressed) http://flof.dk/~jbn/shorewall-status.txt.gz or (uncompressed) http://flof.dk/~jbn/shorewall-status.txt (Don''t be confused that the IP address has changed since my first post, it''s still the same box with the same problems.)> There are entries in /proc/sys/net/ipv4 that affect ICMP latency. On my > gateway (which doesn''t exhibit high ICMP latency), these are set as follows: > > gateway:/etc/shorewall# cat /proc/sys/net/ipv4/icmp_ratelimit > 250 > gateway:/etc/shorewall# cat /proc/sys/net/ipv4/icmp_ratemask > 6168My system have the exact same settings in those two "files".> Again, Shorewall doesn''t alter those settings so given that your firewall > behaves normally until ''shorewall start'', I wouldn''t think that those > settings are involved.Same here.> If you want to see everything that ''shorewall start'' does *besides* configure > Netfilter, look at /var/lib/shorewall/restore-base -- that file contains the > commands other than ''iptables'' that Shorewall executed the last time it was > started.I''m just about to leave for a few hours. I''ll have a deeper look into it later tonight or tomorrow. Thank you very much for taking the time to help. -- Jacob ------------------------------------------------------- SF.Net email is sponsored by: Tame your development challenges with Apache''s Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
On Friday 11 November 2005 09:44, Jacob Bunk Nielsen wrote:> > I''m just about to leave for a few hours. I''ll have a deeper look into > it later tonight or tomorrow. Thank you very much for taking the time > to help.It would surely help if you didn''t log every ICMP echo-request packet. In fact, it appears as though you have decided to log every connection through your firewall (which for the 18 hours since you restarted Shorewall has logged 77596 messages). Netfilter logging is NOT designed for IP traffic auditing. It is best used only for exceptions. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
On Friday 11 November 2005 10:08, Tom Eastep wrote:> On Friday 11 November 2005 09:44, Jacob Bunk Nielsen wrote: > > I''m just about to leave for a few hours. I''ll have a deeper look into > > it later tonight or tomorrow. Thank you very much for taking the time > > to help. > > It would surely help if you didn''t log every ICMP echo-request packet. In > fact, it appears as though you have decided to log every connection through > your firewall (which for the 18 hours since you restarted Shorewall has > logged 77596 messages).Nevertheless, I''m unable to produce any significant latency through logging -- sure makes you vulnerable to DOS through ping flooding though... -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
Tom Eastep <teastep@shorewall.net> writes:> On Friday 11 November 2005 09:44, Jacob Bunk Nielsen wrote: > >> I''m just about to leave for a few hours. I''ll have a deeper look into >> it later tonight or tomorrow. Thank you very much for taking the time >> to help. > > It would surely help if you didn''t log every ICMP echo-request packet. In > fact, it appears as though you have decided to log every connection through > your firewall (which for the 18 hours since you restarted Shorewall has > logged 77596 messages).I tried turning logging off. It didn''t make any difference, just as I expected. The weird thing is that the kernel will use around 50% of the CPU time available if I pull around 400 Mbps of HTTP traffic through the firewall with logging turned on and everything. If we just assume that it''s all 1500 byte packets (which is probably a fairly good assumption), it still adds up to something like 30,000 packets per second. If I send it just 10 small ICMP echo-request packets each second I''ll see the kernel using 16-18 percent of the available CPU time even though that only adds up to around 500 bps that doesn''t even have to be masqueraded. I''m truly puzzled. You are right I have decided to log every connection through the firewall. The few megabytes of logfile doesn''t scare me at all, and I don''t expect performance to degrade that extremely just because I decide to make it write 15 to 20 megabyte of logfile in a day. I do mount the partition where the logfile is stored asynchronously ;-)> Netfilter logging is NOT designed for IP traffic auditing. It is best used > only for exceptions.OK - I could consider setting up snort or something similar, but I like the logfiles for the moment, and I don''t find it reasonable that it should take 20 ms to write one line to a logfile when it''s ICMP and far less when it''s TCP or some other protocol. -- Jacob ------------------------------------------------------- SF.Net email is sponsored by: Tame your development challenges with Apache''s Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
On Friday 11 November 2005 16:00, Jacob Bunk Nielsen wrote:> > If I send it just 10 small ICMP echo-request packets each second I''ll > see the kernel using 16-18 percent of the available CPU time even > though that only adds up to around 500 bps that doesn''t even have to > be masqueraded. I''m truly puzzled.Me too -- I''ve not seen or heard of anything like it. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
Tom Eastep <teastep@shorewall.net> writes:> On Friday 11 November 2005 16:00, Jacob Bunk Nielsen wrote: > >> If I send it just 10 small ICMP echo-request packets each second I''ll >> see the kernel using 16-18 percent of the available CPU time even >> though that only adds up to around 500 bps that doesn''t even have to >> be masqueraded. I''m truly puzzled. > > Me too -- I''ve not seen or heard of anything like it.Now I have tried to compile all the iptables stuff into the kernel in stead of as modules. That didn''t change anything, just as expected. I have also tried doing iptables-save > somefile.txt shorewall stop iptables-restore < somefile.txt That should give me just the iptables rules that shorewall has generated without any changes shorewall might make in other places. It doesn''t make any difference either, so I think the problem is related to some iptables or kernel issue. I''ll keep working on it. -- Jacob ------------------------------------------------------- SF.Net email is sponsored by: Tame your development challenges with Apache''s Geronimo App Server. Download it for free - -and be entered to win a 42" plasma tv or your very own Sony(tm)PSP. Click here to play: http://sourceforge.net/geronimo.php
Tom Eastep <teastep@shorewall.net> writes:> On Friday 11 November 2005 09:44, Jacob Bunk Nielsen wrote: > >> I''m just about to leave for a few hours. I''ll have a deeper look into >> it later tonight or tomorrow. Thank you very much for taking the time >> to help. > > It would surely help if you didn''t log every ICMP echo-request packet. In > fact, it appears as though you have decided to log every connection through > your firewall (which for the 18 hours since you restarted Shorewall has > logged 77596 messages).It would seem that you were right here. Even though you were not able to reproduce this on first try. I can reproduce it simply by doing: iptables -A OUTPUT -p icmp -j LOG on a similar box (2,8 GHz, 1 GB memory) with no other firewall rules and ACCEPT as policy on all chains. Watch this: $ ping -c 5 -q host PING host (10.0.0.27) 56(84) bytes of data. --- host ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3999ms rtt min/avg/max/mdev = 0.153/0.200/0.233/0.029 ms $ sudo iptables -A OUTPUT -p icmp -j LOG $ ping -c 5 -q host PING host (10.0.0.27) 56(84) bytes of data. --- host ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4004ms rtt min/avg/max/mdev = 18.215/20.947/23.873/2.037 ms On another box (2 GHz, 2 GB memory) this has no effect: $ ping -c 5 -q host PING host (10.0.0.27) 56(84) bytes of data. --- host ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 3999ms rtt min/avg/max/mdev = 0.117/0.162/0.251/0.049 ms erdinger:~$ sudo iptables -A OUTPUT -p icmp -j LOG erdinger:~$ ping -c 5 -q host PING host (10.0.0.27) 56(84) bytes of data. --- host ping statistics --- 5 packets transmitted, 5 received, 0% packet loss, time 4000ms rtt min/avg/max/mdev = 0.146/0.193/0.245/0.039 ms Apparently logging a ping request adds around 20 ms to the RTT on some hosts. Not what I would expect. It could seem that the logging happens synchronously, and this is not a ShoreWall issue. Apparently it only happens on some configurations. I have not looked into what exactly triggers this.> Netfilter logging is NOT designed for IP traffic auditing. It is best used > only for exceptions.I have a fairly good box running as firewallñ and only log new connections. I truely had no idea it was this inefficient. Sorry for this very late reply. I didn''t have time to do further debugging until today. Also, thanks again for your help. -- Jacob ------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
On Thursday 15 December 2005 06:22, Jacob Bunk Nielsen wrote:> I have a fairly good box running as firewallñ and only log new > connections. I truely had no idea it was this inefficient. > > Sorry for this very late reply. I didn''t have time to do further > debugging until today. Also, thanks again for your help.Thanks for the update, Jacob -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
Jacob, could you tell us what kernel version you use on the two machines? On my 2.6.14, the extra time using your ping test is less than 0.1 ms. Rune On 12/15/05, Tom Eastep <teastep@shorewall.net> wrote:> On Thursday 15 December 2005 06:22, Jacob Bunk Nielsen wrote: > > > I have a fairly good box running as firewallñ and only log new > > connections. I truely had no idea it was this inefficient. > > > > Sorry for this very late reply. I didn''t have time to do further > > debugging until today. Also, thanks again for your help. > > Thanks for the update, Jacob > > -Tom > -- > Tom Eastep \ Nothing is foolproof to a sufficiently talented fool > Shoreline, \ http://shorewall.net > Washington USA \ teastep@shorewall.net > PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key > > >------------------------------------------------------- This SF.net email is sponsored by: Splunk Inc. Do you grep through log files for problems? Stop! Download the new AJAX search engine that makes searching your log files as easy as surfing the web. DOWNLOAD SPLUNK! http://ads.osdn.com/?ad_idv37&alloc_id865&op=click