Ok, I''ve flogged this issue probably longer than some of you can stand by now. (remember, I''m the nut trying to use a PPro200 to support ~500 users on a 10Mb internet link :o) To appease those who think I''m nuts, I am ordering a new firewall shortly to allow for future growth. (probably a Dell PE750 with P4/2.8 and dual GE nics.) However, since I have yet to prove that processor speed has anything to do with my random slow response times, I have this horrible nightmare that I will build a brand new 2.8Ghz firewall and *have the same problem*! So I won''t bore you with the details, but simply ask that anyone who is using iptables/shorewall on an aging CPU (say from 100-500 Mhz) supporting several hundred clients on a 10Mb link or faster, please let me know, on or off list. I just hate not knowing what is causing our problems, and having them occur on a new, fast firewall would probably push me over the edge.... Thanks. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Shawn Wright, I.T. Manager Shawnigan Lake School http://www.sls.bc.ca swright@sls.bc.ca
On Fri, 2004-12-03 at 12:00 -0800, Shawn Wright wrote:> Ok, I''ve flogged this issue probably longer than some of you can stand > by now. (remember, I''m the nut trying to use a PPro200 to support ~500 > users on a 10Mb internet link :o)Shawn, I just looked at your ''good'' and ''bad'' configs again -- the policy files are different; in the ''bad'' config, you are rate-limiting with lots of traffic being dropped. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
Shawn, I supported aproximately 150 users and around 300 systems/clients on a Dell PII 450Mhz with 792Mb RAM, having 3 zones/NICs and 1 IPSec tunnel. I upgraded this to a dual 2.4Ghz Xeon system with 2Gb RAM. I can honestly say that, while I have not measured the throughput differences, access through the new firewall is noticableably faster all round. I continue to keep the old firewall as a backup because, in a pinch it still works! I hope that helps. If you have other questions, you can contact me off list. Graeme -----Original Message----- From: shorewall-users-bounces@lists.shorewall.net [mailto:shorewall-users-bounces@lists.shorewall.net] On Behalf Of Shawn Wright Sent: Friday, December 03, 2004 3:00 PM To: shorewall-users@lists.shorewall.net Subject: [Shorewall-users] Old, slow firewall users please speak up! Ok, I''ve flogged this issue probably longer than some of you can stand by now. (remember, I''m the nut trying to use a PPro200 to support ~500 users on a 10Mb internet link :o) To appease those who think I''m nuts, I am ordering a new firewall shortly to allow for future growth. (probably a Dell PE750 with P4/2.8 and dual GE nics.) However, since I have yet to prove that processor speed has anything to do with my random slow response times, I have this horrible nightmare that I will build a brand new 2.8Ghz firewall and *have the same problem*! So I won''t bore you with the details, but simply ask that anyone who is using iptables/shorewall on an aging CPU (say from 100-500 Mhz) supporting several hundred clients on a 10Mb link or faster, please let me know, on or off list. I just hate not knowing what is causing our problems, and having them occur on a new, fast firewall would probably push me over the edge.... Thanks. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Shawn Wright, I.T. Manager Shawnigan Lake School http://www.sls.bc.ca swright@sls.bc.ca _______________________________________________ Shorewall-users mailing list Post: Shorewall-users@lists.shorewall.net Subscribe/Unsubscribe: https://lists.shorewall.net/mailman/listinfo/shorewall-users Support: http://www.shorewall.net/support.htm FAQ: http://www.shorewall.net/FAQ.htm
On 3 Dec 2004 at 12:19, Tom Eastep wrote:> On Fri, 2004-12-03 at 12:00 -0800, Shawn Wright wrote: > > Ok, I''ve flogged this issue probably longer than some of you can stand > > by now. (remember, I''m the nut trying to use a PPro200 to support ~500 > > users on a 10Mb internet link :o) > > Shawn, > > I just looked at your ''good'' and ''bad'' configs again -- the policy files > are different; in the ''bad'' config, you are rate-limiting with lots of > traffic being dropped.Tom, The change was made in the first week of troubleshooting in an effort to reduce log sizes (it didn''t help). My understanding was the rate limit only applied to those packets falling through to these default policies that would then be dropped or rejected. I have now just re-read the docs on this, and now I am not sure about this... am I wrong? -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Shawn Wright, I.T. Manager Shawnigan Lake School http://www.sls.bc.ca swright@sls.bc.ca
On Fri, 2004-12-03 at 14:35 -0800, Shawn Wright wrote:> On 3 Dec 2004 at 12:19, Tom Eastep wrote: > > > On Fri, 2004-12-03 at 12:00 -0800, Shawn Wright wrote: > > > Ok, I''ve flogged this issue probably longer than some of you can stand > > > by now. (remember, I''m the nut trying to use a PPro200 to support ~500 > > > users on a 10Mb internet link :o) > > > > Shawn, > > > > I just looked at your ''good'' and ''bad'' configs again -- the policy files > > are different; in the ''bad'' config, you are rate-limiting with lots of > > traffic being dropped. > > Tom, > > The change was made in the first week of troubleshooting in an effort to > reduce log sizes (it didn''t help). My understanding was the rate limit only > applied to those packets falling through to these default policies that > would then be dropped or rejected. I have now just re-read the docs on > this, and now I am not sure about this... am I wrong?You are wrong: The LIMIT:BURST column in /etc/shorewall/policy defines a maximum TCP connection rate for *ACCEPTED* connections. If the first policy to match Z1->Z2 specifies a value in this column, then TCP connections from Z1 to Z2 are aggregated with all other zone pairs that match that policy and the limit is applied to the aggregate. Unfortunately, traffic that exceeds the limit is silently dropped. In your "bad" shorewall status output, we find: Chain @all2all (4 references) pkts bytes target prot opt in out source destination 7674 370K RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 limit: avg 5/sec burst 20 14975 722K DROP all -- * * 0.0.0.0/0 0.0.0.0/0 So approximately 2/3 of the TCP connections being passed to @all2all are being silently dropped. And we also find: Chain sls2net (1 references) pkts bytes target prot opt in out source destination 110K 16M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED 22637 1091K @all2all tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x16/0x02 So all TCP traffic from sls to the internet is being rate-limited by that chain. Your diagnostic measure has actually been contributing to your problem -- likely it has been contributing more than whatever the original problem was. Because rate limiting is a syn-flood protection measure, I chose not to log dropped packets. To try to clue in folks like yourself that stumble into this in the future, I''ve added code to both 2.0 and 2.2 to log the dropped packets at a slow rate (5/min with burst 5). -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
On 3 Dec 2004 at 14:47, Tom Eastep wrote:> You are wrong:Thank you for confirming this.> The LIMIT:BURST column in /etc/shorewall/policy defines a maximum TCP > connection rate for *ACCEPTED* connections. If the first policy to match > Z1->Z2 specifies a value in this column, then TCP connections from Z1 to > Z2 are aggregated with all other zone pairs that match that policy and > the limit is applied to the aggregate. Unfortunately, traffic that > exceeds the limit is silently dropped. > > In your "bad" shorewall status output, we find: > > Chain @all2all (4 references) > pkts bytes target prot opt in out source > destination > 7674 370K RETURN all -- * * 0.0.0.0/0 > 0.0.0.0/0 limit: avg 5/sec burst 20 > 14975 722K DROP all -- * * 0.0.0.0/0 > 0.0.0.0/0 > > So approximately 2/3 of the TCP connections being passed to @all2all are > being silently dropped. And we also find: > > Chain sls2net (1 references) > pkts bytes target prot opt in out source > destination > 110K 16M ACCEPT all -- * * 0.0.0.0/0 > 0.0.0.0/0 state RELATED,ESTABLISHED > 22637 1091K @all2all tcp -- * * 0.0.0.0/0 > 0.0.0.0/0 tcp flags:0x16/0x02 > > So all TCP traffic from sls to the internet is being rate-limited by > that chain. > > Your diagnostic measure has actually been contributing to your problem > -- likely it has been contributing more than whatever the original > problem was. > > Because rate limiting is a syn-flood protection measure, I chose not to > log dropped packets. To try to clue in folks like yourself that stumble > into this in the future, I''ve added code to both 2.0 and 2.2 to log the > dropped packets at a slow rate (5/min with burst 5).Does this code exist in 2.0.10? If so, are the log entries identified as DROPs due to LIMIT or some other way to distinguish them? Would you mind giving an example of how to properly use rate-limit for syn-flood protection? And perhaps how to effectively limit log entries as I was intending to do? I realize now that I have been completely clueless on this feature, partly because I took these two fragments of documentation (yours, and the iptables man page) and pieced them together without understanding the whole process. I see now why our traffic was affected by this in the all2all policy. Thanks for clearing it up. -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Shawn Wright, I.T. Manager Shawnigan Lake School http://www.sls.bc.ca swright@sls.bc.ca
On Fri, 2004-12-03 at 15:45 -0800, Shawn Wright wrote:> > Does this code exist in 2.0.10? If so, are the log entries identified as > DROPs due to LIMIT or some other way to distinguish them?The code exists only in CVS -- it has not been released. The attached patch should apply to 2.0.10 with offsets. You can consult Shorewall FAQ 17 (http://shorewall.net/FAQ.htm#faq17) for information about how to distinguish these messages (the chain name begins with ''@'' as you can see in my previous message). Note that FAQ 17 mentions 2.2.0 Beta 7 but not 2.0.14 -- I need to fix that before releasing 2.0.14.> > Would you mind giving an example of how to properly use rate-limit for > syn-flood protection? And perhaps how to effectively limit log entries as I > was intending to do?I use it in my configuration as follows: net all DROP $LOG 10/sec:40 This limits TCP connections coming from the net to 10 per second but can accomodate bursts as large as 40. For my little server and its slow DSL line, that''s ok. I chose it through some experimentation where I reduced the limit and burst until normal daily traffic started to be dropped. Then I increased it with a little headroom. I issue "shorewall show @net2all" occasionally to check on it. I don''t recommend specifying a rate on the ''all2all'' policy. To limit overall log message rages, use the LOGLIMIT and LOGBURST parameters in /etc/shorewall/shorewall.conf. I hope the documentation is clear enough for you to follow it but if you have questions, don''t hesitate to ask.> Thanks for clearing it up.You''re welcome -- sorry I didn''t see it two days ago :-( -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key
On 3 Dec 2004 at 16:00, Tom Eastep wrote:> On Fri, 2004-12-03 at 15:45 -0800, Shawn Wright wrote: > > > > > Does this code exist in 2.0.10? If so, are the log entries identified as > > DROPs due to LIMIT or some other way to distinguish them? > > The code exists only in CVS -- it has not been released. The attached > patch should apply to 2.0.10 with offsets. > > You can consult Shorewall FAQ 17 (http://shorewall.net/FAQ.htm#faq17) > for information about how to distinguish these messages (the chain name > begins with ''@'' as you can see in my previous message). Note that FAQ 17 > mentions 2.2.0 Beta 7 but not 2.0.14 -- I need to fix that before > releasing 2.0.14. > > > > > Would you mind giving an example of how to properly use rate-limit for > > syn-flood protection? And perhaps how to effectively limit log entries as I > > was intending to do? > > I use it in my configuration as follows: > > net all DROP $LOG 10/sec:40Excellent, I will look into this as time permits. Our outbound traffic is rarely anything to worry about though.> This limits TCP connections coming from the net to 10 per second but can > accomodate bursts as large as 40. For my little server and its slow DSL > line, that''s ok. > > I chose it through some experimentation where I reduced the limit and > burst until normal daily traffic started to be dropped. Then I increased > it with a little headroom. I issue "shorewall show @net2all" > occasionally to check on it. > > I don''t recommend specifying a rate on the ''all2all'' policy.I now see why... :-)> To limit overall log message rages, use the LOGLIMIT and LOGBURST > parameters in /etc/shorewall/shorewall.conf. I hope the documentation is > clear enough for you to follow it but if you have questions, don''t > hesitate to ask. > > > Thanks for clearing it up. > > You''re welcome -- sorry I didn''t see it two days ago :-(Not a problem. Had I actually had time to study this problem without an interruption every 5 minutes, I might have actually found this myself, although I still might not have understood it as I do now. I''m pleased to report the old PPro200 is back in service, and has handled about 1Gb in the past 40 minutes, and is nice and quick. A new server is still in the works, since our recent peak usage means we will likely need a 100Mb feed pretty soon...our users are impatient. :-o) Thanks again, this is a nice way to end the week for a change! :-) -=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=- Shawn Wright, I.T. Manager Shawnigan Lake School http://www.sls.bc.ca swright@sls.bc.ca
On Fri, 2004-12-03 at 17:30 -0800, Shawn Wright wrote:> I''m pleased to report the old PPro200 is back in service, and has handled > about 1Gb in the past 40 minutes, and is nice and quick. A new server is > still in the works, since our recent peak usage means we will likely need a > 100Mb feed pretty soon...our users are impatient. :-o)That''s probably wise -- I suspect that you were experiencing peek-demand performance problems with the old server and that the rest of the problems are a result of your rate-limiting snafu. -Tom -- Tom Eastep \ Nothing is foolproof to a sufficiently talented fool Shoreline, \ http://shorewall.net Washington USA \ teastep@shorewall.net PGP Public Key \ https://lists.shorewall.net/teastep.pgp.key