Hi again, I use some keep-state rules in ipfw, but get the following kernel message: kernel: ipfw: install_state: Too many dynamic rules when presumably my state table reaches its limit (and I effectively get DoS'd). netstat shows tons of connections in FIN_WAIT_2 state, mostly to my webserver. Consequently net.inet.ip.fw.dyn_count is large too. I can increase my net.inet.ip.fw.dyn_max but the new limit will simply be reached later on. I currently get around this with a cronjob that sets net.inet.ip.fw.dyn_keepalive to 0 for just less than 5 minutes every night. If I leave it at 0 for longer or indefinitely then idle ssh sessions and the like are dropped. This works fine for me but it looks like there's some bug with net.inet.ip.fw.dyn_keepalive=1? Or with Apache? I'm using 8.1-STABLE, GENERIC kernel. Experienced the same behaviour on 8.0-RELEASE, but not on 6.1-RELEASE where I had a similar setup. I have a KeepAliveTimeout of 4 in Apache (2.2.16).
On Thu, Sep 09, 2010 at 05:39:02PM +0200, Gareth de Vaux wrote:> Hi again, I use some keep-state rules in ipfw, but get the following > kernel message: > > kernel: ipfw: install_state: Too many dynamic rules > > when presumably my state table reaches its limit (and I effectively > get DoS'd). > > netstat shows tons of connections in FIN_WAIT_2 state, mostly to > my webserver. Consequently net.inet.ip.fw.dyn_count is large too. > > I can increase my net.inet.ip.fw.dyn_max but the new limit will > simply be reached later on. > > I currently get around this with a cronjob that sets > net.inet.ip.fw.dyn_keepalive to 0 for just less than 5 minutes > every night. If I leave it at 0 for longer or indefinitely then > idle ssh sessions and the like are dropped. This works fine for > me but it looks like there's some bug with net.inet.ip.fw.dyn_keepalive=1? > Or with Apache? > > I'm using 8.1-STABLE, GENERIC kernel. Experienced the same behaviour > on 8.0-RELEASE, but not on 6.1-RELEASE where I had a similar setup. I > have a KeepAliveTimeout of 4 in Apache (2.2.16).Firstly, I'm not familiar with dynamic firewall rules in ipfw. I tend to use pf these days, with ALTQ for rate-limiting. pf offers a lot of improvements over ipfw. Secondly, I'm fairly certain HTTP KeepAlive (re: KeepAliveTimeout) are unrelated to TCP keepalives[1]. I mention this because you're focusing on netstat, which will give you indication of TCP session state, not HTTP protocol statefulness. Thirdly, if you feel FIN_WAIT2 is the cause of your problem, then you should consider adjusting the following sysctl: net.inet.tcp.finwait2_timeout Try something like 15000 (15 seconds) instead of the default (60000). Finally, why are you using dynamic firewall rules at all? For what purpose do you need these that, say, pf and its state tracking would not suffice? [1]: http://en.wikipedia.org/wiki/Keepalive -- | Jeremy Chadwick jdc@parodius.com | | Parodius Networking http://www.parodius.com/ | | UNIX Systems Administrator Mountain View, CA, USA | | Making life hard for others since 1977. PGP: 4BD6C0CB |
On Thu, 9 Sep 2010, Gareth de Vaux wrote: > Hi again, I use some keep-state rules in ipfw, but get the following > kernel message: > > kernel: ipfw: install_state: Too many dynamic rules > > when presumably my state table reaches its limit (and I effectively > get DoS'd). > > netstat shows tons of connections in FIN_WAIT_2 state, mostly to > my webserver. Consequently net.inet.ip.fw.dyn_count is large too. > > I can increase my net.inet.ip.fw.dyn_max but the new limit will > simply be reached later on. Try using 'limit' rather than the unlimited 'keep-state' for inbound dynamic connections to your server/s. eg, derived from ipfw(8): ipfw add allow tcp from any to me 80 setup limit src-addr 4 ".. can be placed on a server to make sure that a single client does not use more than 4 simultaneous connections." You could add 'in recv $ext_if' to avoid limiting internal clients. > I currently get around this with a cronjob that sets > net.inet.ip.fw.dyn_keepalive to 0 for just less than 5 minutes > every night. If I leave it at 0 for longer or indefinitely then > idle ssh sessions and the like are dropped. This works fine for > me but it looks like there's some bug with net.inet.ip.fw.dyn_keepalive=1? > Or with Apache? Limiting the number of source connections per source address to what apache is happy to deal with, you mightn't need to fuss with that? cheers, Ian > I'm using 8.1-STABLE, GENERIC kernel. Experienced the same behaviour > on 8.0-RELEASE, but not on 6.1-RELEASE where I had a similar setup. I > have a KeepAliveTimeout of 4 in Apache (2.2.16).
Gareth de Vaux wrote:> Hi again, I use some keep-state rules in ipfw, but get the following > kernel message: > > kernel: ipfw: install_state: Too many dynamic rules > > when presumably my state table reaches its limit (and I effectively > get DoS'd). > > netstat shows tons of connections in FIN_WAIT_2 state, mostly to > my webserver. Consequently net.inet.ip.fw.dyn_count is large too. > > I can increase my net.inet.ip.fw.dyn_max but the new limit will > simply be reached later on. > > I currently get around this with a cronjob that sets > net.inet.ip.fw.dyn_keepalive to 0 for just less than 5 minutes > every night. If I leave it at 0 for longer or indefinitely then > idle ssh sessions and the like are dropped. This works fine for > me but it looks like there's some bug with net.inet.ip.fw.dyn_keepalive=1? > Or with Apache? > > I'm using 8.1-STABLE, GENERIC kernel. Experienced the same behaviour > on 8.0-RELEASE, but not on 6.1-RELEASE where I had a similar setup. I > have a KeepAliveTimeout of 4 in Apache (2.2.16). > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >I wonder, are these dynamic rules really necessary? let's see, a client connects to your web-server and you immediately should create a new dynamic rule, therefore you participate in this DoS attack as well as attacker. ;) may be using ipfw add XXX allow tcp from me 80 to any would be enough? I usually use keep-state rules only for outgoing connections and try to keep number of such rules as few as possible. if you're afraid of somebody trying to attack your servers using unopened connections you may filter out connections that aren't established. you can try to change, of course, net.inet.ip.fw.dyn_*_lifetime variables, but I think that using dynamic rules to filter out web-server answers is not as good practice as it seems. -- SY, Marat
On 09/09/10 17:39, Gareth de Vaux wrote:> Hi again, I use some keep-state rules in ipfw, but get the following > kernel message: > > kernel: ipfw: install_state: Too many dynamic rules > > when presumably my state table reaches its limit (and I effectively > get DoS'd). > > netstat shows tons of connections in FIN_WAIT_2 state, mostly to > my webserver. Consequently net.inet.ip.fw.dyn_count is large too. > > I can increase my net.inet.ip.fw.dyn_max but the new limit will > simply be reached later on.For what it's worth, here's what I've been running: net.inet.ip.fw.dyn_buckets=1024 net.inet.ip.fw.dyn_max=8192 net.inet.ip.fw.dyn_ack_lifetime=60 If in a tight spot, I might reduce dyn_ack_lifetime to 10. There is no way this machine would service 8192 legitimate simultaneous connections so this works for me. If you have the memory I think you can increase dyn_max practically arbitrarily. If under a DDoS attack, you might run out of some other resource, like ephemeral TCP ports for the server side of connections, before running out of ipfw entries.