freebsd stable - May 2008 - Sockets stuck in FIN_WAIT_1

I have a rather busy Apache 2.2 server; tons of small & some large  
requests.  It's a standard Dell 2650 server using the bge (broadcom)  
network driver.

I seem to have a rather strange problem where after just a day or so  
Apache just stops processing new connections.  You can connect to port  
80, but trying to get Apache to process any data just hangs.  There is  
nothing strange in dmesg or in /var/log/messages.

The server has plenty free available physical RAM, swap is untouched,  
CPU load is low, etc.  Apache is setup to handle a max of 100 clients  
using prefork model.

If I stop and restart Apache, it does not help.

What I do notice is 1000's of sockets stuck in "FIN_WAIT_1" in
netstat:

[web0:~] netstat -an | grep FIN_WAIT | wc -l
     1827

These stick around forever.  Some eventually trickle away after hours,  
but the only thing that appears to fix it is to reboot the server.   
Then all is fine for another day or so.  I've tried just about every  
tuning trick out there but to no eval.  I can mitigate the problem by  
increasing available socket buffs and decreasing the tcp.sendspace.   
I've tried different versions of Apache and I've tried with and  
without the accf_http kernel filter.

Here is what I have on the server now:

sysctl.conf:

kern.maxfiles=65535
kern.maxfilesperproc=16384
kern.ipc.maxsockbuf=4194304
kern.ipc.somaxconn=1024
net.inet.tcp.sendspace=8192
net.inet.tcp.recvspace=8192
net.inet.tcp.keepidle=900000
net.inet.tcp.keepintvl=30000
net.inet.tcp.msl=5000
net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.tcp.inflight_enable=1


and loader.conf

accf_http_load="YES"
kern.ipc.nmbclusters=32768
net.inet.tcp.tcbhashsize=4096
kern.ipc.maxsockets=131072


ipfw:

00200 allow tcp from any to me 80 setup
00200 allow icmp from any to me icmptype 0,3,8,11
00200 deny log ip from any to me


ifconfig:
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 1500
	options=3<RXCSUM,TXCSUM>
	inet 1.2.3.4 netmask 0xfffffff8 broadcast 5.6.7.8
	ether 00:06:5b:f7:c8:7b
	media: Ethernet autoselect (1000baseTX <full-duplex>)



Any ideas would be greatly appreciated.

-- 
Robert Blayzor, BOFH
INOC, LLC
rblayzor@inoc.net
http://www.inoc.net/~rblayzor/

Hi--

You didn't mention which version of FreeBSD you are running-- that's  
rather important info.

On May 28, 2008, at 3:13 PM, Robert Blayzor wrote:
> ipfw:
>
> 00200 allow tcp from any to me 80 setup
> 00200 allow icmp from any to me icmptype 0,3,8,11
> 00200 deny log ip from any to me
Also, surely these can't be the only IPFW rules you are using?  If you  
want to use stateful rules, you need a keep-state argument, and you  
shouldn't be combining allow rules and deny rules into the same  
ruleset number...

-- 
-Chuck

On Wed, May 28, 2008 at 06:13:04PM -0400, Robert Blayzor
wrote:
> Here is what I have on the server now:
>
> and loader.conf
>
> accf_http_load="YES"
You shouldn't bother with this.  Let the apache22 rc.d script handle
loading it dynamically.  Use apache22_http_accept_enable="yes" in
rc.conf.

I've read reports in the past how the accept filter can cause FIN_WAIT
issues, but I can't remember if that's FIN_WAIT_2 or FIN_WAIT_1.

I've also read that pf(4) could possibly cause what you're seeing, so if
you have broken firewall rules, those may be responsible for this
behaviour.

Finally, for what it's worth, we don't have this problem on RELENG_6 or
RELENG_7 using Apache 2.2 with the accept filter, and also use pf(4).
We use absolutely no tuning in sysctl.conf for any of the things you
have listed.

-- 
| Jeremy Chadwick                                jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |