freebsd stable - Mar 2015 - Stale TIME_WAIT tcp connections

Hello everyone,

We have a server running 9.3-RELEASE which is exhibiting a high number of
TIME_WAIT tcp connections which are NOT being recycled. That is, netstat
reports them over and over again, no matter how long we wait for them to be
flushed out. Currently this server has been out of rotation for a couple of
hours and I still see the same tcp sockets there. Overall we have:

# netstat -na | grep TIME_WAIT | wc -l
   *30066*

Tracking one particular TCP socket in TIME_WAIT proves that it stays there
all the time.

Another observation is that pfctl shows a very large number of state
entries, even after pfctl -F all, or disable/enable sequence.

# pfctl -si
State Table                          Total             Rate
  current entries                    *59280*

At the same time though:

# pfctl -ss | wc -l
      18

After the problem was discovered we tried tweaking the following settings
without any luck:

net.inet.tcp.fast_finwait2_recycle=1
net.inet.tcp.finwait2_timeout=5000
net.inet.tcp.maxtcptw=50000
net.inet.tcp.msl=100

?So it seems like this system is "stuck" and ?doesn't recycle
those TCP
sockets. Again, the machine is out of rotation and not actively accepting
any traffic. I will keep it like that in case further investigation is
required. Please do let me know if there's anything else you'd like to
know
from the state of the machine or something I could try.

?Regards,
-- 
Rumen Telbizov
Unix Systems Administrator <http://telbizov.com>

On Wed, 04 Mar 2015 01:36:18 +0100, Rumen Telbizov <telbizov at gmail.com>
wrote:
> Hello everyone,
>
> We have a server running 9.3-RELEASE which is exhibiting a high number of
> TIME_WAIT tcp connections which are NOT being recycled. That is, netstat
> reports them over and over again, no matter how long we wait for them to  
> be
> flushed out. Currently this server has been out of rotation for a couple  
> of
> hours and I still see the same tcp sockets there. Overall we have:
>
> # netstat -na | grep TIME_WAIT | wc -l
>    *30066*
>
> Tracking one particular TCP socket in TIME_WAIT proves that it stays  
> there
> all the time.
>
> Another observation is that pfctl shows a very large number of state
> entries, even after pfctl -F all, or disable/enable sequence.
>
> # pfctl -si
> State Table                          Total             Rate
>   current entries                    *59280*
>
> At the same time though:
>
> # pfctl -ss | wc -l
>       18
>
> After the problem was discovered we tried tweaking the following settings
> without any luck:
>
> net.inet.tcp.fast_finwait2_recycle=1
> net.inet.tcp.finwait2_timeout=5000
> net.inet.tcp.maxtcptw=50000
> net.inet.tcp.msl=100
>
> ?So it seems like this system is "stuck" and ?doesn't recycle
those TCP
> sockets. Again, the machine is out of rotation and not actively accepting
> any traffic. I will keep it like that in case further investigation is
> required. Please do let me know if there's anything else you'd like
to
> know
> from the state of the machine or something I could try.
>
> ?Regards,
Are you using any IPSEC?
I observed something similar a while back, haven't checked again since i  
reported this.
https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=194690
Affected 9.2, too.

Michael