freebsd stable - Feb 2010 - Sudden mbuf demand increase and shortage under the load

Hi,

Our company have a FreeBSD based product that consists of the numerous
interconnected processes and it does some high-PPS UDP processing
(30-50K PPS is not uncommon). We are seeing some strange periodic
failures under the load in several such systems, which usually evidences
itself in IPC (even through unix domain sockets) suddenly either
breaking down or pausing and restoring only some time later (like 5-10
minutes). The only sign of failure I managed to find was the increase of
the "requests for mbufs denied" in the netstat -m and number of total
mbuf clusters (nmbclusters) raising up to the limit.

I have tried to raise some network-related limits (most notably maxusers
and nmbclusters), but it has not helped with the issue - it's still
happening from time to time to us. Below you can find output from the
netstat -m few minutes right after that shortage period - you see that
somehow the system has allocated huge amount of memory for the network
(700MB), with only tiny amount of that being actually in use. This is
for the kern.ipc.nmbclusters: 302400. Eventually the system reclaims all
that memory and goes back to its normal use of 30-70MB.

This problem is killing us, so any suggestions are greatly appreciated.
My current hypothesis is that due to some issues either with the network
driver or network subsystem itself, the system goes insane and "eats"
up
all mbufs up to nmbclusters limit. But since mbufs are shared between
network and local IPC, IPC goes down as well.

We observe this issue with systems using both em(4) driver and igb(4)
driver. I believe both drivers share the same design, however I am not
sure if this is some kind of design flaw in the driver or part of a
larger problem with the network subsystem.

This happens on amd64 7.2-RELEASE and 7.3-PRERELEASE alike, with 8GB of
memory. I have not tried upgrading to 8.0, this is production system so
upgrading will not be easy.  I don't believe there are some differences
that let us hope that this problem will go away after upgrade, but I can
try it as the last resort.

As I said, this is very critical issue, so I can provide any additional
debug information upon request. We are ready to go as far as paying
somebody reasonable amount of money for tracking down and resolving the
issue.

Regards,
-- 
Maksym Sobolyev
Sippy Software, Inc.
Internet Telephony (VoIP) Experts
T/F: +1-646-651-1110
Web: http://www.sippysoft.com
MSN: sales@sippysoft.com
Skype: SippySoft


[ssp-root@ds-467 /usr/src]$ netstat -m
17061/417669/434730 mbufs in use (current/cache/total)
10420/291980/302400/302400 mbuf clusters in use (current/cache/total/max)
10420/0 mbuf+clusters out of packet secondary zone in use (current/cache)
19/1262/1281/51200 4k (page size) jumbo clusters in use
(current/cache/total/max)
0/0/0/25600 9k jumbo clusters in use (current/cache/total/max)
0/0/0/12800 16k jumbo clusters in use (current/cache/total/max)
25181K/693425K/718606K bytes allocated to network (current/cache/total)
1246681/129567494/67681640 requests for mbufs denied
(mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

[FEW MINUTES LATER]

[ssp-root@ds-467 /usr/src]$ netstat -m
10001/84574/94575 mbufs in use (current/cache/total)
6899/6931/13830/302400 mbuf clusters in use (current/cache/total/max)
6899/6267 mbuf+clusters out of packet secondary zone in use (current/cache)
2/1151/1153/51200 4k (page size) jumbo clusters in use
(current/cache/total/max)
0/0/0/25600 9k jumbo clusters in use (current/cache/total/max)
0/0/0/12800 16k jumbo clusters in use (current/cache/total/max)
16306K/39609K/55915K bytes allocated to network (current/cache/total)
1246681/129567494/67681640 requests for mbufs denied
(mbufs/clusters/mbuf+clusters)
0/0/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines

On 02/15/10 13:25, Maxim Sobolev wrote:
> Hi,
>
> Our company have a FreeBSD based product that consists of the numerous
> interconnected processes and it does some high-PPS UDP processing
> (30-50K PPS is not uncommon). We are seeing some strange periodic
I have nothing very useful to help you with but maybe you can detect if 
it's a em/igp issue by buying a cheap Realtek gigabit (re) card and 
trying it out. Those can be bought for a few dollars now (e.g. from 
D-Link and many others), and I can confirm that at least the one I tried 
can carry around 50K pps, but not much more (I can tell you the exact 
chip later today if you are interested).
> failures under the load in several such systems, which usually evidences
> itself in IPC (even through unix domain sockets) suddenly either
> breaking down or pausing and restoring only some time later (like 5-10
> minutes). The only sign of failure I managed to find was the increase of
> the "requests for mbufs denied" in the netstat -m and number of
total
> mbuf clusters (nmbclusters) raising up to the limit.
>
> I have tried to raise some network-related limits (most notably maxusers
> and nmbclusters), but it has not helped with the issue - it's still
> happening from time to time to us. Below you can find output from the
> netstat -m few minutes right after that shortage period - you see that
> somehow the system has allocated huge amount of memory for the network
> (700MB), with only tiny amount of that being actually in use. This is
> for the kern.ipc.nmbclusters: 302400. Eventually the system reclaims all
> that memory and goes back to its normal use of 30-70MB.
>
> This problem is killing us, so any suggestions are greatly appreciated.
> My current hypothesis is that due to some issues either with the network
> driver or network subsystem itself, the system goes insane and
"eats" up
> all mbufs up to nmbclusters limit. But since mbufs are shared between
> network and local IPC, IPC goes down as well.
>
> We observe this issue with systems using both em(4) driver and igb(4)
> driver. I believe both drivers share the same design, however I am not
> sure if this is some kind of design flaw in the driver or part of a
> larger problem with the network subsystem.
>
> This happens on amd64 7.2-RELEASE and 7.3-PRERELEASE alike, with 8GB of
> memory. I have not tried upgrading to 8.0, this is production system so
> upgrading will not be easy. I don't believe there are some differences
> that let us hope that this problem will go away after upgrade, but I can
> try it as the last resort.
>
> As I said, this is very critical issue, so I can provide any additional
> debug information upon request. We are ready to go as far as paying
> somebody reasonable amount of money for tracking down and resolving the
> issue.
>
> Regards,

* Maxim Sobolev <sobomax@FreeBSD.org> [100215 04:49]
wrote:
> Hi,
> 
> Our company have a FreeBSD based product that consists of the numerous
> interconnected processes and it does some high-PPS UDP processing
> (30-50K PPS is not uncommon). We are seeing some strange periodic
> failures under the load in several such systems, which usually evidences
> itself in IPC (even through unix domain sockets) suddenly either
> breaking down or pausing and restoring only some time later (like 5-10
> minutes). The only sign of failure I managed to find was the increase of
> the "requests for mbufs denied" in the netstat -m and number of
total
> mbuf clusters (nmbclusters) raising up to the limit.
Hey Maxim

Can you run a process to dump sysctl -a every second or so
and mark the time when you did it?

Other monitoring things would probably be helpful as well (netstat -m)
in a timed log format.  vmstat -i?  (interrupts storm?)

Perhaps ps output (showing interrupt threads, etc) would be good toknow
perhaps some ithreads went off into the weeds...

Any console messages of note?

A few people have suggested that there may be too many packets
on the outgoing interface, I think there should be a limit to the
number of packets queued for outgoing and probably counters to
show how many were dropped due to overflow of the outgoing queue.

You should be able to check these counters to see what is going
on. If the driver is broken and never drops outgoing packets when
the card's queue is full, then those counters should be 0.

I hope this helps.

-Alfred