Tim Chen
2009-Nov-03 08:29 UTC
Increasing number of "requests for jumbo clusters denied" in netstat -m
My machine is an IBM HS21 blade server and the NIC is bce.
bce1: <Broadcom NetXtreme II BCM5708 1000Base-SX (B2)> mem
0xd8000000-0xd9ffffff irq 19 at device 0.0 on pci6
miibus1: <MII bus> on bce1
bce1: Ethernet address: 00:1a:64:34:f2:fa
bce1: [ITHREAD]
bce1: ASIC (0x57081021); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C
(0x03040405) ; Flags( MSI )
That machine is serving as a mail and web server. Recently I found that the
number of "requests for jumbo clusters denied" in netstat -m increases
all
the time.
1031/3469/4500 mbufs in use (current/cache/total)
510/3326/3836/32768 mbuf clusters in use (current/cache/total/max)
510/2278 mbuf+clusters out of packet secondary zone in use (current/cache)
1/1453/1454/8704 4k (page size) jumbo clusters in use
(current/cache/total/max)
510/1086/1596/4352 9k jumbo clusters in use (current/cache/total/max)
0/0/0/2176 16k jumbo clusters in use (current/cache/total/max)
5871K/23105K/28977K bytes allocated to network (current/cache/total)
0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters)
0/4337166/0 requests for jumbo clusters denied (4k/9k/16k)
0/0/0 sfbufs in use (current/peak/max)
0 requests for sfbufs denied
0 requests for sfbufs delayed
0 requests for I/O initiated by sendfile
0 calls to protocol drain routines
ifconfig shows:
bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,
TSO4>
ether 00:1a:64:34:f2:fa
inet 192.168.152.152 netmask 0xffffff00 broadcast 192.168.152.255
media: Ethernet autoselect (1000baseSX <full-duplex>)
status: active
And in rc.conf:
ifconfig_bce1="inet 192.168.152.152 netmask 255.255.255.0 mtu 9000"
Is the problem of increasing "requests for jumbo clusters denied"
resulted
from my "mtu 9000" jumbo frame setting?
Will it be the problem of "bce" nic's driver or hardware?
Most of all, how will my system be efftected by that ""requests for
jumbo
clusters denied" problem? Will it degrade
the performance or it is harmless?
Thanks for your help.
BR,
Tim Chen
Eugene Grosbein
2009-Nov-03 11:02 UTC
Increasing number of "requests for jumbo clusters denied" in netstat -m
Tim Chen wrote:> That machine is serving as a mail and web server. Recently I found that the > number of "requests for jumbo clusters denied" in netstat -m increases all > the time. > > 1031/3469/4500 mbufs in use (current/cache/total) > 510/3326/3836/32768 mbuf clusters in use (current/cache/total/max) > 510/2278 mbuf+clusters out of packet secondary zone in use (current/cache) > 1/1453/1454/8704 4k (page size) jumbo clusters in use > (current/cache/total/max) > 510/1086/1596/4352 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/2176 16k jumbo clusters in use (current/cache/total/max) > 5871K/23105K/28977K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/4337166/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routinesLook at "sysctl kern.ipc | fgrep nmb" Just increase limit (if you have enough memory :-)
Tom Judge
2009-Nov-03 19:23 UTC
Increasing number of "requests for jumbo clusters denied" in netstat -m
Tim Chen wrote:> My machine is an IBM HS21 blade server and the NIC is bce. > > bce1: <Broadcom NetXtreme II BCM5708 1000Base-SX (B2)> mem > 0xd8000000-0xd9ffffff irq 19 at device 0.0 on pci6 > miibus1: <MII bus> on bce1 > bce1: Ethernet address: 00:1a:64:34:f2:fa > bce1: [ITHREAD] > bce1: ASIC (0x57081021); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); B/C > (0x03040405) ; Flags( MSI ) > > That machine is serving as a mail and web server. Recently I found that the > number of "requests for jumbo clusters denied" in netstat -m increases all > the time. > > 1031/3469/4500 mbufs in use (current/cache/total) > 510/3326/3836/32768 mbuf clusters in use (current/cache/total/max) > 510/2278 mbuf+clusters out of packet secondary zone in use (current/cache) > 1/1453/1454/8704 4k (page size) jumbo clusters in use > (current/cache/total/max) > 510/1086/1596/4352 9k jumbo clusters in use (current/cache/total/max) > 0/0/0/2176 16k jumbo clusters in use (current/cache/total/max) > 5871K/23105K/28977K bytes allocated to network (current/cache/total) > 0/0/0 requests for mbufs denied (mbufs/clusters/mbuf+clusters) > 0/4337166/0 requests for jumbo clusters denied (4k/9k/16k) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > 0 calls to protocol drain routines > > ifconfig shows: > bce1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000 > > options=1bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM, > TSO4> > ether 00:1a:64:34:f2:fa > inet 192.168.152.152 netmask 0xffffff00 broadcast 192.168.152.255 > media: Ethernet autoselect (1000baseSX <full-duplex>) > status: active > > And in rc.conf: > ifconfig_bce1="inet 192.168.152.152 netmask 255.255.255.0 mtu 9000" > > Is the problem of increasing "requests for jumbo clusters denied" resulted > from my "mtu 9000" jumbo frame setting? > Will it be the problem of "bce" nic's driver or hardware? > Most of all, how will my system be efftected by that ""requests for jumbo > clusters denied" problem? Will it degrade > the performance or it is harmless? > >Second thread on this bug today :) There is a bug in bce(4) that causes memory fragmentation, which results in denied mbuf requests as you are seeing. You can correct this issue by applying the patch created by this command: svn diff -r 198319:198320 http://svn.freebsd.org/base/head Once you have applied the patch you need to add: options BCE_JUMBO_HDRSPLIT To you kernel config and recompile and install the new kernel. Hope this works for you. Tom