Joerg Pernfuss
2006-Nov-28 10:57 UTC
kernel statistics weirdness on RELENG_6{_1,_2} / 64bit?
Hi, a while ago I noticed something odd on my 64bit FreeBSD 6 systems - two amd64, one alpha. I can't make head or tails of it, so I post it in hope that someone can. Box #1: elessar@snake: ~% uname -srm FreeBSD 6.1-RELEASE-p7 amd64 elessar@snake: ~% grep kern.ipc /boot/loader.conf kern.ipc.nsfbufs="8192" kern.ipc.nmbclusters="49152" elessar@snake: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters kern.ipc.nsfbufs: 0 kern.ipc.nmbclusters: 32768 elessar@snake: ~% netstat -m 350/132/482/49152 mbuf clusters in use (current/cache/total/max) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 2864770 requests for I/O initiated by sendfile Box #2: elessar@forseti: ~% uname -srm FreeBSD 6.2-RC1 alpha elessar@forseti: ~% grep kern.ipc /boot/loader.conf kern.ipc.nsfbufs="4294967300" elessar@forseti: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters kern.ipc.nsfbufs: 0 kern.ipc.nmbclusters: 1664 elessar@forseti: ~% netstat -m 64/80/144/1664 mbuf clusters in use (current/cache/total/max) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 0 requests for I/O initiated by sendfile Box #3: elessar@rivendell: ~% uname -srm FreeBSD 6.1-RELEASE-p10 amd64 elessar@rivendell: ~% grep kern.ipc /boot/loader.conf kern.ipc.nsfbufs="8192" kern.ipc.nmbclusters="49152" elessar@rivendell: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters kern.ipc.nsfbufs: 0 kern.ipc.nmbclusters: 32768 elessar@rivendell: ~% grep NSF /root/kernel/RIVENDELL options NSFBUFS=8192 elessar@rivendell: ~% netstat -m 64/118/182/49152 mbuf clusters in use (current/cache/total/max) 0/0/0 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 35 requests for I/O initiated by sendfile For comparison, Box #4: elessar@web2: ~% uname -srm FreeBSD 6.1-STABLE i386 elessar@web2: ~% grep kern.ipc /boot/loader.conf kern.ipc.nsfbufs="8192" elessar@web2: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters kern.ipc.nsfbufs: 8192 kern.ipc.nmbclusters: 32768 elessar@web2: ~% netstat -m 104/272/376/32768 mbuf clusters in use (current/cache/total/max) 1598/2635/8192 sfbufs in use (current/peak/max) 0 requests for sfbufs denied 0 requests for sfbufs delayed 8938512 requests for I/O initiated by sendfile So, looking at the `netstat -m' output, setting kern.ipc.nmbclusters in the loader.conf seems to work, but the sysctl reports the default value that would be in place, if it wasn't overridden in the loader.conf. For the sendfile buffers, the stats are completely out of whack. The sysctl reports 0 for every value (`netstat-m' just prints the sysctl fields for these values if I understood the source right (unlikely). But sendfile() works and the accounting on the number of sendfile calls seems to work too. So I believe the NSFBUFS kernel option and the loader.conf setting work, but it is quite impossible to properly tune the buffers to the workload. I did not bother trying the NMBCLUSTERS kernel option, it was removed a while ago anyway. The loader.conf kern.ipc.nsfbufs value on box #2 is ridiculously high, yes, I tried to get beyond the boundary of a u_int_32 to see if maybe something good or bad happens. It did not. At this point, I wanted to include links to two mails on amd64@ I found when I first searched for this oddity, both of which had no reply. Instead I found a posting to net@ that occured in the meantime, and a reply to it by Yar Tikhiy two days ago explaining the nsfbufs issue. There are no such buffers on amd64, ia64 and alpha as they aren't needed. http://docs.freebsd.org/cgi/getmsg.cgi?fetch=26091+0+current/freebsd-net Ok, that leaves the question why `sysctl kern.ipc.nmbclusters' reports a different value compared to the `netstat -m' output on box #1 and #3. Thanks for any pointers, Joerg PS: documenting the nsfbufs issue on amd64/ia64/alpha might be a good PS: idea. loader(8) seems a good place. Opinions? -- | /"\ ASCII ribbon | GnuPG Key ID | e86d b753 3deb e749 6c3a | | \ / campaign against | 0xbbcaad24 | 5706 1f7d 6cfd bbca ad24 | | X HTML in email | .the next sentence is true. | | / \ and news | .the previous sentence was a lie. | -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20061128/3937a165/signature.pgp
On Tue, Nov 28, 2006 at 07:54:52PM +0100, Joerg Pernfuss wrote:> Hi, > > a while ago I noticed something odd on my 64bit FreeBSD 6 systems - two > amd64, one alpha. I can't make head or tails of it, so I post it in hope > that someone can. > > Box #1: > elessar@snake: ~% uname -srm > FreeBSD 6.1-RELEASE-p7 amd64 > elessar@snake: ~% grep kern.ipc /boot/loader.conf > kern.ipc.nsfbufs="8192" > kern.ipc.nmbclusters="49152" > elessar@snake: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters > kern.ipc.nsfbufs: 0 > kern.ipc.nmbclusters: 32768 > elessar@snake: ~% netstat -m > 350/132/482/49152 mbuf clusters in use (current/cache/total/max) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 2864770 requests for I/O initiated by sendfile > > Box #2: > elessar@forseti: ~% uname -srm > FreeBSD 6.2-RC1 alpha > elessar@forseti: ~% grep kern.ipc /boot/loader.conf > kern.ipc.nsfbufs="4294967300" > elessar@forseti: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters > kern.ipc.nsfbufs: 0 > kern.ipc.nmbclusters: 1664 > elessar@forseti: ~% netstat -m > 64/80/144/1664 mbuf clusters in use (current/cache/total/max) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 0 requests for I/O initiated by sendfile > > Box #3: > elessar@rivendell: ~% uname -srm > FreeBSD 6.1-RELEASE-p10 amd64 > elessar@rivendell: ~% grep kern.ipc /boot/loader.conf > kern.ipc.nsfbufs="8192" > kern.ipc.nmbclusters="49152" > elessar@rivendell: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters > kern.ipc.nsfbufs: 0 > kern.ipc.nmbclusters: 32768 > elessar@rivendell: ~% grep NSF /root/kernel/RIVENDELL > options NSFBUFS=8192 > elessar@rivendell: ~% netstat -m > 64/118/182/49152 mbuf clusters in use (current/cache/total/max) > 0/0/0 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 35 requests for I/O initiated by sendfile > > For comparison, Box #4: > elessar@web2: ~% uname -srm > FreeBSD 6.1-STABLE i386 > elessar@web2: ~% grep kern.ipc /boot/loader.conf > kern.ipc.nsfbufs="8192" > elessar@web2: ~% sysctl kern.ipc.nsfbufs kern.ipc.nmbclusters > kern.ipc.nsfbufs: 8192 > kern.ipc.nmbclusters: 32768 > elessar@web2: ~% netstat -m > 104/272/376/32768 mbuf clusters in use (current/cache/total/max) > 1598/2635/8192 sfbufs in use (current/peak/max) > 0 requests for sfbufs denied > 0 requests for sfbufs delayed > 8938512 requests for I/O initiated by sendfile > > So, looking at the `netstat -m' output, setting kern.ipc.nmbclusters > in the loader.conf seems to work, but the sysctl reports the default > value that would be in place, if it wasn't overridden in the loader.conf. > > For the sendfile buffers, the stats are completely out of whack. > The sysctl reports 0 for every value (`netstat-m' just prints the sysctl > fields for these values if I understood the source right (unlikely). > But sendfile() works and the accounting on the number of sendfile calls > seems to work too. > So I believe the NSFBUFS kernel option and the loader.conf setting work, > but it is quite impossible to properly tune the buffers to the workload. > I did not bother trying the NMBCLUSTERS kernel option, it was removed > a while ago anyway. > > The loader.conf kern.ipc.nsfbufs value on box #2 is ridiculously high, > yes, I tried to get beyond the boundary of a u_int_32 to see if maybe > something good or bad happens. It did not. > > At this point, I wanted to include links to two mails on amd64@ I > found when I first searched for this oddity, both of which had no > reply. Instead I found a posting to net@ that occured in the meantime, > and a reply to it by Yar Tikhiy two days ago explaining the nsfbufs issue. > There are no such buffers on amd64, ia64 and alpha as they aren't needed. > > http://docs.freebsd.org/cgi/getmsg.cgi?fetch=26091+0+current/freebsd-net > > Ok, that leaves the question why `sysctl kern.ipc.nmbclusters' reports a > different value compared to the `netstat -m' output on box #1 and #3.netstat -m doesn't use the sysctl. It reads the zone limit via memstat_get_countlimit() from libmemstat(3) instead. Perhaps this can account for the difference observed.> Thanks for any pointers, > Joerg > > PS: documenting the nsfbufs issue on amd64/ia64/alpha might be a good > PS: idea. loader(8) seems a good place. Opinions?I've already documented the issue in sendfile(2) in CURRENT. MFC is due. A small note in loader(8) would be good, too -- thanks for the suggestion. -- Yar