freebsd stable - Dec 2004 - netstat fails with memory allocation error and error in kvm_read

Hi

We are trying to gather some debug information for a problem that is difficult
to diagnose, as the machine always ends up hard frozen (does not do anything,
can not break to debugger, does not respond to keyboard, etc.), so we are
dumping output from netstat, vmstat, iostat etc. quite often.

The cron jobs fail ever so often with error messages I do not quite understand,
and I can not find anything relevant in the archives. Can anyone shed some light
on this?

Command:	netstat -r
Error message:	netstat: kvm_read: Bad address
Debug before:	http://www.aub.dk/~jmp/fw/debug-2004.12.09-21.34.41.gz
Debug after:	http://www.aub.dk/~jmp/fw/debug-2004.12.09-21.35.06.gz
# errors:	7

Command:	netstat -an
Error message:	netstat: sysctl: net.inet.udp.pcblist: Cannot allocate memory
Debug before:	http://www.aub.dk/~jmp/fw/debug-2004.12.09-07.38.48.gz
Debug after:	http://www.aub.dk/~jmp/fw/debug-2004.12.09-07.39.04.gz
# errors:	3

Command:	netstat -an
Error message:	netstat: sysctl: net.inet.raw.pcblist: Cannot allocate memory
Debug before:	http://www.aub.dk/~jmp/fw/debug-2004.12.10-22.19.50.gz
Debug after:	http://www.aub.dk/~jmp/fw/debug-2004.12.10-22.20.10.gz
# errors:	6

The number of errors are in the last five days. Kernelconfig (FIREWALL), dmesg,
pf.conf etc. can be found at http://www.aub.dk/~jmp/fw/

The commands we are running are:
	netstat -i, netstat -m, netstat -s, netstat -an, netstat -r
	vmstat, vmstat -i, vmstat -f, vmstat -m, vmstat -z
	iostat
	ps waux
every 15 seconds. We are running the same commands (plus pfctl -vvsa) from
root's cron every minute.

The hard lockups seems to be connected to our use of pf and altq, that is at
least the problem we're trying to gather debug information for.

Thanks in advance
Martin

On Fri, 10 Dec 2004, J. Martin Petersen wrote:
> We are trying to gather some debug information for a problem that is
difficult
> to diagnose, as the machine always ends up hard frozen (does not do
anything,
> can not break to debugger, does not respond to keyboard, etc.), so we are
> dumping output from netstat, vmstat, iostat etc. quite often.
>
> The cron jobs fail ever so often with error messages I do not quite
understand,
> and I can not find anything relevant in the archives. Can anyone shed some
light
> on this?
>
> Command:	netstat -r
> Error message:	netstat: kvm_read: Bad address
> Debug before:	http://www.aub.dk/~jmp/fw/debug-2004.12.09-21.34.41.gz
> Debug after:	http://www.aub.dk/~jmp/fw/debug-2004.12.09-21.35.06.gz
> # errors:	7
>
> Command:	netstat -an
> Error message:	netstat: sysctl: net.inet.udp.pcblist: Cannot allocate
memory
> Debug before:	http://www.aub.dk/~jmp/fw/debug-2004.12.09-07.38.48.gz
> Debug after:	http://www.aub.dk/~jmp/fw/debug-2004.12.09-07.39.04.gz
> # errors:	3
You appear to be running out of kernel memory. Since you're capturing the
output of vmstat -m, you should check that for any bins that are growing
at a high rate of speed.

Seems possible that its in pf :)
>
> Command:	netstat -an
> Error message:	netstat: sysctl: net.inet.raw.pcblist: Cannot allocate
memory
> Debug before:	http://www.aub.dk/~jmp/fw/debug-2004.12.10-22.19.50.gz
> Debug after:	http://www.aub.dk/~jmp/fw/debug-2004.12.10-22.20.10.gz
> # errors:	6
>
> The number of errors are in the last five days. Kernelconfig (FIREWALL),
dmesg,
> pf.conf etc. can be found at http://www.aub.dk/~jmp/fw/
>
> The commands we are running are:
> 	netstat -i, netstat -m, netstat -s, netstat -an, netstat -r
> 	vmstat, vmstat -i, vmstat -f, vmstat -m, vmstat -z
> 	iostat
> 	ps waux
> every 15 seconds. We are running the same commands (plus pfctl -vvsa) from
> root's cron every minute.
>
> The hard lockups seems to be connected to our use of pf and altq, that is
at
> least the problem we're trying to gather debug information for.
>
> Thanks in advance
> Martin
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
"freebsd-stable-unsubscribe@freebsd.org"
>
-- 
Doug White                    |  FreeBSD: The Power to Serve
dwhite@gumbysoft.com          |  www.FreeBSD.org

> I just got another "netstat: kvm_read: Bad address". 
> vm.kvm_free was 675278848 two seconds earlier and a couple of 
> seconds later (and also currently).
We just had another hard hang, and this time I managed to break to the
debugger on the serial console. I got this stack trace
<http://www.aub.dk/~jmp/fw/trace>, which seems to indicate that pf is
messing up somehow. 

I of course didn't have the ddb man or the handbook section on kernel
debugging at hand, so I've just printed it for the next time it happens.

Both 'show witness' and 'show locks' came up empty. What's
the most
effective way to hunt down these deadlocks with the online kernel debugger? 

Best regards
Martin