freebsd stable - Jun 2006 - Requesting advice for panic: vm_thread_new: kstack allocation failed

I currently have a problem that others have had before, but no one  
actually did anything to explain or help.

I have a mail server that is often under high load.  It panics about  
every day or two with "vm_thread_new: kstack allocation failed".  I  
have a dump, but I doubt it is necessary as the details of this  
problem have been documented many times before.

threads/80435: panic on high loads
http://lists.freebsd.org/pipermail/freebsd-threads/2005-April/ 
003025.html
No resolution because reporter wasn't running -STABLE.

panic: vm_thread_new: kstack allocation failed
http://lists.freebsd.org/pipermail/freebsd-current/2006-March/ 
061544.html
No discussion about it other than testing methods.

FreeBSD 5.4-P12 Panic "vm_thread_new: kstack allocation failed" under
load
http://lists.freebsd.org/pipermail/freebsd-questions/2006-March/ 
116290.html
Just a "pass the buck" comment about upgrading to 6.1

I *am* running 6-STABLE.  I have had this particular panic since 5.4- 
RELEASE.
I am using an SMP kernel with only this change to fix a different  
kernel panic before.
   options VM_KMEM_SIZE_MAX=419430400
The machine is a dual AMD Opteron w/ 2G memory.  I have another  
identical machine that this also occurs on but less frequently as it  
does not have as high of a load.
I have tested every piece of hardware as best as I can, especially  
the memory.

I have been fighting this issue for months now.  I am about to give  
up and go back to FreeBSD 4.x.  I am looking for suggestions now.

1. What does this panic really mean?
2. Any ideas for a work around?  (kernel options, patches, anything)
3. Would a bug report actually get anywhere? (bad experiences in the  
past, 93809)

On Jun 5, 2006, at 2:00 PM, Lonnie Olson wrote:
> I have been fighting this issue for months now.  I am about to give  
> up and go back to FreeBSD 4.x.  I am looking for suggestions now.
>
> 1. What does this panic really mean?
I believe it implies that the kernel address space is exhausted, so  
when something tried to start up a new kthread, this failed because  
the kernel does not have enough memory to allocate a new stack for  
that thread.
> 2. Any ideas for a work around?  (kernel options, patches, anything)
Set up a cron job to run "sysctl kern.malloc >
/var/tmp/kern_malloc"
every minute or whatnot, and take a look at the last entry you get  
before the machine panics.  That will help identify what is running  
your kernel out of memory.
> 3. Would a bug report actually get anywhere? (bad experiences in  
> the past, 93809)
Maybe.  You'll need to provide more information than just a dmesg,  
but even if you don't get a specific solution to your problem in the  
short term, having things in the PR database will help over the long  
term...

-- 
-Chuck