freebsd stable - Aug 2009 - Fatal Trap 12 in various processes always at address 0x3030313a

(Sorry, update to subject to be <something>)

3 weeks ago:
I upgraded from 7.1-PRELEASE to -stable and all seemed fine
until I rebooted out of single user mode after doing make
installworld and mergemaster.? At that point, near the
end of the boot sequence I got a core dump, apparently
triggered by devd.

****
Fatal trap 12: page fault while in kernel mode.
cpu id = 0; apic id = 00
fault virtual address = 0x3030313a
fault code = supervisor write, page not present
[snip]
current process = 355 (devd)
****

I redid the *whole* process in single user mode, yet no
difference.? I was out of town for 2.5 weeks after
that, then busy for a few days, which brings me to now:

TODAY:
I booted to my memtest86+ CD, let it run through which it
did with no issues noted.
I blew away my /usr/src and resynced as of about noon
today.? After following through all steps to rebuild, I
now get a fatal trap 12...

****
Fatal trap 12: page fault while in kernel mode.
cpu id = 0; apic id = 00
fault virtual address = 0x3030313a
fault code = supervisor write, page not present
[snip]
current process = 352 (sysctl)
****

Same virtual address, different process.? Umm.?
Interesting.

I'm at a loss.? Google's been of little help, and
searching these lists hasn't turned up much either. 

Does anyone have a further recommendation on what to do,
try, test or change?

BTW, I'm GENERIC.? 

Rich Mahlerwein

Richard Mahlerwein wrote:
> (Sorry, update to subject to be <something>)
> 
> 3 weeks ago:
> I upgraded from 7.1-PRELEASE to -stable and all seemed fine
> until I rebooted out of single user mode after doing make
> installworld and mergemaster.  At that point, near the
> end of the boot sequence I got a core dump, apparently
> triggered by devd.
> 
> ****
> Fatal trap 12: page fault while in kernel mode.
> cpu id = 0; apic id = 00
> fault virtual address = 0x3030313a
> fault code = supervisor write, page not present
> [snip]
> current process = 355 (devd)
> ****
> 
> I redid the *whole* process in single user mode, yet no
> difference.  I was out of town for 2.5 weeks after
> that, then busy for a few days, which brings me to now:
> 
> TODAY:
> I booted to my memtest86+ CD, let it run through which it
> did with no issues noted.
> I blew away my /usr/src and resynced as of about noon
> today.  After following through all steps to rebuild, I
> now get a fatal trap 12...
> 
> ****
> Fatal trap 12: page fault while in kernel mode.
> cpu id = 0; apic id = 00
> fault virtual address = 0x3030313a
> fault code = supervisor write, page not present
> [snip]
> current process = 352 (sysctl)
> ****
> 
> Same virtual address, different process.  Umm. 
> Interesting.
> 
> I'm at a loss.  Google's been of little help, and
> searching these lists hasn't turned up much either. 
> 
> Does anyone have a further recommendation on what to do,
> try, test or change?
> 
> BTW, I'm GENERIC.  
> 
> Rich Mahlerwein
> 
> 
> 
>       
> _______________________________________________
> freebsd-stable@freebsd.org mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-stable
> To unsubscribe, send any mail to
"freebsd-stable-unsubscribe@freebsd.org"
> 
such trap could be triggered by 'floating' memory/cache error. and i 
think that you should try to suspect memory first. memtest helps to 
diagnose most of memory problems, but not all.

-- 
SY, Marat
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3221 bytes
Desc: S/MIME Cryptographic Signature
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20090829/5b95c28d/smime.bin

In the last episode (Aug 29), Richard Mahlerwein said:
> (Sorry, update to subject to be <something>)
> 
> 3 weeks ago:
> I upgraded from 7.1-PRELEASE to -stable and all seemed fine
> until I rebooted out of single user mode after doing make
> installworld and mergemaster.? At that point, near the
> end of the boot sequence I got a core dump, apparently
> triggered by devd.
> 
> ****
> Fatal trap 12: page fault while in kernel mode.
> cpu id = 0; apic id = 00
> fault virtual address = 0x3030313a
> fault code = supervisor write, page not present
> [snip]
> current process = 355 (devd)
> ****
At this point you will want to get a kernel dump and post its stack trace. 
0x3030313a in ASCII is "001:", so some string probably overflowed a
buffer.

http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug.html
http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-gdb.html

-- 
	Dan Nelson
	dnelson@allantgroup.com

On Sat, 29 Aug 2009, Richard Mahlerwein wrote:
> (Sorry, update to subject to be <something>)
>
> 3 weeks ago:
> I upgraded from 7.1-PRELEASE to -stable and all seemed fine
> until I rebooted out of single user mode after doing make
> installworld and mergemaster.? At that point, near the
> end of the boot sequence I got a core dump, apparently
> triggered by devd.
> 
> ****
> Fatal trap 12: page fault while in kernel mode.
> cpu id = 0; apic id = 00
> fault virtual address = 0x3030313a
> fault code = supervisor write, page not present
> [snip]
> current process = 355 (devd)
> ****
> 
> [snip]
> Does anyone have a further recommendation on what to do,
> try, test or change?
Firstly, please set up a dump partition by adding
'dumpdev="AUTO"' to
your rc.conf.

Then, can you compile in the kernel debugger (options KGB / options DDB) 
and when this happens again, please obtain a backtrace from the debugger 
with the "bt" command.  Then, give the "show registers"
command so that we
can establish which register is pointing to the odd address. Finally, 
issue the "call doadump()" command to hopefully save a copy of the
kernel
dump for later analysis.

Thanks,

Gavin

--- On Sun, 8/30/09, Gavin Atkinson <gavin@FreeBSD.org> wrote:
> From: Gavin Atkinson <gavin@FreeBSD.org>
> Subject: Re: Fatal Trap 12 in various processes always at address
0x3030313a
> To: "Richard Mahlerwein" <mahlerrd@yahoo.com>
> Cc: "FreeBSD-Stable" <freebsd-stable@FreeBSD.org>
> Date: Sunday, August 30, 2009, 3:47 PM
> On Sat, 29 Aug 2009, Richard
> Mahlerwein wrote:
> 
> > (Sorry, update to subject to be <something>)
> > 
> > 3 weeks ago:
> > I upgraded from 7.1-PRELEASE to -stable and all seemed
> fine
> > until I rebooted out of single user mode after doing
> make
> > installworld and mergemaster.? At that point, near
> the
> > end of the boot sequence I got a core dump,
> apparently
> > triggered by devd.
> > 
> > ****
> > Fatal trap 12: page fault while in kernel mode.
> > cpu id = 0; apic id = 00
> > fault virtual address = 0x3030313a
> > fault code = supervisor write, page not present
> > [snip]
> > current process = 355 (devd)
> > ****
> > 
> > [snip]
> > Does anyone have a further recommendation on what to
> do,
> > try, test or change?
> 
> Firstly, please set up a dump partition by adding
> 'dumpdev="AUTO"' to your rc.conf.
> 
> Then, can you compile in the kernel debugger (options KGB /
> options DDB) and when this happens again, please obtain a
> backtrace from the debugger with the "bt" command.?
> Then, give the "show registers" command so that we can
> establish which register is pointing to the odd address.
> Finally, issue the "call doadump()" command to hopefully
> save a copy of the kernel dump for later analysis.
> 
> Thanks,
> 
> Gavin
No problem, but for future reference by others reading this thread, the handbook
says KGB should be KDB, which is what I'm enabling in the kernel config. 
DDB is fine.

http://www.freebsd.org/doc/en/books/developers-handbook/kerneldebug-online-ddb.html

> From: Gavin Atkinson <gavin@FreeBSD.org>
> Subject: Re: Fatal Trap 12 in various processes always at address
0x3030313a
> To: "Richard Mahlerwein" <mahlerrd@yahoo.com>
> Cc: "FreeBSD-Stable" <freebsd-stable@FreeBSD.org>
> Date: Sunday, August 30, 2009, 3:47 PM
> On Sat, 29 Aug 2009, Richard
> Mahlerwein wrote:
> 
>> I upgraded from 7.1-PRELEASE to -stable and all seemed fine
>> until I rebooted out of single user mode after doing make
>> installworld and mergemaster.? At that point, near the
>> end of the boot sequence I got a core dump, apparently
>> triggered by devd.
>> 
>> ****
>> Fatal trap 12: page fault while in kernel mode.
>> cpu id = 0; apic id = 00
>> fault virtual address = 0x3030313a
>> fault code = supervisor write, page not present
>> [snip]
>> current process = 355 (devd)
>> ****
>> Does anyone have a further recommendation on what to do,
>> try, test or change?
> 
> Firstly, please set up a dump partition by adding
> 'dumpdev="AUTO"' to your rc.conf.
> 
> Then, can you compile in the kernel debugger (options KGB /
> options DDB) and when this happens again, please obtain a
> backtrace from the debugger with the "bt" command.?
> Then, give the "show registers" command so that we can
> establish which register is pointing to the odd address.
> Finally, issue the "call" command to hopefully
> save a copy of the kernel dump for later analysis.
> 
> Thanks,
> 
> Gavin
Error at virtual address 0x3030313a
current process 352(sysctl) 

bt says (typing blind, but I think I got it):

Tracing pid 352 tid 100044 td 0xc3378480
sysctl_devctl_disable(c0c80ac0,0,0,d82fcba4,d82fcba4...) at
sysctl_devctl_disable+0xaa
sysctl_root(d82fcba4,4,1,d82fcc60,0,...) at sysctl_root+0x187
userland_sysctl(c3378480,d82fcc14,3,0,0,...) at userland_sysctl+0x1c4
__sysctl(c3378480,d82fccfc,18,d82fcd38,d82fcd2c,...) at __sysctl_0x94
syscall(d82fcd38) at syscall_0x335
Xint0x80_syscall() at Xint0x80_syscall_0x20
--- syscall (202,FreeBSD ELF32, __sysctl, eip = 0x2815beaf, esp=0xbfbfe55c, epb
= 0xbfbfe588 ---

show registers :

cs 0x20
ds 0xc1470028
es 0xc1470028
fs 0xc1460008
ss 0x28
eax 0xc0cea0cc devsoftc_0x4c
ecx 0
edx 0x30303132
ebx 0xc3181350
esp 0xd82fcb34
ebp 0xd82fcb54
esi 0
edi 0
eip 0xc08218da sysctl_devctl_disable+0xaa
efl 0x10202
sysctl_devctl_disable+0xaa: movl %eax,0x8(%edx)

I hope that tells someone on the list way more than it tells me.

Thanks again for all the help so far!

-Rich