One of my systems (FreeBSD 7.1-RELEASE-p3/amd64) has panicked a couple
times recently without an identified cause. This most recent time I was
able to obtain a crash dump from the system, but output from kgdb is
garbled.
-------------------- Output #1 --------------------
% pwd
/usr/obj/usr/src/sys/[REDACTED]
% sudo kgdb kernel.debug ~/crash/vmcore.0
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "amd64-marcel-freebsd"...
Unread portion of the kernel message buffer:
kernel trap 12 with interrupts disabled
Fatal trap 12: page fault while in kernel mode
cpuid = 3; apic id = 07
fault virtual addresske rn= el0 xt60r
afapul t 1co2de w= isutpehrv isiorn twerritreu pdtasta
d,i spaagbe lnoet dpres
ent
instruction pointer = 0x8:0xffffffff80424561
s
tack
pFoianttera l = 0x10t:0xfffrffaffpfac057af0
frame pointer = 0x10:0xffffff00010f86e0
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = resume, IOPL = 0
current process = 11 (idle: cpu3)
trap number = 12
panic: page fault
cpuid = 3
Uptime: 40d10h35m18s
Physical memory: 8176 MB
Dumping 691 MB: 676 660 644 628 612 596 580 564 548 532 516 500 484 468
452 436 420 404 388 372 356 340 324 308 292 276 260 244 228 212 196 180
164 148 132 116 100 84 68 52 36 20 4
Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from
/boot/kernel/daemon_saver.ko.symbols...done.
done.
Loaded symbols for /boot/kernel/daemon_saver.ko
#0 doadump () at pcpu.h:195
195 __asm __volatile("movq %%gs:0,%0" : "=r"
(td));
(kgdb) list *0x8:0xffffffff80424561
A syntax error in expression, near `:0xffffffff80424561'.
------------------ End output #1 ------------------
I've seen a thread on garbled serial console output and have seen
symptoms of this on several of my systems (including this one), but
that's been more of an annoyance than anything about which I'm actually
worried... until now. (I've actually suspected hardware/BIOS issues
relating to serial port access, which is why I've stayed out of that
thread.) That the crash dump includes similar corruption suggests to me
that it's not the serial device, but something a bit closer to FreeBSD
itself.
Any ideas what's causing the garbled output in kgdb?
--
Alan Amesbury
OIT Security and Assurance
University of Minnesota
On Wed, Apr 29, 2009 at 03:28:09PM -0500, Alan Amesbury wrote:> One of my systems (FreeBSD 7.1-RELEASE-p3/amd64) has panicked a couple > times recently without an identified cause. This most recent time I was > able to obtain a crash dump from the system, but output from kgdb is > garbled. > Fatal trap 12: page fault while in kernel mode > cpuid = 3; apic id = 07 > fault virtual addresske rn= el0 xt60r > afapul t 1co2de w= isutpehrv isiorn twerritreu pdtasta > d,i spaagbe lnoet dpres > ent > instruction pointer = 0x8:0xffffffff80424561 > s > tack > pFoianttera l = 0x10t:0xfffrffaffpfac057af0 > > ------------------ End output #1 ------------------ > > > I've seen a thread on garbled serial console output and have seen > symptoms of this on several of my systems (including this one), but > that's been more of an annoyance than anything about which I'm actually > worried... until now. (I've actually suspected hardware/BIOS issues > relating to serial port access, which is why I've stayed out of that > thread.) That the crash dump includes similar corruption suggests to me > that it's not the serial device, but something a bit closer to FreeBSD > itself. > > Any ideas what's causing the garbled output in kgdb?Lack of buffering within kernel threads; two or more CPUs/threads are trying to log messages concurrently, one character at a time. You can actually decode some of it with a bit of attention: virtual address = 0 x 601 kernel t r a p f a u l t ... or something of the kind. The issue is not really to do with serial console vs. video console, which is why you can see it even in the kernel message buffer following the crash dump. -- Clifton -- Clifton Royston -- cliftonr@iandicomputing.com / cliftonr@lava.net President - I and I Computing * http://www.iandicomputing.com/ Custom programming, network design, systems and network consulting services
On Wednesday 29 April 2009 4:28:09 pm Alan Amesbury wrote:> One of my systems (FreeBSD 7.1-RELEASE-p3/amd64) has panicked a couple > times recently without an identified cause. This most recent time I was > able to obtain a crash dump from the system, but output from kgdb is > garbled. > > -------------------- Output #1 -------------------- > % pwd > /usr/obj/usr/src/sys/[REDACTED] > % sudo kgdb kernel.debug ~/crash/vmcore.0 > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain > conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd"... > > Unread portion of the kernel message buffer: > kernel trap 12 with interrupts disabled > > > Fatal trap 12: page fault while in kernel mode > cpuid = 3; apic id = 07 > fault virtual addresske rn= el0 xt60r > afapul t 1co2de w= isutpehrv isiorn twerritreu pdtasta > d,i spaagbe lnoet dpres > ent > instruction pointer = 0x8:0xffffffff80424561 > s > tack > pFoianttera l = 0x10t:0xfffrffaffpfac057af0 > frame pointer = 0x10:0xffffff00010f86e0 > code segment = base 0x0, limit 0xfffff, type 0x1b > = DPL 0, pres 1, long 1, def32 0, gran 1 > processor eflags = resume, IOPL = 0 > current process = 11 (idle: cpu3) > trap number = 12 > panic: page fault > cpuid = 3 > Uptime: 40d10h35m18s > Physical memory: 8176 MB > Dumping 691 MB: 676 660 644 628 612 596 580 564 548 532 516 500 484 468 > 452 436 420 404 388 372 356 340 324 308 292 276 260 244 228 212 196 180 > 164 148 132 116 100 84 68 52 36 20 4 > > Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from > /boot/kernel/daemon_saver.ko.symbols...done. > done. > Loaded symbols for /boot/kernel/daemon_saver.ko > #0 doadump () at pcpu.h:195 > 195 __asm __volatile("movq %%gs:0,%0" : "=r" (td)); > (kgdb) list *0x8:0xffffffff80424561 > A syntax error in expression, near `:0xffffffff80424561'.Drop the '0x8:' from this and it will work better. Also, 'bt' output would be good. -- John Baldwin