Chris Chou
2007-Oct-28 00:32 UTC
FreeBSD 7.0 crashed when running super-smack upon PostgreSQL
Dear all,
FreeBSD RELENG_7.0 crashed when running super-smack benchmark test upon
jailed postgresql.
uname -a
FreeBSD mercury 7.0-BETA1 FreeBSD 7.0-BETA1 #4: Fri Oct 26 23:49:24 CST
2007 chris@mercury:/usr/obj/usr/src/sys/MERCURY i386
I have got a dumped core, and the following backtrace:
kgdb: kvm_nlist(_stopped_cpus):
kgdb: kvm_nlist(_stoppcbs):
[GDB will not be able to debug user-mode threads:
/usr/lib/libthread_db.so: Undefined symbol "ps_pglobal_lookup"]
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "i386-marcel-freebsd".
Unread portion of the kernel message buffer:
Sleeping thread (tid 100100, pid 9284) owns a non-sleepable lock
panic: sleeping thread
Uptime: 15h3m6s
Physical memory: 1011 MB
Dumping 223 MB: 208 192 176 160 144 128 112 96 80 64 48 32 16
#0 doadump () at pcpu.h:195
195 __asm __volatile("movl %%fs:0,%0" : "=r" (td));
(kgdb) backtrace
#0 doadump () at pcpu.h:195
#1 0xc0612624 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409
#2 0xc0612824 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:563
#3 0xc064131b in propagate_priority (td=0xc46d6210)
at /usr/src/sys/kern/subr_turnstile.c:222
#4 0xc0641cf8 in turnstile_wait (ts=0xc40b5b90, owner=0xc46d6210,
queue=Variable "queue" is not available.
)
at /usr/src/sys/kern/subr_turnstile.c:739
#5 0xc0606dcd in _mtx_lock_sleep (m=0xc09617ac, tid=3295513696, opts=0,
file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:394
#6 0xc0750c64 in tcp_usr_attach (so=0xc49014a4, proto=0, td=0xc46d8c60)
at /usr/src/sys/netinet/tcp_usrreq.c:1421
#7 0xc0661da4 in socreate (dom=2, aso=0xe6924c70, type=1, proto=0,
cred=0xc4971900, td=0xc46d8c60) at /usr/src/sys/kern/uipc_socket.c:376
#8 0xc0667b1b in socket (td=0xc46d8c60, uap=0xe6924cfc)
at /usr/src/sys/kern/uipc_syscalls.c:178
#9 0xc0873c35 in syscall (frame=0xe6924d38)
at /usr/src/sys/i386/i386/trap.c:1008
#10 0xc085deb0 in Xint0x80_syscall () at
/usr/src/sys/i386/i386/exception.s:196
#11 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
(kgdb)
The super-smack is compiled from port benchmarks/super-smack with option
WITH_POSTGRESQL enabled.
The Postgresql is in a jail environment, packages are
postgresql-client-8.2.4
postgresql-server-8.2.4_1
The crash occurred only once, and I can not reproduce it after rebooting.
Best regards,
Chris Chou
Robert Watson
2007-Oct-28 03:54 UTC
FreeBSD 7.0 crashed when running super-smack upon PostgreSQL
On Sun, 28 Oct 2007, Chris Chou wrote:> Dear all, > > FreeBSD RELENG_7.0 crashed when running super-smack benchmark test upon > jailed postgresql.Chris, Thanks for this report. Sounds like a bug somewhere :-). Is there any chance I can get remote access to a box holding the synchronized kernel, kernel debugging symbols, source code, and core dump? Could I ask you also to be careful not to delete the kernel with symbols or change the source code on the box so that the core and associated parts all remain in sync? I don't strictly need access to the core directly, since we may well be able to reasonably debug this without that, so here are some directions to try and see how far we get: It sounds like TCP in a user thread is stumbling over a violation of system invariants (don't sleep while holding a mutex) performed by another thread. We need to track down the original thread and figure out why it's sleeping while holding that lock -- perhaps it's a user thread performing a copyin/copyout holding the lock, or perhaps an ithread or other software interrupt thread acquiring a lock of an inappropriate type (such as an sx lock) while holding the lock. The debugging output, 'tid 100100, pid 9284', tells us which thread/process the culprit may be. We need to get a kernel stack trace of that thread and that should shed some light on matters. "info threads" and "thread" can be used in kgdb to list threads and select a thread. kgdb has its own thread ID scheme so you'll need to use info threads to track down the right kgdb thread id before you can select the identified kernel thread/process. Once it's selected, "bt" should print the backtrace of that thread, and that may give us a good starting point. Could you let me know what, if any, firewall packages you are using? Robert N M Watson Computer Laboratory University of Cambridge> > uname -a > FreeBSD mercury 7.0-BETA1 FreeBSD 7.0-BETA1 #4: Fri Oct 26 23:49:24 CST 2007 > chris@mercury:/usr/obj/usr/src/sys/MERCURY i386 > > I have got a dumped core, and the following backtrace: > > kgdb: kvm_nlist(_stopped_cpus): > kgdb: kvm_nlist(_stoppcbs): > [GDB will not be able to debug user-mode threads: /usr/lib/libthread_db.so: > Undefined symbol "ps_pglobal_lookup"] > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "i386-marcel-freebsd". > > Unread portion of the kernel message buffer: > Sleeping thread (tid 100100, pid 9284) owns a non-sleepable lock > panic: sleeping thread > Uptime: 15h3m6s > Physical memory: 1011 MB > Dumping 223 MB: 208 192 176 160 144 128 112 96 80 64 48 32 16 > > #0 doadump () at pcpu.h:195 > 195 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); > (kgdb) backtrace > #0 doadump () at pcpu.h:195 > #1 0xc0612624 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:409 > #2 0xc0612824 in panic (fmt=Variable "fmt" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:563 > #3 0xc064131b in propagate_priority (td=0xc46d6210) > at /usr/src/sys/kern/subr_turnstile.c:222 > #4 0xc0641cf8 in turnstile_wait (ts=0xc40b5b90, owner=0xc46d6210, > queue=Variable "queue" is not available. > ) > at /usr/src/sys/kern/subr_turnstile.c:739 > #5 0xc0606dcd in _mtx_lock_sleep (m=0xc09617ac, tid=3295513696, opts=0, > file=0x0, line=0) at /usr/src/sys/kern/kern_mutex.c:394 > #6 0xc0750c64 in tcp_usr_attach (so=0xc49014a4, proto=0, td=0xc46d8c60) > at /usr/src/sys/netinet/tcp_usrreq.c:1421 > #7 0xc0661da4 in socreate (dom=2, aso=0xe6924c70, type=1, proto=0, > cred=0xc4971900, td=0xc46d8c60) at /usr/src/sys/kern/uipc_socket.c:376 > #8 0xc0667b1b in socket (td=0xc46d8c60, uap=0xe6924cfc) > at /usr/src/sys/kern/uipc_syscalls.c:178 > #9 0xc0873c35 in syscall (frame=0xe6924d38) > at /usr/src/sys/i386/i386/trap.c:1008 > #10 0xc085deb0 in Xint0x80_syscall () at > /usr/src/sys/i386/i386/exception.s:196 > #11 0x00000033 in ?? () > Previous frame inner to this frame (corrupt stack?) > (kgdb) > > The super-smack is compiled from port benchmarks/super-smack with option > WITH_POSTGRESQL enabled. > > The Postgresql is in a jail environment, packages are > postgresql-client-8.2.4 > postgresql-server-8.2.4_1 > > The crash occurred only once, and I can not reproduce it after rebooting. > > Best regards, > > Chris Chou > _______________________________________________ > freebsd-stable@freebsd.org mailing list > http://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscribe@freebsd.org" >