Hi, got this panic today on an SMP system. Modifications to the kernel are polling and altq support for the sf driver and removed SMP ifdefs from sys/kern/kern_poll.c. FreeBSD haakonia.hitnet.RWTH-Aachen.DE 5.3-STABLE FreeBSD 5.3-STABLE #61: Sat Nov 27 02:59:01 CET 2004 chris@haakonia.hitnet.RWTH-Aachen.DE:/usr/home/build/usr/home/build/src/sys/LORIEN i386 Sources are from November 22nd. Crashdump and debug kernel are available for further investigation. panic: lock (sleep mutex) inp not locked @ /usr/home/build/src/sys/netinet/tcp_input.c:2521 cpuid = 1 KDB: enter: panic [thread 100034] Stopped at kdb_enter+0x2b: nop db> tr kdb_enter(c06c6c54) at kdb_enter+0x2b panic(c06caa02,c06db3d0,c06d57ce,c06d2493,9d9) at panic+0x127 witness_unlock(c3f4d57c,8,c06d2493,9d9) at witness_unlock+0xbc _mtx_unlock_flags(c3f4d57c,0,c06d2493,9d9,0) at _mtx_unlock_flags+0x5b tcp_input(c44bb000,14,5cb5e289,0,0) at tcp_input+0x2fb0 ip_input(c44bb000) at ip_input+0x52d netisr_dispatch(2,c44bb000,0,c44b0800,c1a93c00) at netisr_dispatch+0x58 ether_demux(c1a93c00,c44bb000,297,96,d74f6400) at ether_demux+0x292 ether_input(c1a93c00,c44bb000,c1a93e84,0,c0833d13) at ether_input+0x269 sf_rxeof(c1a93c00,c1a93c00,c1a93e84,c07276c0,d543ecbc) at sf_rxeof+0x183 sf_poll_locked(c1a93c00,0,5,c1a93e84,0) at sf_poll_locked+0x6b sf_poll(c1a93c00,0,5) at sf_poll+0x2e netisr_poll(0) at netisr_poll+0x188 swi_net(0) at swi_net+0x81 ithread_loop(c19e1080,d543ed48,c19e1080,c051aa34,0) at ithread_loop+0x124 fork_exit(c051aa34,c19e1080,d543ed48) at fork_exit+0xa4 fork_trampoline() at fork_trampoline+0x8 --- trap 0x1, eip = 0, esp = 0xd543ed7c, ebp = 0 --- db> show locks exclusive sleep mutex tcp r = 0 (0xc0755dac) locked @ /usr/home/build/src/sys/netinet/tcp_input.c:617 exclusive sleep mutex Giant r = 1 (0xc0728320) locked @ /usr/home/build/src/sys/kern/kern_poll.c:332 db> show pcpu cpuid = 1 curthread = 0xc19e3640: pid 38 "swi1: net" curpcb = 0xd543eda0 fpcurthread = none idlethread = 0xc19a84b0: pid 11 "idle: cpu1" APIC ID = 1 currentldt = 0x28 spin locks held: db> - Christian -- Christian Brueffer chris@unixpages.org brueffer@FreeBSD.org GPG Key: http://people.freebsd.org/~brueffer/brueffer.key.asc GPG Fingerprint: A5C8 2099 19FF AACA F41B B29B 6C76 178C A0ED 982D -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 187 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20041129/c3427ee2/attachment.bin
On 11/28/04 17:07, Christian Brueffer wrote:> Hi, > > got this panic today on an SMP system. Modifications to the kernel are > polling and altq support for the sf driver and removed SMP ifdefs from > sys/kern/kern_poll.c. > > FreeBSD haakonia.hitnet.RWTH-Aachen.DE 5.3-STABLE FreeBSD 5.3-STABLE #61: Sat Nov 27 02:59:01 CET 2004 > chris@haakonia.hitnet.RWTH-Aachen.DE:/usr/home/build/usr/home/build/src/sys/LORIEN i386 > > Sources are from November 22nd. > > Crashdump and debug kernel are available for further investigation. > > panic: lock (sleep mutex) inp not locked @ /usr/home/build/src/sys/netinet/tcp_input.c:2521 > cpuid = 1 > KDB: enter: panic > [thread 100034] > Stopped at kdb_enter+0x2b: nop > db> tr > kdb_enter(c06c6c54) at kdb_enter+0x2b > panic(c06caa02,c06db3d0,c06d57ce,c06d2493,9d9) at panic+0x127 > witness_unlock(c3f4d57c,8,c06d2493,9d9) at witness_unlock+0xbc > _mtx_unlock_flags(c3f4d57c,0,c06d2493,9d9,0) at _mtx_unlock_flags+0x5b > tcp_input(c44bb000,14,5cb5e289,0,0) at tcp_input+0x2fb0 > ip_input(c44bb000) at ip_input+0x52d > netisr_dispatch(2,c44bb000,0,c44b0800,c1a93c00) at netisr_dispatch+0x58 > ether_demux(c1a93c00,c44bb000,297,96,d74f6400) at ether_demux+0x292 > ether_input(c1a93c00,c44bb000,c1a93e84,0,c0833d13) at ether_input+0x269 > sf_rxeof(c1a93c00,c1a93c00,c1a93e84,c07276c0,d543ecbc) at sf_rxeof+0x183 > sf_poll_locked(c1a93c00,0,5,c1a93e84,0) at sf_poll_locked+0x6b > sf_poll(c1a93c00,0,5) at sf_poll+0x2e > netisr_poll(0) at netisr_poll+0x188 > swi_net(0) at swi_net+0x81 > ithread_loop(c19e1080,d543ed48,c19e1080,c051aa34,0) at > ithread_loop+0x124 > fork_exit(c051aa34,c19e1080,d543ed48) at fork_exit+0xa4 > fork_trampoline() at fork_trampoline+0x8 > --- trap 0x1, eip = 0, esp = 0xd543ed7c, ebp = 0 --- > db> show locks > exclusive sleep mutex tcp r = 0 (0xc0755dac) locked @ > /usr/home/build/src/sys/netinet/tcp_input.c:617 > exclusive sleep mutex Giant r = 1 (0xc0728320) locked @ > /usr/home/build/src/sys/kern/kern_poll.c:332 > db> show pcpu > cpuid = 1 > curthread = 0xc19e3640: pid 38 "swi1: net" > curpcb = 0xd543eda0 > fpcurthread = none > idlethread = 0xc19a84b0: pid 11 "idle: cpu1" > APIC ID = 1 > currentldt = 0x28 > spin locks held: > db>I successfully used polling on my SMP machine a while back, but it pessimized performance for my workloads so I disabled it. All I did was to wrap the SMP error in /sys/kern/kern_poll.c with #ifdef 0...#endif and everything worked. In any case, it seems rwatson@ is interested in polling on SMP; check out his netperf page (http://www.watson.org/~robert/freebsd/netperf/) for more info. Specifically, the following entry in his 20041103 patch is of interest: "A mutex to protect device polling state is introduced, pollmtx, and used to protect the global variables present in kern_poll.c. As a result, Giant is removed from the polling netisr code, and the #ifdef causing polling build to fail on SMP is removed. This has not been tested." No idea whether or not it works, but I thought I'd give you a heads up. Jon
On Mon, 29 Nov 2004, Christian Brueffer wrote:> got this panic today on an SMP system. Modifications to the kernel are > polling and altq support for the sf driver and removed SMP ifdefs from > sys/kern/kern_poll.c. > > FreeBSD haakonia.hitnet.RWTH-Aachen.DE 5.3-STABLE FreeBSD 5.3-STABLE > #61: Sat Nov 27 02:59:01 CET 2004 > chris@haakonia.hitnet.RWTH-Aachen.DE:/usr/home/build/usr/home/build/src/sys/LORIEN > i386 > > Sources are from November 22nd. > > Crashdump and debug kernel are available for further investigation.Sorry I didn't see this post before; thanks for the pointer. Could you tell me the revision of tcp_input.c, and the line number for tcp_input+0x2fb0? I have a number of locking tweaks in HEAD for TCP that are on the MFC path, so it might already be fixed there. Thanks! Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Principal Research Scientist, McAfee Research> > panic: lock (sleep mutex) inp not locked @ /usr/home/build/src/sys/netinet/tcp_input.c:2521 > cpuid = 1 > KDB: enter: panic > [thread 100034] > Stopped at kdb_enter+0x2b: nop > db> tr > kdb_enter(c06c6c54) at kdb_enter+0x2b > panic(c06caa02,c06db3d0,c06d57ce,c06d2493,9d9) at panic+0x127 > witness_unlock(c3f4d57c,8,c06d2493,9d9) at witness_unlock+0xbc > _mtx_unlock_flags(c3f4d57c,0,c06d2493,9d9,0) at _mtx_unlock_flags+0x5b > tcp_input(c44bb000,14,5cb5e289,0,0) at tcp_input+0x2fb0 > ip_input(c44bb000) at ip_input+0x52d > netisr_dispatch(2,c44bb000,0,c44b0800,c1a93c00) at netisr_dispatch+0x58 > ether_demux(c1a93c00,c44bb000,297,96,d74f6400) at ether_demux+0x292 > ether_input(c1a93c00,c44bb000,c1a93e84,0,c0833d13) at ether_input+0x269 > sf_rxeof(c1a93c00,c1a93c00,c1a93e84,c07276c0,d543ecbc) at sf_rxeof+0x183 > sf_poll_locked(c1a93c00,0,5,c1a93e84,0) at sf_poll_locked+0x6b > sf_poll(c1a93c00,0,5) at sf_poll+0x2e > netisr_poll(0) at netisr_poll+0x188 > swi_net(0) at swi_net+0x81 > ithread_loop(c19e1080,d543ed48,c19e1080,c051aa34,0) at > ithread_loop+0x124 > fork_exit(c051aa34,c19e1080,d543ed48) at fork_exit+0xa4 > fork_trampoline() at fork_trampoline+0x8 > --- trap 0x1, eip = 0, esp = 0xd543ed7c, ebp = 0 --- > db> show locks > exclusive sleep mutex tcp r = 0 (0xc0755dac) locked @ > /usr/home/build/src/sys/netinet/tcp_input.c:617 > exclusive sleep mutex Giant r = 1 (0xc0728320) locked @ > /usr/home/build/src/sys/kern/kern_poll.c:332 > db> show pcpu > cpuid = 1 > curthread = 0xc19e3640: pid 38 "swi1: net" > curpcb = 0xd543eda0 > fpcurthread = none > idlethread = 0xc19a84b0: pid 11 "idle: cpu1" > APIC ID = 1 > currentldt = 0x28 > spin locks held: > db> > > > - Christian > > -- > Christian Brueffer chris@unixpages.org brueffer@FreeBSD.org > GPG Key: http://people.freebsd.org/~brueffer/brueffer.key.asc > GPG Fingerprint: A5C8 2099 19FF AACA F41B B29B 6C76 178C A0ED 982D >