Hi, I updated my server at home from a -CURRENT from june 7 to 5.3-RELEASE, and now I'm seeing this panic whenever someone behind this gateway starts emule and opens a lot of connections: http://www.piwebs.com/freebsd/pagefault-network.jpg The panic doesn't occur with the -CURRENT kernel from june 7. Setting debug.mpsafenet=0 doesn't help. Does anyone have a suggestion on how to avoid this panic? The panic is very well reproduceable, so if you need more details, please tell me what to do. For now, I'll just downgrade to an older -CURRENT. Best regards, Arjan
On Tue, 9 Nov 2004, Arjan Van Leeuwen wrote:> I updated my server at home from a -CURRENT from june 7 to 5.3-RELEASE, > and now I'm seeing this panic whenever someone behind this gateway > starts emule and opens a lot of connections: > > http://www.piwebs.com/freebsd/pagefault-network.jpg > > The panic doesn't occur with the -CURRENT kernel from june 7. Setting > debug.mpsafenet=0 doesn't help. > > Does anyone have a suggestion on how to avoid this panic? > > The panic is very well reproduceable, so if you need more details, > please tell me what to do. For now, I'll just downgrade to an older > -CURRENT.Could you send a copy of your dmesg? Could you also use gdb on a kernel with debug symbols or addr2line to convert the function+offsets in the trace to file and line number in the source? This is a NULL pointer dereference, so presumably somewhere there is a poor assumption about memory allocation or the like. Thanks, Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Principal Research Scientist, McAfee Research
On Tue, 9 Nov 2004 23:33:15 +0000 (GMT), Robert Watson <rwatson@freebsd.org> wrote:> > > > On Tue, 9 Nov 2004, Arjan Van Leeuwen wrote: >(...)> > Could you send a copy of your dmesg? Could you also use gdb on a kernel > with debug symbols or addr2line to convert the function+offsets in the > trace to file and line number in the source? This is a NULL pointer > dereference, so presumably somewhere there is a poor assumption about > memory allocation or the like.dmesg is attached. # addr2line -e kernel.debug.20041109 0xc06544a0 /usr/src/sys/sys/libkern.h:56 How do I get the function+offsets? Thanks, Arjan> > Thanks, > > Robert N M Watson FreeBSD Core Team, TrustedBSD Projects > robert@fledge.watson.org Principal Research Scientist, McAfee Research > >-------------- next part -------------- A non-text attachment was scrubbed... Name: dmesg Type: application/octet-stream Size: 6073 bytes Desc: not available Url : http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20041110/06989e75/dmesg.obj
On Wed, 10 Nov 2004 10:15:31 +0000 (GMT), Robert Watson <rwatson@freebsd.org> wrote:> > Could you recompile your kernel with "options INVARIANTS"? It looks like > whatever is happening should be triggering one of the assertions in > m_copydata(), if not icmp_error(), and it would be helpful to track it a > little earlier before it faults. It looks like m_copydata() is alking off > of the end of an mbuf chain, since min()'s only pointer dereference is to > get the length out of an mbuf. However, under those circumstances it > shouldn't need to walk a chain. > > Another thing we may want to do for debugging purposes, subject to this > being something you can put the system through, is add an additional > warning in icmp_error() if the mbuf comes in fragmented. Something like > the following would generate a warning instead of panicking: > (...)I've applied the patch and enabled INVARIANTS. The result was that I did indeed see the warning once, but _not_ when I was trying to create a panic; I saw the warning shortly after starting the machine, I didn't yet start emule to get the panic. When I did start emule, I got a panic that looked a bit different. In this screenshot, you can see both the warning message (that appeared a lot earlier, independent of the panic) and the new panic: http://www.piwebs.com/freebsd/panic.jpg Here is what addr2line tells me: winston% addr2line -f -e kernel.debug.20041110 0xC074CF44 uma_dbg_free /usr/src/sys/vm/uma_dbg.c:308 winston% addr2line -f -e kernel.debug.20041110 0xC074BD13 uma_zfree_arg /usr/src/sys/vm/uma_core.c:2237 winston% addr2line -f -e kernel.debug.20041110 0xC0646C9A m_freem /usr/src/sys/vm/uma.h:302 winston% addr2line -f -e kernel.debug.20041110 0xC0460D11 fr_check /usr/src/sys/contrib/ipfilter/netinet/fil.c:1402 winston% addr2line -f -e kernel.debug.20041110 0xC0462052 fr_check_wrapper /usr/src/sys/contrib/ipfilter/netinet/ip_fil.c:344 winston% addr2line -f -e kernel.debug.20041110 0xC067F11D pfil_run_hooks /usr/src/sys/net/pfil.c:137 winston% addr2line -f -e kernel.debug.20041110 0xC069BE95 ip_input /usr/src/sys/netinet/ip_input.c:439 winston% addr2line -f -e kernel.debug.20041110 0xC067DB92 netisr_processqueue /usr/src/sys/net/netisr.c:229 winston% addr2line -f -e kernel.debug.20041110 0xC067DD76 swi_net /usr/src/sys/net/netisr.c:346 winston% addr2line -f -e kernel.debug.20041110 0xC06051C4 ithread_loop /usr/src/sys/kern/kern_intr.c:548 Thanks for your help, Arjan
On Wed, 10 Nov 2004, Arjan Van Leeuwen wrote:> > Could you send a copy of your dmesg? Could you also use gdb on a kernel > > with debug symbols or addr2line to convert the function+offsets in the > > trace to file and line number in the source? This is a NULL pointer > > dereference, so presumably somewhere there is a poor assumption about > > memory allocation or the like. > > dmesg is attached.Could you say a little about how ipfilter is being used on the box; would it be possible to test with it disabled? Robert N M Watson FreeBSD Core Team, TrustedBSD Projects robert@fledge.watson.org Principal Research Scientist, McAfee Research