Hi, I've encountered a strange problem while using FreeBSD 6.0 for our local mirror (mirror.math.uni-bielefeld.de) and thus is providing access via ftp, http, rsync and cvsup (all local and remote). The system crashes periodically with a kernel panic (panic: sbdrop). The uptimes between two crashes are going from a few hours to a few weeks. The system is a i386, Intel Pentium 4 based with 512MB ram and a 3ware-7000 (twe) raid controller containig 1 raid 5 set with approx. 1.9TB. The kernel is a GENERIC kernel without changes of the config. These are the kernel dumps: Unread portion of the kernel message buffer: panic: sbdrop Uptime: 22h22m7s Dumping 503 MB (2 chunks) chunk 0: 1MB (159 pages) ... ok chunk 1: 503MB (128752 pages) 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7 #0 doadump () at pcpu.h:165 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); (kgdb) backtrace #0 doadump () at pcpu.h:165 #1 0xc068d10e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399 #2 0xc068d680 in panic (fmt=0xc090c16a "sbdrop") at /usr/src/sys/kern/kern_shutdown.c:555 #3 0xc06d266c in sbdrop_locked (sb=0xc20aca84, len=1) at /usr/src/sys/kern/uipc_socket2.c:1157 #4 0xc06d3d93 in sbdrop (sb=0xc20aca84, len=0) at /usr/src/sys/kern/uipc_socket2.c:1208 #5 0xc0748a7d in tcp_input (m=0xc1c09100, off0=-1039845124) at /usr/src/sys/netinet/tcp_input.c:1201 #6 0xc0740147 in ip_input (m=0xc1c09100) at /usr/src/sys/netinet/ip_input.c:778 #7 0xc07171ff in netisr_processqueue (ni=0xc09ca4f8) at /usr/src/sys/net/netisr.c:236 #8 0xc07174be in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349 #9 0xc06740e5 in ithread_loop (arg=0xc19c3280) at /usr/src/sys/kern/kern_intr.c:547 #10 0xc0673110 in fork_exit (callout=0xc067402c <ithread_loop>, arg=0x0, frame=0x0) at /usr/src/sys/kern/kern_fork.c:789 #11 0xc0894a1c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 Now the output of bt full: (kgdb) bt full #0 doadump () at pcpu.h:165 No locals. #1 0xc068d10e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399 first_buf_printf = 1 #2 0xc068d680 in panic (fmt=0xc090c16a "sbdrop") at /usr/src/sys/kern/kern_shutdown.c:555 bootopt = 260 newpanic = 0 buf = "sbdrop", '\0' <repeats 249 times> #3 0xc06d266c in sbdrop_locked (sb=0xc20aca84, len=1) at /usr/src/sys/kern/uipc_socket2.c:1157 m = (struct mbuf *) 0x0 next = (struct mbuf *) 0x0 #4 0xc06d3d93 in sbdrop (sb=0xc20aca84, len=0) at /usr/src/sys/kern/uipc_socket2.c:1208 No locals. #5 0xc0748a7d in tcp_input (m=0xc1c09100, off0=-1039845124) at /usr/src/sys/netinet/tcp_input.c:1201 dbuf = "\024\000\000\000\000????\033\b???\211?X?\231??\033\b?x\034\b??M\211?G\000\000\000\b\000\000\000(\000\b?(\000l?" sbuf = "\0003\234?l\033\b?\200???\0003\234?\224\033\b??\201\211?\0003\234?\000\000\000\000\000\000\000\000\000?\226?\027\000\000\000\020(??" th = (struct tcphdr *) 0xc1b2f824 ip = (struct ip *) 0xc1b2f810 inp = (struct inpcb *) 0xc3ec5ca8 optp = (u_char *) 0xc1b2f838 "\001\001\b\n:\r?\027\004?\236?#E?W" optlen = 12 len = 69 tlen = 0 off = 32 drop_hdrlen = 52 tp = (struct tcpcb *) 0xc20538fc thflags = 16 so = (struct socket *) 0xc20ac9bc todrop = 69 acked = 69 ourfinisacked = 0 needoutput = 0 tiwin = 5840 to = {to_flags = 1, to_tsval = 973977367, to_tsecr = 80518898, to_mss = 0, to_requested_s_scale = 0 '\0', to_nsacks = 0 '\0', to_sacks = 0x0} headlocked = 0 rstreason = 69 ip6 = (struct ip6_hdr *) 0x0 isipv6 = 0 #6 0xc0740147 in ip_input (m=0xc1c09100) at /usr/src/sys/netinet/ip_input.c:778 ip = (struct ip *) 0xc1b2f810 ia = (struct in_ifaddr *) 0xc1c0bb00 ifa = (struct ifaddr *) 0xc1c0bb00 checkif = 0 hlen = 20 sum = 0 dchg = 0 odst = {s_addr = 3250633472} #7 0xc07171ff in netisr_processqueue (ni=0xc09ca4f8) at /usr/src/sys/net/netisr.c:236 m = (struct mbuf *) 0xc1c09100 #8 0xc07174be in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349 ni = (struct netisr *) 0xc09ca4f8 bits = 0 i = 0 #9 0xc06740e5 in ithread_loop (arg=0xc19c3280) at /usr/src/sys/kern/kern_intr.c:547 ih = (struct intrhand *) 0xc19c1080 p = (struct proc *) 0xc19b0624 count = 0 warned = 0 hlen = 20 sum = 0 dchg = 0 odst = {s_addr = 3250633472} #7 0xc07171ff in netisr_processqueue (ni=0xc09ca4f8) at /usr/src/sys/net/netisr.c:236 m = (struct mbuf *) 0xc1c09100 #8 0xc07174be in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349 ni = (struct netisr *) 0xc09ca4f8 bits = 0 i = 0 #9 0xc06740e5 in ithread_loop (arg=0xc19c3280) at /usr/src/sys/kern/kern_intr.c:547 ih = (struct intrhand *) 0xc19c1080 p = (struct proc *) 0xc19b0624 count = 0 warned = 0 #10 0xc0673110 in fork_exit (callout=0xc067402c <ithread_loop>, arg=0x0, frame=0x0) at /usr/src/sys/kern/kern_fork.c:789 p = (struct proc *) 0xc19b0624 #11 0xc0894a1c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 No locals. I hope that helps. If you need further information or if you have some hints or directions for me, please send me a mail. -- Best regards, Konstantin Saurbier ------------------------------------------------------ Konstantin Saurbier Tel.: 0521 106 3861 Computerlabor Mathematik U5-138 Universitaet Bielefeld Universitaetsstr.25 33501 Bielefeld email: saurbier@math.uni-bielefeld.de ------------------------------------------------------ -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 189 bytes Desc: not available Url : lists.freebsd.org/pipermail/freebsd-stable/attachments/20060411/fcbc60ea/attachment.pgp
Maybe your reboots are related to a problem I had with several Dell 1750's and 2850's. I added:> [rutger@darwin rutger]$ cat /boot/loader.conf > debug.mpsafenet=0to some of the affected systems and they have not crashed yet in about 8 days. Your dump seems to indicate a failure somewhere in netinet code, maybe it's locking related as well... Met vriendelijke groet / Kind Regards, Rutger Bevaart illian.networks B.V.> Hi, > > I've encountered a strange problem while using FreeBSD 6.0 for our > local mirror (mirror.math.uni-bielefeld.de) and thus is providing > access via ftp, http, rsync and cvsup (all local and remote). > The system crashes periodically with a kernel panic (panic: sbdrop). > The uptimes between two crashes are going from a few hours to a few > weeks. > > The system is a i386, Intel Pentium 4 based with 512MB ram and a > 3ware-7000 (twe) raid controller containig 1 raid 5 set with > approx. 1.9TB. > The kernel is a GENERIC kernel without changes of the config.
On Tue, 11 Apr 2006, Konstantin Saurbier wrote:> I've encountered a strange problem while using FreeBSD 6.0 for our local > mirror (mirror.math.uni-bielefeld.de) and thus is providing access via ftp, > http, rsync and cvsup (all local and remote). The system crashes > periodically with a kernel panic (panic: sbdrop). The uptimes between two > crashes are going from a few hours to a few weeks. > > The system is a i386, Intel Pentium 4 based with 512MB ram and a 3ware-7000 > (twe) raid controller containig 1 raid 5 set with approx. 1.9TB. The kernel > is a GENERIC kernel without changes of the config. These are the kernel > dumps:There have been one or more long-term bugs we've been attempting to track down that result in socket buffer corruption discovered only on socket close (hence in sbdrop() when we flush the cover). We've had a lot of trouble tracking it down, and it's not clear that it's actually a single bug, since the sbdrop() panic is a sanity check that can detect a number of types of problems. We could, for example, be looking at a network interface driver bug. There are changes in progress in the 7.x branch to further clean up the socket code on SMP, and they might fix some outstanding problems. There are a couple of instances of this bug report in the PR database, but if you could file the below details, it would be helpful. I'm on travel currently, but will take another stab at this when back. Robert N M Watson> > > Unread portion of the kernel message buffer: > panic: sbdrop > Uptime: 22h22m7s > Dumping 503 MB (2 chunks) > chunk 0: 1MB (159 pages) ... ok > chunk 1: 503MB (128752 pages) 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7 > > #0 doadump () at pcpu.h:165 > 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); > > > (kgdb) backtrace > #0 doadump () at pcpu.h:165 > #1 0xc068d10e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399 > #2 0xc068d680 in panic (fmt=0xc090c16a "sbdrop") > at /usr/src/sys/kern/kern_shutdown.c:555 > #3 0xc06d266c in sbdrop_locked (sb=0xc20aca84, len=1) > at /usr/src/sys/kern/uipc_socket2.c:1157 > #4 0xc06d3d93 in sbdrop (sb=0xc20aca84, len=0) > at /usr/src/sys/kern/uipc_socket2.c:1208 > #5 0xc0748a7d in tcp_input (m=0xc1c09100, off0=-1039845124) > at /usr/src/sys/netinet/tcp_input.c:1201 > #6 0xc0740147 in ip_input (m=0xc1c09100) > at /usr/src/sys/netinet/ip_input.c:778 > #7 0xc07171ff in netisr_processqueue (ni=0xc09ca4f8) > at /usr/src/sys/net/netisr.c:236 > #8 0xc07174be in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349 > #9 0xc06740e5 in ithread_loop (arg=0xc19c3280) > at /usr/src/sys/kern/kern_intr.c:547 > #10 0xc0673110 in fork_exit (callout=0xc067402c <ithread_loop>, arg=0x0, > frame=0x0) at /usr/src/sys/kern/kern_fork.c:789 > #11 0xc0894a1c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 > > > > Now the output of bt full: > > (kgdb) bt full > #0 doadump () at pcpu.h:165 > No locals. > #1 0xc068d10e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399 > first_buf_printf = 1 > #2 0xc068d680 in panic (fmt=0xc090c16a "sbdrop") > at /usr/src/sys/kern/kern_shutdown.c:555 > bootopt = 260 > newpanic = 0 > buf = "sbdrop", '\0' <repeats 249 times> > #3 0xc06d266c in sbdrop_locked (sb=0xc20aca84, len=1) > at /usr/src/sys/kern/uipc_socket2.c:1157 > m = (struct mbuf *) 0x0 > next = (struct mbuf *) 0x0 > #4 0xc06d3d93 in sbdrop (sb=0xc20aca84, len=0) > at /usr/src/sys/kern/uipc_socket2.c:1208 > No locals. > #5 0xc0748a7d in tcp_input (m=0xc1c09100, off0=-1039845124) > at /usr/src/sys/netinet/tcp_input.c:1201 > dbuf = "\024\000\000\000\000????\033\b???\211?X?\231??\033\b?x\034\b??M\211?G\000\000\000\b\000\000\000(\000\b?(\000l?" > sbuf = "\0003\234?l\033\b?\200???\0003\234?\224\033\b??\201\211?\0003\234?\000\000\000\000\000\000\000\000\000?\226?\027\000\000\000\020(??" > th = (struct tcphdr *) 0xc1b2f824 > ip = (struct ip *) 0xc1b2f810 > inp = (struct inpcb *) 0xc3ec5ca8 > optp = (u_char *) 0xc1b2f838 "\001\001\b\n:\r?\027\004?\236?#E?W" > optlen = 12 > len = 69 > tlen = 0 > off = 32 > drop_hdrlen = 52 > tp = (struct tcpcb *) 0xc20538fc > thflags = 16 > so = (struct socket *) 0xc20ac9bc > todrop = 69 > acked = 69 > ourfinisacked = 0 > needoutput = 0 > tiwin = 5840 > to = {to_flags = 1, to_tsval = 973977367, to_tsecr = 80518898, > to_mss = 0, to_requested_s_scale = 0 '\0', to_nsacks = 0 '\0', > to_sacks = 0x0} > headlocked = 0 > rstreason = 69 > ip6 = (struct ip6_hdr *) 0x0 > isipv6 = 0 > #6 0xc0740147 in ip_input (m=0xc1c09100) > at /usr/src/sys/netinet/ip_input.c:778 > ip = (struct ip *) 0xc1b2f810 > ia = (struct in_ifaddr *) 0xc1c0bb00 > ifa = (struct ifaddr *) 0xc1c0bb00 > checkif = 0 > hlen = 20 > sum = 0 > dchg = 0 > odst = {s_addr = 3250633472} > #7 0xc07171ff in netisr_processqueue (ni=0xc09ca4f8) > at /usr/src/sys/net/netisr.c:236 > m = (struct mbuf *) 0xc1c09100 > #8 0xc07174be in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349 > ni = (struct netisr *) 0xc09ca4f8 > bits = 0 > i = 0 > #9 0xc06740e5 in ithread_loop (arg=0xc19c3280) > at /usr/src/sys/kern/kern_intr.c:547 > ih = (struct intrhand *) 0xc19c1080 > p = (struct proc *) 0xc19b0624 > count = 0 > warned = 0 > hlen = 20 > sum = 0 > dchg = 0 > odst = {s_addr = 3250633472} > #7 0xc07171ff in netisr_processqueue (ni=0xc09ca4f8) > at /usr/src/sys/net/netisr.c:236 > m = (struct mbuf *) 0xc1c09100 > #8 0xc07174be in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349 > ni = (struct netisr *) 0xc09ca4f8 > bits = 0 > i = 0 > #9 0xc06740e5 in ithread_loop (arg=0xc19c3280) > at /usr/src/sys/kern/kern_intr.c:547 > ih = (struct intrhand *) 0xc19c1080 > p = (struct proc *) 0xc19b0624 > count = 0 > warned = 0 > #10 0xc0673110 in fork_exit (callout=0xc067402c <ithread_loop>, arg=0x0, > frame=0x0) at /usr/src/sys/kern/kern_fork.c:789 > p = (struct proc *) 0xc19b0624 > #11 0xc0894a1c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 > No locals. > > > I hope that helps. If you need further information or if you have some hints or directions for me, please send me a mail. > -- > > Best regards, > > Konstantin Saurbier > > ------------------------------------------------------ > Konstantin Saurbier Tel.: 0521 106 3861 > Computerlabor Mathematik U5-138 > Universitaet Bielefeld Universitaetsstr.25 > 33501 Bielefeld > email: saurbier@math.uni-bielefeld.de > ------------------------------------------------------ > >
Reasonably Related Threads
- Fatal trap 12: page fault while in kernel mode / current process=12 (swi1: net)
- Strange reboot since 9.1
- serious networking (em) performance (ggate and NFS) problem
- RELENG_7 panic under load: vm_page_unwire: invalid wire count: 0
- panic in callout_reset: bad link in callwheel