Hi,
I've encountered a strange problem while using FreeBSD 6.0 for our local
mirror (mirror.math.uni-bielefeld.de) and thus is providing access via ftp,
http, rsync and cvsup (all local and remote).
The system crashes periodically with a kernel panic (panic: sbdrop).
The uptimes between two crashes are going from a few hours to a few weeks.
The system is a i386, Intel Pentium 4 based with 512MB ram and a 3ware-7000
(twe) raid controller containig 1 raid 5 set with approx. 1.9TB.
The kernel is a GENERIC kernel without changes of the config.
These are the kernel dumps:
Unread portion of the kernel message buffer:
panic: sbdrop
Uptime: 22h22m7s
Dumping 503 MB (2 chunks)
chunk 0: 1MB (159 pages) ... ok
chunk 1: 503MB (128752 pages) 487 471 455 439 423 407 391 375 359 343 327 311
295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7
#0 doadump () at pcpu.h:165
165 __asm __volatile("movl %%fs:0,%0" : "=r"
(td));
(kgdb) backtrace
#0 doadump () at pcpu.h:165
#1 0xc068d10e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399
#2 0xc068d680 in panic (fmt=0xc090c16a "sbdrop")
at /usr/src/sys/kern/kern_shutdown.c:555
#3 0xc06d266c in sbdrop_locked (sb=0xc20aca84, len=1)
at /usr/src/sys/kern/uipc_socket2.c:1157
#4 0xc06d3d93 in sbdrop (sb=0xc20aca84, len=0)
at /usr/src/sys/kern/uipc_socket2.c:1208
#5 0xc0748a7d in tcp_input (m=0xc1c09100, off0=-1039845124)
at /usr/src/sys/netinet/tcp_input.c:1201
#6 0xc0740147 in ip_input (m=0xc1c09100)
at /usr/src/sys/netinet/ip_input.c:778
#7 0xc07171ff in netisr_processqueue (ni=0xc09ca4f8)
at /usr/src/sys/net/netisr.c:236
#8 0xc07174be in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349
#9 0xc06740e5 in ithread_loop (arg=0xc19c3280)
at /usr/src/sys/kern/kern_intr.c:547
#10 0xc0673110 in fork_exit (callout=0xc067402c <ithread_loop>, arg=0x0,
frame=0x0) at /usr/src/sys/kern/kern_fork.c:789
#11 0xc0894a1c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208
Now the output of bt full:
(kgdb) bt full
#0 doadump () at pcpu.h:165
No locals.
#1 0xc068d10e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399
first_buf_printf = 1
#2 0xc068d680 in panic (fmt=0xc090c16a "sbdrop")
at /usr/src/sys/kern/kern_shutdown.c:555
bootopt = 260
newpanic = 0
buf = "sbdrop", '\0' <repeats 249 times>
#3 0xc06d266c in sbdrop_locked (sb=0xc20aca84, len=1)
at /usr/src/sys/kern/uipc_socket2.c:1157
m = (struct mbuf *) 0x0
next = (struct mbuf *) 0x0
#4 0xc06d3d93 in sbdrop (sb=0xc20aca84, len=0)
at /usr/src/sys/kern/uipc_socket2.c:1208
No locals.
#5 0xc0748a7d in tcp_input (m=0xc1c09100, off0=-1039845124)
at /usr/src/sys/netinet/tcp_input.c:1201
dbuf =
"\024\000\000\000\000????\033\b???\211?X?\231??\033\b?x\034\b??M\211?G\000\000\000\b\000\000\000(\000\b?(\000l?"
sbuf =
"\0003\234?l\033\b?\200???\0003\234?\224\033\b??\201\211?\0003\234?\000\000\000\000\000\000\000\000\000?\226?\027\000\000\000\020(??"
th = (struct tcphdr *) 0xc1b2f824
ip = (struct ip *) 0xc1b2f810
inp = (struct inpcb *) 0xc3ec5ca8
optp = (u_char *) 0xc1b2f838
"\001\001\b\n:\r?\027\004?\236?#E?W"
optlen = 12
len = 69
tlen = 0
off = 32
drop_hdrlen = 52
tp = (struct tcpcb *) 0xc20538fc
thflags = 16
so = (struct socket *) 0xc20ac9bc
todrop = 69
acked = 69
ourfinisacked = 0
needoutput = 0
tiwin = 5840
to = {to_flags = 1, to_tsval = 973977367, to_tsecr = 80518898,
to_mss = 0, to_requested_s_scale = 0 '\0', to_nsacks = 0 '\0',
to_sacks = 0x0}
headlocked = 0
rstreason = 69
ip6 = (struct ip6_hdr *) 0x0
isipv6 = 0
#6 0xc0740147 in ip_input (m=0xc1c09100)
at /usr/src/sys/netinet/ip_input.c:778
ip = (struct ip *) 0xc1b2f810
ia = (struct in_ifaddr *) 0xc1c0bb00
ifa = (struct ifaddr *) 0xc1c0bb00
checkif = 0
hlen = 20
sum = 0
dchg = 0
odst = {s_addr = 3250633472}
#7 0xc07171ff in netisr_processqueue (ni=0xc09ca4f8)
at /usr/src/sys/net/netisr.c:236
m = (struct mbuf *) 0xc1c09100
#8 0xc07174be in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349
ni = (struct netisr *) 0xc09ca4f8
bits = 0
i = 0
#9 0xc06740e5 in ithread_loop (arg=0xc19c3280)
at /usr/src/sys/kern/kern_intr.c:547
ih = (struct intrhand *) 0xc19c1080
p = (struct proc *) 0xc19b0624
count = 0
warned = 0
hlen = 20
sum = 0
dchg = 0
odst = {s_addr = 3250633472}
#7 0xc07171ff in netisr_processqueue (ni=0xc09ca4f8)
at /usr/src/sys/net/netisr.c:236
m = (struct mbuf *) 0xc1c09100
#8 0xc07174be in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349
ni = (struct netisr *) 0xc09ca4f8
bits = 0
i = 0
#9 0xc06740e5 in ithread_loop (arg=0xc19c3280)
at /usr/src/sys/kern/kern_intr.c:547
ih = (struct intrhand *) 0xc19c1080
p = (struct proc *) 0xc19b0624
count = 0
warned = 0
#10 0xc0673110 in fork_exit (callout=0xc067402c <ithread_loop>, arg=0x0,
frame=0x0) at /usr/src/sys/kern/kern_fork.c:789
p = (struct proc *) 0xc19b0624
#11 0xc0894a1c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208
No locals.
I hope that helps. If you need further information or if you have some hints or
directions for me, please send me a mail.
--
Best regards,
Konstantin Saurbier
------------------------------------------------------
Konstantin Saurbier Tel.: 0521 106 3861
Computerlabor Mathematik U5-138
Universitaet Bielefeld Universitaetsstr.25
33501 Bielefeld
email: saurbier@math.uni-bielefeld.de
------------------------------------------------------
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20060411/fcbc60ea/attachment.pgp
Maybe your reboots are related to a problem I had with several Dell 1750's and 2850's. I added:> [rutger@darwin rutger]$ cat /boot/loader.conf > debug.mpsafenet=0to some of the affected systems and they have not crashed yet in about 8 days. Your dump seems to indicate a failure somewhere in netinet code, maybe it's locking related as well... Met vriendelijke groet / Kind Regards, Rutger Bevaart illian.networks B.V.> Hi, > > I've encountered a strange problem while using FreeBSD 6.0 for our > local mirror (mirror.math.uni-bielefeld.de) and thus is providing > access via ftp, http, rsync and cvsup (all local and remote). > The system crashes periodically with a kernel panic (panic: sbdrop). > The uptimes between two crashes are going from a few hours to a few > weeks. > > The system is a i386, Intel Pentium 4 based with 512MB ram and a > 3ware-7000 (twe) raid controller containig 1 raid 5 set with > approx. 1.9TB. > The kernel is a GENERIC kernel without changes of the config.
On Tue, 11 Apr 2006, Konstantin Saurbier wrote:> I've encountered a strange problem while using FreeBSD 6.0 for our local > mirror (mirror.math.uni-bielefeld.de) and thus is providing access via ftp, > http, rsync and cvsup (all local and remote). The system crashes > periodically with a kernel panic (panic: sbdrop). The uptimes between two > crashes are going from a few hours to a few weeks. > > The system is a i386, Intel Pentium 4 based with 512MB ram and a 3ware-7000 > (twe) raid controller containig 1 raid 5 set with approx. 1.9TB. The kernel > is a GENERIC kernel without changes of the config. These are the kernel > dumps:There have been one or more long-term bugs we've been attempting to track down that result in socket buffer corruption discovered only on socket close (hence in sbdrop() when we flush the cover). We've had a lot of trouble tracking it down, and it's not clear that it's actually a single bug, since the sbdrop() panic is a sanity check that can detect a number of types of problems. We could, for example, be looking at a network interface driver bug. There are changes in progress in the 7.x branch to further clean up the socket code on SMP, and they might fix some outstanding problems. There are a couple of instances of this bug report in the PR database, but if you could file the below details, it would be helpful. I'm on travel currently, but will take another stab at this when back. Robert N M Watson> > > Unread portion of the kernel message buffer: > panic: sbdrop > Uptime: 22h22m7s > Dumping 503 MB (2 chunks) > chunk 0: 1MB (159 pages) ... ok > chunk 1: 503MB (128752 pages) 487 471 455 439 423 407 391 375 359 343 327 311 295 279 263 247 231 215 199 183 167 151 135 119 103 87 71 55 39 23 7 > > #0 doadump () at pcpu.h:165 > 165 __asm __volatile("movl %%fs:0,%0" : "=r" (td)); > > > (kgdb) backtrace > #0 doadump () at pcpu.h:165 > #1 0xc068d10e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399 > #2 0xc068d680 in panic (fmt=0xc090c16a "sbdrop") > at /usr/src/sys/kern/kern_shutdown.c:555 > #3 0xc06d266c in sbdrop_locked (sb=0xc20aca84, len=1) > at /usr/src/sys/kern/uipc_socket2.c:1157 > #4 0xc06d3d93 in sbdrop (sb=0xc20aca84, len=0) > at /usr/src/sys/kern/uipc_socket2.c:1208 > #5 0xc0748a7d in tcp_input (m=0xc1c09100, off0=-1039845124) > at /usr/src/sys/netinet/tcp_input.c:1201 > #6 0xc0740147 in ip_input (m=0xc1c09100) > at /usr/src/sys/netinet/ip_input.c:778 > #7 0xc07171ff in netisr_processqueue (ni=0xc09ca4f8) > at /usr/src/sys/net/netisr.c:236 > #8 0xc07174be in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349 > #9 0xc06740e5 in ithread_loop (arg=0xc19c3280) > at /usr/src/sys/kern/kern_intr.c:547 > #10 0xc0673110 in fork_exit (callout=0xc067402c <ithread_loop>, arg=0x0, > frame=0x0) at /usr/src/sys/kern/kern_fork.c:789 > #11 0xc0894a1c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 > > > > Now the output of bt full: > > (kgdb) bt full > #0 doadump () at pcpu.h:165 > No locals. > #1 0xc068d10e in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:399 > first_buf_printf = 1 > #2 0xc068d680 in panic (fmt=0xc090c16a "sbdrop") > at /usr/src/sys/kern/kern_shutdown.c:555 > bootopt = 260 > newpanic = 0 > buf = "sbdrop", '\0' <repeats 249 times> > #3 0xc06d266c in sbdrop_locked (sb=0xc20aca84, len=1) > at /usr/src/sys/kern/uipc_socket2.c:1157 > m = (struct mbuf *) 0x0 > next = (struct mbuf *) 0x0 > #4 0xc06d3d93 in sbdrop (sb=0xc20aca84, len=0) > at /usr/src/sys/kern/uipc_socket2.c:1208 > No locals. > #5 0xc0748a7d in tcp_input (m=0xc1c09100, off0=-1039845124) > at /usr/src/sys/netinet/tcp_input.c:1201 > dbuf = "\024\000\000\000\000????\033\b???\211?X?\231??\033\b?x\034\b??M\211?G\000\000\000\b\000\000\000(\000\b?(\000l?" > sbuf = "\0003\234?l\033\b?\200???\0003\234?\224\033\b??\201\211?\0003\234?\000\000\000\000\000\000\000\000\000?\226?\027\000\000\000\020(??" > th = (struct tcphdr *) 0xc1b2f824 > ip = (struct ip *) 0xc1b2f810 > inp = (struct inpcb *) 0xc3ec5ca8 > optp = (u_char *) 0xc1b2f838 "\001\001\b\n:\r?\027\004?\236?#E?W" > optlen = 12 > len = 69 > tlen = 0 > off = 32 > drop_hdrlen = 52 > tp = (struct tcpcb *) 0xc20538fc > thflags = 16 > so = (struct socket *) 0xc20ac9bc > todrop = 69 > acked = 69 > ourfinisacked = 0 > needoutput = 0 > tiwin = 5840 > to = {to_flags = 1, to_tsval = 973977367, to_tsecr = 80518898, > to_mss = 0, to_requested_s_scale = 0 '\0', to_nsacks = 0 '\0', > to_sacks = 0x0} > headlocked = 0 > rstreason = 69 > ip6 = (struct ip6_hdr *) 0x0 > isipv6 = 0 > #6 0xc0740147 in ip_input (m=0xc1c09100) > at /usr/src/sys/netinet/ip_input.c:778 > ip = (struct ip *) 0xc1b2f810 > ia = (struct in_ifaddr *) 0xc1c0bb00 > ifa = (struct ifaddr *) 0xc1c0bb00 > checkif = 0 > hlen = 20 > sum = 0 > dchg = 0 > odst = {s_addr = 3250633472} > #7 0xc07171ff in netisr_processqueue (ni=0xc09ca4f8) > at /usr/src/sys/net/netisr.c:236 > m = (struct mbuf *) 0xc1c09100 > #8 0xc07174be in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349 > ni = (struct netisr *) 0xc09ca4f8 > bits = 0 > i = 0 > #9 0xc06740e5 in ithread_loop (arg=0xc19c3280) > at /usr/src/sys/kern/kern_intr.c:547 > ih = (struct intrhand *) 0xc19c1080 > p = (struct proc *) 0xc19b0624 > count = 0 > warned = 0 > hlen = 20 > sum = 0 > dchg = 0 > odst = {s_addr = 3250633472} > #7 0xc07171ff in netisr_processqueue (ni=0xc09ca4f8) > at /usr/src/sys/net/netisr.c:236 > m = (struct mbuf *) 0xc1c09100 > #8 0xc07174be in swi_net (dummy=0x0) at /usr/src/sys/net/netisr.c:349 > ni = (struct netisr *) 0xc09ca4f8 > bits = 0 > i = 0 > #9 0xc06740e5 in ithread_loop (arg=0xc19c3280) > at /usr/src/sys/kern/kern_intr.c:547 > ih = (struct intrhand *) 0xc19c1080 > p = (struct proc *) 0xc19b0624 > count = 0 > warned = 0 > #10 0xc0673110 in fork_exit (callout=0xc067402c <ithread_loop>, arg=0x0, > frame=0x0) at /usr/src/sys/kern/kern_fork.c:789 > p = (struct proc *) 0xc19b0624 > #11 0xc0894a1c in fork_trampoline () at /usr/src/sys/i386/i386/exception.s:208 > No locals. > > > I hope that helps. If you need further information or if you have some hints or directions for me, please send me a mail. > -- > > Best regards, > > Konstantin Saurbier > > ------------------------------------------------------ > Konstantin Saurbier Tel.: 0521 106 3861 > Computerlabor Mathematik U5-138 > Universitaet Bielefeld Universitaetsstr.25 > 33501 Bielefeld > email: saurbier@math.uni-bielefeld.de > ------------------------------------------------------ > >
Possibly Parallel Threads
- Fatal trap 12: page fault while in kernel mode / current process=12 (swi1: net)
- Strange reboot since 9.1
- serious networking (em) performance (ggate and NFS) problem
- RELENG_7 panic under load: vm_page_unwire: invalid wire count: 0
- panic in callout_reset: bad link in callwheel