freebsd stable - Apr 2004 - panic on 4.x using star to /dev/nsa0

Hello,

I've experienced a panic when I was trying to backup /home using star 
(archivers/star) to sa0:

sa0 at ahc0 bus 0 target 6 lun 0
sa0: <HP C1557A U812> Removable Sequential Access SCSI-2 device
sa0: 10.000MB/s transfers (10.000MHz, offset 15)

ahc0 is the following:

ahc0: <Adaptec 2940 SCSI adapter> port 0xd400-0xd4ff mem 
0xef800000-0xef800fff irq 9 at device 10.0 on pci2

/home is on a Seagate SCSI disk (da1) sitting on a Tekram controller 
(sym0). The system is a 4.9-STABLE box:

FreeBSD number6.magda.ca 4.9-STABLE FreeBSD 4.9-STABLE #2: Tue Mar  2 
15:58:20 EST 2004     
dave<at>number6.magda.ca:/usr/obj/usr/src/sys/MYKERNELE  i386

I've setup the system to dump core in these situations so running 
gdb(1) gives the following results:

(no debugging symbols found)...
IdlePTD at physical address 0x005af000
initial pcb at physical address 0x0030a0e0
panicstr: Waiting List traversal

panic messages:
---
panic: Waiting List traversal

[...]
7 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 
13 12 11 10 9 8 7 6 5 4 3 2 1 0
<<<<<<<<<<<<< Dump Card State Ends
 >>>>>>>>>>>>>>>>>>
ahc0: ahc_intr - referenced scb not valid during SELTO scb(0, 0)
 >>>>>>>>>>>>>>>>>> Dump
Card State Begins
<<<<<<<<<<<<<<<<<
ahc0: Dumping Card State while idle, at SEQADDR 0x18
Card was paused
ACCUM = 0x8, SINDEX = 0x20, DINDEX = 0xc0, ARG_2 = 0x2
HCNT = 0x0 SCBPTR = 0x0
SCSISIGI[0x0] ERROR[0x0] SCSIBUSL[0x0] LASTPHASE[0x1]
SCSISEQ[0x0] SBLKCTL[0x0] SCSIRATE[0x0] SEQCTL[0x10]
SEQ_FLAGS[0xc0] SSTAT0[0x5] SSTAT1[0x2] SSTAT2[0x0]
SSTAT3[0x0] SIMODE0[0x0] SIMODE1[0xa4] SXFRCTL0[0x80]
DFCNTRL[0x0] DFSTATUS[0x8]
STACK: 0x0 0x160 0x190 0x17
SCB count = 10
Kernel NEXTQSCB = 7
Card NEXTQSCB = 31
QINFIFO entries:
Waiting Queue entries: 0:0
Disconnected Queue entries:
QOUTFIFO entries:
Sequencer Free SCB List: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Sequencer SCB Info:
   0 SCB_CONTROL[0x0] SCB_SCSIID[0x10] SCB_LUN[0x0] SCB_TAG[0x0]
   1 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
   2 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
   3 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
   4 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
   5 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
   6 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
   7 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
   8 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
   9 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
  10 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
  11 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
  12 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
  13 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
  14 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
  15 SCB_CONTROL[0x0] SCB_SCSIID[0xff] SCB_LUN[0xff] SCB_TAG[0xff]
Pending list:
   8 SCB_CONTROL[0x40] SCB_SCSIID[0x67] SCB_LUN[0x0]
Kernel Free SCB list: 9 6 5 4 3 2 1 0
Untagged Q(6): 8

<<<<<<<<<<<<<<<<< Dump Card
State Ends
>>>>>>>>>>>>>>>>>>
ahc0: ahc_intr - referenced scb not valid during SELTO scb(0, 0)
 >>>>>>>>>>>>>>>>>> Dump
Card State Begins
<<<<<<<<<<<<<<<<<
ahc0: Dumping Card State while idle, at SEQADDR 0x18
Card was paused
ACCUM = 0x8, SINDEX = 0x20, DINDEX = 0xc0, ARG_2 = 0x2
HCNT = 0x0 SCBPTR = 0x0
[...]

The above is repeated over and over again. A 'bt' command in the 
debugger gives the following:

(kgdb) bt
#0  0xc01822ce in dumpsys ()
#1  0xc0182098 in boot ()
#2  0xc01824e5 in panic ()
#3  0xc0152c8f in ahc_search_qinfifo ()
#4  0xc0155690 in ahc_recover_commands ()
#5  0xc01573cf in ahc_recovery_thread ()

This has only happened once so far so I haven't filed a PR in case it's 
not reproducible. Since I have the kernel dump I can give more 
information if needed if someone would be willing to tell me what to 
type into the debugger's CLI. Don't have much experience in this since 
I haven't had many panics under FreeBSD. :> (Is there a document 
describing this?  A kind of "what happens if the system panics" 
FAQ/HOWTO.)

I'll try reproducing it tomorrow (Saturday).

P.S. Please CC me as I'm not currently subscribed to freebsd-stable@.

On Fri, Apr 09, 2004 at 09:12:33PM -0400, David Magda wrote:
> This has only happened once so far so I haven't filed a PR in case
it's
> not reproducible. Since I have the kernel dump I can give more 
> information if needed if someone would be willing to tell me what to 
> type into the debugger's CLI. Don't have much experience in this
since
> I haven't had many panics under FreeBSD. :> (Is there a document 
> describing this?  A kind of "what happens if the system panics" 
> FAQ/HOWTO.)
The chapter on kernel debugging in the Developers Handbook explains
how to build a debugging kernel and use it to extract a traceback
suitable for someone to diagnose a kernel panic.

Kris
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 187 bytes
Desc: not available
Url :
http://lists.freebsd.org/pipermail/freebsd-stable/attachments/20040409/6e3f0544/attachment.bin

David Magda <dmagda <at> ee.ryerson.ca> writes:
[...]
> This has only happened once so far so I haven't filed a PR in case
it's
> not reproducible. Since I have the kernel dump I can give more 
[...]

It's happened again doing the same thing. I've installed a newer kernel
with
the debugging options following Kris Kennaway's advice. The new kernel
(which I
compiled a while ago but never installed) is:

FreeBSD number6.magda.ca 4.9-STABLE FreeBSD 4.9-STABLE #3: Wed Mar 17 16:24:30
EST 2004     dave<at>number6.magda.ca:/usr/obj/usr/src/sys/MYKERNELE  i386

Here's the output of starting gdb and of running 'bt':


number6:/usr/obj/usr/src/sys/MYKERNELE# gdb -k kernel.debug /var/crash/vmcore.6
GNU gdb 4.18 (FreeBSD)
Copyright 1998 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for
details.
This GDB was configured as "i386-unknown-freebsd"... \
    Deprecated bfd_read called at /usr/src/gnu/
    usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c line 2627 \
    in elfstab_build_psymtabs
Deprecated bfd_read called at \
    /usr/src/gnu/usr.bin/binutils/gdb/../../../../contrib/gdb/gdb/dbxread.c \
    line 933 in fill_symbuf

IdlePTD at physical address 0x005af000
initial pcb at physical address 0x0030a100
panicstr: Waiting List traversal

panic messages:
---
panic: Waiting List traversal


syncing disks... 
done
Uptime: 1h5m20s

dumping to dev #da/0x20001, offset 180392
dump 
511 510 509 508 507 506 505 504 503 502 501 500
[snip]
 212 211 210 20
---
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
487             if (dumping++) {
(kgdb) bt
#0  dumpsys () at /usr/src/sys/kern/kern_shutdown.c:487
#1  0xc018209c in boot (howto=256) at /usr/src/sys/kern/kern_shutdown.c:316
#2  0xc01824e9 in panic (fmt=0xc028fe7b "Waiting List traversal\n")
    at /usr/src/sys/kern/kern_shutdown.c:595
#3  0xc0152c8f in ahc_search_qinfifo (ahc=0xc21d2400, target=6, 
    channel=65 'A', lun=0, tag=7, role=ROLE_INITIATOR, status=0, 
    action=SEARCH_COUNT) at /usr/src/sys/dev/aic7xxx/aic7xxx.c:5506
#4  0xc0155690 in ahc_recover_commands (ahc=0xc21d2400)
    at /usr/src/sys/dev/aic7xxx/aic7xxx.c:7046
#5  0xc01573cf in ahc_recovery_thread (arg=0xc21d2400)
    at /usr/src/sys/dev/aic7xxx/aic_osm_lib.c:142

Again, this was a result of running star 1.4.3 to backup my /home onto DDS3
tapes. The tapes are kind of old so I though I'd run one more backup on them
for archival purposes before putting in new ones.

Anything else I can do to help things out?