Panagiotis Christias
2008-Oct-14 22:24 UTC
FreeBSD 7-STABLE, isp(4), QLE2462: panic & deadlocks
Hello everybody,
we recently got three Dell PowerEdge servers equipped with Qlogic
QLE2462 cards (dual FC 4Gbps ports) and an EMC CLARiiON CX3-40 SAN.
The servers with the FC cards were successfully and extensively
tested with an older SUN StorEdge T3 SAN.
However, when we connect them to the CX3-40, create and mount a new
partition and then do something as simple as "tar -C /san -xf
ports.tgz"
the system panics and deadlocks. We have tried several FreeBSD versions
(6.3 i386/adm64, 7.0 i386/adm64, 7.1 i386/adm64 and lastly 7-STABLE i386
- we also tried the latest 8-CURRENT snapshot but it panicked too soon).
The result is always the same; panic and deadlock.
In case it helps, we have also tried all available connection speeds
(4, 2 and 1Gbps), the QLE2462 cards have installed the latest firwmare
version (4.03.10) and we have verified our hardware setup by testing
it with Linux (CentOS 5.2), although we would try anything in order
to keep the whole of our server base to FreeBSD (we have succeeded
till now).
Currently, we have installed the latest version of 7-STABLE (csuped
today) along with a debug-enabled kernel and collected as much output
as we can. See below.
We are open to any suggestion and any kind of tests in order to help
resolving the problem.
Thanks!
COLLECTED DATA:
1. uname -a:
FreeBSD 7x86.noc.ntua.gr 7.1-PRERELEASE FreeBSD 7.1-PRERELEASE #0: Tue Oct 14
21:12:57 EEST 2008 root@7x86.noc.ntua.gr:/usr/obj/usr/src/sys/GENERIC-DEBUG
i386
---------------------------------------------------------------
2. Kernel configuration file /sys/i386/conf/GENERIC-DEBUG:
include GENERIC
ident GENERIC-DEBUG
# force isp firmware to fully loaded
# device ispfw
options KDB
options DDB
options INVARIANTS
options INVARIANT_SUPPORT
options WITNESS
options DEBUG_LOCKS
options DEBUG_VFS_LOCKS
options DIAGNOSTIC
options CAMDEBUG
options CAM_DEBUG_BUS=-1
options CAM_DEBUG_TARGET=-1
options CAM_DEBUG_LUN=-1
options CAM_DEBUG_FLAGS=CAM_DEBUG_INFO
---------------------------------------------------------------
3. dmesg output, also available at
http://noc.ntua.gr/~christia/san/7-STABLE.dmesg.out :
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.1-PRERELEASE #0: Tue Oct 14 21:12:57 EEST 2008
root@7x86.noc.ntua.gr:/usr/obj/usr/src/sys/GENERIC-DEBUG
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU E5405 @ 2.00GHz (1997.03-MHz 686-class CPU)
Origin = "GenuineIntel" Id = 0x10676 Stepping = 6
Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2=0xce33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,PDCM,DCA,<b19>>
AMD Features=0x20100000<NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 4
real memory = 3484745728 (3323 MB)
avail memory = 3405070336 (3247 MB)
ACPI APIC Table: <DELL PE_SC3 >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
cpu4 (AP): APIC ID: 4
cpu5 (AP): APIC ID: 5
cpu6 (AP): APIC ID: 6
cpu7 (AP): APIC ID: 7
WITNESS: spin lock cpuset not in order list
WITNESS: spin lock intrcnt not in order list
ioapic0: Changing APIC ID to 8
ioapic1: Changing APIC ID to 9
ioapic2: Changing APIC ID to 10
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 160-183 on motherboard
ioapic2 <Version 2.0> irqs 224-247 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
acpi0: <DELL PE_SC3> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on
acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci4: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> at device 0.0 on pci4
pci5: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 0.0 on pci5
pci6: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> at device 0.0 on pci6
pci7: <PCI bus> on pcib4
bce0: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem
0xf4000000-0xf5ffffff irq 16 at device 0.0 on pci7
miibus0: <MII bus> on bce0
brgphy0: <BCM5708C 10/100/1000baseTX PHY> PHY 1 on miibus0
brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bce0: Ethernet address: 00:1e:c9:e8:13:e9
bce0: [ITHREAD]
bce0: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W
(0x04000305); Flags( MFW MSI )
pcib5: <ACPI PCI-PCI bridge> at device 1.0 on pci5
pci8: <ACPI PCI bus> on pcib5
pcib6: <PCI-PCI bridge> at device 0.3 on pci4
pci9: <PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci1: <ACPI PCI bus> on pcib7
mfi0: <Dell PERC 6> port 0xec00-0xecff mem
0xfc780000-0xfc7bffff,0xfc740000-0xfc77ffff irq 16 at device 0.0 on pci1
mfi0: Megaraid SAS driver Ver 2.00
mfi0: 3299 (277344669s/0x0020/info) - Shutdown command received from host
mfi0: 3300 (boot + 3s/0x0020/info) - Firmware initialization started (PCI ID
0060/1000/1f0c/1028)
mfi0: 3301 (boot + 3s/0x0020/info) - Firmware version 1.11.82-0473
mfi0: 3302 (boot + 3s/0x0008/info) - Battery Present
mfi0: 3303 (boot + 3s/0x0020/info) - Package version 6.0.3-0002
mfi0: 3304 (boot + 23s/0x0004/info) - Enclosure (SES) discovered on PD 20(c
None/p0)
mfi0: 3305 (boot + 23s/0x0002/info) - Unexpected sense: Encl PD 20, CDB: 12 00
00 00 00 00, Sense: 70 00 06 00 00 00 00 0a 00 00 00 00 29 00 00 00 00 00
mfi0: 3306 (boot + 24s/0x0002/info) - Inserted: Encl PD 20
mfi0: 3307 (boot + 24s/0x0002/info) - Inserted: PD 20(c None/p0) Info:
enclPd=20, scsiType=d, portMap=09, sasAddr=5001e090d62bfb00,0000000000000000
mfi0: 3308 (boot + 24s/0x0002/info) - Inserted: PD 00(e0x20/s0)
mfi0: 3309 (boot + 24s/0x0002/info) - Inserted: PD 00(e0x20/s0) Info: enclPd=20,
scsiType=0, portMap=00, sasAddr=5000cca005b1bf89,0000000000000000
mfi0: 3310 (boot + 24s/0x0002/info) - Inserted: PD 01(e0x20/s1)
mfi0: 3311 (boot + 24s/0x0002/info) - Inserted: PD 01(e0x20/s1) Info: enclPd=20,
scsiType=0, portMap=01, sasAddr=5000cca005b1427d,0000000000000000
mfi0: 3312 (boot + 24s/0x0002/info) - Inserted: PD 02(e0x20/s2)
mfi0: 3313 (boot + 24s/0x0002/info) - Inserted: PD 02(e0x20/s2) Info: enclPd=20,
scsiType=0, portMap=02, sasAddr=5000cca005b13c61,0000000000000000
mfi0: 3314 (boot + 24s/0x0042/info) - Global Hot Spare created on PD
01(e0x20/s1) (global)
mfi0: 3315 (boot + 24s/0x0002/info) - State change on PD 01(e0x20/s1) from
UNCONFIGURED_GOOD(0) to HOT SPARE(2)
mfi0: 3316 (boot + 68s/0x0008/info) - Battery temperature is normal
mfi0: 3317 (boot + 68s/0x0008/info) - Battery started charging
mfi0: 3318 (boot + 68s/0x0008/info) - Current capacity of the battery is above
threshold
mfi0: [ITHREAD]
pcib8: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci10: <ACPI PCI bus> on pcib8
isp0: <Qlogic ISP 2432 PCI FC-AL Adapter> port 0xdc00-0xdcff mem
0xfc5bc000-0xfc5bffff irq 16 at device 0.0 on pci10
firmware_get: failed to load firmware image isp_2400
isp0: [ITHREAD]
isp0: Board Type 2422, Chip Revision 0x3, resident F/W Revision 4.3.1
isp1: <Qlogic ISP 2432 PCI FC-AL Adapter> port 0xd800-0xd8ff mem
0xfc5b8000-0xfc5bbfff irq 17 at device 0.1 on pci10
firmware_get: failed to load firmware image isp_2400
isp1: [ITHREAD]
isp1: Board Type 2422, Chip Revision 0x3, resident F/W Revision 4.3.1
pcib9: <PCI-PCI bridge> at device 5.0 on pci0
pci11: <PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci12: <ACPI PCI bus> on pcib10
pcib11: <ACPI PCI-PCI bridge> at device 0.0 on pci12
pci13: <ACPI PCI bus> on pcib11
pcib12: <ACPI PCI-PCI bridge> at device 0.2 on pci12
pci14: <ACPI PCI bus> on pcib12
pcib13: <PCI-PCI bridge> at device 7.0 on pci0
pci15: <PCI bus> on pcib13
pcib14: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib14
pcib15: <PCI-PCI bridge> at device 0.0 on pci2
pci3: <PCI bus> on pcib15
bce1: <Broadcom NetXtreme II BCM5708 1000Base-T (B2)> mem
0xf8000000-0xf9ffffff irq 16 at device 0.0 on pci3
miibus1: <MII bus> on bce1
brgphy1: <BCM5708C 10/100/1000baseTX PHY> PHY 1 on miibus1
brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT,
1000baseT-FDX, auto
bce1: Ethernet address: 00:1e:c9:e8:13:e7
bce1: [ITHREAD]
bce1: ASIC (0x57081020); Rev (B2); Bus (PCI-X, 64-bit, 133MHz); F/W
(0x04000305); Flags( MFW MSI )
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port
0xbce0-0xbcff irq 21 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port
0xbcc0-0xbcdf irq 20 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port
0xbca0-0xbcbf irq 21 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
uhci3: <Intel 631XESB/632XESB/3100 USB controller USB-4> port
0xbc80-0xbc9f irq 20 at device 29.3 on pci0
uhci3: [GIANT-LOCKED]
uhci3: [ITHREAD]
usb3: <Intel 631XESB/632XESB/3100 USB controller USB-4> on uhci3
usb3: USB revision 1.0
uhub3: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb3
uhub3: 2 ports with 2 removable, self powered
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xfc800000-0xfc8003ff irq 21
at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb4: EHCI version 1.0
usb4: companion controllers, 2 ports each: usb0 usb1 usb2 usb3
usb4: <Intel 63XXESB USB 2.0 controller> on ehci0
usb4: USB revision 2.0
uhub4: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb4
uhub4: 8 ports with 8 removable, self powered
uhub5: <vendor 0x413c product 0xa001, class 9/0, rev 2.00/0.00, addr 2> on
uhub4
uhub5: multiple transaction translators
uhub5: 2 ports with 2 removable, self powered
ehci_device_intr_done: not done, ex=0xc699e800
ukbd0: <Dell DRAC5, class 0/0, rev 1.10/0.00, addr 3> on uhub5
ehci_device_intr_done: not done, ex=0xc699e000
kbd2 at ukbd0
ums0: <Dell DRAC5, class 0/0, rev 1.10/0.00, addr 3> on uhub5
ums0: X report 0x0002 not supported
device_attach: ums0 attach returned 6
umass0: <DELL INC. DRAC5 VIRTUAL MEDIA, class 0/0, rev 2.00/0.00, addr
4> on uhub5
umass1: <DELL INC. DRAC5 VIRTUAL MEDIA, class 0/0, rev 2.00/0.00, addr
4> on uhub5
uhub6: <vendor 0x04b4 product 0x6560, class 9/0, rev 2.00/0.0b, addr 5> on
uhub4
uhub6: multiple transaction translators
uhub6: 4 ports with 4 removable, self powered
ehci_device_intr_done: not done, ex=0xc69a9400
pcib16: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci16: <ACPI PCI bus> on pcib16
vgapci0: <VGA-compatible display> port 0xcc00-0xccff mem
0xd8000000-0xdfffffff,0xfc2d0000-0xfc2dffff irq 19 at device 13.0 on pci16
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfc00-0xfc0f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 flags 0x10 on
acpi0
sio0: type 16550A, console
sio0: [FILTER]
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 on acpi0
sio1: type 16550A
sio1: [FILTER]
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
p4tcc3: <CPU Frequency Thermal Control> on cpu3
cpu4: <ACPI CPU> on acpi0
p4tcc4: <CPU Frequency Thermal Control> on cpu4
cpu5: <ACPI CPU> on acpi0
p4tcc5: <CPU Frequency Thermal Control> on cpu5
cpu6: <ACPI CPU> on acpi0
p4tcc6: <CPU Frequency Thermal Control> on cpu6
cpu7: <ACPI CPU> on acpi0
p4tcc7: <CPU Frequency Thermal Control> on cpu7
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0
fdc0: does not respond
device_attach: fdc0 attach returned 6
pmtimer0 on isa0
orm0: <ISA Option ROMs> at iomem
0xc0000-0xc8fff,0xc9000-0xc9fff,0xd4800-0xd57ff,0xec000-0xeffff pnpid ORM0000 on
isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
ppc0: parallel port not found.
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Timecounters tick every 1.000 msec
bce0: link state changed to DOWN
bce1: link state changed to DOWN
acd0: DVDROM <TEAC DVD-ROM DV28EV/D.AE> at ata0-master UDMA33
mfi0: 3319 (277344766s/0x0020/info) - Time established as 10/15/08 0:12:46; (71
seconds since power on)
Expensive timeout(9) function: 0xc06f9de0(0xc0c52520) 0.113017715 s
mfid0: <MFI Logical Disk> on mfi0
mfid0: 69376MB (142082048 sectors) RAID volume '' is optimal
(probe376:isp0:0:376:0): cam completion status 0x4a
(probe376:isp0:0:376:0): cam completion status 0x4a
(probe395:isp0:0:395:0): cam completion status 0x4a
(probe395:isp0:0:395:0): cam completion status 0x4a
(probe400:isp0:0:400:0): cam completion status 0x4a
(probe400:isp0:0:400:0): cam completion status 0x4a
(probe411:isp0:0:411:0): cam completion status 0x4a
(probe411:isp0:0:411:0): cam completion status 0x4a
(probe423:isp0:0:423:0): cam completion status 0x4a
(probe423:isp0:0:423:0): cam completion status 0x4a
(probe437:isp0:0:437:0): cam completion status 0x4a
(probe437:isp0:0:437:0): cam completion status 0x4a
(probe449:isp0:0:449:0): cam completion status 0x4a
(probe449:isp0:0:449:0): cam completion status 0x4a
(probe459:isp0:0:459:0): cam completion status 0x4a
(probe459:isp0:0:459:0): cam completion status 0x4a
(probe464:isp0:0:464:0): cam completion status 0x4a
(probe464:isp0:0:464:0): cam completion status 0x4a
(probe471:isp0:0:471:0): cam completion status 0x4a
(probe471:isp0:0:471:0): cam completion status 0x4a
(probe478:isp0:0:478:0): cam completion status 0x4a
(probe478:isp0:0:478:0): cam completion status 0x4a
(probe488:isp0:0:488:0): cam completion status 0x4a
(probe488:isp0:0:488:0): cam completion status 0x4a
(probe504:isp0:0:504:0): cam completion status 0x4a
(probe504:isp0:0:504:0): cam completion status 0x4a
(probe505:isp0:0:505:0): cam completion status 0x4a
(probe505:isp0:0:505:0): cam completion status 0x4a
(probe506:isp0:0:506:0): cam completion status 0x4a
(probe506:isp0:0:506:0): cam completion status 0x4a
(probe507:isp0:0:507:0): cam completion status 0x4a
(probe507:isp0:0:507:0): cam completion status 0x4a
(probe508:isp0:0:508:0): cam completion status 0x4a
(probe508:isp0:0:508:0): cam completion status 0x4a
(probe509:isp0:0:509:0): cam completion status 0x4a
(probe509:isp0:0:509:0): cam completion status 0x4a
(probe510:isp0:0:510:0): cam completion status 0x4a
(probe510:isp0:0:510:0): cam completion status 0x4a
(probe511:isp0:0:511:0): cam completion status 0x4a
(probe511:isp0:0:511:0): cam completion status 0x4a
(probe1:isp0:0:1:0): cam completion status 0x4a
(probe1:isp0:0:1:0): cam completion status 0x4a
(probe2:isp0:0:2:0): cam completion status 0x4a
(probe2:isp0:0:2:0): cam completion status 0x4a
(probe3:isp0:0:3:0): cam completion status 0x4a
(probe0:isp0:0:0:0): cam completion status 0xcc
(probe3:isp0:0:3:0): cam completion status 0x4a
(probe4:isp0:0:4:0): cam completion status 0x4a
(probe4:isp0:0:4:0): cam completion status 0x4a
(probe5:isp0:0:5:0): cam completion status 0x4a
(probe5:isp0:0:5:0): cam completion status 0x4a
(probe7:isp0:0:7:0): cam completion status 0x4a
(probe7:isp0:0:7:0): cam completion status 0x4a
(probe8:isp0:0:8:0): cam completion status 0x4a
(probe8:isp0:0:8:0): cam completion status 0x4a
(probe9:isp0:0:9:0): cam completion status 0x4a
(probe9:isp0:0:9:0): cam completion status 0x4a
(probe10:isp0:0:10:0): cam completion status 0x4a
(probe10:isp0:0:10:0): cam completion status 0x4a
(probe11:isp0:0:11:0): cam completion status 0x4a
(probe11:isp0:0:11:0): cam completion status 0x4a
(probe12:isp0:0:12:0): cam completion status 0x4a
(probe12:isp0:0:12:0): cam completion status 0x4a
(probe13:isp0:0:13:0): cam completion status 0x4a
(probe13:isp0:0:13:0): cam completion status 0x4a
(probe14:isp0:0:14:0): cam completion status 0x4a
(probe14:isp0:0:14:0): cam completion status 0x4a
(probe15:isp0:0:15:0): cam completion status 0x4a
(probe15:isp0:0:15:0): cam completion status 0x4a
(probe16:isp0:0:16:0): cam completion status 0x4a
(probe16:isp0:0:16:0): cam completion status 0x4a
(probe17:isp0:0:17:0): cam completion status 0x4a
(probe17:isp0:0:17:0): cam completion status 0x4a
(probe18:isp0:0:18:0): cam completion status 0x4a
(probe18:isp0:0:18:0): cam completion status 0x4a
(probe19:isp0:0:19:0): cam completion status 0x4a
(probe19:isp0:0:19:0): cam completion status 0x4a
(probe20:isp0:0:20:0): cam completion status 0x4a
(probe20:isp0:0:20:0): cam completion status 0x4a
(probe21:isp0:0:21:0): cam completion status 0x4a
(probe21:isp0:0:21:0): cam completion status 0x4a
(probe22:isp0:0:22:0): cam completion status 0x4a
(probe22:isp0:0:22:0): cam completion status 0x4a
(probe23:isp0:0:23:0): cam completion status 0x4a
(probe23:isp0:0:23:0): cam completion status 0x4a
(probe24:isp0:0:24:0): cam completion status 0x4a
(probe24:isp0:0:24:0): cam completion status 0x4a
(probe26:isp0:0:26:0): cam completion status 0x4a
(probe26:isp0:0:26:0): cam completion status 0x4a
(probe27:isp0:0:27:0): cam completion status 0x4a
(probe27:isp0:0:27:0): cam completion status 0x4a
(probe28:isp0:0:28:0): cam completion status 0x4a
(probe28:isp0:0:28:0): cam completion status 0x4a
(probe29:isp0:0:29:0): cam completion status 0x4a
(probe29:isp0:0:29:0): cam completion status 0x4a
(probe30:isp0:0:30:0): cam completion status 0x4a
(probe30:isp0:0:30:0): cam completion status 0x4a
(probe31:isp0:0:31:0): cam completion status 0x4a
(probe31:isp0:0:31:0): cam completion status 0x4a
(probe32:isp0:0:32:0): cam completion status 0x4a
(probe32:isp0:0:32:0): cam completion status 0x4a
(probe33:isp0:0:33:0): cam completion status 0x4a
(probe33:isp0:0:33:0): cam completion status 0x4a
(probe34:isp0:0:34:0): cam completion status 0x4a
(probe34:isp0:0:34:0): cam completion status 0x4a
(probe35:isp0:0:35:0): cam completion status 0x4a
(probe35:isp0:0:35:0): cam completion status 0x4a
(probe36:isp0:0:36:0): cam completion status 0x4a
(probe36:isp0:0:36:0): cam completion status 0x4a
(probe37:isp0:0:37:0): cam completion status 0x4a
(probe37:isp0:0:37:0): cam completion status 0x4a
(probe38:isp0:0:38:0): cam completion status 0x4a
(probe38:isp0:0:38:0): cam completion status 0x4a
(probe39:isp0:0:39:0): cam completion status 0x4a
(probe39:isp0:0:39:0): cam completion status 0x4a
(probe40:isp0:0:40:0): cam completion status 0x4a
(probe40:isp0:0:40:0): cam completion status 0x4a
(probe41:isp0:0:41:0): cam completion status 0x4a
(probe41:isp0:0:41:0): cam completion status 0x4a
(probe42:isp0:0:42:0): cam completion status 0x4a
(probe42:isp0:0:42:0): cam completion status 0x4a
(probe43:isp0:0:43:0): cam completion status 0x4a
(probe43:isp0:0:43:0): cam completion status 0x4a
(probe44:isp0:0:44:0): cam completion status 0x4a
(probe44:isp0:0:44:0): cam completion status 0x4a
(probe45:isp0:0:45:0): cam completion status 0x4a
(probe45:isp0:0:45:0): cam completion status 0x4a
(probe46:isp0:0:46:0): cam completion status 0x4a
(probe46:isp0:0:46:0): cam completion status 0x4a
(probe47:isp0:0:47:0): cam completion status 0x4a
(probe47:isp0:0:47:0): cam completion status 0x4a
(probe48:isp0:0:48:0): cam completion status 0x4a
(probe48:isp0:0:48:0): cam completion status 0x4a
(probe49:isp0:0:49:0): cam completion status 0x4a
(probe49:isp0:0:49:0): cam completion status 0x4a
(probe50:isp0:0:50:0): cam completion status 0x4a
(probe50:isp0:0:50:0): cam completion status 0x4a
(probe51:isp0:0:51:0): cam completion status 0x4a
(probe51:isp0:0:51:0): cam completion status 0x4a
(probe52:isp0:0:52:0): cam completion status 0x4a
(probe52:isp0:0:52:0): cam completion status 0x4a
(probe53:isp0:0:53:0): cam completion status 0x4a
(probe53:isp0:0:53:0): cam completion status 0x4a
(probe54:isp0:0:54:0): cam completion status 0x4a
(probe54:isp0:0:54:0): cam completion status 0x4a
(probe55:isp0:0:55:0): cam completion status 0x4a
(probe55:isp0:0:55:0): cam completion status 0x4a
(probe56:isp0:0:56:0): cam completion status 0x4a
(probe56:isp0:0:56:0): cam completion status 0x4a
(probe57:isp0:0:57:0): cam completion status 0x4a
(probe57:isp0:0:57:0): cam completion status 0x4a
(probe58:isp0:0:58:0): cam completion status 0x4a
(probe58:isp0:0:58:0): cam completion status 0x4a
(probe59:isp0:0:59:0): cam completion status 0x4a
(probe59:isp0:0:59:0): cam completion status 0x4a
(probe60:isp0:0:60:0): cam completion status 0x4a
(probe60:isp0:0:60:0): cam completion status 0x4a
(probe61:isp0:0:61:0): cam completion status 0x4a
(probe61:isp0:0:61:0): cam completion status 0x4a
(probe63:isp0:0:63:0): cam completion status 0x4a
(probe63:isp0:0:63:0): cam completion status 0x4a
(probe64:isp0:0:64:0): cam completion status 0x4a
(probe64:isp0:0:64:0): cam completion status 0x4a
(probe65:isp0:0:65:0): cam completion status 0x4a
(probe65:isp0:0:65:0): cam completion status 0x4a
(probe66:isp0:0:66:0): cam completion status 0x4a
(probe66:isp0:0:66:0): cam completion status 0x4a
(probe67:isp0:0:67:0): cam completion status 0x4a
(probe67:isp0:0:67:0): cam completion status 0x4a
(probe68:isp0:0:68:0): cam completion status 0x4a
(probe68:isp0:0:68:0): cam completion status 0x4a
(probe69:isp0:0:69:0): cam completion status 0x4a
(probe69:isp0:0:69:0): cam completion status 0x4a
(probe70:isp0:0:70:0): cam completion status 0x4a
(probe70:isp0:0:70:0): cam completion status 0x4a
(probe71:isp0:0:71:0): cam completion status 0x4a
(probe71:isp0:0:71:0): cam completion status 0x4a
(probe72:isp0:0:72:0): cam completion status 0x4a
(probe72:isp0:0:72:0): cam completion status 0x4a
(probe73:isp0:0:73:0): cam completion status 0x4a
(probe73:isp0:0:73:0): cam completion status 0x4a
(probe75:isp0:0:75:0): cam completion status 0x4a
(probe75:isp0:0:75:0): cam completion status 0x4a
(probe76:isp0:0:76:0): cam completion status 0x4a
(probe76:isp0:0:76:0): cam completion status 0x4a
(probe77:isp0:0:77:0): cam completion status 0x4a
(probe77:isp0:0:77:0): cam completion status 0x4a
(probe78:isp0:0:78:0): cam completion status 0x4a
(probe78:isp0:0:78:0): cam completion status 0x4a
(probe79:isp0:0:79:0): cam completion status 0x4a
(probe79:isp0:0:79:0): cam completion status 0x4a
(probe81:isp0:0:81:0): cam completion status 0x4a
(probe81:isp0:0:81:0): cam completion status 0x4a
(probe82:isp0:0:82:0): cam completion status 0x4a
(probe82:isp0:0:82:0): cam completion status 0x4a
(probe83:isp0:0:83:0): cam completion status 0x4a
(probe83:isp0:0:83:0): cam completion status 0x4a
(probe84:isp0:0:84:0): cam completion status 0x4a
(probe84:isp0:0:84:0): cam completion status 0x4a
(probe85:isp0:0:85:0): cam completion status 0x4a
(probe85:isp0:0:85:0): cam completion status 0x4a
(probe86:isp0:0:86:0): cam completion status 0x4a
(probe86:isp0:0:86:0): cam completion status 0x4a
(probe87:isp0:0:87:0): cam completion status 0x4a
(probe87:isp0:0:87:0): cam completion status 0x4a
(probe88:isp0:0:88:0): cam completion status 0x4a
(probe88:isp0:0:88:0): cam completion status 0x4a
(probe89:isp0:0:89:0): cam completion status 0x4a
(probe89:isp0:0:89:0): cam completion status 0x4a
(probe90:isp0:0:90:0): cam completion status 0x4a
(probe90:isp0:0:90:0): cam completion status 0x4a
(probe91:isp0:0:91:0): cam completion status 0x4a
(probe91:isp0:0:91:0): cam completion status 0x4a
(probe92:isp0:0:92:0): cam completion status 0x4a
(probe92:isp0:0:92:0): cam completion status 0x4a
(probe93:isp0:0:93:0): cam completion status 0x4a
(probe93:isp0:0:93:0): cam completion status 0x4a
(probe94:isp0:0:94:0): cam completion status 0x4a
(probe94:isp0:0:94:0): cam completion status 0x4a
(probe95:isp0:0:95:0): cam completion status 0x4a
(probe95:isp0:0:95:0): cam completion status 0x4a
(probe96:isp0:0:96:0): cam completion status 0x4a
(probe96:isp0:0:96:0): cam completion status 0x4a
(probe97:isp0:0:97:0): cam completion status 0x4a
(probe97:isp0:0:97:0): cam completion status 0x4a
(probe98:isp0:0:98:0): cam completion status 0x4a
(probe98:isp0:0:98:0): cam completion status 0x4a
(probe99:isp0:0:99:0): cam completion status 0x4a
(probe99:isp0:0:99:0): cam completion status 0x4a
(probe100:isp0:0:100:0): cam completion status 0x4a
(probe100:isp0:0:100:0): cam completion status 0x4a
(probe101:isp0:0:101:0): cam completion status 0x4a
(probe101:isp0:0:101:0): cam completion status 0x4a
(probe102:isp0:0:102:0): cam completion status 0x4a
(probe102:isp0:0:102:0): cam completion status 0x4a
(probe103:isp0:0:103:0): cam completion status 0x4a
(probe103:isp0:0:103:0): cam completion status 0x4a
(probe104:isp0:0:104:0): cam completion status 0x4a
(probe104:isp0:0:104:0): cam completion status 0x4a
(probe105:isp0:0:105:0): cam completion status 0x4a
(probe105:isp0:0:105:0): cam completion status 0x4a
(probe106:isp0:0:106:0): cam completion status 0x4a
(probe106:isp0:0:106:0): cam completion status 0x4a
(probe107:isp0:0:107:0): cam completion status 0x4a
(probe107:isp0:0:107:0): cam completion status 0x4a
(probe108:isp0:0:108:0): cam completion status 0x4a
(probe108:isp0:0:108:0): cam completion status 0x4a
(probe110:isp0:0:110:0): cam completion status 0x4a
(probe110:isp0:0:110:0): cam completion status 0x4a
(probe111:isp0:0:111:0): cam completion status 0x4a
(probe111:isp0:0:111:0): cam completion status 0x4a
(probe112:isp0:0:112:0): cam completion status 0x4a
(probe112:isp0:0:112:0): cam completion status 0x4a
(probe113:isp0:0:113:0): cam completion status 0x4a
(probe113:isp0:0:113:0): cam completion status 0x4a
(probe114:isp0:0:114:0): cam completion status 0x4a
(probe114:isp0:0:114:0): cam completion status 0x4a
(probe115:isp0:0:115:0): cam completion status 0x4a
(probe115:isp0:0:115:0): cam completion status 0x4a
(probe116:isp0:0:116:0): cam completion status 0x4a
(probe116:isp0:0:116:0): cam completion status 0x4a
(probe117:isp0:0:117:0): cam completion status 0x4a
(probe117:isp0:0:117:0): cam completion status 0x4a
(probe118:isp0:0:118:0): cam completion status 0x4a
(probe118:isp0:0:118:0): cam completion status 0x4a
(probe119:isp0:0:119:0): cam completion status 0x4a
(probe119:isp0:0:119:0): cam completion status 0x4a
(probe120:isp0:0:120:0): cam completion status 0x4a
(probe120:isp0:0:120:0): cam completion status 0x4a
(probe121:isp0:0:121:0): cam completion status 0x4a
(probe121:isp0:0:121:0): cam completion status 0x4a
(probe122:isp0:0:122:0): cam completion status 0x4a
(probe122:isp0:0:122:0): cam completion status 0x4a
(probe123:isp0:0:123:0): cam completion status 0x4a
(probe123:isp0:0:123:0): cam completion status 0x4a
(probe124:isp0:0:124:0): cam completion status 0x4a
(probe124:isp0:0:124:0): cam completion status 0x4a
(probe125:isp0:0:125:0): cam completion status 0x4a
(probe125:isp0:0:125:0): cam completion status 0x4a
(probe126:isp0:0:126:0): cam completion status 0x4a
(probe126:isp0:0:126:0): cam completion status 0x4a
(probe127:isp0:0:127:0): cam completion status 0x4a
(probe127:isp0:0:127:0): cam completion status 0x4a
(probe128:isp0:0:128:0): cam completion status 0x4a
(probe128:isp0:0:128:0): cam completion status 0x4a
(probe129:isp0:0:129:0): cam completion status 0x4a
(probe129:isp0:0:129:0): cam completion status 0x4a
(probe130:isp0:0:130:0): cam completion status 0x4a
(probe130:isp0:0:130:0): cam completion status 0x4a
(probe132:isp0:0:132:0): cam completion status 0x4a
(probe132:isp0:0:132:0): cam completion status 0x4a
(probe133:isp0:0:133:0): cam completion status 0x4a
(probe133:isp0:0:133:0): cam completion status 0x4a
(probe134:isp0:0:134:0): cam completion status 0x4a
(probe134:isp0:0:134:0): cam completion status 0x4a
(probe135:isp0:0:135:0): cam completion status 0x4a
(probe135:isp0:0:135:0): cam completion status 0x4a
(probe136:isp0:0:136:0): cam completion status 0x4a
(probe136:isp0:0:136:0): cam completion status 0x4a
(probe137:isp0:0:137:0): cam completion status 0x4a
(probe137:isp0:0:137:0): cam completion status 0x4a
(probe138:isp0:0:138:0): cam completion status 0x4a
(probe138:isp0:0:138:0): cam completion status 0x4a
(probe139:isp0:0:139:0): cam completion status 0x4a
(probe139:isp0:0:139:0): cam completion status 0x4a
(probe140:isp0:0:140:0): cam completion status 0x4a
(probe140:isp0:0:140:0): cam completion status 0x4a
(probe141:isp0:0:141:0): cam completion status 0x4a
(probe141:isp0:0:141:0): cam completion status 0x4a
(probe142:isp0:0:142:0): cam completion status 0x4a
(probe142:isp0:0:142:0): cam completion status 0x4a
(probe143:isp0:0:143:0): cam completion status 0x4a
(probe143:isp0:0:143:0): cam completion status 0x4a
(probe144:isp0:0:144:0): cam completion status 0x4a
(probe144:isp0:0:144:0): cam completion status 0x4a
(probe145:isp0:0:145:0): cam completion status 0x4a
(probe145:isp0:0:145:0): cam completion status 0x4a
(probe146:isp0:0:146:0): cam completion status 0x4a
(probe146:isp0:0:146:0): cam completion status 0x4a
(probe148:isp0:0:148:0): cam completion status 0x4a
(probe148:isp0:0:148:0): cam completion status 0x4a
(probe149:isp0:0:149:0): cam completion status 0x4a
(probe149:isp0:0:149:0): cam completion status 0x4a
(probe150:isp0:0:150:0): cam completion status 0x4a
(probe150:isp0:0:150:0): cam completion status 0x4a
(probe151:isp0:0:151:0): cam completion status 0x4a
(probe151:isp0:0:151:0): cam completion status 0x4a
(probe152:isp0:0:152:0): cam completion status 0x4a
(probe152:isp0:0:152:0): cam completion status 0x4a
(probe153:isp0:0:153:0): cam completion status 0x4a
(probe153:isp0:0:153:0): cam completion status 0x4a
(probe154:isp0:0:154:0): cam completion status 0x4a
(probe154:isp0:0:154:0): cam completion status 0x4a
(probe155:isp0:0:155:0): cam completion status 0x4a
(probe155:isp0:0:155:0): cam completion status 0x4a
(probe156:isp0:0:156:0): cam completion status 0x4a
(probe156:isp0:0:156:0): cam completion status 0x4a
(probe157:isp0:0:157:0): cam completion status 0x4a
(probe157:isp0:0:157:0): cam completion status 0x4a
(probe158:isp0:0:158:0): cam completion status 0x4a
(probe158:isp0:0:158:0): cam completion status 0x4a
(probe159:isp0:0:159:0): cam completion status 0x4a
(probe159:isp0:0:159:0): cam completion status 0x4a
(probe160:isp0:0:160:0): cam completion status 0x4a
(probe160:isp0:0:160:0): cam completion status 0x4a
(probe161:isp0:0:161:0): cam completion status 0x4a
(probe161:isp0:0:161:0): cam completion status 0x4a
(probe162:isp0:0:162:0): cam completion status 0x4a
(probe162:isp0:0:162:0): cam completion status 0x4a
(probe163:isp0:0:163:0): cam completion status 0x4a
(probe163:isp0:0:163:0): cam completion status 0x4a
(probe164:isp0:0:164:0): cam completion status 0x4a
(probe164:isp0:0:164:0): cam completion status 0x4a
(probe165:isp0:0:165:0): cam completion status 0x4a
(probe165:isp0:0:165:0): cam completion status 0x4a
(probe166:isp0:0:166:0): cam completion status 0x4a
(probe166:isp0:0:166:0): cam completion status 0x4a
(probe167:isp0:0:167:0): cam completion status 0x4a
(probe167:isp0:0:167:0): cam completion status 0x4a
(probe168:isp0:0:168:0): cam completion status 0x4a
(probe168:isp0:0:168:0): cam completion status 0x4a
(probe169:isp0:0:169:0): cam completion status 0x4a
(probe169:isp0:0:169:0): cam completion status 0x4a
(probe170:isp0:0:170:0): cam completion status 0x4a
(probe170:isp0:0:170:0): cam completion status 0x4a
(probe171:isp0:0:171:0): cam completion status 0x4a
(probe171:isp0:0:171:0): cam completion status 0x4a
(probe172:isp0:0:172:0): cam completion status 0x4a
(probe172:isp0:0:172:0): cam completion status 0x4a
(probe173:isp0:0:173:0): cam completion status 0x4a
(probe173:isp0:0:173:0): cam completion status 0x4a
(probe174:isp0:0:174:0): cam completion status 0x4a
(probe174:isp0:0:174:0): cam completion status 0x4a
(probe175:isp0:0:175:0): cam completion status 0x4a
(probe175:isp0:0:175:0): cam completion status 0x4a
(probe176:isp0:0:176:0): cam completion status 0x4a
(probe176:isp0:0:176:0): cam completion status 0x4a
(probe177:isp0:0:177:0): cam completion status 0x4a
(probe177:isp0:0:177:0): cam completion status 0x4a
(probe178:isp0:0:178:0): cam completion status 0x4a
(probe178:isp0:0:178:0): cam completion status 0x4a
(probe179:isp0:0:179:0): cam completion status 0x4a
(probe179:isp0:0:179:0): cam completion status 0x4a
(probe180:isp0:0:180:0): cam completion status 0x4a
(probe180:isp0:0:180:0): cam completion status 0x4a
(probe181:isp0:0:181:0): cam completion status 0x4a
(probe181:isp0:0:181:0): cam completion status 0x4a
(probe182:isp0:0:182:0): cam completion status 0x4a
(probe182:isp0:0:182:0): cam completion status 0x4a
(probe183:isp0:0:183:0): cam completion status 0x4a
(probe183:isp0:0:183:0): cam completion status 0x4a
(probe184:isp0:0:184:0): cam completion status 0x4a
(probe184:isp0:0:184:0): cam completion status 0x4a
(probe185:isp0:0:185:0): cam completion status 0x4a
(probe185:isp0:0:185:0): cam completion status 0x4a
(probe186:isp0:0:186:0): cam completion status 0x4a
(probe186:isp0:0:186:0): cam completion status 0x4a
(probe187:isp0:0:187:0): cam completion status 0x4a
(probe187:isp0:0:187:0): cam completion status 0x4a
(probe188:isp0:0:188:0): cam completion status 0x4a
(probe188:isp0:0:188:0): cam completion status 0x4a
(probe189:isp0:0:189:0): cam completion status 0x4a
(probe189:isp0:0:189:0): cam completion status 0x4a
(probe190:isp0:0:190:0): cam completion status 0x4a
(probe190:isp0:0:190:0): cam completion status 0x4a
(probe191:isp0:0:191:0): cam completion status 0x4a
(probe191:isp0:0:191:0): cam completion status 0x4a
(probe192:isp0:0:192:0): cam completion status 0x4a
(probe192:isp0:0:192:0): cam completion status 0x4a
(probe193:isp0:0:193:0): cam completion status 0x4a
(probe193:isp0:0:193:0): cam completion status 0x4a
(probe194:isp0:0:194:0): cam completion status 0x4a
(probe194:isp0:0:194:0): cam completion status 0x4a
(probe195:isp0:0:195:0): cam completion status 0x4a
(probe195:isp0:0:195:0): cam completion status 0x4a
(probe196:isp0:0:196:0): cam completion status 0x4a
(probe196:isp0:0:196:0): cam completion status 0x4a
(probe197:isp0:0:197:0): cam completion status 0x4a
(probe197:isp0:0:197:0): cam completion status 0x4a
(probe198:isp0:0:198:0): cam completion status 0x4a
(probe198:isp0:0:198:0): cam completion status 0x4a
(probe199:isp0:0:199:0): cam completion status 0x4a
(probe199:isp0:0:199:0): cam completion status 0x4a
(probe200:isp0:0:200:0): cam completion status 0x4a
(probe200:isp0:0:200:0): cam completion status 0x4a
(probe201:isp0:0:201:0): cam completion status 0x4a
(probe201:isp0:0:201:0): cam completion status 0x4a
(probe202:isp0:0:202:0): cam completion status 0x4a
(probe202:isp0:0:202:0): cam completion status 0x4a
(probe203:isp0:0:203:0): cam completion status 0x4a
(probe203:isp0:0:203:0): cam completion status 0x4a
(probe204:isp0:0:204:0): cam completion status 0x4a
(probe204:isp0:0:204:0): cam completion status 0x4a
(probe205:isp0:0:205:0): cam completion status 0x4a
(probe205:isp0:0:205:0): cam completion status 0x4a
(probe207:isp0:0:207:0): cam completion status 0x4a
(probe207:isp0:0:207:0): cam completion status 0x4a
(probe208:isp0:0:208:0): cam completion status 0x4a
(probe208:isp0:0:208:0): cam completion status 0x4a
(probe209:isp0:0:209:0): cam completion status 0x4a
(probe209:isp0:0:209:0): cam completion status 0x4a
(probe210:isp0:0:210:0): cam completion status 0x4a
(probe210:isp0:0:210:0): cam completion status 0x4a
(probe211:isp0:0:211:0): cam completion status 0x4a
(probe211:isp0:0:211:0): cam completion status 0x4a
(probe212:isp0:0:212:0): cam completion status 0x4a
(probe212:isp0:0:212:0): cam completion status 0x4a
(probe213:isp0:0:213:0): cam completion status 0x4a
(probe213:isp0:0:213:0): cam completion status 0x4a
(probe214:isp0:0:214:0): cam completion status 0x4a
(probe214:isp0:0:214:0): cam completion status 0x4a
(probe215:isp0:0:215:0): cam completion status 0x4a
(probe215:isp0:0:215:0): cam completion status 0x4a
(probe216:isp0:0:216:0): cam completion status 0x4a
(probe216:isp0:0:216:0): cam completion status 0x4a
(probe217:isp0:0:217:0): cam completion status 0x4a
(probe217:isp0:0:217:0): cam completion status 0x4a
(probe218:isp0:0:218:0): cam completion status 0x4a
(probe218:isp0:0:218:0): cam completion status 0x4a
(probe219:isp0:0:219:0): cam completion status 0x4a
(probe219:isp0:0:219:0): cam completion status 0x4a
(probe220:isp0:0:220:0): cam completion status 0x4a
(probe220:isp0:0:220:0): cam completion status 0x4a
(probe221:isp0:0:221:0): cam completion status 0x4a
(probe221:isp0:0:221:0): cam completion status 0x4a
(probe222:isp0:0:222:0): cam completion status 0x4a
(probe222:isp0:0:222:0): cam completion status 0x4a
(probe223:isp0:0:223:0): cam completion status 0x4a
(probe223:isp0:0:223:0): cam completion status 0x4a
(probe224:isp0:0:224:0): cam completion status 0x4a
(probe224:isp0:0:224:0): cam completion status 0x4a
(probe225:isp0:0:225:0): cam completion status 0x4a
(probe225:isp0:0:225:0): cam completion status 0x4a
(probe226:isp0:0:226:0): cam completion status 0x4a
(probe226:isp0:0:226:0): cam completion status 0x4a
(probe227:isp0:0:227:0): cam completion status 0x4a
(probe227:isp0:0:227:0): cam completion status 0x4a
(probe228:isp0:0:228:0): cam completion status 0x4a
(probe228:isp0:0:228:0): cam completion status 0x4a
(probe229:isp0:0:229:0): cam completion status 0x4a
(probe229:isp0:0:229:0): cam completion status 0x4a
(probe230:isp0:0:230:0): cam completion status 0x4a
(probe230:isp0:0:230:0): cam completion status 0x4a
(probe231:isp0:0:231:0): cam completion status 0x4a
(probe231:isp0:0:231:0): cam completion status 0x4a
(probe232:isp0:0:232:0): cam completion status 0x4a
(probe232:isp0:0:232:0): cam completion status 0x4a
(probe233:isp0:0:233:0): cam completion status 0x4a
(probe233:isp0:0:233:0): cam completion status 0x4a
(probe234:isp0:0:234:0): cam completion status 0x4a
(probe234:isp0:0:234:0): cam completion status 0x4a
(probe235:isp0:0:235:0): cam completion status 0x4a
(probe235:isp0:0:235:0): cam completion status 0x4a
(probe236:isp0:0:236:0): cam completion status 0x4a
(probe236:isp0:0:236:0): cam completion status 0x4a
(probe237:isp0:0:237:0): cam completion status 0x4a
(probe237:isp0:0:237:0): cam completion status 0x4a
(probe238:isp0:0:238:0): cam completion status 0x4a
(probe238:isp0:0:238:0): cam completion status 0x4a
(probe239:isp0:0:239:0): cam completion status 0x4a
(probe239:isp0:0:239:0): cam completion status 0x4a
(probe240:isp0:0:240:0): cam completion status 0x4a
(probe240:isp0:0:240:0): cam completion status 0x4a
(probe241:isp0:0:241:0): cam completion status 0x4a
(probe241:isp0:0:241:0): cam completion status 0x4a
(probe242:isp0:0:242:0): cam completion status 0x4a
(probe242:isp0:0:242:0): cam completion status 0x4a
(probe243:isp0:0:243:0): cam completion status 0x4a
(probe243:isp0:0:243:0): cam completion status 0x4a
(probe244:isp0:0:244:0): cam completion status 0x4a
(probe244:isp0:0:244:0): cam completion status 0x4a
(probe245:isp0:0:245:0): cam completion status 0x4a
(probe245:isp0:0:245:0): cam completion status 0x4a
(probe246:isp0:0:246:0): cam completion status 0x4a
(probe246:isp0:0:246:0): cam completion status 0x4a
(probe247:isp0:0:247:0): cam completion status 0x4a
(probe247:isp0:0:247:0): cam completion status 0x4a
(probe248:isp0:0:248:0): cam completion status 0x4a
(probe248:isp0:0:248:0): cam completion status 0x4a
(probe249:isp0:0:249:0): cam completion status 0x4a
(probe249:isp0:0:249:0): cam completion status 0x4a
(probe250:isp0:0:250:0): cam completion status 0x4a
(probe250:isp0:0:250:0): cam completion status 0x4a
(probe251:isp0:0:251:0): cam completion status 0x4a
(probe251:isp0:0:251:0): cam completion status 0x4a
(probe252:isp0:0:252:0): cam completion status 0x4a
(probe252:isp0:0:252:0): cam completion status 0x4a
(probe253:isp0:0:253:0): cam completion status 0x4a
(probe253:isp0:0:253:0): cam completion status 0x4a
(probe254:isp0:0:254:0): cam completion status 0x4a
(probe254:isp0:0:254:0): cam completion status 0x4a
(probe255:isp0:0:255:0): cam completion status 0x4a
(probe255:isp0:0:255:0): cam completion status 0x4a
(probe256:isp0:0:256:0): cam completion status 0x4a
(probe256:isp0:0:256:0): cam completion status 0x4a
(probe257:isp0:0:257:0): cam completion status 0x4a
(probe257:isp0:0:257:0): cam completion status 0x4a
(probe258:isp0:0:258:0): cam completion status 0x4a
(probe258:isp0:0:258:0): cam completion status 0x4a
(probe259:isp0:0:259:0): cam completion status 0x4a
(probe259:isp0:0:259:0): cam completion status 0x4a
(probe260:isp0:0:260:0): cam completion status 0x4a
(probe260:isp0:0:260:0): cam completion status 0x4a
(probe261:isp0:0:261:0): cam completion status 0x4a
(probe261:isp0:0:261:0): cam completion status 0x4a
(probe262:isp0:0:262:0): cam completion status 0x4a
(probe262:isp0:0:262:0): cam completion status 0x4a
(probe263:isp0:0:263:0): cam completion status 0x4a
(probe263:isp0:0:263:0): cam completion status 0x4a
(probe264:isp0:0:264:0): cam completion status 0x4a
(probe264:isp0:0:264:0): cam completion status 0x4a
(probe265:isp0:0:265:0): cam completion status 0x4a
(probe265:isp0:0:265:0): cam completion status 0x4a
(probe266:isp0:0:266:0): cam completion status 0x4a
(probe266:isp0:0:266:0): cam completion status 0x4a
(probe267:isp0:0:267:0): cam completion status 0x4a
(probe267:isp0:0:267:0): cam completion status 0x4a
(probe268:isp0:0:268:0): cam completion status 0x4a
(probe268:isp0:0:268:0): cam completion status 0x4a
(probe269:isp0:0:269:0): cam completion status 0x4a
(probe269:isp0:0:269:0): cam completion status 0x4a
(probe270:isp0:0:270:0): cam completion status 0x4a
(probe270:isp0:0:270:0): cam completion status 0x4a
(probe271:isp0:0:271:0): cam completion status 0x4a
(probe271:isp0:0:271:0): cam completion status 0x4a
(probe272:isp0:0:272:0): cam completion status 0x4a
(probe272:isp0:0:272:0): cam completion status 0x4a
(probe273:isp0:0:273:0): cam completion status 0x4a
(probe273:isp0:0:273:0): cam completion status 0x4a
(probe274:isp0:0:274:0): cam completion status 0x4a
(probe274:isp0:0:274:0): cam completion status 0x4a
(probe275:isp0:0:275:0): cam completion status 0x4a
(probe275:isp0:0:275:0): cam completion status 0x4a
(probe276:isp0:0:276:0): cam completion status 0x4a
(probe276:isp0:0:276:0): cam completion status 0x4a
(probe277:isp0:0:277:0): cam completion status 0x4a
(probe277:isp0:0:277:0): cam completion status 0x4a
(probe278:isp0:0:278:0): cam completion status 0x4a
(probe278:isp0:0:278:0): cam completion status 0x4a
(probe279:isp0:0:279:0): cam completion status 0x4a
(probe279:isp0:0:279:0): cam completion status 0x4a
(probe280:isp0:0:280:0): cam completion status 0x4a
(probe280:isp0:0:280:0): cam completion status 0x4a
(probe281:isp0:0:281:0): cam completion status 0x4a
(probe281:isp0:0:281:0): cam completion status 0x4a
(probe282:isp0:0:282:0): cam completion status 0x4a
(probe282:isp0:0:282:0): cam completion status 0x4a
(probe283:isp0:0:283:0): cam completion status 0x4a
(probe283:isp0:0:283:0): cam completion status 0x4a
(probe284:isp0:0:284:0): cam completion status 0x4a
(probe284:isp0:0:284:0): cam completion status 0x4a
(probe285:isp0:0:285:0): cam completion status 0x4a
(probe285:isp0:0:285:0): cam completion status 0x4a
(probe286:isp0:0:286:0): cam completion status 0x4a
(probe286:isp0:0:286:0): cam completion status 0x4a
(probe287:isp0:0:287:0): cam completion status 0x4a
(probe287:isp0:0:287:0): cam completion status 0x4a
(probe288:isp0:0:288:0): cam completion status 0x4a
(probe288:isp0:0:288:0): cam completion status 0x4a
(probe289:isp0:0:289:0): cam completion status 0x4a
(probe289:isp0:0:289:0): cam completion status 0x4a
(probe290:isp0:0:290:0): cam completion status 0x4a
(probe290:isp0:0:290:0): cam completion status 0x4a
(probe291:isp0:0:291:0): cam completion status 0x4a
(probe291:isp0:0:291:0): cam completion status 0x4a
(probe292:isp0:0:292:0): cam completion status 0x4a
(probe292:isp0:0:292:0): cam completion status 0x4a
(probe293:isp0:0:293:0): cam completion status 0x4a
(probe293:isp0:0:293:0): cam completion status 0x4a
(probe294:isp0:0:294:0): cam completion status 0x4a
(probe294:isp0:0:294:0): cam completion status 0x4a
(probe295:isp0:0:295:0): cam completion status 0x4a
(probe295:isp0:0:295:0): cam completion status 0x4a
(probe296:isp0:0:296:0): cam completion status 0x4a
(probe296:isp0:0:296:0): cam completion status 0x4a
(probe297:isp0:0:297:0): cam completion status 0x4a
(probe297:isp0:0:297:0): cam completion status 0x4a
(probe298:isp0:0:298:0): cam completion status 0x4a
(probe298:isp0:0:298:0): cam completion status 0x4a
(probe299:isp0:0:299:0): cam completion status 0x4a
(probe299:isp0:0:299:0): cam completion status 0x4a
(probe300:isp0:0:300:0): cam completion status 0x4a
(probe300:isp0:0:300:0): cam completion status 0x4a
(probe301:isp0:0:301:0): cam completion status 0x4a
(probe301:isp0:0:301:0): cam completion status 0x4a
(probe302:isp0:0:302:0): cam completion status 0x4a
(probe302:isp0:0:302:0): cam completion status 0x4a
(probe303:isp0:0:303:0): cam completion status 0x4a
(probe303:isp0:0:303:0): cam completion status 0x4a
(probe304:isp0:0:304:0): cam completion status 0x4a
(probe304:isp0:0:304:0): cam completion status 0x4a
(probe305:isp0:0:305:0): cam completion status 0x4a
(probe305:isp0:0:305:0): cam completion status 0x4a
(probe306:isp0:0:306:0): cam completion status 0x4a
(probe306:isp0:0:306:0): cam completion status 0x4a
(probe307:isp0:0:307:0): cam completion status 0x4a
(probe307:isp0:0:307:0): cam completion status 0x4a
(probe308:isp0:0:308:0): cam completion status 0x4a
(probe308:isp0:0:308:0): cam completion status 0x4a
(probe309:isp0:0:309:0): cam completion status 0x4a
(probe309:isp0:0:309:0): cam completion status 0x4a
(probe310:isp0:0:310:0): cam completion status 0x4a
(probe310:isp0:0:310:0): cam completion status 0x4a
(probe311:isp0:0:311:0): cam completion status 0x4a
(probe311:isp0:0:311:0): cam completion status 0x4a
(probe312:isp0:0:312:0): cam completion status 0x4a
(probe312:isp0:0:312:0): cam completion status 0x4a
(probe313:isp0:0:313:0): cam completion status 0x4a
(probe313:isp0:0:313:0): cam completion status 0x4a
(probe314:isp0:0:314:0): cam completion status 0x4a
(probe314:isp0:0:314:0): cam completion status 0x4a
(probe315:isp0:0:315:0): cam completion status 0x4a
(probe315:isp0:0:315:0): cam completion status 0x4a
(probe316:isp0:0:316:0): cam completion status 0x4a
(probe316:isp0:0:316:0): cam completion status 0x4a
(probe317:isp0:0:317:0): cam completion status 0x4a
(probe317:isp0:0:317:0): cam completion status 0x4a
(probe318:isp0:0:318:0): cam completion status 0x4a
(probe318:isp0:0:318:0): cam completion status 0x4a
(probe319:isp0:0:319:0): cam completion status 0x4a
(probe319:isp0:0:319:0): cam completion status 0x4a
(probe320:isp0:0:320:0): cam completion status 0x4a
(probe320:isp0:0:320:0): cam completion status 0x4a
(probe321:isp0:0:321:0): cam completion status 0x4a
(probe321:isp0:0:321:0): cam completion status 0x4a
(probe322:isp0:0:322:0): cam completion status 0x4a
(probe322:isp0:0:322:0): cam completion status 0x4a
(probe323:isp0:0:323:0): cam completion status 0x4a
(probe323:isp0:0:323:0): cam completion status 0x4a
(probe324:isp0:0:324:0): cam completion status 0x4a
(probe324:isp0:0:324:0): cam completion status 0x4a
(probe325:isp0:0:325:0): cam completion status 0x4a
(probe325:isp0:0:325:0): cam completion status 0x4a
(probe326:isp0:0:326:0): cam completion status 0x4a
(probe326:isp0:0:326:0): cam completion status 0x4a
(probe327:isp0:0:327:0): cam completion status 0x4a
(probe327:isp0:0:327:0): cam completion status 0x4a
(probe328:isp0:0:328:0): cam completion status 0x4a
(probe328:isp0:0:328:0): cam completion status 0x4a
(probe329:isp0:0:329:0): cam completion status 0x4a
(probe329:isp0:0:329:0): cam completion status 0x4a
(probe330:isp0:0:330:0): cam completion status 0x4a
(probe330:isp0:0:330:0): cam completion status 0x4a
(probe331:isp0:0:331:0): cam completion status 0x4a
(probe331:isp0:0:331:0): cam completion status 0x4a
(probe332:isp0:0:332:0): cam completion status 0x4a
(probe332:isp0:0:332:0): cam completion status 0x4a
(probe333:isp0:0:333:0): cam completion status 0x4a
(probe333:isp0:0:333:0): cam completion status 0x4a
(probe334:isp0:0:334:0): cam completion status 0x4a
(probe334:isp0:0:334:0): cam completion status 0x4a
(probe335:isp0:0:335:0): cam completion status 0x4a
(probe335:isp0:0:335:0): cam completion status 0x4a
(probe336:isp0:0:336:0): cam completion status 0x4a
(probe336:isp0:0:336:0): cam completion status 0x4a
(probe337:isp0:0:337:0): cam completion status 0x4a
(probe337:isp0:0:337:0): cam completion status 0x4a
(probe338:isp0:0:338:0): cam completion status 0x4a
(probe338:isp0:0:338:0): cam completion status 0x4a
(probe339:isp0:0:339:0): cam completion status 0x4a
(probe339:isp0:0:339:0): cam completion status 0x4a
(probe340:isp0:0:340:0): cam completion status 0x4a
(probe340:isp0:0:340:0): cam completion status 0x4a
(probe341:isp0:0:341:0): cam completion status 0x4a
(probe341:isp0:0:341:0): cam completion status 0x4a
(probe342:isp0:0:342:0): cam completion status 0x4a
(probe342:isp0:0:342:0): cam completion status 0x4a
(probe343:isp0:0:343:0): cam completion status 0x4a
(probe343:isp0:0:343:0): cam completion status 0x4a
(probe344:isp0:0:344:0): cam completion status 0x4a
(probe344:isp0:0:344:0): cam completion status 0x4a
(probe345:isp0:0:345:0): cam completion status 0x4a
(probe345:isp0:0:345:0): cam completion status 0x4a
(probe346:isp0:0:346:0): cam completion status 0x4a
(probe346:isp0:0:346:0): cam completion status 0x4a
(probe347:isp0:0:347:0): cam completion status 0x4a
(probe347:isp0:0:347:0): cam completion status 0x4a
(probe348:isp0:0:348:0): cam completion status 0x4a
(probe348:isp0:0:348:0): cam completion status 0x4a
(probe349:isp0:0:349:0): cam completion status 0x4a
(probe349:isp0:0:349:0): cam completion status 0x4a
(probe350:isp0:0:350:0): cam completion status 0x4a
(probe350:isp0:0:350:0): cam completion status 0x4a
(probe351:isp0:0:351:0): cam completion status 0x4a
(probe351:isp0:0:351:0): cam completion status 0x4a
(probe352:isp0:0:352:0): cam completion status 0x4a
(probe352:isp0:0:352:0): cam completion status 0x4a
(probe353:isp0:0:353:0): cam completion status 0x4a
(probe353:isp0:0:353:0): cam completion status 0x4a
(probe354:isp0:0:354:0): cam completion status 0x4a
(probe354:isp0:0:354:0): cam completion status 0x4a
(probe355:isp0:0:355:0): cam completion status 0x4a
(probe355:isp0:0:355:0): cam completion status 0x4a
(probe356:isp0:0:356:0): cam completion status 0x4a
(probe356:isp0:0:356:0): cam completion status 0x4a
(probe357:isp0:0:357:0): cam completion status 0x4a
(probe357:isp0:0:357:0): cam completion status 0x4a
(probe358:isp0:0:358:0): cam completion status 0x4a
(probe358:isp0:0:358:0): cam completion status 0x4a
(probe359:isp0:0:359:0): cam completion status 0x4a
(probe359:isp0:0:359:0): cam completion status 0x4a
(probe360:isp0:0:360:0): cam completion status 0x4a
(probe360:isp0:0:360:0): cam completion status 0x4a
(probe361:isp0:0:361:0): cam completion status 0x4a
(probe361:isp0:0:361:0): cam completion status 0x4a
(probe362:isp0:0:362:0): cam completion status 0x4a
(probe362:isp0:0:362:0): cam completion status 0x4a
(probe363:isp0:0:363:0): cam completion status 0x4a
(probe363:isp0:0:363:0): cam completion status 0x4a
(probe364:isp0:0:364:0): cam completion status 0x4a
(probe364:isp0:0:364:0): cam completion status 0x4a
(probe365:isp0:0:365:0): cam completion status 0x4a
(probe365:isp0:0:365:0): cam completion status 0x4a
(probe366:isp0:0:366:0): cam completion status 0x4a
(probe366:isp0:0:366:0): cam completion status 0x4a
(probe367:isp0:0:367:0): cam completion status 0x4a
(probe367:isp0:0:367:0): cam completion status 0x4a
(probe368:isp0:0:368:0): cam completion status 0x4a
(probe368:isp0:0:368:0): cam completion status 0x4a
(probe369:isp0:0:369:0): cam completion status 0x4a
(probe369:isp0:0:369:0): cam completion status 0x4a
(probe370:isp0:0:370:0): cam completion status 0x4a
(probe370:isp0:0:370:0): cam completion status 0x4a
(probe371:isp0:0:371:0): cam completion status 0x4a
(probe371:isp0:0:371:0): cam completion status 0x4a
(probe372:isp0:0:372:0): cam completion status 0x4a
(probe372:isp0:0:372:0): cam completion status 0x4a
(probe373:isp0:0:373:0): cam completion status 0x4a
(probe373:isp0:0:373:0): cam completion status 0x4a
(probe374:isp0:0:374:0): cam completion status 0x4a
(probe374:isp0:0:374:0): cam completion status 0x4a
(probe375:isp0:0:375:0): cam completion status 0x4a
(probe375:isp0:0:375:0): cam completion status 0x4a
(probe377:isp0:0:377:0): cam completion status 0x4a
(probe377:isp0:0:377:0): cam completion status 0x4a
(probe378:isp0:0:378:0): cam completion status 0x4a
(probe378:isp0:0:378:0): cam completion status 0x4a
(probe379:isp0:0:379:0): cam completion status 0x4a
(probe379:isp0:0:379:0): cam completion status 0x4a
(probe380:isp0:0:380:0): cam completion status 0x4a
(probe380:isp0:0:380:0): cam completion status 0x4a
(probe381:isp0:0:381:0): cam completion status 0x4a
(probe381:isp0:0:381:0): cam completion status 0x4a
(probe382:isp0:0:382:0): cam completion status 0x4a
(probe382:isp0:0:382:0): cam completion status 0x4a
(probe383:isp0:0:383:0): cam completion status 0x4a
(probe383:isp0:0:383:0): cam completion status 0x4a
(probe384:isp0:0:384:0): cam completion status 0x4a
(probe384:isp0:0:384:0): cam completion status 0x4a
(probe385:isp0:0:385:0): cam completion status 0x4a
(probe385:isp0:0:385:0): cam completion status 0x4a
(probe386:isp0:0:386:0): cam completion status 0x4a
(probe386:isp0:0:386:0): cam completion status 0x4a
(probe387:isp0:0:387:0): cam completion status 0x4a
(probe387:isp0:0:387:0): cam completion status 0x4a
(probe388:isp0:0:388:0): cam completion status 0x4a
(probe388:isp0:0:388:0): cam completion status 0x4a
(probe389:isp0:0:389:0): cam completion status 0x4a
(probe389:isp0:0:389:0): cam completion status 0x4a
(probe390:isp0:0:390:0): cam completion status 0x4a
(probe390:isp0:0:390:0): cam completion status 0x4a
(probe391:isp0:0:391:0): cam completion status 0x4a
(probe391:isp0:0:391:0): cam completion status 0x4a
(probe392:isp0:0:392:0): cam completion status 0x4a
(probe392:isp0:0:392:0): cam completion status 0x4a
(probe393:isp0:0:393:0): cam completion status 0x4a
(probe393:isp0:0:393:0): cam completion status 0x4a
(probe394:isp0:0:394:0): cam completion status 0x4a
(probe394:isp0:0:394:0): cam completion status 0x4a
(probe396:isp0:0:396:0): cam completion status 0x4a
(probe396:isp0:0:396:0): cam completion status 0x4a
(probe397:isp0:0:397:0): cam completion status 0x4a
(probe397:isp0:0:397:0): cam completion status 0x4a
(probe398:isp0:0:398:0): cam completion status 0x4a
(probe398:isp0:0:398:0): cam completion status 0x4a
(probe399:isp0:0:399:0): cam completion status 0x4a
(probe399:isp0:0:399:0): cam completion status 0x4a
(probe401:isp0:0:401:0): cam completion status 0x4a
(probe401:isp0:0:401:0): cam completion status 0x4a
(probe402:isp0:0:402:0): cam completion status 0x4a
(probe402:isp0:0:402:0): cam completion status 0x4a
(probe403:isp0:0:403:0): cam completion status 0x4a
(probe403:isp0:0:403:0): cam completion status 0x4a
(probe404:isp0:0:404:0): cam completion status 0x4a
(probe404:isp0:0:404:0): cam completion status 0x4a
(probe405:isp0:0:405:0): cam completion status 0x4a
(probe405:isp0:0:405:0): cam completion status 0x4a
(probe406:isp0:0:406:0): cam completion status 0x4a
(probe406:isp0:0:406:0): cam completion status 0x4a
(probe407:isp0:0:407:0): cam completion status 0x4a
(probe407:isp0:0:407:0): cam completion status 0x4a
(probe408:isp0:0:408:0): cam completion status 0x4a
(probe408:isp0:0:408:0): cam completion status 0x4a
(probe409:isp0:0:409:0): cam completion status 0x4a
(probe409:isp0:0:409:0): cam completion status 0x4a
(probe410:isp0:0:410:0): cam completion status 0x4a
(probe410:isp0:0:410:0): cam completion status 0x4a
(probe412:isp0:0:412:0): cam completion status 0x4a
(probe412:isp0:0:412:0): cam completion status 0x4a
(probe413:isp0:0:413:0): cam completion status 0x4a
(probe413:isp0:0:413:0): cam completion status 0x4a
(probe414:isp0:0:414:0): cam completion status 0x4a
(probe414:isp0:0:414:0): cam completion status 0x4a
(probe415:isp0:0:415:0): cam completion status 0x4a
(probe415:isp0:0:415:0): cam completion status 0x4a
(probe416:isp0:0:416:0): cam completion status 0x4a
(probe416:isp0:0:416:0): cam completion status 0x4a
(probe417:isp0:0:417:0): cam completion status 0x4a
(probe417:isp0:0:417:0): cam completion status 0x4a
(probe418:isp0:0:418:0): cam completion status 0x4a
(probe418:isp0:0:418:0): cam completion status 0x4a
(probe419:isp0:0:419:0): cam completion status 0x4a
(probe419:isp0:0:419:0): cam completion status 0x4a
(probe420:isp0:0:420:0): cam completion status 0x4a
(probe420:isp0:0:420:0): cam completion status 0x4a
(probe421:isp0:0:421:0): cam completion status 0x4a
(probe421:isp0:0:421:0): cam completion status 0x4a
(probe422:isp0:0:422:0): cam completion status 0x4a
(probe422:isp0:0:422:0): cam completion status 0x4a
(probe424:isp0:0:424:0): cam completion status 0x4a
(probe424:isp0:0:424:0): cam completion status 0x4a
(probe425:isp0:0:425:0): cam completion status 0x4a
(probe425:isp0:0:425:0): cam completion status 0x4a
(probe426:isp0:0:426:0): cam completion status 0x4a
(probe426:isp0:0:426:0): cam completion status 0x4a
(probe427:isp0:0:427:0): cam completion status 0x4a
(probe427:isp0:0:427:0): cam completion status 0x4a
(probe428:isp0:0:428:0): cam completion status 0x4a
(probe428:isp0:0:428:0): cam completion status 0x4a
(probe429:isp0:0:429:0): cam completion status 0x4a
(probe429:isp0:0:429:0): cam completion status 0x4a
(probe430:isp0:0:430:0): cam completion status 0x4a
(probe430:isp0:0:430:0): cam completion status 0x4a
(probe431:isp0:0:431:0): cam completion status 0x4a
(probe431:isp0:0:431:0): cam completion status 0x4a
(probe432:isp0:0:432:0): cam completion status 0x4a
(probe432:isp0:0:432:0): cam completion status 0x4a
(probe433:isp0:0:433:0): cam completion status 0x4a
(probe433:isp0:0:433:0): cam completion status 0x4a
(probe434:isp0:0:434:0): cam completion status 0x4a
(probe434:isp0:0:434:0): cam completion status 0x4a
(probe435:isp0:0:435:0): cam completion status 0x4a
(probe435:isp0:0:435:0): cam completion status 0x4a
(probe436:isp0:0:436:0): cam completion status 0x4a
(probe436:isp0:0:436:0): cam completion status 0x4a
(probe438:isp0:0:438:0): cam completion status 0x4a
(probe438:isp0:0:438:0): cam completion status 0x4a
(probe439:isp0:0:439:0): cam completion status 0x4a
(probe439:isp0:0:439:0): cam completion status 0x4a
(probe440:isp0:0:440:0): cam completion status 0x4a
(probe440:isp0:0:440:0): cam completion status 0x4a
(probe441:isp0:0:441:0): cam completion status 0x4a
(probe441:isp0:0:441:0): cam completion status 0x4a
(probe442:isp0:0:442:0): cam completion status 0x4a
(probe442:isp0:0:442:0): cam completion status 0x4a
(probe443:isp0:0:443:0): cam completion status 0x4a
(probe443:isp0:0:443:0): cam completion status 0x4a
(probe444:isp0:0:444:0): cam completion status 0x4a
(probe444:isp0:0:444:0): cam completion status 0x4a
(probe445:isp0:0:445:0): cam completion status 0x4a
(probe445:isp0:0:445:0): cam completion status 0x4a
(probe446:isp0:0:446:0): cam completion status 0x4a
(probe446:isp0:0:446:0): cam completion status 0x4a
(probe447:isp0:0:447:0): cam completion status 0x4a
(probe447:isp0:0:447:0): cam completion status 0x4a
(probe448:isp0:0:448:0): cam completion status 0x4a
(probe448:isp0:0:448:0): cam completion status 0x4a
(probe450:isp0:0:450:0): cam completion status 0x4a
(probe450:isp0:0:450:0): cam completion status 0x4a
(probe451:isp0:0:451:0): cam completion status 0x4a
(probe451:isp0:0:451:0): cam completion status 0x4a
(probe452:isp0:0:452:0): cam completion status 0x4a
(probe452:isp0:0:452:0): cam completion status 0x4a
(probe453:isp0:0:453:0): cam completion status 0x4a
(probe453:isp0:0:453:0): cam completion status 0x4a
(probe454:isp0:0:454:0): cam completion status 0x4a
(probe454:isp0:0:454:0): cam completion status 0x4a
(probe455:isp0:0:455:0): cam completion status 0x4a
(probe455:isp0:0:455:0): cam completion status 0x4a
(probe456:isp0:0:456:0): cam completion status 0x4a
(probe456:isp0:0:456:0): cam completion status 0x4a
(probe457:isp0:0:457:0): cam completion status 0x4a
(probe457:isp0:0:457:0): cam completion status 0x4a
(probe458:isp0:0:458:0): cam completion status 0x4a
(probe458:isp0:0:458:0): cam completion status 0x4a
(probe460:isp0:0:460:0): cam completion status 0x4a
(probe460:isp0:0:460:0): cam completion status 0x4a
(probe461:isp0:0:461:0): cam completion status 0x4a
(probe461:isp0:0:461:0): cam completion status 0x4a
(probe462:isp0:0:462:0): cam completion status 0x4a
(probe462:isp0:0:462:0): cam completion status 0x4a
(probe463:isp0:0:463:0): cam completion status 0x4a
(probe463:isp0:0:463:0): cam completion status 0x4a
(probe465:isp0:0:465:0): cam completion status 0x4a
(probe465:isp0:0:465:0): cam completion status 0x4a
(probe466:isp0:0:466:0): cam completion status 0x4a
(probe466:isp0:0:466:0): cam completion status 0x4a
(probe467:isp0:0:467:0): cam completion status 0x4a
(probe467:isp0:0:467:0): cam completion status 0x4a
(probe468:isp0:0:468:0): cam completion status 0x4a
(probe468:isp0:0:468:0): cam completion status 0x4a
(probe469:isp0:0:469:0): cam completion status 0x4a
(probe469:isp0:0:469:0): cam completion status 0x4a
(probe470:isp0:0:470:0): cam completion status 0x4a
(probe470:isp0:0:470:0): cam completion status 0x4a
(probe472:isp0:0:472:0): cam completion status 0x4a
(probe472:isp0:0:472:0): cam completion status 0x4a
(probe473:isp0:0:473:0): cam completion status 0x4a
(probe473:isp0:0:473:0): cam completion status 0x4a
(probe474:isp0:0:474:0): cam completion status 0x4a
(probe474:isp0:0:474:0): cam completion status 0x4a
(probe475:isp0:0:475:0): cam completion status 0x4a
(probe475:isp0:0:475:0): cam completion status 0x4a
(probe476:isp0:0:476:0): cam completion status 0x4a
(probe476:isp0:0:476:0): cam completion status 0x4a
(probe477:isp0:0:477:0): cam completion status 0x4a
(probe477:isp0:0:477:0): cam completion status 0x4a
(probe479:isp0:0:479:0): cam completion status 0x4a
(probe479:isp0:0:479:0): cam completion status 0x4a
(probe480:isp0:0:480:0): cam completion status 0x4a
(probe480:isp0:0:480:0): cam completion status 0x4a
(probe481:isp0:0:481:0): cam completion status 0x4a
(probe481:isp0:0:481:0): cam completion status 0x4a
(probe482:isp0:0:482:0): cam completion status 0x4a
(probe482:isp0:0:482:0): cam completion status 0x4a
(probe483:isp0:0:483:0): cam completion status 0x4a
(probe483:isp0:0:483:0): cam completion status 0x4a
(probe484:isp0:0:484:0): cam completion status 0x4a
(probe484:isp0:0:484:0): cam completion status 0x4a
(probe485:isp0:0:485:0): cam completion status 0x4a
(probe485:isp0:0:485:0): cam completion status 0x4a
(probe486:isp0:0:486:0): cam completion status 0x4a
(probe486:isp0:0:486:0): cam completion status 0x4a
(probe487:isp0:0:487:0): cam completion status 0x4a
(probe487:isp0:0:487:0): cam completion status 0x4a
(probe489:isp0:0:489:0): cam completion status 0x4a
(probe489:isp0:0:489:0): cam completion status 0x4a
(probe490:isp0:0:490:0): cam completion status 0x4a
(probe490:isp0:0:490:0): cam completion status 0x4a
(probe491:isp0:0:491:0): cam completion status 0x4a
(probe491:isp0:0:491:0): cam completion status 0x4a
(probe492:isp0:0:492:0): cam completion status 0x4a
(probe492:isp0:0:492:0): cam completion status 0x4a
(probe493:isp0:0:493:0): cam completion status 0x4a
(probe493:isp0:0:493:0): cam completion status 0x4a
(probe494:isp0:0:494:0): cam completion status 0x4a
(probe494:isp0:0:494:0): cam completion status 0x4a
(probe495:isp0:0:495:0): cam completion status 0x4a
(probe495:isp0:0:495:0): cam completion status 0x4a
(probe496:isp0:0:496:0): cam completion status 0x4a
(probe496:isp0:0:496:0): cam completion status 0x4a
(probe497:isp0:0:497:0): cam completion status 0x4a
(probe497:isp0:0:497:0): cam completion status 0x4a
(probe498:isp0:0:498:0): cam completion status 0x4a
(probe498:isp0:0:498:0): cam completion status 0x4a
(probe499:isp0:0:499:0): cam completion status 0x4a
(probe499:isp0:0:499:0): cam completion status 0x4a
(probe500:isp0:0:500:0): cam completion status 0x4a
(probe500:isp0:0:500:0): cam completion status 0x4a
(probe501:isp0:0:501:0): cam completion status 0x4a
(probe501:isp0:0:501:0): cam completion status 0x4a
(probe502:isp0:0:502:0): cam completion status 0x4a
(probe502:isp0:0:502:0): cam completion status 0x4a
(probe503:isp0:0:503:0): cam completion status 0x4a
(probe503:isp0:0:503:0): cam completion status 0x4a
(probe6:isp0:0:6:0): cam completion status 0x4a
(probe6:isp0:0:6:0): cam completion status 0x4a
(probe25:isp0:0:25:0): cam completion status 0x4a
(probe25:isp0:0:25:0): cam completion status 0x4a
(probe62:isp0:0:62:0): cam completion status 0x4a
(probe62:isp0:0:62:0): cam completion status 0x4a
(probe74:isp0:0:74:0): cam completion status 0x4a
(probe74:isp0:0:74:0): cam completion status 0x4a
(probe80:isp0:0:80:0): cam completion status 0x4a
(probe80:isp0:0:80:0): cam completion status 0x4a
(probe109:isp0:0:109:0): cam completion status 0x4a
(probe109:isp0:0:109:0): cam completion status 0x4a
(probe131:isp0:0:131:0): cam completion status 0x4a
(probe131:isp0:0:131:0): cam completion status 0x4a
(probe147:isp0:0:147:0): cam completion status 0x4a
(probe147:isp0:0:147:0): cam completion status 0x4a
(probe206:isp0:0:206:0): cam completion status 0x4a
(probe206:isp0:0:206:0): cam completion status 0x4a
da0 at isp0 bus 0 target 0 lun 0
da0: <DGC RAID 5 0326> Fixed Direct Access SCSI-4 device
da0: 100.000MB/s transfers
da0: Command Queueing Enabled
da0: 512000MB (1048576000 512 byte sectors: 255H 63S/T 65270C)
cd0 at umass-sim0 bus 0 target 0 lun 0
cd0: <Dell Virtual CDROM 123> Removable CD-ROM SCSI-0 device
cd0: 40.000MB/s transfers
cd0: Attempt to query device size failed: NOT READY, Medium not present
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #4 Launched!
WARNING: WITNESS option enabled, expect reduced performance.
WARNING: DIAGNOSTIC option enabled, expect reduced performance.
da1 at umass-sim1 bus 1 target 0 lun 0
da1: <Dell Virtual Floppy 123> Removable Direct Access SCSI-0 device
da1: 40.000MB/s transfers
da1: Attempt to query device size failed: NOT READY, Medium not present
Trying to mount root from ufs:/dev/mfid0s1a
bce0: link state changed to UP
mfi0: 3320 (277345023s/0x0008/info) - Battery charge complete
---------------------------------------------------------------
4. panic data and ddb output, also available at
http://noc.ntua.gr/~christia/san/7-STABLE.deadlock-ddb.out :
KKDDpanic: ufs_dirbad: /a: bad dir ino 3370734 at ofBBfset 0: mangled entry
cpuid = 1
KDB: enter: panic
[thread pid 768 tid 100104 ]
Stopped at kdb_enter_why+0x3a: movl $0,kdb_why
db> trace
Tracing pid 768 tid 100104 td 0xc74c6690
kdb_enter_why(c0b3b0a1,c0b3b0a1,c0b5d0ec,ed2dd85c,1,...) at kdb_enter_why+0x3a
panic(c0b5d0ec,c70365c0,336eee,0,c0b5d164,...) at panic+0x136
ufs_dirbad(c9c40180,0,c0b5d164,0,ed2dd904,...) at ufs_dirbad+0x49
ufs_lookup(ed2dd94c,c0b6f08d,cb090160,cb090160,ed2ddba4,...) at ufs_lookup+0x377
VOP_CACHEDLOOKUP_APV(c0c0c1a0,ed2dd94c,ed2ddba4,ed2ddb90,c75dae00,...) at
VOP_CACHEDLOOKUP_APV+0xc5
vfs_cache_lookup(ed2dd9d0,c0b471e4,cb090160,2,cb090160,...) at
vfs_cache_lookup+0xd3
VOP_LOOKUP_APV(c0c0c1a0,ed2dd9d0,c74c6690,c0b46172,19b,...) at
VOP_LOOKUP_APV+0xe5
lookup(ed2ddb7c,c0b46172,c6,bf,c703a72c,...) at lookup+0x59e
namei(ed2ddb7c,c74c6690,ed2dda74,246,c0c61ab8,...) at namei+0x35e
vn_open_cred(ed2ddb7c,ed2ddc78,1a4,c75dae00,c74871c8,...) at vn_open_cred+0x62
vn_open(ed2ddb7c,ed2ddc78,1a4,c74871c8,0,...) at vn_open+0x33
kern_open(c74c6690,8164100,0,a02,1a4,...) at kern_open+0xe7
open(c74c6690,ed2ddcfc,c,c0b40d50,c0beb278,...) at open+0x30
syscall(ed2ddd38) at syscall+0x2d3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (5, FreeBSD ELF32, open), eip = 0x281a161b, esp = 0xbfbfe8cc, ebp =
0xbfbfe8f8 ---
db> ps
pid ppid pgrp uid state wmesg wchan cmd
768 728 768 0 R+ CPU 1 bsdtar
766 728 766 0 R+ CPU 4 bsdtar
728 724 728 0 Ss+ ttyin 0xc68ce810 csh
724 645 724 0 Ss select 0xc0ca821c sshd
720 711 720 0 S+ ttyin 0xc69a7810 csh
719 1 719 0 Ss+ ttyin 0xc69db410 getty
718 1 718 0 Ss+ ttyin 0xc69dc810 getty
717 1 717 0 Ss+ ttyin 0xc69de410 getty
716 1 716 0 Ss+ ttyin 0xc69d3810 getty
715 1 715 0 Ss+ ttyin 0xc69d5810 getty
714 1 714 0 Ss+ ttyin 0xc69d5c10 getty
713 1 713 0 Ss+ ttyin 0xc69dc010 getty
712 1 712 0 Ss+ ttyin 0xc69db010 getty
711 1 711 0 Ss+ wait 0xc6feb2b8 login
672 1 672 0 Ss nanslp 0xc0c56844 cron
666 1 666 25 Ss pause 0xc7049b40 sendmail
660 1 660 0 Ss select 0xc0ca821c sendmail
645 1 645 0 Ss select 0xc0ca821c sshd
527 1 527 0 Ss select 0xc0ca821c syslogd
473 1 473 0 Ss select 0xc0ca821c devd
143 1 143 0 Ss pause 0xc704ab40 adjkerntz
56 0 0 0 SL sdflush 0xc0cb4668 [softdepflush]
55 0 0 0 SL syncer 0xc0c5666c [syncer]
54 0 0 0 SL vlruwt 0xc6fec000 [vnlru]
53 0 0 0 RL CPU 5 [bufdaemon]
52 0 0 0 SL pgzero 0xc0cb5220 [pagezero]
51 0 0 0 SL psleep 0xc0cb4e38 [vmdaemon]
50 0 0 0 SL psleep 0xc0cb4e00 [pagedaemon]
49 0 0 0 SL waiting_ 0xc0caa434 [sctp_iterator]
48 0 0 0 WL [irq1: atkbd0]
47 0 0 0 WL [swi0: sio]
46 0 0 0 WL [irq15: ata1]
45 0 0 0 WL [irq14: ata0]
44 0 0 0 SL usbevt 0xc68a8210 [usb4]
43 0 0 0 SL usbevt 0xc6994210 [usb3]
42 0 0 0 SL usbevt 0xc6977210 [usb2]
41 0 0 0 SL usbevt 0xc697c210 [usb1]
40 0 0 0 WL [irq20: uhci1 uhci3]
39 0 0 0 SL usbtsk 0xc0c53e14 [usbtask-dr]
38 0 0 0 SL usbtsk 0xc0c53e00 [usbtask-hc]
37 0 0 0 SL usbevt 0xc692d210 [usb0]
36 0 0 0 WL [irq21: uhci0 uhci+]
35 0 0 0 WL [irq259: bce1]
34 0 0 0 SL ispf 0xc67e14d4 [isp1: fc_thrd]
33 0 0 0 WL [irq258: isp1]
32 0 0 0 SL ispf 0xc67e06d4 [isp0: fc_thrd]
31 0 0 0 RL [irq257: isp0]
30 0 0 0 WL [irq16: mfi0]
29 0 0 0 WL [irq256: bce0]
28 0 0 0 WL [irq9: acpi0]
27 0 0 0 SL - 0xc685e900 [thread taskq]
26 0 0 0 WL [swi5: +]
9 0 0 0 SL - 0xc685eb00 [kqueue taskq]
8 0 0 0 SL - 0xc685eb80 [acpi_task_2]
7 0 0 0 SL - 0xc685eb80 [acpi_task_1]
6 0 0 0 SL - 0xc685eb80 [acpi_task_0]
25 0 0 0 WL [swi2: cambio]
5 0 0 0 SL ccb_scan 0xc0c24394 [xpt_thrd]
24 0 0 0 WL [swi6: task queue]
23 0 0 0 WL [swi6: Giant taskq]
22 0 0 0 SL - 0xc0c56674 [yarrow]
4 0 0 0 SL - 0xc0c5420c [g_down]
3 0 0 0 RL CPU 3 [g_up]
2 0 0 0 SL - 0xc0c54200 [g_event]
21 0 0 0 WL [swi3: vm]
20 0 0 0 WL [swi4: clock sio]
19 0 0 0 WL [swi1: net]
18 0 0 0 RL CPU 0 [idle: cpu0]
17 0 0 0 RL [idle: cpu1]
16 0 0 0 RL CPU 2 [idle: cpu2]
15 0 0 0 RL [idle: cpu3]
14 0 0 0 RL [idle: cpu4]
13 0 0 0 RL [idle: cpu5]
12 0 0 0 RL CPU 6 [idle: cpu6]
11 0 0 0 RL CPU 7 [idle: cpu7]
1 0 1 0 SLs wait 0xc6793ae0 [init]
10 0 0 0 SL audit_wo 0xc0cb40a4 [audit]
0 0 0 0 SLs sched 0xc0c542c0 [swapper]
db> show pcpu
cpuid = 1
curthread = 0xc74c6690: pid 768 "bsdtar"
curpcb = 0xed2ddd90
fpcurthread = none
idlethread = 0xc67a7460: pid 17 "idle: cpu1"
APIC ID = 1
currentldt = 0x50
spin locks held:
db> show allpcpu
Current CPU: 1
cpuid = 0
curthread = 0xc67a7230: pid 18 "idle: cpu0"
curpcb = 0xe5424d90
fpcurthread = none
idlethread = 0xc67a7230: pid 18 "idle: cpu0"
APIC ID = 0
currentldt = 0x50
spin locks held:
cpuid = 1
curthread = 0xc74c6690: pid 768 "bsdtar"
curpcb = 0xed2ddd90
fpcurthread = none
idlethread = 0xc67a7460: pid 17 "idle: cpu1"
APIC ID = 1
currentldt = 0x50
spin locks held:
cpuid = 2
curthread = 0xc6795000: pid 16 "idle: cpu2"
curpcb = 0xe541ed90
fpcurthread = none
idlethread = 0xc6795000: pid 16 "idle: cpu2"
APIC ID = 2
currentldt = 0x50
spin locks held:
cpuid = 3
curthread = 0xc6797690: pid 3 "g_up"
curpcb = 0xe6e36d90
fpcurthread = none
idlethread = 0xc6795230: pid 15 "idle: cpu3"
APIC ID = 3
currentldt = 0x50
spin locks held:
cpuid = 4
curthread = 0xc75c3690: pid 766 "bsdtar"
curpcb = 0xed348d90
fpcurthread = none
idlethread = 0xc6795460: pid 14 "idle: cpu4"
APIC ID = 4
currentldt = 0x50
spin locks held:
cpuid = 5
curthread = 0xc6fea8c0: pid 53 "bufdaemon"
curpcb = 0xeafead90
fpcurthread = none
idlethread = 0xc6795690: pid 13 "idle: cpu5"
APIC ID = 5
currentldt = 0x50
spin locks held:
cpuid = 6
curthread = 0xc67958c0: pid 12 "idle: cpu6"
curpcb = 0xe5412d90
fpcurthread = none
idlethread = 0xc67958c0: pid 12 "idle: cpu6"
APIC ID = 6
currentldt = 0x50
spin locks held:
cpuid = 7
curthread = 0xc6795af0: pid 11 "idle: cpu7"
curpcb = 0xe540fd90
fpcurthread = none
idlethread = 0xc6795af0: pid 11 "idle: cpu7"
APIC ID = 7
currentldt = 0x50
spin locks held:
db> show locks
db> show alllocks
Process 4 (g_down) thread 0xc6797460 (100015)
exclusive sleep mutex isp r = 0 (0xc67e0690) locked @
/usr/src/sys/cam/cam_periph.h:182
db> show lockedvnods
Locked vnodes
0xcb090160: tag ufs, type VDIR
usecount 1, writecount 0, refcount 3 mountedhere 0
flags ()
v_object 0xcaa70c1c ref 0 pages 1
lock type ufs: EXCL (count 1) by thread 0xc74c6690 (pid 768)#0 0xc077f6c0
at _lockmgr+0x560
#1 0xc099cb51 at ffs_lock+0x91
#2 0xc0aafcc5 at VOP_LOCK1_APV+0xa5
#3 0xc081f7b8 at _vn_lock+0xf8
#4 0xc0813c20 at vget+0x110
#5 0xc08029b7 at cache_lookup+0x4e7
#6 0xc0802aea at vfs_cache_lookup+0xaa
#7 0xc0ab0e65 at VOP_LOOKUP_APV+0xe5
#8 0xc0808d2e at lookup+0x59e
#9 0xc08098de at namei+0x35e
#10 0xc081f022 at vn_open_cred+0x62
#11 0xc081f513 at vn_open+0x33
#12 0xc081e177 at kern_open+0xe7
#13 0xc081e660 at open+0x30
#14 0xc0a98853 at syscall+0x2d3
#15 0xc0a7ea80 at Xint0x80_syscall+0x20
ino 3370734, on dev da0s1d
0xcb5906e0: tag ufs, type VDIR
usecount 0, writecount 0, refcount 2 mountedhere 0
flags ()
v_object 0xc9b65a2c ref 0 pages 0
lock type ufs: EXCL (count 1) by thread 0xc6fea8c0 (pid 53)#0 0xc077f6c0 at
_lockmgr+0x560
#1 0xc099cb51 at ffs_lock+0x91
#2 0xc0aafcc5 at VOP_LOCK1_APV+0xa5
#3 0xc081f7b8 at _vn_lock+0xf8
#4 0xc07ff5d8 at flushbufqueues+0x568
#5 0xc07ff8f0 at buf_daemon+0xc0
#6 0xc0770ea8 at fork_exit+0xb8
#7 0xc0a7ea90 at fork_trampoline+0x8
ino 24168915, on dev da0s1d
db> trace 768
Tracing pid 768 tid 100104 td 0xc74c6690
kdb_enter_why(c0b3b0a1,c0b3b0a1,c0b5d0ec,ed2dd85c,1,...) at kdb_enter_why+0x3a
panic(c0b5d0ec,c70365c0,336eee,0,c0b5d164,...) at panic+0x136
ufs_dirbad(c9c40180,0,c0b5d164,0,ed2dd904,...) at ufs_dirbad+0x49
ufs_lookup(ed2dd94c,c0b6f08d,cb090160,cb090160,ed2ddba4,...) at ufs_lookup+0x377
VOP_CACHEDLOOKUP_APV(c0c0c1a0,ed2dd94c,ed2ddba4,ed2ddb90,c75dae00,...) at
VOP_CACHEDLOOKUP_APV+0xc5
vfs_cache_lookup(ed2dd9d0,c0b471e4,cb090160,2,cb090160,...) at
vfs_cache_lookup+0xd3
VOP_LOOKUP_APV(c0c0c1a0,ed2dd9d0,c74c6690,c0b46172,19b,...) at
VOP_LOOKUP_APV+0xe5
lookup(ed2ddb7c,c0b46172,c6,bf,c703a72c,...) at lookup+0x59e
namei(ed2ddb7c,c74c6690,ed2dda74,246,c0c61ab8,...) at namei+0x35e
vn_open_cred(ed2ddb7c,ed2ddc78,1a4,c75dae00,c74871c8,...) at vn_open_cred+0x62
vn_open(ed2ddb7c,ed2ddc78,1a4,c74871c8,0,...) at vn_open+0x33
kern_open(c74c6690,8164100,0,a02,1a4,...) at kern_open+0xe7
open(c74c6690,ed2ddcfc,c,c0b40d50,c0beb278,...) at open+0x30
syscall(ed2ddd38) at syscall+0x2d3
Xint0x80_syscall() at Xint0x80_syscall+0x20
--- syscall (5, FreeBSD ELF32, open), eip = 0x281a161b, esp = 0xbfbfe8cc, ebp =
0xbfbfe8f8 ---
db> trace 55
Tracing pid 55 tid 100054 td 0xc6fea460
sched_switch(c6fea460,0,1,17c,29bc06da,...) at sched_switch+0x49b
mi_switch(1,0,c0b3ec35,1cb,c6fea460,...) at mi_switch+0x223
sleepq_switch(c6fea460,0,c0b3ec35,243,68,...) at sleepq_switch+0x182
sleepq_wait(c0c5666c,0,c0b3ba36,d8,0,...) at sleepq_wait+0x60
_sleep(c0c5666c,c0ca8988,68,c0b47461,0,...) at _sleep+0x369
sched_sync(0,eaff0d38,c0b37584,31c,c6febae0,...) at sched_sync+0x752
fork_exit(c0814220,0,eaff0d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xeaff0d70, ebp = 0 ---
db> trace 4
Tracing pid 4 tid 100015 td 0xc6797460
sched_switch(c6797460,0,1,17c,5512f4a0,...) at sched_switch+0x49b
mi_switch(1,0,c0b3ec35,1cb,4c,...) at mi_switch+0x223
sleepq_switch(c6797460,0,c0b3ec35,266,4c,...) at sleepq_switch+0x182
sleepq_timedwait(c0c5420c,64,c0b32247,0,0,...) at sleepq_timedwait+0x68
_sleep(c0c5420c,c0c54128,24c,c0b32247,64,...) at _sleep+0x345
g_io_schedule_down(c6797460,0,c0b339ab,74,0,...) at g_io_schedule_down+0x6b
g_down_procbody(0,e6e39d38,c0b37584,31c,c67eb828,...) at g_down_procbody+0x8d
fork_exit(c073d540,0,e6e39d38) at fork_exit+0xb8
fork_trampoline() at fork_trampoline+0x8
--- trap 0, eip = 0, esp = 0xe6e39d70, ebp = 0 ---
db> continue
::Uptime: 7m21s
ssttPhysical memory: 3310 MB
Dumping 225 MB:aacckk bbaacckkttrraaccee::
ddbb__ttrraaccee__sseellff__wwrrappppeerr((cc00bb33ee003388,,eeadf3e4a8ab400c,,cc00881100333333,,00,,eeadf3e4a8ab400c,,......))
aatt ddbb__ttrraaccee__sseellff__wwrraappppeerr++00xx2266
kkddbb__bbaacckkttrraaccee((00,,ee
4a8ab400c,,cc00aaaacc99bb55,,eeadf3e4a8ab510c,,
c
b4623806dec00,,......)) aatt kkddbb__bbaacckkttrraaccee++00xx2299
vv
f
bbaaddlloocckk((c00cc00cc11aa00,,eeadf3e4a8ab510c,,cc00cc223388aa00,,ccbb4623806dc00,,00))
aatt vvffss__bbaaddlloocckk++00xx2233
aasssseerrtt__vvoopp__lloocckkeedd((ccbb4623806dec00,,cc
) aatt
aasssseerrtt__vvoopp__lloocckkeedd++00xx5500
vvoopp__lloocckk__ppoosstt((eeda3f4e8aba7a8c,,00,,11020,21,21,0c0b242c8b66e300,d.c.0.,).
.a.t) vaotp _vloopc_kl_opcoks_tp+o0sxt3+60
x
P
O
_VLOOPC_KL1O_CAKP1V_(AcP0Vc(0cc01ca00c,1eaa0f,eeada3a48,bc708b,4c709bb427,93b222,,3c2b24,2c8b76b340,e.9.4.,).
.a.t) VaOtP _VLOOPC_KL1O_CAKP1V_x0PxVc+10
nc_1v
__lvonc_kl(occbk4(2c8b66e300,d1c20,,c160f0e2a,
c
05,cc306b9404,7cc04b,487a93b,2.,.2.4)2 ,a.t. ._)v na_tl o_
+
lxofc8k
h0fxlfu8s
0bvunf_qwureiutees((cc760f5e2c62d48,,e0d,3c40bc46407,cc47,58d0abe,0604,,0.,.c.7)5
ca3t6 9f0l,u.s.h.b)u faqtu uvens_+w0rxi5t6e8+
dxb1u6fa_
adeomfoinl(e0w,reiatfee(aedd3384,8cc06b03,7f5f8f4f,f3f1fcf,,cf6fffefcf2bf8f,,.0.,.c)7
0a5t2 6bdu4f,_.d.a.e)m oant+ 0dxocf0
i
efworrikt_ee+x0ixfr843_0w,r0i,teeavf(eca7d53c83)6 9a0t,
4f,oerdk3_4e8xci6t0+,08x1b281
o0f0o,r1k9_at,r.a.m.p)o laitn ek(e)r na_tw rfiotrekv_+t0rxa5m8p
5lwirniet+e0(xc87
4c-3-6-9 0t,readp3 408,c feci,pc ,=c 00b,4 0edscp5 ,=c 00bxeeba2f6e0a,d.7.0.,)
eabtp w=r i0t e-+-0-x
OfV
tPs_yLsOcCaKl:l (0exdc3b4482d8368e)0 aits snyostc allolc+k0exd2 db3u
y Xsihnotu0lxd8 0b_es
sKc
+ aBl:l (e)n taetr :X ilnotc0kx 8v0i_oslyastciaolnl
[thread pid 53 tid 100052 ]
Stopped at kdb_enter_why+0x3a: movl $0,kdb_why
db> <Enter>
xD2B0:
s-t-a-c ks ybsaccakltlr a(c4:,
Fdrbe_etBrSaDc eE_LsFe3l2f,_ wwrraiptpee)r,( c0ibp3 e=0 308x,2e8a1fbe1a793e3c,,
ce0s8p1 0=3 303x,b0f,befaef9e2ac9,e ce,b.p. .=) 0axtb fdbbf_etlr7a8ce-_-s-e
6fV_OwPr_aLpOpCeKr:+ 00xx2c6b
,3k0ddbc_0baicsk tnroatc el(o0c,keeadf ebau9te cs,hco0ualadc 9bbe5
deKaDfBe:a 9efnct,ecrb:4 2l8o6cek ,v.i.o.l)a taito nk
[thread pid 766 tid 100137 ]
Stopped at kdb_enter_why+0x3a: movl $0,kdb_why
db> <Enter>
r_DbBa:c ksttraaccke +b0axc2k9t
bavcfes:_
addbl_otcrka(cce0_cs0ecl1f
c
_w,reaapfpeear9(fcc0,bc30ec023388,ae0d,3c4b84a2d8c6,ec00,801)0 3a3t3
,vcf7s5_cb3a6d9l0o,cekd+304x82a3d
,,a.s.s.e)r ta_tv odpb__ltorcakceed_(scebl4f2_8w6rea0p,pce0rb+60fx22465
cck0dbb3_9bca4c9k,tdraa9c3e3(0c1705,c43,6.9.0.,)e da3t4
8aasdsce,rct0_avaocp9_bl5o,cekde3d4+80axe5c0,
bV6O3P0_dScT0R,A.T.E.G)Y _aAtP Vk(dcb0_c
0
, c1kat0r,aecaef+e0axa239c
cvbf4
) 2_b7abd4l,occ0kc(2c30cca00c,1cab04,2e8d63e408,a.e.c.,)c 0act2
3V8OaP0_,ScTbR6A3T0EdGcY0_,AcP7V5+c03x699b0
bautf svtfrsa_tbeag
, yl(occbk4+20x72e3c
daas9s3e3r0t1_0v
,
c 0_0e,lco0cckae8d6(3c0b,6d3a09d3c300,7c00,b.6.f.4)8 0a,te
db3u4f8sbt2rca,ce0g8y1+20dx6601,
e0bcu0fcw1rai0t,e.(.d.a)9 3a3t0 1a0s,seearfte_avao9p0_,ecl0oacakfecde+10,xe5a3f
aV
OaPc_,W0R,I.T.E._)A PaVt(
c 0ucf0wcr1iat0e,+e0dx314981b
4v,fcs7_5bci3o6_9a0w,rci0tbe4(7d9ab923,320420,,.0.,.c)0 ba4t4
7VcO4P,_8WaR4I,T8Ea3A,P.V.+.0)x 9a8t
3vvfns__wbriiot_ea(wcr7i0t5e2+60dx43,2e8d
48lcu6s0h,bcu7f5qduaeeu0e0s,(0c,6cf7e5cc23b689,00,,.
.
4a7tc 4v,n8_0wbr,
t
6t4e,+.0.x.2)3 1a
f dfolfuislhebwurfiqtuee(ueeds3+408xc5690c,
fbfufff_fd
f
b fonf(f0f,fefaff,e0a,dc3780,5c206bd347,5.8.4.,)3 1act,
cd6offeicl2ebw8r,i.t.e.+)0 xa9t5
5ukfe_rdna_ewmroint+e0vx(c c07
02cf3o6r9k0_,e4x,ietd(3c4087cf6f08,3801,201,4e0a0f,e1a9da3,8.). .a)t afto
rkke_renx_iwtr+i0txebv8+
kxf5o8r
0_wtrriatmep(ocl7i5nce3(6)9 0a,te df3o4r8kc_ftcr,acm,pco0lbi4n0ed+c05x,8c
=b-e-b-2 6t0r,a.p. .0), aeti pw r=i t0e,+ 0exs4pf
c s0yxsecaaflela(d1e7d03,04 8edb3p8 )= a0t -s-y-s
EaVlOlP+_0SxT2RdA3T
0GXYi:n t00xxc8b04_2s8y6sec0a lils( )n oatt lXoicnkte0dx 8b0u_ts ysshcoaullld+
0bxe2
K
LD-B-:- esnytsecra:l ll o(c4k, vFiroeleaBtSiDo nE
[thread pid 53 tid 100052 ]
Stopped at kdb_enter_why+0x3a: movl $0,kdb_why
db> panic
[no answer... let's power cycle the server once more]
--
Panagiotis J. Christias Network Management Center
P.Christias@noc.ntua.gr National Technical Univ. of Athens, GREECE
Hi! On Wed, 2008-10-15 at 01:23 +0300, Panagiotis Christias wrote:> However, when we connect them to the CX3-40, create and mount a new > partition and then do something as simple as "tar -C /san -xf ports.tgz" > the system panics and deadlocks. We have tried several FreeBSD versions > (6.3 i386/adm64, 7.0 i386/adm64, 7.1 i386/adm64 and lastly 7-STABLE i386 > - we also tried the latest 8-CURRENT snapshot but it panicked too soon). > The result is always the same; panic and deadlock.Try reducing the number of "tagged openings" with 'camcontrol tags' down to 46. If it doesn't work try reducing it further to 2. Also be advised that I've seen panics with geom_multipath in FreeBSD-7, unfortunately I had no time to test it in -current. -- Oleg Sharoiko. Software and Network Engineer Computer Center of Rostov State University.