Hi,
While doing an rsync from a zfs filesystem to an external usb hd (also
zfs), the rsync processes hung in zfs state. I could not kill these
processes, although the rest of the server seemingly continued to run
fine. The reboot command did not work. Next I tried a shutdown now
command. This caused a panic:
...
Stopping dhcpd.
Shutting down local daemons:.
Stopping named.
Waiting for PIDS: 106830 second watchdog timeout expired. Shutdown
terminated.
Tue May 13 21:02:50 CEST 2008
May 13 21:02:50 <auth.alert> mars init: /bin/sh on /etc/rc.shutdown
terminated abnormally, going to single user mode
panic: vrele: negative ref cnt
cpuid = 6
Uptime: 8d3h8m21s
Physical memory: 8178 MB
Dumping 3679 MB: 3664 3648 3632 3616 3600 3584 3568 3552 3536 3520
3504 3488 3472 3456 3440 3424 3408 3392 3376 3360 3344 3328 3312 3296
3280 3264 3248
atl
trap 12: page fault while in kernel mode
cpuid = 2; apic id = 02
fau
=Fta tvailr tduoaulb laed dfraeuslst
= r0ixp8
v f0axuflftf fcfofdfef 8 0=6 esbuap9e3r
girssopr =r e0axdf fifnfsftfrfufcftbi5o4n8,f fp0a
er bnpo t= p0rxe1s0e0n0t
xipnusitdr u=c t0i;o na ppioci nitde r= =0 00
8:0x8
stack pointer = 0x10:0xfffffffff8c50a50
frame pointer = 0x10:0xfffffffffb54c450
code segment = base 0x0, limit 0xfffff, type 0x1b
= DPL 0, pres 1, long 1, def32 0, gran 1
processor eflags = interrupt enabled, resume, IOPL = 0
current process = 44 (irq19: uhci1+)
trap number = 12
Yes, it printed all that gibberish. ("double fault" I think it says?)
At this point, the server completely hung, and I could not do a bt
unfortunately. I had to reset the server. After reboot, savecore did
not find a kernel dump. I ran the rsync again and could not reproduce
the problem. Although I've had zfs related problems before, this one
was new to me. Any idea what happened here?
Other info on this machine follows below.
Thanks,
Ben
[mars:~]133: uname -a
FreeBSD mars.altus-escon.com 7.0-STABLE FreeBSD 7.0-STABLE #2: Mon Apr
21 08:45:56 CEST 2008 root@mars.altus-escon.com:/usr/obj/usr/src/
sys/MARS amd64
[mars:~]134: cat /boot/loader.conf
console="comconsole"
zfs_load="YES"
vfs.root.mountfrom="zfs:tank"
vfs.zfs.prefetch_disable=1
vm.kmem_size="1536M"
vm.kmem_size_max="1536M"
vfs.zfs.arc_max="768M"
[mars:~]135: dmesg
Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 7.0-STABLE #2: Mon Apr 21 08:45:56 CEST 2008
root@mars.altus-escon.com:/usr/obj/usr/src/sys/MARS
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Xeon(R) CPU E5335 @ 2.00GHz (2000.08-MHz K8-
class CPU)
Origin = "GenuineIntel" Id = 0x6f7 Stepping = 7
Features
=
0xbfebfbff
<
FPU
,VME
,DE
,PSE
,TSC
,MSR
,PAE
,MCE
,CX8
,APIC
,SEP
,MTRR
,PGE
,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Features2
=0x4e33d<SSE3,RSVD2,MON,DS_CPL,VMX,TM2,SSSE3,CX16,xTPR,PDCM,DCA>
AMD Features=0x20100800<SYSCALL,NX,LM>
AMD Features2=0x1<LAHF>
Cores per package: 4
usable memory = 8575598592 (8178 MB)
avail memory = 8285937664 (7902 MB)
ACPI APIC Table: <PTLTD APIC >
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
cpu0 (BSP): APIC ID: 0
cpu1 (AP): APIC ID: 1
cpu2 (AP): APIC ID: 2
cpu3 (AP): APIC ID: 3
cpu4 (AP): APIC ID: 4
cpu5 (AP): APIC ID: 5
cpu6 (AP): APIC ID: 6
cpu7 (AP): APIC ID: 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
kbd1 at kbdmux0
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413,
RF5413)
acpi0: <PTLTD RSDT> on motherboard
acpi0: [ITHREAD]
acpi0: Power Button (fixed)
Timecounter "ACPI-fast" frequency 3579545 Hz quality 1000
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1008-0x100b on acpi0
cpu0: <ACPI CPU> on acpi0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
cpu1: <ACPI CPU> on acpi0
p4tcc1: <CPU Frequency Thermal Control> on cpu1
cpu2: <ACPI CPU> on acpi0
p4tcc2: <CPU Frequency Thermal Control> on cpu2
cpu3: <ACPI CPU> on acpi0
p4tcc3: <CPU Frequency Thermal Control> on cpu3
cpu4: <ACPI CPU> on acpi0
p4tcc4: <CPU Frequency Thermal Control> on cpu4
cpu5: <ACPI CPU> on acpi0
p4tcc5: <CPU Frequency Thermal Control> on cpu5
cpu6: <ACPI CPU> on acpi0
p4tcc6: <CPU Frequency Thermal Control> on cpu6
cpu7: <ACPI CPU> on acpi0
p4tcc7: <CPU Frequency Thermal Control> on cpu7
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 2.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci1
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 0.0 on pci3
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 18 at device 2.0 on pci2
pci5: <ACPI PCI bus> on pcib5
em0: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0x2000-0x201f mem 0xd8000000-0xd801ffff irq 18 at device 0.0 on pci5
em0: Using MSI interrupt
em0: Ethernet address: 00:30:48:7b:88:7e
em0: [FILTER]
em1: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0x2020-0x203f mem 0xd8020000-0xd803ffff irq 19 at device 0.1 on pci5
em1: Using MSI interrupt
em1: Ethernet address: 00:30:48:7b:88:7f
em1: [FILTER]
pcib6: <ACPI PCI-PCI bridge> at device 0.3 on pci1
pci6: <ACPI PCI bus> on pcib6
puc0: <NetMos NM9835 Dual UART and 1284 Printer port> port
0x3030-0x3037,0x3028-0x302f,0x3020-0x3027,0x3018-0x301f,
0x3010-0x3017,0x3000-0x300f irq 24 at device 1.0 on pci6
puc0: [FILTER]
uart0: <Non-standard ns8250 class UART with FIFOs> on puc0
uart0: [FILTER]
uart1: <Non-standard ns8250 class UART with FIFOs> on puc0
uart1: [FILTER]
ppc0: <Parallel port> on puc0
ppc0: Generic chipset (EPP/NIBBLE) in COMPATIBLE mode
ppbus0: <Parallel port bus> on ppc0
plip0: cannot reserve interrupt, failed.
lpt0: <Printer> on ppbus0
lpt0: Polled port
ppi0: <Parallel I/O> on ppbus0
pcib7: <ACPI PCI-PCI bridge> at device 4.0 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> at device 6.0 on pci0
pci8: <ACPI PCI bus> on pcib8
pci0: <base peripheral> at device 8.0 (no driver attached)
uhci0: <Intel 631XESB/632XESB/3100 USB controller USB-1> port
0x1800-0x181f irq 17 at device 29.0 on pci0
uhci0: [GIANT-LOCKED]
uhci0: [ITHREAD]
usb0: <Intel 631XESB/632XESB/3100 USB controller USB-1> on uhci0
usb0: USB revision 1.0
uhub0: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb0
uhub0: 2 ports with 2 removable, self powered
uhci1: <Intel 631XESB/632XESB/3100 USB controller USB-2> port
0x1820-0x183f irq 19 at device 29.1 on pci0
uhci1: [GIANT-LOCKED]
uhci1: [ITHREAD]
usb1: <Intel 631XESB/632XESB/3100 USB controller USB-2> on uhci1
usb1: USB revision 1.0
uhub1: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb1
uhub1: 2 ports with 2 removable, self powered
uhci2: <Intel 631XESB/632XESB/3100 USB controller USB-3> port
0x1840-0x185f irq 18 at device 29.2 on pci0
uhci2: [GIANT-LOCKED]
uhci2: [ITHREAD]
usb2: <Intel 631XESB/632XESB/3100 USB controller USB-3> on uhci2
usb2: USB revision 1.0
uhub2: <Intel UHCI root hub, class 9/0, rev 1.00/1.00, addr 1> on usb2
uhub2: 2 ports with 2 removable, self powered
ehci0: <Intel 63XXESB USB 2.0 controller> mem 0xd8500000-0xd85003ff
irq 17 at device 29.7 on pci0
ehci0: [GIANT-LOCKED]
ehci0: [ITHREAD]
usb3: EHCI version 1.0
usb3: companion controllers, 2 ports each: usb0 usb1 usb2
usb3: <Intel 63XXESB USB 2.0 controller> on ehci0
usb3: USB revision 2.0
uhub3: <Intel EHCI root hub, class 9/0, rev 2.00/1.00, addr 1> on usb3
uhub3: 6 ports with 6 removable, self powered
umass0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2>
on uhub3
ums0: <Peppercon AG Multidevice, class 0/0, rev 2.00/0.01, addr 2> on
uhub3
ums0: X report 0x0022 not supported
device_attach: ums0 attach returned 6
pcib9: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci9: <ACPI PCI bus> on pcib9
vgapci0: <VGA-compatible display> port 0x4000-0x40ff mem
0xd0000000-0xd7ffffff,0xd8220000-0xd822ffff irq 18 at device 1.0 on pci9
em2: <Intel(R) PRO/1000 Network Connection Version - 6.7.3> port
0x4400-0x443f mem 0xd8200000-0xd821ffff irq 16 at device 2.0 on pci9
em2: Ethernet address: 00:30:48:44:1a:ab
em2: [FILTER]
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 63XXESB2 UDMA100 controller> port
0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0x1860-0x186f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata0: [ITHREAD]
ata1: <ATA channel 1> on atapci0
ata1: [ITHREAD]
atapci1: <Intel 63XXESB2 SATA300 controller> port
0x1890-0x1897,0x1884-0x1887,0x1888-0x188f,0x1880-0x1883,0x1870-0x187f
mem 0xd8500400-0xd85007ff irq 19 at device 31.2 on pci0
atapci1: [ITHREAD]
ata2: <ATA channel 0> on atapci1
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci1
ata3: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
acpi_button0: <Power Button> on acpi0
atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: configured irq 4 not in bitmap of probed irqs 0
sio0: port may not be enabled
sio0: <16550A-compatible COM port> port 0x3f8-0x3ff irq 4 on acpi0
sio0: type 16550A
sio0: [FILTER]
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
sio1: <16550A-compatible COM port> port 0x2f8-0x2ff irq 3 flags 0x10
on acpi0
sio1: type 16550A, console
sio1: [FILTER]
fdc0: <floppy drive controller> port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on
acpi0
fdc0: [FILTER]
fd0: <1440-KB 3.5" drive> on fdc0 drive 0
orm0: <ISA Option ROM> at iomem 0xc0000-0xcafff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x100>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on
isa0
WARNING: ZFS is considered to be an experimental feature in FreeBSD.
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat loadable, rule-based
forwarding disabled, default to deny, logging limited to 50 packets/
entry by default
ZFS filesystem version 6
ZFS storage pool version 6
acd0: DMA limited to UDMA33, controller found non-ATA66 cable
acd0: DVDROM <MATSHITADVD-ROM SR-8178/PZ16> at ata0-slave UDMA33
ad4: 143089MB <WDC WD1500ADFD-00NLR5 21.07QR5> at ata2-master SATA150
ad6: 143089MB <WDC WD1500ADFD-00NLR5 21.07QR5> at ata3-master SATA150
SMP: AP CPU #2 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #7 Launched!
da0 at umass-sim0 bus 0 target 0 lun 0
da0: <PepperC Virtual Disc 1 0.01> Removable Direct Access SCSI-3 device
da0: 40.000MB/s transfers
da0: Attempt to query device size failed: NOT READY, Medium not present
Trying to mount root from zfs:tank
em1: link state changed to UP
em0: link state changed to UP
em0: link state changed to DOWN
em0: link state changed to UP
umass1: <LaCie LaCie Hard Drive USB, class 0/0, rev 2.00/0.00, addr 3>
on uhub3
da1 at umass-sim1 bus 1 target 0 lun 0
da1: <SAMSUNG HM250JI > Fixed Direct Access SCSI-2 device
da1: 40.000MB/s transfers
da1: 238475MB (488397168 512 byte sectors: 255H 63S/T 30401C)