Yesterday I tried to clone a xen dom0 zfs root filesystem and hit this panic
(probably Bug ID 6580715):
System is running last week''s opensolaris bits (but I''m also
accessing the zpool
using the xen snv_66 bits).
files/s11-root-xen: is an existing version 1 zfs
files/s11-root-xen at uppc: new snapshot
files/s11-root-xen-uppc: clone for files/s11-root-xen at uppc
- initially the files/s11-root-xen at uppc snapshot couldn''t be
created,
because files/s11-root-xen (zfs with legacy mount / not mounted) was
"busy"
This should be bug 6462803 or 6482985.
Workaround: manually mount files/s11-root-xen and umount it -
this clears the unplayed log
- created files/s11-root-xen at uppc and cloned it as files/s11-root-xen-uppc,
set files/s11-root-xen-uppc mountpoint as legacy
- mount files/s11-root-xen-uppc and edited a few files using vi,
after writing back one of them and leaving vi, system crashed
Looks like the new zfs filesystem is using log blocks, that are not allocated?
(see the zdb output below)
=======================================================================
Details for the initial panic:
> ::status
debugging crash dump vmcore.6 (64-bit) from moritz
operating system: 5.11 wos_b73 (i86pc)
panic message: freeing free segment (vdev=0 offset=11c14df000 size=1000)
dump content: kernel pages only
> $c
vpanic()
vcmn_err+0x28(3, fffffffff812d818, ffffff0004850798)
zfs_panic_recover+0xb6()
metaslab_free_dva+0x1a2(ffffff01487ec580, ffffff0162231b20, 20b236c, 0)
metaslab_free+0x97(ffffff01487ec580, ffffff0162231b20, 20b236c, 0)
zio_free_blk+0x4c(ffffff01487ec580, ffffff0162231b20, 20b236c)
zil_sync+0x334(ffffff015b7d94c0, ffffff015689d180)
dmu_objset_sync+0x18e(ffffff014ff39c40, ffffff017c500d58, ffffff015689d180)
dsl_dataset_sync+0x5d(ffffff01571efa00, ffffff017c500d58, ffffff015689d180)
dsl_pool_sync+0xb5(ffffff014f4ace00, 20b236c)
spa_sync+0x1c5(ffffff01487ec580, 20b236c)
txg_sync_thread+0x19a(ffffff014f4ace00)
thread_start+8()
> ::msgbuf
MESSAGE
zfs0 is /pseudo/zfs at 0
pcplusmp: pci-ide (pci-ide) instance #1 vector 0x17 ioapic 0x2 intin 0x17 is bou
nd to cpu 0
IDE device at targ 0, lun 0 lastlun 0x0
model SAMSUNG HD300LJ
ATA/ATAPI-7 supported, majver 0xfe minver 0x21
PCI Express-device: ide at 0, ata2
ata2 is /pci at 0,0/pci-ide at 5/ide at 0
UltraDMA mode 6 selected
Disk0: <Vendor ''Gen-ATA '' Product ''SAMSUNG
HD300LJ ''>
cmdk0 at ata2 target 0 lun 0
cmdk0 is /pci at 0,0/pci-ide at 5/ide at 0/cmdk at 0,0
NOTICE: nge0: Using FIXED interrupt type
NOTICE: IRQ20 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: nge0 registered
NOTICE: nge0 link up, 100 Mbps, full duplex
NOTICE: cpqhpc: 64-bit driver module not found
UltraDMA mode 6 selected
dump on /dev/dsk/c1d0s1 size 2055 MB
UltraDMA mode 6 selected
pseudo-device: devinfo0
devinfo0 is /pseudo/devinfo at 0
iscsi0 at root
iscsi0 is /iscsi
xsvc0 at root: space 0 offset 0
xsvc0 is /xsvc at 0,0
pseudo-device: pseudo1
pseudo1 is /pseudo/zconsnex at 1
pcplusmp: fdc (fdc) instance 0 vector 0x6 ioapic 0x2 intin 0x6 is bound to cpu 0
ISA-device: fdc0
pseudo-device: ramdisk1024
ramdisk1024 is /pseudo/ramdisk at 1024
pcplusmp: lp (ecpp) instance 0 vector 0x7 ioapic 0x2 intin 0x7 is bound to cpu 1
ISA-device: ecpp0
ecpp0 is /isa/lp at 1,378
fd0 at fdc0
fd0 is /isa/fdc at 1,3f0/fd at 0,0
NOTICE: audiohd0: codec info: vid=0x11d4198b, sid=0x00000000, rev=0x00100200
NOTICE: IRQ21 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
PCI Express-device: pci1043,81f6 at 6,1, audiohd0
audiohd0 is /pci at 0,0/pci1043,81f6 at 6,1
pcplusmp: ide (ata) instance 0 vector 0xe ioapic 0x2 intin 0xe is bound to cpu 0
ATAPI device at targ 1, lun 0 lastlun 0x0
model _NEC DVD_RW ND-4550A
PCI Express-device: ide at 0, ata0
ata0 is /pci at 0,0/pci-ide at 4/ide at 0
UltraDMA mode 2 selected
UltraDMA mode 2 selected
UltraDMA mode 2 selected
PCI-device: pci1274,2000 at 7, audioens0
audioens0 is /pci at 0,0/pci10de,370 at 6/pci1274,2000 at 7
pseudo-device: lockstat0
lockstat0 is /pseudo/lockstat at 0
pseudo-device: llc10
llc10 is /pseudo/llc1 at 0
pseudo-device: lofi0
lofi0 is /pseudo/lofi at 0
pseudo-device: profile0
profile0 is /pseudo/profile at 0
pseudo-device: systrace0
systrace0 is /pseudo/systrace at 0
pseudo-device: fbt0
fbt0 is /pseudo/fbt at 0
pseudo-device: sdt0
sdt0 is /pseudo/sdt at 0
pseudo-device: fasttrap0
fasttrap0 is /pseudo/fasttrap at 0
pseudo-device: power0
power0 is /pseudo/power at 0
pseudo-device: fcp0
fcp0 is /pseudo/fcp at 0
pseudo-device: fcsm0
fcsm0 is /pseudo/fcsm at 0
pseudo-device: ucode0
ucode0 is /pseudo/ucode at 0
pseudo-device: lx_systrace0
lx_systrace0 is /pseudo/lx_systrace at 0
pseudo-device: fssnap0
fssnap0 is /pseudo/fssnap at 0
pseudo-device: winlock0
winlock0 is /pseudo/winlock at 0
pseudo-device: pm0
pm0 is /pseudo/pm at 0
pseudo-device: rsm0
rsm0 is /pseudo/rsm at 0
pseudo-device: pool0
pool0 is /pseudo/pool at 0
IP Filter: v4.1.9, running.
NOTICE: kqemu accelerator v0.2 (kqemu version 1.3.0)
pcplusmp: asy (asy) instance 0 vector 0x4 ioapic 0x2 intin 0x4 is bound to cpu 1
ISA-device: asy0
asy0 is /isa/asy at 1,3f8
pseudo-device: dtrace0
dtrace0 is /pseudo/dtrace at 0
sd0 at ata0: target 1 lun 0
sd0 is /pci at 0,0/pci-ide at 4/ide at 0/sd at 1,0
pseudo-device: nvidia255
nvidia255 is /pseudo/nvidia at 255
NOTICE: nge1: Using FIXED interrupt type
NOTICE: IRQ22 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: nge1 registered
UltraDMA mode 2 selected
UltraDMA mode 2 selected
UltraDMA mode 2 selected
UltraDMA mode 6 selected
UltraDMA mode 6 selected
NOTICE: nge1 unregistered
pseudo-device: devinfo0
devinfo0 is /pseudo/devinfo at 0
xsvc0 at root: space 0 offset 0
xsvc0 is /xsvc at 0,0
pseudo-device: nvidia255
nvidia255 is /pseudo/nvidia at 255
PCI-device: pci1274,2000 at 7, audioens0
audioens0 is /pci at 0,0/pci10de,370 at 6/pci1274,2000 at 7
NOTICE: audiohd0: codec info: vid=0x11d4198b, sid=0x00000000, rev=0x00100200
NOTICE: IRQ21 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
PCI Express-device: pci1043,81f6 at 6,1, audiohd0
audiohd0 is /pci at 0,0/pci1043,81f6 at 6,1
panic[cpu0]/thread=ffffff0004850c80:
freeing free segment (vdev=0 offset=11c14df000 size=1000)
ffffff00048506d0 genunix:vcmn_err+28 ()
ffffff00048507c0 zfs:zfs_panic_recover+b6 ()
ffffff0004850840 zfs:metaslab_free_dva+1a2 ()
ffffff00048508b0 zfs:metaslab_free+97 ()
ffffff0004850900 zfs:zio_free_blk+4c ()
ffffff0004850a00 zfs:zil_sync+334 ()
ffffff0004850a80 zfs:dmu_objset_sync+18e ()
ffffff0004850ad0 zfs:dsl_dataset_sync+5d ()
ffffff0004850b40 zfs:dsl_pool_sync+b5 ()
ffffff0004850bd0 zfs:spa_sync+1c5 ()
ffffff0004850c60 zfs:txg_sync_thread+19a ()
ffffff0004850c70 unix:thread_start+8 ()
syncing file systems...
done
dumping to /dev/dsk/c1d0s1, offset 431030272, content: kernel
=======================================================================
Now, when trying to mount files/s11-root-xen-uppc, it panics in the same way:
> ::status
debugging crash dump vmcore.7 (64-bit) from moritz
operating system: 5.11 wos_b73 (i86pc)
panic message: freeing free segment (vdev=0 offset=11c14df000 size=1000)
dump content: kernel pages only
> $C
ffffff00049535c0 vpanic()
ffffff00049535f0 vcmn_err+0x28(3, fffffffff812d818, ffffff00049536b8)
ffffff00049536e0 zfs_panic_recover+0xb6()
ffffff0004953760 metaslab_free_dva+0x1a2(ffffff01487feac0, ffffff00049538b0,
20b239c, 0)
ffffff00049537d0 metaslab_free+0x97(ffffff01487feac0, ffffff00049538b0, 20b239c
, 0)
ffffff0004953820 zio_free_blk+0x4c(ffffff01487feac0, ffffff00049538b0, 20b239c)
ffffff0004953860 zil_free_log_block+0x39(ffffff015a87b240, ffffff00049538b0,
ffffff015aa40f00, 20b2365)
ffffff0004953990 zil_parse+0x175(ffffff015a87b240, fffffffff8108d40,
fffffffff8108d80, ffffff015aa40f00, 20b2365)
ffffff00049539f0 zil_destroy+0xd7(ffffff015a87b240, 0)
ffffff0004953a90 zil_replay+0xf2(ffffff01581fb208, ffffff0157905c00,
ffffff0157905c30, ffffffffc0276650)
ffffff0004953b50 zfs_domount+0x243(ffffff0151d530c0, ffffff0157e9edc0,
ffffff01525ab0d8)
ffffff0004953c70 zfs_mount+0x1b6(ffffff0151d530c0, ffffff015b9c9980,
ffffff0004953e30, ffffff01525ab0d8)
ffffff0004953ca0 fsop_mount+0x21(ffffff0151d530c0, ffffff015b9c9980,
ffffff0004953e30, ffffff01525ab0d8)
ffffff0004953e00 domount+0x906(0, ffffff0004953e30, ffffff015b9c9980,
ffffff01525ab0d8, ffffff0004953e28)
ffffff0004953e80 mount+0xd2(ffffff014d9ed978, ffffff0004953eb8)
ffffff0004953ec0 syscall_ap+0x8f()
ffffff0004953f10 sys_syscall32+0x101()
> ::msgbuf
MESSAGE
pcplusmp: ide (ata) instance 0 vector 0xe ioapic 0x2 intin 0xe is bound to cpu 0
ATAPI device at targ 1, lun 0 lastlun 0x0
model _NEC DVD_RW ND-4550A
PCI Express-device: ide at 0, ata0
ata0 is /pci at 0,0/pci-ide at 4/ide at 0
UltraDMA mode 2 selected
UltraDMA mode 2 selected
UltraDMA mode 2 selected
pseudo-device: lockstat0
lockstat0 is /pseudo/lockstat at 0
PCI-device: pci1274,2000 at 7, audioens0
audioens0 is /pci at 0,0/pci10de,370 at 6/pci1274,2000 at 7
pseudo-device: llc10
llc10 is /pseudo/llc1 at 0
pseudo-device: lofi0
lofi0 is /pseudo/lofi at 0
pseudo-device: profile0
profile0 is /pseudo/profile at 0
pseudo-device: systrace0
systrace0 is /pseudo/systrace at 0
pseudo-device: fbt0
fbt0 is /pseudo/fbt at 0
pseudo-device: sdt0
sdt0 is /pseudo/sdt at 0
pseudo-device: fasttrap0
fasttrap0 is /pseudo/fasttrap at 0
pseudo-device: power0
power0 is /pseudo/power at 0
pseudo-device: fcp0
fcp0 is /pseudo/fcp at 0
pseudo-device: fcsm0
fcsm0 is /pseudo/fcsm at 0
pseudo-device: ucode0
ucode0 is /pseudo/ucode at 0
pseudo-device: lx_systrace0
lx_systrace0 is /pseudo/lx_systrace at 0
pseudo-device: fssnap0
fssnap0 is /pseudo/fssnap at 0
pseudo-device: winlock0
winlock0 is /pseudo/winlock at 0
pseudo-device: pm0
pm0 is /pseudo/pm at 0
pseudo-device: rsm0
rsm0 is /pseudo/rsm at 0
pseudo-device: pool0
pool0 is /pseudo/pool at 0
IP Filter: v4.1.9, running.
NOTICE: kqemu accelerator v0.2 (kqemu version 1.3.0)
pseudo-device: dtrace0
dtrace0 is /pseudo/dtrace at 0
pcplusmp: asy (asy) instance 0 vector 0x4 ioapic 0x2 intin 0x4 is bound to cpu 1
ISA-device: asy0
asy0 is /isa/asy at 1,3f8
sd0 at ata0: target 1 lun 0
sd0 is /pci at 0,0/pci-ide at 4/ide at 0/sd at 1,0
pseudo-device: nvidia255
nvidia255 is /pseudo/nvidia at 255
NOTICE: nge1: Using FIXED interrupt type
NOTICE: IRQ22 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: nge1 registered
UltraDMA mode 2 selected
UltraDMA mode 6 selected
UltraDMA mode 2 selected
UltraDMA mode 2 selected
UltraDMA mode 6 selected
NOTICE: nge1 unregistered
pcplusmp: asy (asy) instance 0 vector 0x4 ioapic 0x2 intin 0x4 is bound to cpu 0
ISA-device: asy0
asy0 is /isa/asy at 1,3f8
sd0 at ata0: target 1 lun 0
sd0 is /pci at 0,0/pci-ide at 4/ide at 0/sd at 1,0
pcplusmp: lp (ecpp) instance 0 vector 0x7 ioapic 0x2 intin 0x7 is bound to cpu 1
ISA-device: ecpp0
ecpp0 is /isa/lp at 1,378
fd0 at fdc0
fd0 is /isa/fdc at 1,3f0/fd at 0,0
pseudo-device: devinfo0
devinfo0 is /pseudo/devinfo at 0
PCI-device: pci1274,2000 at 7, audioens0
audioens0 is /pci at 0,0/pci10de,370 at 6/pci1274,2000 at 7
NOTICE: audiohd0: codec info: vid=0x11d4198b, sid=0x00000000, rev=0x00100200
NOTICE: IRQ21 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
PCI Express-device: pci1043,81f6 at 6,1, audiohd0
audiohd0 is /pci at 0,0/pci1043,81f6 at 6,1
NOTICE: nge1: Using FIXED interrupt type
NOTICE: IRQ22 is being shared by drivers with different interrupt levels.
This may result in reduced system performance.
NOTICE: nge1 registered
pseudo-device: nvidia255
nvidia255 is /pseudo/nvidia at 255
pseudo-device: ramdisk1024
ramdisk1024 is /pseudo/ramdisk at 1024
pseudo-device: lockstat0
lockstat0 is /pseudo/lockstat at 0
pseudo-device: lofi0
lofi0 is /pseudo/lofi at 0
pseudo-device: profile0
profile0 is /pseudo/profile at 0
pseudo-device: systrace0
systrace0 is /pseudo/systrace at 0
pseudo-device: fbt0
fbt0 is /pseudo/fbt at 0
pseudo-device: sdt0
sdt0 is /pseudo/sdt at 0
pseudo-device: fcp0
fcp0 is /pseudo/fcp at 0
pseudo-device: fcsm0
fcsm0 is /pseudo/fcsm at 0
pseudo-device: ucode0
ucode0 is /pseudo/ucode at 0
pseudo-device: lx_systrace0
lx_systrace0 is /pseudo/lx_systrace at 0
pseudo-device: fssnap0
fssnap0 is /pseudo/fssnap at 0
pseudo-device: winlock0
winlock0 is /pseudo/winlock at 0
pseudo-device: pm0
pm0 is /pseudo/pm at 0
pseudo-device: rsm0
rsm0 is /pseudo/rsm at 0
NOTICE: kqemu accelerator v0.2 (kqemu version 1.3.0)
panic[cpu0]/thread=ffffff015a974520:
freeing free segment (vdev=0 offset=11c14df000 size=1000)
ffffff00049535f0 genunix:vcmn_err+28 ()
ffffff00049536e0 zfs:zfs_panic_recover+b6 ()
ffffff0004953760 zfs:metaslab_free_dva+1a2 ()
ffffff00049537d0 zfs:metaslab_free+97 ()
ffffff0004953820 zfs:zio_free_blk+4c ()
ffffff0004953860 zfs:zil_free_log_block+39 ()
ffffff0004953990 zfs:zil_parse+175 ()
ffffff00049539f0 zfs:zil_destroy+d7 ()
ffffff0004953a90 zfs:zil_replay+f2 ()
ffffff0004953b50 zfs:zfs_domount+243 ()
ffffff0004953c70 zfs:zfs_mount+1b6 ()
ffffff0004953ca0 genunix:fsop_mount+21 ()
ffffff0004953e00 genunix:domount+906 ()
ffffff0004953e80 genunix:mount+d2 ()
ffffff0004953ec0 genunix:syscall_ap+8f ()
ffffff0004953f10 unix:brand_sys_syscall32+1a3 ()
syncing file systems...
done
dumping to /dev/dsk/c1d0s1, offset 431030272, content: kernel
=======================================================================
moritz 14# zdb -ivv files/s11-root-xen
Dataset files/s11-root-xen [ZPL], ID 148, cr_txg 34132547, 1.14G, 28335 objects
moritz 15# zdb -ivv files/s11-root-xen at uppc
Dataset files/s11-root-xen at uppc [ZPL], ID 141, cr_txg 34284363, 1.14G, 28335
objects
moritz 16# zdb -ivv files/s11-root-xen-uppc
Dataset files/s11-root-xen-uppc [ZPL], ID 153, cr_txg 34284364, 1.14G, 28335
objects
ZIL header: claim_txg 34284389, seq 0
TX_TRUNCATE len 56, txg 34284396, seq 1
foid 23407, offset 0x0, length 0x0
TX_WRITE len 448, txg 34284396, seq 2
foid 23407, offset 0x0, length 0xff, blkoff 0x0
Total 2
TX_CREATE 0
TX_MKDIR 0
TX_MKXATTR 0
TX_SYMLINK 0
TX_REMOVE 0
TX_RMDIR 0
TX_LINK 0
TX_RENAME 0
TX_WRITE 1
TX_TRUNCATE 1
TX_SETATTR 0
TX_ACL 0
moritz 17# zdb -ivvvv files/s11-root-xen-uppc
Dataset files/s11-root-xen-uppc [ZPL], ID 153, cr_txg 34284364, 1.14G, 28335
objects
ZIL header: claim_txg 34284389, seq 0
first block: [L0 ZIL intent log] 1000L/1000P
DVA[0]=<0:11c14df000:1000> zilog uncompressed LE contiguous birth=34284371
fill=0 cksum=3dba7e1a8cf4ade2:9238d510ee995f8b:99:1
Block seqno 1, already claimed, [L0 ZIL intent log] 1000L/1000P
DVA[0]=<0:11c14df000:1000> zilog uncompressed LE contiguous birth=34284371
fill=0 cksum=3dba7e1a8cf4ade2:9238d510ee995f8b:99:1
TX_TRUNCATE len 56, txg 34284396, seq 1
foid 23407, offset 0x0, length 0x0
TX_WRITE len 448, txg 34284396, seq 2
foid 23407, offset 0x0, length 0xff, blkoff 0x0
# ! / b i n / c s h A As e t p a t h
Block seqno 2, already claimed, [L0 ZIL intent log] 1000L/1000P
DVA[0]=<0:11c14e9000:1000> zilog uncompressed LE contiguous birth=34284396
fill=0 cksum=3dba7e1a8cf4ade2:9238d510ee995f8b:99:2
Total 2
TX_CREATE 0
TX_MKDIR 0
TX_MKXATTR 0
TX_SYMLINK 0
TX_REMOVE 0
TX_RMDIR 0
TX_LINK 0
TX_RENAME 0
TX_WRITE 1
TX_TRUNCATE 1
TX_SETATTR 0
TX_ACL 0
=======================================================================
zfs destroy files/s11-root-xen-uppc> ::status
debugging crash dump vmcore.8 (64-bit) from moritz
operating system: 5.11 wos_b73 (i86pc)
panic message: freeing free segment (vdev=0 offset=11c14df000 size=1000)
dump content: kernel pages only> $c
vpanic()
vcmn_err+0x28(3, fffffffff812d818, ffffff000404b8a8)
zfs_panic_recover+0xb6()
metaslab_free_dva+0x1a2(ffffff01487ea040, ffffff000404baa0, 20b27cf, 0)
metaslab_free+0x97(ffffff01487ea040, ffffff000404baa0, 20b27cf, 0)
zio_free_blk+0x4c(ffffff01487ea040, ffffff000404baa0, 20b27cf)
zil_free_log_block+0x39(ffffff015be3c900, ffffff000404baa0, ffffff015b316940,
20b2365)
zil_parse+0x175(ffffff015be3c900, fffffffff8108d40, fffffffff8108d80,
ffffff015b316940, 20b2365)
zil_destroy+0xd7(ffffff015be3c900, 0)
dmu_objset_destroy+0x40(ffffff015b225000)
zfs_ioc_destroy+0x42(ffffff015b225000)
zfsdev_ioctl+0x10d(b600000000, 5a18, 8046a68, 100003, ffffff015379d248,
ffffff000404be9c)
cdev_ioctl+0x48(b600000000, 5a18, 8046a68, 100003, ffffff015379d248,
ffffff000404be9c)
spec_ioctl+0x86(ffffff014d9a8a40, 5a18, 8046a68, 100003, ffffff015379d248,
ffffff000404be9c)
fop_ioctl+0x6e(ffffff014d9a8a40, 5a18, 8046a68, 100003, ffffff015379d248,
ffffff000404be9c)
ioctl+0x16b(3, 5a18, 8046a68)
sys_syscall32+0x101()
This message posted from opensolaris.org