György Vilmos
2009-Jun-11 12:13 UTC
.zfs snapshot dir disappears and a crash later on, while umounting
Hi All! (please keep me on cc) FreeBSD 7/amd64 stable compiled with GENERIC on May 23 22:22:00 CEST 2009 (csuped sources around that time) on an IBM xSomething. I have a single disk zfs pool for backup purposes, the fs is compressed and the stuff goes onto it every night and then snapshotted once. I have a nullfs mount from that to a UFS partition, like this: bkp on /bkp (zfs, local) bkp@20090607 on /bkp/.zfs/snapshot/20090607 (zfs, local, noatime, read- only) bkp@20090610 on /bkp/.zfs/snapshot/20090610 (zfs, local, noatime, read- only) bkp@20090609 on /bkp/.zfs/snapshot/20090609 (zfs, local, noatime, read- only) bkp@20090608 on /bkp/.zfs/snapshot/20090608 (zfs, local, noatime, read- only) /bkp/hosting on /data/jail/hosting/bkp (nullfs, local) The strange thing is that while did the above, I lost .zfs, so an ls wrote this: ls /bkp/.zfs/snapshot ls: /bkp/.zfs/snapshot: Bad file descriptor df also didn't show the snapshot dirs after it, but I could do snapshots and zfs list -t snapshot showed them, I just couldn't access them. Today, I've tried to umount the nullfs mount and the zfs pool to see whether it helps the situation, but it didn't, I was awarded with a crashdump. What I did is: umount /data/jail/hosting/bkp (went fine, I haven't got back the .zfs snapshot dir) then: zfs export bkp (crash) The crashdump says: Fatal trap 12: page fault while in kernel mode cpuid = 1; apic id = 01 fault virtual address = 0xc0 fault code = supervisor write data, page not present instruction pointer = 0x8:0xffffffff80517925 stack pointer = 0x10:0xffffff80780249d0 frame pointer = 0x10:0xffffff8078024a50 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 35021 (zpool) trap number = 12 panic: page fault cpuid = 1 Uptime: 14d22h20m56s Physical memory: 4082 MB bt says: (kgdb) bt #0 doadump () at pcpu.h:195 #1 0x0000000000000004 in ?? () #2 0xffffffff8050ff29 in boot (howto=260) at /usr/src/sys/kern/ kern_shutdown.c:418 #3 0xffffffff80510332 in panic (fmt=0x104 <Address 0x104 out of bounds>) at /usr/src/sys/kern/kern_shutdown.c:574 #4 0xffffffff807d5a33 in trap_fatal (frame=0xffffff0020214390, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:757 #5 0xffffffff807d5e05 in trap_pfault (frame=0xffffff8078024920, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:673 #6 0xffffffff807d6744 in trap (frame=0xffffff8078024920) at /usr/src/ sys/amd64/amd64/trap.c:444 #7 0xffffffff807ba96e in calltrap () at /usr/src/sys/amd64/amd64/ exception.S:209 #8 0xffffffff80517925 in _sx_xlock (sx=0xa0, opts=0, file=0xffffffff80f045e8 "/usr/src/sys/modules/zfs/../../cddl/contrib/ opensolaris/uts/common/fs/zfs/zfs_ctldir.c", line=1288) at atomic.h:143 #9 0xffffffff80e97e55 in zfsctl_umount_snapshots (vfsp=Variable "vfsp" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ common/fs/zfs/zfs_ctldir.c:1288 #10 0xffffffff80ea3560 in zfs_umount (vfsp=0xffffff00048b7380, fflag=0, td=Variable "td" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/ common/fs/zfs/zfs_vfsops.c:1003 #11 0xffffffff8058e1a7 in dounmount (mp=0xffffff00048b7380, flags=0, td=0xffffff0020214390) at /usr/src/sys/kern/vfs_mount.c:1290 #12 0xffffffff8058e975 in unmount (td=0xffffff0020214390, uap=0xffffff8078024bf0) at /usr/src/sys/kern/vfs_mount.c:1186 #13 0xffffffff807d6087 in syscall (frame=0xffffff8078024c80) at /usr/ src/sys/amd64/amd64/trap.c:900 #14 0xffffffff807bab7b in Xfast_syscall () at /usr/src/sys/amd64/amd64/ exception.S:330 #15 0x000000080102e5cc in ?? () Previous frame inner to this frame (corrupt stack?) I have the dump, if any further info could help. Thank you in advance.
Thomas Quinot
2009-Nov-06 00:36 UTC
.zfs snapshot dir disappears and a crash later on, while umounting (8.0-RC1)
* Gy?rgy Vilmos, 2009-06-11 :> Subject: .zfs snapshot dir disappears and a crash later on, while > umountingFor the record, I got the same symptom here on 8.0-RC1.> #7 0xffffffff807ba96e in calltrap () at /usr/src/sys/amd64/amd64/ > exception.S:209 > #8 0xffffffff80517925 in _sx_xlock (sx=0xa0, opts=0, > file=0xffffffff80f045e8 "/usr/src/sys/modules/zfs/../../cddl/contrib/ > opensolaris/uts/common/fs/zfs/zfs_ctldir.c", > line=1288) at atomic.h:143 > #9 0xffffffff80e97e55 in zfsctl_umount_snapshots (vfsp=Variable > "vfsp" is not available. > ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/Clearly looks like something in the .zfs structure gets botched at some point... I'll keep the crash dump for a while, if more information is needed please let me know. Thomas. melamine.cuivre.fr.eu.org dumped core - see /var/crash/vmcore.0 Fri Nov 6 01:23:21 CET 2009 FreeBSD melamine.cuivre.fr.eu.org 8.0-RC1 FreeBSD 8.0-RC1 #0: Tue Oct 6 19:43:29 UTC 2009 root@melamine2.cuivre.fr.eu.org:/usr/obj/usr/src/sys/GENERIC amd64 panic: page fault GNU gdb 6.1.1 [FreeBSD] Copyright 2004 Free Software Foundation, Inc. GDB is free software, covered by the GNU General Public License, and you are welcome to change it and/or distribute copies of it under certain conditions. Type "show copying" to see the conditions. There is absolutely no warranty for GDB. Type "show warranty" for details. This GDB was configured as "amd64-marcel-freebsd"... Unread portion of the kernel message buffer: <6>ifa_del_loopback_route: deletion failed <5>tun2: link state changed to DOWN <118>Nov 6 00:16:38 melamine syslogd: exiting on signal 15 info: [drm] Resetting GPU Waiting (max 60 seconds) for system process `vnlru' to stop...done Waiting (max 60 seconds) for system process `bufdaemon' to stop...done Waiting (max 60 seconds) for system process `syncer' to stop... Syncing disks, vnodes remaining...0 0 0 0 done All buffers synced. Fatal trap 12: page fault while in kernel mode cpuid = 0; apic id = 00 fault virtual address = 0xa8 fault code = supervisor write data, page not present instruction pointer = 0x20:0xffffffff8058cbf5 stack pointer = 0x28:0xffffff815d480970 frame pointer = 0x28:0xffffff815d480980 code segment = base 0x0, limit 0xfffff, type 0x1b = DPL 0, pres 1, long 1, def32 0, gran 1 processor eflags = interrupt enabled, resume, IOPL = 0 current process = 41907 (reboot) trap number = 12 panic: page fault cpuid = 0 Uptime: 15d10h25m49s Physical memory: 12270 MB Dumping 5655 MB: 5640 5624 5608 5592 5576 5560 5544 5528 5512 5496 5480 5464 5448 5432 5416 5400 5384 5368 5352 5336 5320 5304 5288 5272 5256 5240 5224 5208 5192 5176 5160 5144 5128 5112 5096 5080 5064 5048 5032 5016 5000 4984 4968 4952 4936 4920 4904 4888 4872 4856 4840 4824 4808 4792 4776 4760 4744 4728 4712 4696 4680 4664 4648 4632 4616 4600 4584 4568 4552 4536 4520 4504 4488 4472 4456 4440 4424 4408 4392 4376 4360 4344 4328 4312 4296 4280 4264 4248 4232 4216 4200 4184 4168 4152 4136 4120 4104 4088 4072 4056 4040 4024 4008 3992 3976 3960 3944 3928 3912 3896 3880 3864 3848 3832 3816 3800 3784 3768 3752 3736 3720 3704 3688 3672 3656 3640 3624 3608 3592 3576 3560 3544 3528 3512 3496 3480 3464 3448 3432 3416 3400 3384 3368 3352 3336 3320 3304 3288 3272 3256 3240 3224 3208 3192 3176 3160 3144 3128 3112 3096 3080 3064 3048 3032 3016 3000 2984 2968 2952 2936 2920 2904 2888 2872 2856 2840 2824 2808 2792 2776 2760 2744 2728 2712 2696 2680 2664 2648 2632 2616 2600 2584 2568 2552 2536 2520 2504 2488 2472 2456 2440 2424 2408 2392 2376 2360 2344 2328 2312 2296 2280 2264 2248 2232 2216 2200 2184 2168 2152 2136 2120 2104 2088 2072 2056 2040 2024 2008 1992 1976 1960 1944 1928 1912 1896 1880 1864 1848 1832 1816 1800 1784 1768 1752 1736 1720 1704 1688 1672 1656 1640 1624 1608 1592 1576 1560 1544 1528 1512 1496 1480 1464 1448 1432 1416 1400 1384 1368 1352 1336 1320 1304 1288 1272 1256 1240 1224 1208 1192 1176 1160 1144 1128 1112 1096 1080 1064 1048 1032 1016 1000 984 968 952 936 920 904 888 872 856 840 824 808 792 776 760 744 728 712 696 680 664 648 632 616 600 584 568 552 536 520 504 488 472 456 440 424 408 392 376 360 344 328 312 296 280 264 248 232 216 200 184 168 152 136 120 104 88 72 56 40 24 8 Reading symbols from /boot/kernel/zfs.ko...Reading symbols from /boot/kernel/zfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/zfs.ko Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from /boot/kernel/opensolaris.ko.symbols...done. done. Loaded symbols for /boot/kernel/opensolaris.ko Reading symbols from /boot/kernel/atapicam.ko...Reading symbols from /boot/kernel/atapicam.ko.symbols...done. done. Loaded symbols for /boot/kernel/atapicam.ko Reading symbols from /boot/kernel/linux.ko...Reading symbols from /boot/kernel/linux.ko.symbols...done. done. Loaded symbols for /boot/kernel/linux.ko Reading symbols from /boot/kernel/radeon.ko...Reading symbols from /boot/kernel/radeon.ko.symbols...done. done. Loaded symbols for /boot/kernel/radeon.ko Reading symbols from /boot/kernel/drm.ko...Reading symbols from /boot/kernel/drm.ko.symbols...done. done. Loaded symbols for /boot/kernel/drm.ko Reading symbols from /boot/kernel/linprocfs.ko...Reading symbols from /boot/kernel/linprocfs.ko.symbols...done. done. Loaded symbols for /boot/kernel/linprocfs.ko Reading symbols from /boot/kernel/snd_ich.ko...Reading symbols from /boot/kernel/snd_ich.ko.symbols...done. done. Loaded symbols for /boot/kernel/snd_ich.ko Reading symbols from /boot/kernel/sound.ko...Reading symbols from /boot/kernel/sound.ko.symbols...done. done. Loaded symbols for /boot/kernel/sound.ko Reading symbols from /boot/kernel/snd_hda.ko...Reading symbols from /boot/kernel/snd_hda.ko.symbols...done. done. Loaded symbols for /boot/kernel/snd_hda.ko Reading symbols from /boot/kernel/accf_http.ko...Reading symbols from /boot/kernel/accf_http.ko.symbols...done. done. Loaded symbols for /boot/kernel/accf_http.ko Reading symbols from /boot/kernel/accf_data.ko...Reading symbols from /boot/kernel/accf_data.ko.symbols...done. done. Loaded symbols for /boot/kernel/accf_data.ko #0 doadump () at pcpu.h:223 223 pcpu.h: No such file or directory. in pcpu.h (kgdb) #0 doadump () at pcpu.h:223 #1 0xffffffff80584f19 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:416 #2 0xffffffff8058534c in panic (fmt=0xffffffff8092c24c "%s") at /usr/src/sys/kern/kern_shutdown.c:579 #3 0xffffffff80865ad8 in trap_fatal (frame=0xffffff000625c390, eva=Variable "eva" is not available. ) at /usr/src/sys/amd64/amd64/trap.c:852 #4 0xffffffff80865ea4 in trap_pfault (frame=0xffffff815d4808c0, usermode=0) at /usr/src/sys/amd64/amd64/trap.c:768 #5 0xffffffff80866794 in trap (frame=0xffffff815d4808c0) at /usr/src/sys/amd64/amd64/trap.c:494 #6 0xffffffff8084cb33 in calltrap () at /usr/src/sys/amd64/amd64/exception.S:224 #7 0xffffffff8058cbf5 in _sx_xlock (sx=0x90, opts=0, file=0xffffffff80efcca8 "/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c", line=1336) at atomic.h:147 #8 0xffffffff80e8ec75 in zfsctl_umount_snapshots (vfsp=Variable "vfsp" is not available. ) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_ctldir.c:1336 #9 0xffffffff80e9b559 in zfs_umount (vfsp=0xffffff0006a50000, fflag=524288) at /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vfsops.c:1017 #10 0xffffffff80609ffa in dounmount (mp=0xffffff0006a50000, flags=524288, td=Variable "td" is not available. ) at /usr/src/sys/kern/vfs_mount.c:1290 #11 0xffffffff8060d962 in vfs_unmountall () at /usr/src/sys/kern/vfs_subr.c:3141 #12 0xffffffff8058519a in boot (howto=0) at /usr/src/sys/kern/kern_shutdown.c:401 #13 0xffffffff8058543c in reboot (td=0xffffff000625c390, uap=0xffffff815d480bf0) at /usr/src/sys/kern/kern_shutdown.c:173 #14 0xffffffff80866116 in syscall (frame=0xffffff815d480c80) at /usr/src/sys/amd64/amd64/trap.c:984 #15 0xffffffff8084ce11 in Xfast_syscall () at /usr/src/sys/amd64/amd64/exception.S:373 #16 0x000000080078f83c in ?? () Previous frame inner to this frame (corrupt stack?) (kgdb)