Dmitry Morozovsky
2009-Apr-02 13:32 UTC
RELENG_7/i386: ZFS constant panic on file system writes
Pawel,
could you please help me a bit with *very* unpleasant situation: one of my
servers with very large ZFS reboots on most write requests to one (largest,
which effectively prohibits recreating) ZFS file system with
panic: avl_find() succeeded inside avl_add()
(kgdb) bt
#0 doadump () at pcpu.h:196
#1 0xc0533227 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418
#2 0xc0533535 in panic (fmt=Variable "fmt" is not available.
) at /usr/src/sys/kern/kern_shutdown.c:574
#3 0xc0836a20 in avl_add (tree=Variable "tree" is not available.
) at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/avl/avl.c:635
#4 0xc088c39f in zap_lockdir (os=0xc555a590, obj=6108, tx=0x0, lti=RW_READER,
fatreader=1, zapp=0xfc6907f8)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:231
#5 0xc088cc0f in zap_lookup (os=0xc555a590, zapobj=6108, name=0xfc6908bc
"daily.20080701.gz", integer_size=8, num_integers=1, buf=0xfc69083c)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:509
#6 0xc089e25d in zfs_dirent_lock (dlpp=0xfc690878, dzp=0xc709f570,
name=0xfc6908bc "daily.20080701.gz", zpp=0xfc690874, flag=Variable
"flag" is
not available.
)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:173
#7 0xc089e43e in zfs_dirlook (dzp=0xc709f570, name=0xfc6908bc
"daily.20080701.gz", vpp=0xfc690b5c)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:271
#8 0xc08a8653 in zfs_freebsd_lookup (ap=0xfc690a00)
at
/usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1080
#9 0xc06dab42 in VOP_CACHEDLOOKUP_APV (vop=0xc08ba7e0, a=0xfc690a00) at
vnode_if.c:153
#10 0xc05a402c in vfs_cache_lookup (ap=0xfc690a84) at vnode_if.h:83
#11 0xc06dc816 in VOP_LOOKUP_APV (vop=0xc08ba7e0, a=0xfc690a84) at
vnode_if.c:99
#12 0xc05aa681 in lookup (ndp=0xfc690b48) at vnode_if.h:57
#13 0xc05ab308 in namei (ndp=0xfc690b48) at /usr/src/sys/kern/vfs_lookup.c:215
#14 0xc05ba07f in kern_lstat (td=0xc5186af0, path=0xbfbfd088 <Address
0xbfbfd088 out of bounds>, pathseg=UIO_USERSPACE, sbp=0xfc690c18)
at /usr/src/sys/kern/vfs_syscalls.c:2184
#15 0xc05ba22f in lstat (td=0xc5186af0, uap=0xfc690cfc) at
/usr/src/sys/kern/vfs_syscalls.c:2167
#16 0xc06d0288 in syscall (frame=0xfc690d38) at
/usr/src/sys/i386/i386/trap.c:1090
#17 0xc06b5bc0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255
#18 0x00000033 in ?? ()
Previous frame inner to this frame (corrupt stack?)
this is fresh RELENG_7/i386 with (I suppose, unrelated) patch to ata from mav@
Thanks in advance.
--
Sincerely,
D.Marck [DM5020, MCK-RIPE, DM3-RIPN]
[ FreeBSD committer: marck@FreeBSD.org ]
------------------------------------------------------------------------
*** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru ***
------------------------------------------------------------------------
Marius Nünnerich
2009-Apr-03 02:53 UTC
RELENG_7/i386: ZFS constant panic on file system writes
On Thu, Apr 2, 2009 at 22:32, Dmitry Morozovsky <marck@rinet.ru> wrote:> Pawel, > > could you please help me a bit with *very* unpleasant situation: one of my > servers with very large ZFS reboots on most write requests to one (largest, > which effectively prohibits recreating) ZFS file system with > > panic: avl_find() succeeded inside avl_add() > > (kgdb) bt > #0 ?doadump () at pcpu.h:196 > #1 ?0xc0533227 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 > #2 ?0xc0533535 in panic (fmt=Variable "fmt" is not available. > ) at /usr/src/sys/kern/kern_shutdown.c:574 > #3 ?0xc0836a20 in avl_add (tree=Variable "tree" is not available. > ) at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/avl/avl.c:635 > #4 ?0xc088c39f in zap_lockdir (os=0xc555a590, obj=6108, tx=0x0, lti=RW_READER, > fatreader=1, zapp=0xfc6907f8) > ? ?at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:231 > #5 ?0xc088cc0f in zap_lookup (os=0xc555a590, zapobj=6108, name=0xfc6908bc > "daily.20080701.gz", integer_size=8, num_integers=1, buf=0xfc69083c) > ? ?at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:509 > #6 ?0xc089e25d in zfs_dirent_lock (dlpp=0xfc690878, dzp=0xc709f570, > name=0xfc6908bc "daily.20080701.gz", zpp=0xfc690874, flag=Variable "flag" is > not available. > ) > ? ?at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:173 > #7 ?0xc089e43e in zfs_dirlook (dzp=0xc709f570, name=0xfc6908bc > "daily.20080701.gz", vpp=0xfc690b5c) > ? ?at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:271 > #8 ?0xc08a8653 in zfs_freebsd_lookup (ap=0xfc690a00) > ? ?at > /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1080 > #9 ?0xc06dab42 in VOP_CACHEDLOOKUP_APV (vop=0xc08ba7e0, a=0xfc690a00) at > vnode_if.c:153 > #10 0xc05a402c in vfs_cache_lookup (ap=0xfc690a84) at vnode_if.h:83 > #11 0xc06dc816 in VOP_LOOKUP_APV (vop=0xc08ba7e0, a=0xfc690a84) at > vnode_if.c:99 > #12 0xc05aa681 in lookup (ndp=0xfc690b48) at vnode_if.h:57 > #13 0xc05ab308 in namei (ndp=0xfc690b48) at /usr/src/sys/kern/vfs_lookup.c:215 > #14 0xc05ba07f in kern_lstat (td=0xc5186af0, path=0xbfbfd088 <Address > 0xbfbfd088 out of bounds>, pathseg=UIO_USERSPACE, sbp=0xfc690c18) > ? ?at /usr/src/sys/kern/vfs_syscalls.c:2184 > #15 0xc05ba22f in lstat (td=0xc5186af0, uap=0xfc690cfc) at > /usr/src/sys/kern/vfs_syscalls.c:2167 > #16 0xc06d0288 in syscall (frame=0xfc690d38) at > /usr/src/sys/i386/i386/trap.c:1090 > #17 0xc06b5bc0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 > #18 0x00000033 in ?? () > Previous frame inner to this frame (corrupt stack?) > > this is fresh RELENG_7/i386 with (I suppose, unrelated) patch to ata from mav@ > > Thanks in advance.Hi Dmitry, I think the line numbers are misleading, see: http://fxr.watson.org/fxr/source/cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c?v=FREEBSD7;im=bigexcerpts#L262 Could you build a kernel with -O1 or -O0 perhaps that will help. I have no other clue about your situation except maybe try 8-current? - Marius
Dmitry Morozovsky
2009-Apr-07 03:00 UTC
RELENG_7/i386: ZFS constant panic on file system writes
On Fri, 3 Apr 2009, Dmitry Morozovsky wrote: DM> Pawel, DM> DM> could you please help me a bit with *very* unpleasant situation: one of my DM> servers with very large ZFS reboots on most write requests to one (largest, DM> which effectively prohibits recreating) ZFS file system with DM> DM> panic: avl_find() succeeded inside avl_add() Is there a way I can clear the directory in question? Even the latest -current panics when I try to access the directory containing this file. DM> DM> (kgdb) bt DM> #0 doadump () at pcpu.h:196 DM> #1 0xc0533227 in boot (howto=260) at /usr/src/sys/kern/kern_shutdown.c:418 DM> #2 0xc0533535 in panic (fmt=Variable "fmt" is not available. DM> ) at /usr/src/sys/kern/kern_shutdown.c:574 DM> #3 0xc0836a20 in avl_add (tree=Variable "tree" is not available. DM> ) at DM> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/common/avl/avl.c:635 DM> #4 0xc088c39f in zap_lockdir (os=0xc555a590, obj=6108, tx=0x0, lti=RW_READER, DM> fatreader=1, zapp=0xfc6907f8) DM> at DM> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:231 DM> #5 0xc088cc0f in zap_lookup (os=0xc555a590, zapobj=6108, name=0xfc6908bc DM> "daily.20080701.gz", integer_size=8, num_integers=1, buf=0xfc69083c) DM> at DM> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zap_micro.c:509 DM> #6 0xc089e25d in zfs_dirent_lock (dlpp=0xfc690878, dzp=0xc709f570, DM> name=0xfc6908bc "daily.20080701.gz", zpp=0xfc690874, flag=Variable "flag" is DM> not available. DM> ) DM> at DM> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:173 DM> #7 0xc089e43e in zfs_dirlook (dzp=0xc709f570, name=0xfc6908bc DM> "daily.20080701.gz", vpp=0xfc690b5c) DM> at DM> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_dir.c:271 DM> #8 0xc08a8653 in zfs_freebsd_lookup (ap=0xfc690a00) DM> at DM> /usr/src/sys/modules/zfs/../../cddl/contrib/opensolaris/uts/common/fs/zfs/zfs_vnops.c:1080 DM> #9 0xc06dab42 in VOP_CACHEDLOOKUP_APV (vop=0xc08ba7e0, a=0xfc690a00) at DM> vnode_if.c:153 DM> #10 0xc05a402c in vfs_cache_lookup (ap=0xfc690a84) at vnode_if.h:83 DM> #11 0xc06dc816 in VOP_LOOKUP_APV (vop=0xc08ba7e0, a=0xfc690a84) at DM> vnode_if.c:99 DM> #12 0xc05aa681 in lookup (ndp=0xfc690b48) at vnode_if.h:57 DM> #13 0xc05ab308 in namei (ndp=0xfc690b48) at /usr/src/sys/kern/vfs_lookup.c:215 DM> #14 0xc05ba07f in kern_lstat (td=0xc5186af0, path=0xbfbfd088 <Address DM> 0xbfbfd088 out of bounds>, pathseg=UIO_USERSPACE, sbp=0xfc690c18) DM> at /usr/src/sys/kern/vfs_syscalls.c:2184 DM> #15 0xc05ba22f in lstat (td=0xc5186af0, uap=0xfc690cfc) at DM> /usr/src/sys/kern/vfs_syscalls.c:2167 DM> #16 0xc06d0288 in syscall (frame=0xfc690d38) at DM> /usr/src/sys/i386/i386/trap.c:1090 DM> #17 0xc06b5bc0 in Xint0x80_syscall () at /usr/src/sys/i386/i386/exception.s:255 DM> #18 0x00000033 in ?? () DM> Previous frame inner to this frame (corrupt stack?) DM> DM> this is fresh RELENG_7/i386 with (I suppose, unrelated) patch to ata from mav@ DM> DM> Thanks in advance. DM> DM> -- Sincerely, D.Marck [DM5020, MCK-RIPE, DM3-RIPN] [ FreeBSD committer: marck@FreeBSD.org ] ------------------------------------------------------------------------ *** Dmitry Morozovsky --- D.Marck --- Wild Woozle --- marck@rinet.ru *** ------------------------------------------------------------------------