On 10/11/2016 17:20, Henri Hennebert wrote:> On 11/10/2016 15:00, Andriy Gapon wrote: >> Interesting. I can not spot any suspicious thread that would hold the vnode >> lock. Could you please run kgdb (just like that, no arguments), then execute >> 'bt' command and then select a frame when _vn_lock is called with 'fr N' >> command. Then please 'print *vp' and share the result. >> > I Think I miss something in your request:Oh, sorry! The very first step should be 'tid 101112' to switch to the correct context.> [root at avoriaz ~]# kgdb > GNU gdb 6.1.1 [FreeBSD] > Copyright 2004 Free Software Foundation, Inc. > GDB is free software, covered by the GNU General Public License, and you are > welcome to change it and/or distribute copies of it under certain conditions. > Type "show copying" to see the conditions. > There is absolutely no warranty for GDB. Type "show warranty" for details. > This GDB was configured as "amd64-marcel-freebsd"... > Reading symbols from /boot/kernel/zfs.ko...Reading symbols from > /usr/lib/debug//boot/kernel/zfs.ko.debug...done. > done. > Loaded symbols for /boot/kernel/zfs.ko > Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from > /usr/lib/debug//boot/kernel/opensolaris.ko.debug...done. > done. > > --- clip --- > > Loaded symbols for /boot/kernel/accf_data.ko > Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from > /usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done. > done. > Loaded symbols for /boot/kernel/daemon_saver.ko > #0 sched_switch (td=0xfffff8001131da00, newtd=0xfffff800762a8500, flags=<value > optimized out>) > at /usr/src/sys/kern/sched_ule.c:1973 > 1973 cpuid = PCPU_GET(cpuid); > (kgdb) bt > #0 sched_switch (td=0xfffff8001131da00, newtd=0xfffff800762a8500, flags=<value > optimized out>) > at /usr/src/sys/kern/sched_ule.c:1973 > #1 0xffffffff80566b15 in tc_fill_vdso_timehands32 (vdso_th32=0x0) at > /usr/src/sys/kern/kern_tc.c:2121 > #2 0xffffffff80555227 in timekeep_push_vdso () at > /usr/src/sys/kern/kern_sharedpage.c:174 > #3 0xffffffff80566226 in tc_windup () at /usr/src/sys/kern/kern_tc.c:1426 > #4 0xffffffff804eaa41 in hardclock_cnt (cnt=1, usermode=<value optimized out>) > at /usr/src/sys/kern/kern_clock.c:589 > #5 0xffffffff808fac74 in handleevents (now=<value optimized out>, fake=0) at > /usr/src/sys/kern/kern_clocksource.c:223 > #6 0xffffffff808fb1d7 in timercb (et=0xffffffff8100cf20, arg=<value optimized > out>) at /usr/src/sys/kern/kern_clocksource.c:352 > #7 0xfffff800b6429a00 in ?? () > #8 0xffffffff81051080 in vm_page_array () > #9 0xffffffff81051098 in vm_page_queue_free_mtx () > #10 0xfffffe0101818920 in ?? () > #11 0xffffffff805399c0 in __mtx_lock_sleep (c=<value optimized out>, tid=Error > accessing memory address 0xffffffffffffffac: Bad add\ > ress. > ) at /usr/src/sys/kern/kern_mutex.c:590 > Previous frame inner to this frame (corrupt stack?) > Current language: auto; currently minimal > (kgdb) q > [root at avoriaz ~]# > > I don't find the requested frame > > Henri-- Andriy Gapon
On 11/10/2016 16:54, Andriy Gapon wrote:> On 10/11/2016 17:20, Henri Hennebert wrote: >> On 11/10/2016 15:00, Andriy Gapon wrote: >>> Interesting. I can not spot any suspicious thread that would hold the vnode >>> lock. Could you please run kgdb (just like that, no arguments), then execute >>> 'bt' command and then select a frame when _vn_lock is called with 'fr N' >>> command. Then please 'print *vp' and share the result. >>> >> I Think I miss something in your request: > > Oh, sorry! The very first step should be 'tid 101112' to switch to the correct > context. >(kgdb) fr 7 #7 0xffffffff8063c5b3 in _vn_lock (vp=<value optimized out>, flags=2121728, file=<value optimized out>, line=<value optimized out>) at vnode_if.h:859 859 vnode_if.h: No such file or directory. in vnode_if.h (kgdb) print *vp $1 = {v_tag = 0xffffffff80faeb78 "?~\231\200????", v_op = 0xfffff80009a41000, v_data = 0x0, v_mount = 0xfffff80009a41010, v_nmntvnodes = {tqe_next = 0x0, tqe_prev = 0xffffffff80edc088}, v_un = {vu_mount = 0x0, vu_socket = 0x0, vu_cdev = 0x0, vu_fifoinfo = 0x0}, v_hashlist = {le_next = 0xfffff80009466e90, le_prev = 0x0}, v_cache_src = {lh_first = 0xfffffe010186d768}, v_cache_dst = {tqh_first = 0x0, tqh_last = 0xfffffe0000b8a7c0}, v_cache_dd = 0xfffff8000284f000, v_lock = {lock_object = { lo_name = 0xfffff8002c00ee80 "", lo_flags = 0, lo_data = 0, lo_witness = 0xfffff800068bd480}, lk_lock = 18446735277770268056, lk_exslpfail = 153715840, lk_timo = -2048, lk_pri = 0}, v_interlock = {lock_object = { lo_name = 0x18af8ffffffff <Error reading address 0x18af8ffffffff: Bad address>, lo_flags = 0, lo_data = 0, lo_witness = 0x0}, mtx_lock = 0}, v_vnlock = 0x0, v_actfreelist = {tqe_next = 0x0, tqe_prev = 0xfffff80009ba05c0}, v_bufobj = {bo_lock = {lock_object = {lo_name = 0xfffff80009a41000 "", lo_flags = 1, lo_data = 0, lo_witness = 0x4000000ff}, rw_lock = 2}, bo_ops = 0x100000000, bo_object = 0xfffff80049c2c068, bo_synclist = {le_next = 0xffffffff813be535, le_prev = 0x1000000000000}, bo_private = 0x0, __bo_vnode = 0x0, bo_clean = {bv_hd = {tqh_first = 0x0, tqh_last = 0x0}, bv_root = {pt_root = 0}, bv_cnt = 0}, bo_dirty = {bv_hd = {tqh_first = 0xfffff80088ac8d00, tqh_last = 0xfffff8003cc5b600}, bv_root = {pt_root = 2553161591}, bv_cnt = -1741805705}, bo_numoutput = 31, bo_flag = 0, bo_bsize = 0}, v_pollinfo = 0x0, v_label = 0x0, v_lockf = 0x0, v_rl = {rl_waiters = {tqh_first = 0xf88, tqh_last = 0x19cc}, rl_currdep = 0x3f8}, v_cstart = 16256, v_lasta = 679, v_lastw = 0, v_clen = 0, v_holdcnt = 0, v_usecount = 2369, v_iflag = 0, v_vflag = 0, v_writecount = 0, v_hash = 0, v_type = VNON} (kgdb) Thanks for your time Henri