On 11/10/2016 15:00, Andriy Gapon wrote:> On 10/11/2016 12:30, Henri Hennebert wrote:
>> On 11/10/2016 11:21, Andriy Gapon wrote:
>>> On 09/11/2016 15:58, Eric van Gyzen wrote:
>>>> On 11/09/2016 07:48, Henri Hennebert wrote:
>>>>> I encounter a strange deadlock on
>>>>>
>>>>> FreeBSD avoriaz.restart.bel 11.0-RELEASE-p3 FreeBSD
11.0-RELEASE-p3 #0 r308260:
>>>>> Fri Nov 4 02:51:33 CET 2016
>>>>> root at avoriaz.restart.bel:/usr/obj/usr/src/sys/AVORIAZ
amd64
>>>>>
>>>>> This system is exclusively running on zfs.
>>>>>
>>>>> After 3 or 4 days, `periodic daily` is locked in the
directory
>>>>> /usr/local/news/bin
>>>>>
>>>>> [root at avoriaz ~]# ps xa|grep find
>>>>> 85656 - D 0:01.13 find / ( ! -fstype local -o
-fstype rdonly ) -prune
>>>>> -o ( -name [#,]* -o -name .#* -o -name a.out -o -nam
>>>>> 462 1 S+ 0:00.00 grep find
>>>>> [root at avoriaz ~]# procstat -f 85656
>>>>> PID COMM FD T V FLAGS REF OFFSET PRO
NAME
>>>>> 85656 find text v r r------- - - -
/usr/bin/find
>>>>> 85656 find cwd v d r------- - - -
/usr/local/news/bin
>>>>> 85656 find root v d r------- - - - /
>>>>> 85656 find 0 v c r------- 3 0 -
/dev/null
>>>>> 85656 find 1 p - rw------ 1 0 - -
>>>>> 85656 find 2 v r -w------ 7 17 - -
>>>>> 85656 find 3 v d r------- 1 0 -
/home/root
>>>>> 85656 find 4 v d r------- 1 0 -
/home/root
>>>>> 85656 find 5 v d r----n-- 1 533545184 -
/usr/local/news/bin
>>>>> [root at avoriaz ~]#
>>>>>
>>>>> If I try `ls /usr/local/news/bin` it is also locked.
>>>>>
>>>>> After `shutdown -r now` the system remain locked after the
line '0 0 0 0 0 0'
>>>>>
>>>>> After a reset and reboot I can access /usr/local/news/bin.
>>>>>
>>>>> I delete this directory and reinstall the package
`portupgrade -fu news/inn`
>>>>>
>>>>> 5 days later `periodic daily`is locked on the same
directory :-o
>>>>>
>>>>> Any idea?
>>>>
>>>> I can't help with the deadlock, but someone who _can_ help
will probably ask for
>>>> the output of "procstat -kk PID" with the PID of the
"find" process.
>>>
>>> In fact, it's procstat -kk -a. With just one thread we would
see that a thread
>>> is blocked on something, but we won't see why that something
can not be acquired.
>>>
>>>
>> I attach the result,
>
> Interesting. I can not spot any suspicious thread that would hold the
vnode
> lock. Could you please run kgdb (just like that, no arguments), then
execute
> 'bt' command and then select a frame when _vn_lock is called with
'fr N'
> command. Then please 'print *vp' and share the result.
>
I Think I miss something in your request:
[root at avoriaz ~]# kgdb
GNU gdb 6.1.1 [FreeBSD]
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain
conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for
details.
This GDB was configured as "amd64-marcel-freebsd"...
Reading symbols from /boot/kernel/zfs.ko...Reading symbols from
/usr/lib/debug//boot/kernel/zfs.ko.debug...done.
done.
Loaded symbols for /boot/kernel/zfs.ko
Reading symbols from /boot/kernel/opensolaris.ko...Reading symbols from
/usr/lib/debug//boot/kernel/opensolaris.ko.debug...done.
done.
--- clip ---
Loaded symbols for /boot/kernel/accf_data.ko
Reading symbols from /boot/kernel/daemon_saver.ko...Reading symbols from
/usr/lib/debug//boot/kernel/daemon_saver.ko.debug...done.
done.
Loaded symbols for /boot/kernel/daemon_saver.ko
#0 sched_switch (td=0xfffff8001131da00, newtd=0xfffff800762a8500,
flags=<value optimized out>)
at /usr/src/sys/kern/sched_ule.c:1973
1973 cpuid = PCPU_GET(cpuid);
(kgdb) bt
#0 sched_switch (td=0xfffff8001131da00, newtd=0xfffff800762a8500,
flags=<value optimized out>)
at /usr/src/sys/kern/sched_ule.c:1973
#1 0xffffffff80566b15 in tc_fill_vdso_timehands32 (vdso_th32=0x0) at
/usr/src/sys/kern/kern_tc.c:2121
#2 0xffffffff80555227 in timekeep_push_vdso () at
/usr/src/sys/kern/kern_sharedpage.c:174
#3 0xffffffff80566226 in tc_windup () at /usr/src/sys/kern/kern_tc.c:1426
#4 0xffffffff804eaa41 in hardclock_cnt (cnt=1, usermode=<value
optimized out>) at /usr/src/sys/kern/kern_clock.c:589
#5 0xffffffff808fac74 in handleevents (now=<value optimized out>,
fake=0) at /usr/src/sys/kern/kern_clocksource.c:223
#6 0xffffffff808fb1d7 in timercb (et=0xffffffff8100cf20, arg=<value
optimized out>) at /usr/src/sys/kern/kern_clocksource.c:352
#7 0xfffff800b6429a00 in ?? ()
#8 0xffffffff81051080 in vm_page_array ()
#9 0xffffffff81051098 in vm_page_queue_free_mtx ()
#10 0xfffffe0101818920 in ?? ()
#11 0xffffffff805399c0 in __mtx_lock_sleep (c=<value optimized out>,
tid=Error accessing memory address 0xffffffffffffffac: Bad add\
ress.
) at /usr/src/sys/kern/kern_mutex.c:590
Previous frame inner to this frame (corrupt stack?)
Current language: auto; currently minimal
(kgdb) q
[root at avoriaz ~]#
I don't find the requested frame
Henri